Body Wash, Maritime Shipping, and El Niño: The Absent Layer in Product Experimentation

A bottle of body wash, billions in revenue, and the limit of brute force

During the summer of 2018 I worked at Unilever's R&D campus in Trumbull, Connecticut, on a body wash formulation problem. It is still the cleanest example I have for a thing product organizations keep getting wrong: confusing a moved number with a justified attribution.

This is one of those problems you've experienced but probably haven't thought much about. It ties into questions like: why does my facial cleanser produce almost no suds, this hand soap produce a meringue-like lather, and that dish soap bubble up a good few inches? Why does one leave a scent lingering hours longer than the others?

Lather is just a hook. Even though more lather does not actually get you cleaner, consumers correlate sud volume with cleaning ability, so it becomes a meaningful sensory factor in repeat purchase behavior. That half-second sniff at the bottle's neck is another. Even the firmness of the cap when you flip it open contributes something to the equation. When billions of people use a product, there is an enormous amount of revenue stacked on top of small sensory details, and the work of consumer R&D is to optimize each of them under the constraint that the formulation also has to be stable, cost-controlled, and rheologically intact (it must not separate into a sludge before arriving at your local grocery store).

My specific project focused on fragrance deposition. Indeed, I spent my summer optimizing the amount of fragrance deposited on the skin during your brief contact with it in the shower. It involves formulating the products with cellulose-derived polymers substituted with varying degrees of cationic and hydrophobic components. One complicating detail is that "fragrance" is typically a mixture of chemicals that are heterogeneous across hydrophobicity. Within a single body wash, you can have highly hydrophobic terpenes (limonene, pinene) that partition into the oil phase of a surfactant micelle, and more polar alcohols and esters (citronellol, phenylethyl alcohol) that partition more toward the water phase. There's also a litany of intermediate components in between. The chemistry for getting fragrance molecules to deposit on skin during a few seconds of rinsing involves a coacervate, a polymer-surfactant-fragrance network, that precipitates onto the substrate as the water dilutes through it. The polymer is the structural mediator. The surfactant decides the micelle environment the fragrance starts in. The fragrance composition decides which components partition where.

If you wanted to test every combination of three surfactant systems against two co-surfactants against six candidate polymers (each carrying different molecular weights, charge densities, and hydrophobic modifications) against seven fragrance components at two levels each, you are looking at well over a thousand formulations. Each one has to be batched at a lab bench, panel-tested, run through a silk-fabric wash that approximates skin, and then quantified by headspace gas chromatography. The budget covered a fraction of that. We needed to know which fraction would give us the most information about the system.

The methodology that picks the right fraction and tells you when the data is too noisy to confidently adopt a winner is called design of experiments (DoE). It is a discipline that asks, before you spend the budget: which subset of all possible configurations gives you maximum information about the system? And after you spend it: which factors and interactions actually moved the response, with what confidence, and what should we avoid committing to because the data does not support the inference?

The math is mature. Box, Hunter and Hunter published the standard textbook in 1978 (second edition 2005).¹ The screening design we used was published by Plackett and Burman in 1946.² Experimental design is fundamental to how chemistry, pharmaceutical bioprocess optimization, and consumer product R&D have been done for decades.³

It has not landed in product.

Many product teams have adopted traffic allocation and optimization tooling without importing the methodology required to make the resulting inferences trustworthy. The vendor surface often reflects that same gap.

Same situation, different domain

When you've studied a problem in one domain, I've found it's hard to miss it when it appears elsewhere, and the first thing I do in this situation is evaluate whether transposing a methodology to the new context resonates. For what's next in this writing, I'll provide one caveat: the adoption of design-of-experiments thinking is not uniform inside every domain nor every operating scale. That's the structural gap I'm describing. I'm not presenting an absolutist claim.

Drug discovery

In pharmaceutical research and development,⁴ the same combinatorial pressure appears in formulation, process optimization, analytical method development, and some medicinal-chemistry work. A team optimizing a molecule, assay, or process faces candidate variables like substituent choice, solvent system, process temperature, concentration, pH, or excipient ratio. Exhaustive enumeration is rarely feasible. Formal DoE is routine in pharmaceutical product, process, and method development. While adoption varies by organization size and stage of development, formulation, process, and assay work see more routine use than early medicinal chemistry.⁵ The defensible claim is not that every SAR program is a textbook fractional factorial. It is that mature technical disciplines already know the pressure: too many candidate factors, expensive observations, noisy responses, and a need to separate signal from interaction.

Product experimentation & AI features

A modern AI product surface has many of the same combinatorial properties. The prompt has a system message, a tool list, a temperature, a context window, a retrieval depth, an output format, a critic loop. Each of those has three to ten reasonable settings. The full combinatorial space is operationally unrunnable. The product manager who needs to know which configuration is actually winning faces the same problem as the medicinal chemist facing the receptor and the formulator facing the deposition coacervate. Brute force is off the table.

Planning maritime shipping routes

A fleet operator allocating ships across the North Atlantic during an El Niño/Southern Oscillation half-cycle faces several factors that could drive fuel-adjusted transit time: route choice, departure timing relative to ENSO state, cruising speed, fuel grade, ballast condition, weather-routing service, hull-fouling state, and the captain's discretionary margin. Major operators run sophisticated weather-routing optimization, isochrone methods, dynamic programming on routing graphs, multi-objective evolutionary algorithms, and increasingly Bayesian-network inference over casualty and delay risk.⁶ The DoE-style factor-screening layer that asks "of these eight candidates, which three actually move the response, and which interactions are confounded with which main effects?" is not the dominant analytical practice in fleet routing. Is that part of the gap?

Three different domains, three different levels of sophistication, and one underlying problem. The same family of methods can be made useful in each, if the design is properly constructed.

What product-experimentation tools ship today, and what they don't

I evaluated ten product-experimentation platforms in May 2026 against one question: does this platform ship the more nuanced methodological details that turn a multi-factor result into a defensible causal inference?⁷

Three platforms, Optimizely, VWO, and Convert, let the user construct multi-factor multivariate tests that explore combinations of independent elements rather than single-factor A/B/n testing. The applied definition of "multi-factor" appears to differ in each of their shipped offerings.

For the other seven, PostHog, Statsig, Eppo, GrowthBook, Amplitude, LaunchDarkly, and Split/Harness FME, I did not find public documentation related to a multi-factor experiment feature. Their "multivariate" surface is usually a single-factor design with N variants of one element. That is not a factorial design; it is an A/B test with C, D, and more tacked on.⁸

Some of the analytical machinery that lifts a multi-factor result into a defensible inference is missing. This audit reflects public documentation and product surfaces available at the time of review; private enterprise features, roadmap items, or undocumented statistical behavior may differ. The comparison table below documents the per-platform state. The methodology audit follows.

The structural audit, restated against the table:

1. Alias structure

When you run a fractional factorial, certain main effects are confounded with two-factor interactions. The design's resolution tells you which. I did not find public documentation showing that any of the ten surfaces alias patterns, defining relations, or design resolution to the user. The three multi-factor platforms (Optimizely, VWO, Convert) implement "partial factorial" as either manual combination exclusion (Optimizely) or vendor-selected optimal arrays without alias disclosure (VWO). The classical DoE machinery, defining relation, generator words, resolution number, alias chain enumeration, is not part of the public product surface.

2. Screening designs

Plackett-Burman screens up to eleven factors in twelve runs, or up to nineteen factors in twenty runs, and surfaces the Pareto-dominant three or four. The math has been settled since 1946. I did not find public documentation showing that any of the ten ships a screening-design surface. Optimizely's 64-combination cap is the closest the reviewed market gets to first-class multi-factor experimentation, and even that is full-factorial allocation up to a budget ceiling.

3. Response-surface methodology

Central composite and Box-Behnken designs fit the local curvature of a response near an interesting point. For the narrow product contexts where factors are continuous and the response is approximately smooth (recommender-weight tuning, ad-bidding parameters, AI hyperparameter sweeps), RSM finds local optima that screening designs and pure factorials cannot. I did not find this surfaced as a user-facing experiment design family in the ten platforms reviewed.

4. Mixture designs

Scheffé designs²⁶ apply when settings sum to a fixed budget: route-share allocation across a fleet, user attention across surfaces, capital across strategic initiatives. Resource-allocation problems are common in product strategy. I did not find a reviewed platform modeling experiments this way. Attention and capital are entered as separate factors the platform treats as independent. The mismatch is not cosmetic. It is a data-model problem.

5. Validity gates

A formal check at ingest that refuses to publish an attribution when realized telemetry violates the design's assumptions. Sample-ratio mismatch²⁷ is detected on most platforms in some form. In the public surfaces I reviewed, SRM is a diagnostic, not a hard attribution gate. Where a platform detects sample-ratio mismatch but does not gate interpretation behind it, the UI can still leave the user with a decision-ready result despite compromised telemetry.

Why the gap persists

Some vendor education content explains DoE-adjacent concepts. Operational adoption does not follow. Four structural forces hold the gap open:

The skill-gap incentive. Most product managers have not been trained on design resolution, alias structure, or screening. Surfacing "Resolution III" in the UI without the surrounding discipline produces confusion, not insight. Vendors who try this without solving the teaching problem at the same time will lose user-research battles inside their own product orgs. The three platforms that already ship factorial MVT have chosen, plausibly correctly, not to expose the analytical machinery.

The always-give-a-number incentive. The hard-gate on SRM is methodologically correct and commercially awkward. A team running thirty experiments a quarter does not want one in five blocked at result time on a telemetry diagnostic. The commercial incentive points toward publishing a number.

The data-model mismatch. The platform's input is a feature flag with N variants. The mathematics of factorial design is independent factors at orthogonal levels. The two are not the same. To ship factorial as a first-class object, the platform has to change how experiments are entered, not just how results are computed. That is an input-layer migration, not an analytics feature.

Many PMs are not familiar with experimental design. The discipline of enumerating one's candidate variables as orthogonal independent factors is itself a skill. Without it, factor designs are not useful because the factors are not actually independent. A button color is a factor; a "new onboarding flow" is a bundle of factors masquerading as one. Most product teams do not yet have the language to disambiguate.

The implication is that adding a factorial constructor on top of an existing experimentation platform will not close the gap. A new layer is required.

We don't need a new platform

The single most useful question for this section is: how do you preserve methodological rigor when the user does not know the methodology vocabulary? The answer that survives the four forces above is an LLM-plus-deterministic-engine architecture where the LLM is the translator and the engine is the math.

The LLM is not decoration. It is the bridge that lets a PM describe what they want to test in natural language, and have a designed experiment with proper alias structure come back. The engine is the mathematics; the LLM is the doorway.

A product manager prompts: "test whether changing recommendation weight, retrieval depth, and critic-loop count, in combination with two reranker variants, improves my AI-powered email personalization feature's retention." The LLM emits a typed specification of what experiment to run.

{
  "family": "fractional_factorial",
  "factors": ["recommendation_weight", "retrieval_depth",
              "critic_loop_count", "reranker_variant"],
  "levels": [2, 2, 2, 2],
  "runs": 8,
  "resolution": "IV"
}

The deterministic engine takes that specification and does the work: constructs the design matrix, surfaces the alias structure, validates the telemetry as it returns, refuses to attribute when the realized data violates the design assumptions, and returns an attribution with confidence intervals. Design construction has hard mathematical properties (orthogonality of the design matrix, balance across cells, alias structure, resolution, D-efficiency) that you do not want a language model constructing token by token. The LLM should call the function, not be the function.

Qualitative inputs land in the same envelope.

If a maritime operator's captain weather logs, port inspection narratives, and insurance rate tables feed into the same routing decision, the LLM reads them and emits a typed observation:

{
    "event_type": "rogue_wave_probability_elevated",
    "spatial_extent": {"lat_range": [50, 60], "lon_range": [-30, -10]},
    "temporal_window": {"start": "2026-05-14", "duration_days": 21},
    "confidence": 0.78,
    "basis": ["captain_log:vessel_imo_9123456:2026-05-14",
              "casualty_narrative:case_2026_0521"]
}

The engine consumes the observation, elevates the named factor as a candidate for the next screening design, and proceeds. The model never constructs the experiment. The model never runs the attribution. It paraphrases the engine's output back to the user at the end.

The validity gate, in plain terms.

Four checks fire the moment realized telemetry comes back from production:

1. Did the realized allocation match the design? If you told the platform to put 12.5% of traffic into each of eight cells of a 2³ factorial and one came back with 4% because of a routing bug, the chi-squared test fires.
2. Do comparable cells balance against each other? If two cells should look statistically identical on baseline covariates but they do not, the engine flags the imbalance.
3. Did each cell get enough data to support an inference? Below a floor set by the experimenter, the engine emits a diagnostic, not a number.
4. Did underlying conditions hold steady across the run? Did the ENSO state phase-attenuate mid-experiment? Did engineering push a new model behind the personalization API? A Kolmogorov-Smirnov test can detect distributional shift; the engine partitions the analysis instead of averaging responses across the phase transition.

On failure, the engine does not publish an attribution. It returns a diagnostic with the affected cells and an actionable recommendation. This is the structural inversion of how the current platform stack handles SRM. Where platforms detect sample-ratio mismatch but do not gate interpretation behind it, the user can still walk away with a result that looks more decision-ready than the telemetry justifies.

Who would adopt this, and how it composes with what they already run.

The honest audience for a methodology layer is not the broad product-manager market. It is the segment of product organizations that has outgrown warning-and-continue experimentation: technical PMs at AI-native companies who already feel the inference problem; ML and applied-science teams that bridge to JMP, R, or custom notebooks; internal experimentation teams at companies already building proprietary platforms because the public vendor surface does not fit their work.

None of those audiences is a mass market. None requires one. The methodology layer composes on top of the existing experimentation infrastructure rather than replacing it. The architecture reads exposure events and metric data from whichever platform the org already runs (Optimizely, Statsig, Eppo, an internal pipeline), constructs the designed experiment over the cells the org executes, and writes the validated attribution back. The platform stays the system of record for traffic allocation and telemetry. The methodology layer is the inference tier above it.

A worked example: routing the Atlantic across the 2023–2024 El Niño

The 2023–2024 ENSO cycle entered strong El Niño mid-2023, peaked in November–December 2023 at a Niño-3.4 anomaly of approximately 2.0°C, and decayed to neutral by April–June 2024.²⁸ For domain scale, 2023 also recorded an industry-wide low of 221 containers lost at sea, the lowest count since the World Shipping Council survey began in 2008, out of approximately 250 million containers transported, with around 33% recovery on the lost units.²⁹ The container-loss figure is offered here as scale context, not as an ENSO-attributable outcome — the WSC report attributes the low to improving securing and lashing practices, not weather phase. The maritime example below uses the ENSO timing as a known nonstationarity to demonstrate the methodology, not to explain the container statistic.

The example below is hypothetical. Numbers are constructed to be plausible against the named domain, not measured from real fleet operations. The point is the architecture: screening, mixed follow-up modeling, validity gate.

Setup. Imagine a fleet operator allocating 192 transatlantic voyages across a 12-cell screen in Q4 2023 (Boston/NYC to Antwerp/Hamburg/Rotterdam) during peak El Niño. Eight candidate factors are nominated:

• F1 — Route: Great Circle vs. Rhumb Line
• F2 — Departure-day phase: bi-weekly cycle position 1 vs. position 2
• F3 — Cruising speed: Eco vs. Standard
• F4 — Fuel grade: VLSFO vs. LSMGO
• F5 — Ballast condition: Light vs. Full
• F6 — Weather-routing service: Vendor A vs. Vendor B
• F7 — Hull-fouling state: 0–6 mo since drydock vs. 6–12 mo
• F8 — Captain's discretionary margin: tight (±10 nm) vs. wide (±60 nm)

Step 1 — Plackett-Burman screen, 12 runs over 8 factors. The screen runs 12 voyage configurations as orthogonal contrasts. Response: fuel-adjusted transit time in hours. After 12 cells × 16 voyages each = 192 voyages logged, a hypothetical main-effect ranking looks like this:

Factor                       Effect (hrs)   SE     Pareto rank
F1 (Route)                       -8.4       1.2        1
F6 (Weather-routing service)     -6.1       1.3        2
F2 (Departure phase)             +4.7       1.4        3
F3 (Cruising speed)              -1.8       1.5        4
F7 (Hull fouling)                +1.4       1.4        5
F4, F5, F8                       < 1.0      1.4    below noise floor

Three factors (F1, F2, F6) dominate. Five fall below the screen's resolvable threshold and are dropped from the follow-up. Without methodology, the dashboard can still reward whichever bundle looks best in aggregate. The screen does the less theatrical thing: it asks which factors the data can actually support.

Step 2 — Mixed follow-up model on surviving three factors. Because route and routing-service choice are categorical while departure timing is ordered and time-varying, the follow-up is not a textbook central composite design. A defensible follow-up would be a 2×2 factorial on F1 × F6 with F2 modeled as a smoothed phase/timing term. The point is not that the model discovers a real Atlantic routing rule. It is that the method can test whether departure timing behaves monotonically or whether the local optimum moves as ENSO declines from its November–December peak.

Step 3 — Validity gate fires. Suppose one follow-up cell contains four voyages after the ENSO state begins to attenuate from its November–December 2023 peak. A Kolmogorov-Smirnov test on transit-time distribution within that cell rejects stationarity at p < 0.01. The engine refuses to publish a single attribution over that cell and partitions the analysis into pre- and post-attenuation segments.

Step 4 — Recommendation delta.

Without the methodology layer: aggregated Q4 transit times show weather-routing service B with a 7.2% mean speed advantage over A. The operator is tempted to commit Q1 budget to service B.

With the methodology layer: the partitioned analysis treats B's advantage as conditional on the strong-El-Niño window and much smaller after attenuation. The same data that produced "B wins, commit Q1" without the layer produces "B may win under a phase condition that has already passed" with it. The Q1 commit does not happen against the wrong assumption.

I do not claim this stack would have avoided the 221 containers lost at sea in 2023. I claim something narrower and more useful: it would make it harder to commit a quarter's routing budget against a lift that only held under a phase condition that had already passed.

Conclusion

If you build product at an AI-native company, the gap I described is the one you are already feeling but possibly not naming. The post-training-run noise in your eval set looks like the noise in a body-wash panel; the candidate factors driving your AI feature look like the polymers I parameterized in 2018; the El Niño phase looks like the model migration that lands mid-experiment. The math is the same. The methodology is the same. The vendor stack does not ship it.

The methodology layer is the part that determines whether the inference is trustworthy. The body-wash deposition problem, the drug-discovery problem, the AI-feature problem, and the maritime-routing problem are all versions of the same structure: a combinatorial space too large to brute-force, a response that depends on factors and interactions, limited experimental budget, and data conditions that may not remain stable during the run.

The experimentation stack adopted optimization before it adopted methodology. The methodology already exists. Much of product never imported it.