Proposal: Browser-assisted performance ablation experiments

Measuring site performance benefits from a close connection with publisher business outcomes. Establishing this connection is difficult and often confounded by concurrent changes to site UX, content and product offerings.

Causal connections between performance and business outcomes require randomized A/B testing on live traffic. Sites, frameworks and analytics providers may support such experiments within the scope of the components they control, e.g. by varying the image or javascript payloads, or delaying server response. However, these methods do not always have a direct relationship to user-observable performance and/or require large interventions affecting many users to produce measurable effects.

Internally, Chrome has used browser-based experiments that modify memory use and establish causal connections between memory use and user experience. These studies are known as ablation studies, and are common in other black- or grey-box testing environments like machine learning.

This is a proposal for a low-level mechanism to support ablation experiments when measuring web site performance.

We think this proposal could be useful for a CDN (like Akamai), as CDNs are often both in control of HTTP headers for their customers and are already collecting performance metrics for the same properties.

Akamai’s mPulse already predicts on how various performance metrics affect business metrics (positively or negatively), and this API could help an interested customer validate some of those predictions.