Adaptive Experimentation at NeurIPS Systems for ML Workshop

Eytan Bakshy and I presented “AE: A domain-agnostic platform for adaptive experimentation” at the NeurIPS 2018 Systems for ML workshop. This is joint work with Lili Dworkin, Brian Karrer, Benjamin Letham, Ashwin Murthy, and Shaun Singh.

Here’s the abstract:

We describe AE, a machine learning platform for adaptive experimentation (e.g., Bayesian optimization, bandit optimization) that automates the process of sequential experimentation. Unlike existing solutions that are oriented primarily towards optimizing ML hyperparameters and simulations, AE is designed with online experimentation (A/B tests) in mind. Motivated by real-world examples from Facebook, we present a design for ML-assisted experimentation with multiple objectives, noisy, non-stationary measurements, and data from multiple experimentation modalities.