A/B Test Beefy Upload UI

[johngruen]

In order to inaugurate a new era of A/B testing we propose the following test:

Hypothesis

The primary CTA/Interaction points on the send home page are too small and faint, and that making them easier to see will increase overall usage of the service.

Test

This can be proven by conducting an A/B test wherein the control branch is the current UI and the experiment branch makes the following changes:

• The drag drop box border width is changed to 3px
• The Select a file to upload button goes is made to be 60px and its font size will be set to 20px

Current

Beefy

New Event Information

Visit

Triggered when a user first vists the site. We need to implement in order to determine bounce rate since we don’t use the tracking pixel

• ec: send page download page other depending on entry point (we might add to this when we implement password input stuff for receivers)

All events should be added to a new content experiment:
The following keys should be added to all event pings

• xid: beefy-upload
• xvar: control or exp1

Note that if a user visits from a download page we will have to pull the state of these fields from the server.
More info on content experiments here:
Measurement Protocol Parameter Reference  |  Analytics Measurement Protocol  |  Google Developers

Proof metrics

To compare branches we will observe three metrics:

1. The relative rate of upload-started events relative to visits of the upload page in control and experiment
2. The relative rate of successful uploads to downloads for control and experiment

Cohort Size

Since we may want to do more than one experiment we should limit experiment and control cohorts to 10% of users apiece.

[johngruen]

Note I haven’t had time to add the content experiment to GA. We might want to assign @clouserw to that