Skip to content
This repository has been archived by the owner on May 22, 2021. It is now read-only.

Adds metrics documentation (closes #5). #201

Merged
merged 1 commit into from
Jul 18, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

[![CircleCI](https://circleci.com/gh/mozilla/send.svg?style=svg)](https://circleci.com/gh/mozilla/send)

**Docs:** [Docker](docs/docker.md), [Metrics](docs/metrics.md)

## What it does

A P2P file sharing experiment which allows you to send encrypted files to other users.
Expand Down
136 changes: 136 additions & 0 deletions docs/metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# Send Metrics
The metrics collection and analysis plan for Send, a forthcoming Test Pilot experiment.

## Analysis
Data collected by Send will be used to answer the following high-level questions:

- Do users send files?
- How often? How many?
- What is the retention?
- What is the distribution of senders?
- How do recipients interact with promotional UI elements?
- Are file recipients converted to file senders?
- Are non-Firefox users converted to Firefox users?
- Where does it go wrong?
- How often are there errors in uploading or downloading files?
- What types of errors to users commonly see?
- At what point do errors affect retention?

## Collection
Data will be collected with Google Analytics and follow [Test Pilot standards](https://github.com/mozilla/testpilot/blob/master/docs/experiments/ga.md) for reporting.

### Custom Metrics
- `cm1` - the size of the file, in bytes.
- `cm2` - the amount of time it took to complete the file transfer, in milliseconds. Only include if the file completed transferring (ref: `cd2`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "transfer" here? Upload? Download? Both?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. GA limits the numbers of metrics and dimensions, so it befits you to reuse them in this way.

- `cm3` - the rate of the file transfer, in bytes per second. This is computed by dividing `cm1` by `cm2`, not by monitoring transfer speeds. Only include if the file completed transferring (ref: `cd2`).
- `cm4` - the amount of time until the file will expire, in milliseconds.
- `cm5` - the number of files the user has ever uploaded.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is based on a cookie?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

localStorage, as I understand it. Up to the team on how to persist this data.

- `cm6` - the number of unexpired files the user has uploaded.
- `cm7` - the number of files the user has ever downloaded.

### Custom Dimensions
- `cd1` - the method by which the user initiated an upload. One of `drag`, `click`.
- `cd2` - the reason that the file transfer stopped. One of `completed`, `errored`, `cancelled`.
- `cd3` - the destination of a link click. One of `experiment-page`, `download-firefox`, `twitter`, `github`, `cookies`, `terms`, `privacy`, `about`, `legal`, `mozilla`.
- `cd4` - from where the URL for a file was copied. One of `finished-screen`, `file-list`.
- `cd5` - the referring location. One of `completed-download`, `errored-download`, `cancelled-download`, `completed-upload`, `errored-upload`, `cancelled-upload`, `testpilot`, `external`.
- `cd6` - the location from which the user copied the URL to an upload file. One of `success-screen`, `upload-list`.
- `cd7` - identifying information about an error. Exclude if there is no error involved. **TODO:** enumerate a list of possibilities.

### Events

_NB:_ due to how files are being tracked, there are no events indicating file expiry. This carries some risk: most notably, we can only derive expiration rates by looking at download rates, which is prone to skew if there are problems in data collection.

#### `upload-started`
Triggered whenever a user begins uploading a file. Includes:

- `ec` - `sender`
- `ea` - `upload-started`
- `cm1`
- `cm5`
- `cm6`
- `cm7`
- `cd1`
- `cd5`

#### `upload-stopped`
Triggered whenever a user stops uploading a file. Includes:

- `ec` - `sender`
- `ea` - `upload-stopped`
- `cm1`
- `cm2`
- `cm3`
- `cm5`
- `cm6`
- `cm7`
- `cd1`
- `cd2`
- `cd7`

#### `download-started`
Triggered whenever a user begins downloading a file. Includes:

- `ec` - `recipient`
- `ea` - `download-started`
- `cm1`
- `cm4`
- `cm5`
- `cm6`
- `cm7`

#### `download-stopped`
Triggered whenever a user stops downloading a file.

- `ec` - `recipient`
- `ea` - `download-stopped`
- `cm1`
- `cm2` (if possible and applicable)
- `cm3` (if possible and applicable)
- `cm5`
- `cm6`
- `cm7`
- `cd2`
- `cd7`

#### `exited`
Fired whenever a user follows a link external to Send.

- `ec` - `recipient`, `sender`, or `other`, as applicable.
- `ea` - `exited`
- `cd3`

#### `upload-deleted`
Fired whenever a user deletes a file they’ve uploaded.

- `ec` - `sender`
- `ea` - `upload-deleted`
- `cm1`
- `cm2`
- `cm3`
- `cm4`
- `cm5`
- `cm6`
- `cm7`
- `cd1`

#### `copied`
Fired whenever a user copies the URL of an upload file.

- `ec` - `sender`
- `ea` - `copied`
- `cd6`

#### `restarted`
Fired whenever the user interrupts any part of funnel to return to the start of it (e.g. with a “send another file” or “send your own files” link).

- `ec` - `recipient`, `sender`, or `other`, as applicable.
- `ea` - `restarted`
- `cd2`

#### `unsupported`
Fired whenever a user is presented a message saying that their browser is unsupported due to missing crypto APIs.

- `ec` - `sender`
- `ea` - `unsupported`
- `cd7`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General thought... I'm not very clever, and metrics is confusing, but by the time I've scrolled down to here, I've already forgotten what a cd7 is.

Not sure how ludicrous it would be to wrap some of these in an <code title="cd 7 -- identifying information about an error. Exclude if there is no error involved.">cd7</code> so you can just hover over the custom dimension to get a description. Markdown+HTML are hard. It appears GFM doesn't support ye olde <abbr> tag.

Example (hover for tooltip):

cd7

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't read well, agreed. But this is a document largely written for the implementers, so I'm less concerned about the clarity of a narrative. Plus, I'm really uninterested in creating 10 different places to update the spec if a metric's wording is tweaked before implementation.