Consider documenting how cloud architectures might work in practice

[LPardue]

There seems to be some inherent assumptions in the specification about where the test server(s) reside, and how clients communicate with them. These assumptions might not hold for certain kinds of scaling deployment (e.g. clouds or CDNs).

For example, it seems reasonable that a client making several independent HTTP/2 connections to the same authority would land on the same Internet path. However, load balancing after that system boundary could add a lot of undetectable timing variation to servicing of the connection and the requests within that connection.

The testing strategy seems resilent to these forms of architectural differences. But I wonder if we need to be clearer for client implementers on how the measurements could be affected, and whether the aggregation calculation needs to be a bit more advanced to accommodate such cases.

[cpaasch]

We should add to the json test_endpoint, which allows to make a measurement sticky to a CDN-node.

@randall - can you add that?

[randall]

We originally looked at alt-svc for this, but had issues with particular client implementations.

For our current production implementation, there's a field in the config response test_endpoint that specifies a host to remain sticky to.

[cpaasch]

Randall pushed #64 to introduce test_endpoint

[cpaasch]

I think that closes this issue, right?

[LPardue]

Sorry, no I don't think this address the problem - you can't reliably equate a name to any specific server or IP address.

The way these distribution architectures operate in practice means clients don't need to care much about server specifics - that's really the killer feature of CDN or clouds. Fetch $foo and make it snappy.

But in this specification, clients are looking very closely at details about transport, TLS and HTTP connections, which become more stateful.

And so the problem remains that if someone comes and writes a responsiveness client, and doesn't care about the details of serverland (entirely in their rights), they might not understand why the numbers they are getting out are so variable on a hour-by-hour or day-by-day basis.

On the other side of the coin, maybe this specification wants to provide some guidance for server operators that they traditional horizontal scaling might affect RPM and that they could optimize their deployment by making sticky endpoints. But then, isn't that edging away from the real deployment that we are trying to characterise? Not sure.

[LPardue]

ps. definately not able to re-open issues myself :-)

[cpaasch]

Copy from slack:

The text currently says:

  All of the fields in the sample configuration are required except "test\_endpoint".
  If the test server provider can pin all of the requests for a test run to a specific
  host in the service (for a particular run), they can specify that host name in the
  "test\_endpoint" field.

Meaning, the hostname must be a unicast hostname to "pin all of the requests for a test run to a specific host". Shouldn't that address your comment, @LPardue ?