Fix failed websocket handshake leaving connection hanging

[gjcarneiro]

What do these changes do?

It fixes the bug described in #3380, caused by this sequence of events:

1. clients initiates a websocket upgrade
2. server returns a normal HTTP response instead of a websocket response
3. client, with the same connection open, sends further HTTP requests.

The fix involves: if we detect a normal HTTP response instead of a websocket response, in web_protocol, then we:

1. Reset _upgraded flag in the HttpParser to the default value (False)
2. Reset the _uprade flag in web_protocol back to False;
3. We feed any buffered data, originally intended for the websocket payload parser, back to the the http parser, so that the next normal HTTP request can be processed.

This way, if websocket request results in a non-websocket response, we go back to normal HTTP mode, for that socket.

Are there changes in behavior for the user?

Related issue number

#3380

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes
• If you provide code modification, please add yourself to CONTRIBUTORS.txt
  • The format is <Name> <Surname>.
  • Please keep alphabetical order, the file is sorted by names.
• Add a new news fragment into the CHANGES folder
  • name it <issue_id>.<type> for example (588.bugfix)
  • if you don't have an issue_id change it to the pr id after creating the pr
  • ensure type is one of the following:
    • .feature: Signifying a new feature.
    • .bugfix: Signifying a bug fix.
    • .doc: Signifying a documentation improvement.
    • .removal: Signifying a deprecation or removal of public API.
    • .misc: A ticket has been closed, but it is not of interest to users.
  • Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

[codecov-io]

Codecov Report

Merging #3971 into master will decrease coverage by 0.05%.
The diff coverage is 62.5%.

@@            Coverage Diff             @@
##           master    #3971      +/-   ##
==========================================
- Coverage   97.77%   97.71%   -0.06%     
==========================================
  Files          43       43              
  Lines        8763     8771       +8     
  Branches     1375     1377       +2     
==========================================
+ Hits         8568     8571       +3     
- Misses         83       86       +3     
- Partials      112      114       +2

Impacted Files	Coverage Δ
aiohttp/http_parser.py	97.1% <100%> (+0.01%)	⬆️
aiohttp/web_protocol.py	91.59% <50%> (-0.77%)	⬇️
aiohttp/web_fileresponse.py	96.59% <0%> (-1.14%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38a1ec4...c5385aa. Read the comment docs.

[asvetlov]

I see the bug described here and agree that protocol should not set upgraded flag itself.
The proposed fix is not enough I think, sorry.

Let me describe how things work now.
HTTP parser processes incoming requests. If the parser encounters Upgrade header it a) saves msg.upgrade flag and b) sets parser._upgraded flag to true.

parser._upgraded flag is used to prevent parsing next in-flight messages when WebSocket handshake is requested but not completed yet (parser._payload_parser is not set).

This is a rudiment of added-but-removed-later HTTP pipelining support: the parser should stop handling incoming messages on protocol upgrade request.

Your change postpones setting parser._upgraded until the message is handled by user code. So, if a client sends websocket data just after sending websocket handshake request (a little strange behavior but it doesn't violate WebSocket protocol) this data is parsed as HTTP message if the data arrives before WS handshake establishing.

A possible solution is splitting parser._upgraded state into _upgrading (upgrade requested) and _upgraded (upgrade is done).

In upgrading state the transport should stop reading, collect in-flight data into internal binary buffer before parsing and wait for the transition into upgraded state (or back to normal).
In upgdaing there is no parser, it can be HTTP or WebSocket depending on upgrading result.
The unprocessed data can be parsed only after establishing the next parser.

Sounds complicated a little :(

[gjcarneiro]

I think I understand the problem. Unlikely as it may be, I'm glad you caught it. OK, I will work on this some more.

[gjcarneiro]

Travis CI taking forever, but, assuming build passes (it passes on my devbox), what I did was start with the suggested _upgrading + _upgraded flags. But then I realised I never used _upgraded flag. In the end made significant simplifications and the PR is much smaller, while still fixing the original bug.

If you're happy with, I would just like to squash the commits before allowing you to merge.

[gjcarneiro]

Rebased / squashed commits.

[gjcarneiro]

I think this is ready to merge, if you have no more feedback.

[asvetlov]

Thanks!