ResourceWarning when accessing response content

[RyadIso]

ResourceWarning when accessing response content

When trying to access the response content, aiohttp will always return a ResourceWarning:

Simple reproducer:

#!/usr/bin/env python3

import aiohttp
import asyncio
import warnings

warnings.simplefilter('default', category=ResourceWarning)


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://github.com/") as r:
            await r.content.read()
            print("Status code: %d" % r.status)


asyncio.run(main())

This script will output a ResourceWarning:

Status code: 200
/usr/lib/python3.8/asyncio/selector_events.py:694: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Environment:
Ubuntu 20.04
Python 3.8.2

Package versions (in a minimal venv with just aiohttp + dependencies installed):

$ pip freeze
aiohttp==3.6.2
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
idna==2.9
multidict==4.7.6
yarl==1.4.2

[cjrh]

Is this a duplicate of #1115?

[RyadIso]

#1115 is an old bug reported in 2016 and "fixed" (actually not fixed, just documented a workaround) in 2017. I didn't even look at bugs fixed that long before since I am using recent software and shouldn't be affected by bugs fixed several years ago. So the thing I reported with the reproducer may actually be a duplicate of #1115.

That being said the issue I'm actually facing is getting these ResourceWarnings in the middle of a long-running program with many parallel workers (each of them creating its own ClientSession), long before the program finishes (and could do the await asyncio.sleep(0.25) workaround). Can't share the full program and I didn't yet manage to make a short reproducer for that behavior.

[JulienPalard]

Can be reproduced with a HEAD request, I checked, it's the same leak:

import aiohttp
import asyncio


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.head("https://github.com/") as r:
            ...

asyncio.run(main())

gives the same traceback that the one with the get and the read:

$ PYTHONTRACEMALLOC=10 python3 foo.py
/home/mdk/.local/lib/python3.10/asyncio/selector_events.py:704: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
Object allocated at (most recent call last):
  File "./foo.py", lineno 7
    async with session.head("https://github.com/") as r:
  File "./.venv/lib/python3.10/site-packages/aiohttp/client.py", lineno 1117
    self._resp = await self._coro
  File "./.venv/lib/python3.10/site-packages/aiohttp/client.py", lineno 520
    conn = await self._connector.connect(
  File "./.venv/lib/python3.10/site-packages/aiohttp/connector.py", lineno 535
    proto = await self._create_connection(req, traces, timeout)
  File "./.venv/lib/python3.10/site-packages/aiohttp/connector.py", lineno 892
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "./.venv/lib/python3.10/site-packages/aiohttp/connector.py", lineno 1020
    transp, proto = await self._wrap_create_connection(
  File "./.venv/lib/python3.10/site-packages/aiohttp/connector.py", lineno 969
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/home/mdk/.local/lib/python3.10/asyncio/base_events.py", lineno 1080
    transport, protocol = await self._create_connection_transport(
  File "/home/mdk/.local/lib/python3.10/asyncio/base_events.py", lineno 1102
    transport = self._make_ssl_transport(
  File "/home/mdk/.local/lib/python3.10/asyncio/selector_events.py", lineno 78
    _SelectorSocketTransport(self, rawsock, ssl_protocol,

[Dreamsorcerer]

Isn't that a cpython bug?

The code in your traceback points to https://github.com/python/cpython/blob/main/Lib/asyncio/selector_events.py#L78
Where it creates _SelectorSocketTransport without assigning it to a variable or cleaning it up in any way.

[JulienPalard]

Owwww, this is known and documented:

https://docs.aiohttp.org/en/stable/client_advanced.html#graceful-shutdown

This is the underlying SSL protocol having to shutdown gracefully. aiohttp calls "close()" on it, but it won't immediatly close the underlying socket: it starts a shutdown procedure first, and close it later, sleeping a few milliseconds often suffice for the socket to actually be closed.

[Dreamsorcerer]

Right, forgot all about that. Duplicate of #1925 which is fixed in (unreleased) v4.