bpo-36373: Fix deprecation warnings

[asvetlov]

https://bugs.python.org/issue36373

Automerge-Triggered-By: @asvetlov

[asvetlov]

@tirkarthi that's what I mean

[tirkarthi]

Just curious why we moved it here. Is there a difference in constructing it in _setupAsyncioLoop itself without passing loop and having it here?

[asvetlov]

Well, it is possible to just drop the loop parameter from Queue construction because we installed the default event loop a few lines above but I did two steps further.

Instantiation on asyncio loop coupled objects from an async function is more idiomatic, it always grabs a currently executed loop (the loop which is used to run a coroutine).
It prevents things like this:

import asyncio
queue = asyncio.Queue()

asyncio.run(...)  # a code that works with queue

In the example queue variable is instantiated with the default asyncio loop but run() creates a new loop version. So the queue doesn't work and hangs eventually because a future created by one loop cannot be awaited by another one.

In Python 3.9 I have a plan to add another deprecation warning if not self._loop.is_running() for queues and locks and eventually replace get_event_loop() with get_running_loop() as a final fix.

[tirkarthi]

LGTM. As I understand previously we were always passing self._loop which would always be initialized with a loop object and would mean we are passing the loop value to the other callers though the initial caller had loop=None only. This looks more correct and simple to me given asyncio.Queue() is a valid call. Thanks Andrew.

[1st1]

I thought we were going to be more subtle about this.

We have two scenarios:

1. Somebody creates an instance of asyncio.Lock outside of a coroutine, also manually creating the loop, etc. In this case the user has to pass the lock argument.

2. Somebody creates an instance of asyncio.Lock in a coroutine (that is, wheh asyncio.get_running_loop() returns a loop). In this case passing the loop argument is an error.

The (1) approach is how people were used to writing asyncio programs before asyncio.run() (that was the only way actually).

The (2) approach is how we want people to write asyncio programs.

There's a subtle difference between things like asyncio.gather() and asyncio.Lock. Passing loop to the former is just nonsensical. Passing loop to the latter can be a valid thing, it's the (1).

If we remove the loop parameter entirely from classes like asyncio.Lock we're making (1) impossible. I'm not sure that is what we are ready to do now.

@asvetlov thoughts?

[1st1]

Please don't merge this until we discuss it.

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[asvetlov]

asyncio.run() is a nice API, everybody should use it (and people do as I see).

On Python 3.6 where asyncio.run() doesn't exist the difference is not drastic though, instead of asyncio.run(main()) you can use loop.run_until_complete(main()).
The loop preparation and especially finalization is missing here but roughly the code is the same.

Also, please read comment above. In aiohttp we forbade the creation of ClientSession outside of async function context step by step because we have a long gory list of reports when "aiohttp hangs for some reason", and the reason was misusage of a loop and an object that belongs to a different loop by accident.

[1st1]

Alright.

[miss-islington]

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[bedevere-bot]

GH-15896 is a backport of this pull request to the 3.8 branch.

[miss-islington]

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

[miss-islington]

Sorry @asvetlov, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 7264e92b718d307cc499b2f10eab7644b00f0499 3.8

[bedevere-bot]

GH-15901 is a backport of this pull request to the 3.8 branch.

[dstansby]

Is there a drop in replacement for code that was previously using the loop parameter to Queue? I'm struggling to understand how to modify code to have the same behaviour without using the loop argument.

[aaugustin]

I had the same issue and found this workaround.

     # Create an event loop that will run in a background thread.
     loop = asyncio.new_event_loop()

+    # Due to zealous removal of the loop parameter in the Queue constructor,
+    # we need a factory coroutine to run in the freshly created event loop.
+    async def queue_factory() -> asyncio.Queue[str]:
+        return asyncio.Queue()
+
     # Create a queue of user inputs. There's no need to limit its size.
-    inputs: asyncio.Queue[str] = asyncio.Queue(loop=loop)
+    inputs: asyncio.Queue[str] = loop.run_until_complete(queue_factory())

     # Create a stop condition when receiving SIGINT or SIGTERM.
     stop: asyncio.Future[None] = loop.create_future()

Forcing the introduction of factory functions counts as progress towards Java, I guess...

[cladmi]

I ended up here after following deprecation warning -> git blame -> ticket -> comments.
So reviving it as may be useful for others following the path.

I have a case of an asynchronous loop running on another thread already, and a synchronous program that creates tasks to run on it. Basically like background workers.

Trying to go a generic way here. Maybe they would be some existing helpers but I did not find it yet.

async def as_coroutine(cb, *args, **kwargs):
    return cb(*args, **kwargs)


def call_on_loop(cb, *args, loop, **kwargs):
    """Run a callback on the async loop and return its return value."""
    coro = as_coroutine(cb, *args, **kwargs)
    return asyncio.run_coroutine_threadsafe(coro, loop).result()


# Loop is running already somewhere
queue = call_on_loop(asyncio.Queue, loop=loop)