Actual workload memory leak from python 3.9.2

[ilya-lt]

🌍 Environment

• Your operating system and version: Docker, python 3.9.2 image
• Your python version: 3.9.2
• How did you install python (e.g. apt or pyenv)? docker
• Did you use a virtualenv?: yes
• Your Rust version (rustc --version): 0.51.0
• Your PyO3 version: 0.13.2
• Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: No

💥 Reproducing

I didn't try to replicate this memory leak in a test, but I encountered a memory leak running a workload that requires calling a callback object:

let should_process_args = (schema_id, subject_name.as_str());

should_process.insert(
    schema_id,
    callback.call_method1("should_process", should_process_args)?.is_true()?
);

schema_id is u32, subject_name is String, callback is a python object passed as an argument.

After about 20-30 million iterations of this, the program eats 4gb of memory and crushes, unfortunately I'm not skilled enough in rust to try and construct a test case out of it, but thought it might still help.

[ilya-lt]

forgot to mention on the python side, this is a simple:

if "something" in subject_name:
_return True
return False

function, so it really shouldn't allocate anything

[birkenfeld]

The function call will still allocate Python objects, since the u32 and string (and the tuple) will have to be represented as Python objects.

Do you by chance never release the GIL between your iterations? PyO3 stores its Python references in a pool, which are only freed when the GIL is released, because only then can we be certain that they aren't used anymore by Rust (their lifetime having expired).

[ilya-lt]

ok, that makes sense, but I haven't found a useful example of how to release a pool when I'm running rust as a python library rather than invoking python from rust.

Can I release the GIL without crushing the main python thread?
How would I go about doing that?

thanks for the quick reply btw.

[davidhewitt]

@ilya-lt can you provide a more specific minimal repro? It would be useful to understand if there's a legitimate bug here.

[ilya-lt]

Hi, I worked on it a bit, refactor a single iteration into it's own function that has it's own pool and Python object that is dropped at the end of each iteration. Now the memory footprint of the python app seems consistent.

Thanks for the help, I don't think there's an actual bug here.