Memory leak in NeXus/NOMAD HDF5 file parsing pipeline?

[mkuehbach]

No description provided.

[mkuehbach]

Observations from Johannes Dallmann
...
I work with API calls made with a python script and the request package. I tried both creating an upload per file as well as using a PUT request to add files to an existing upload, both with the same result. I always make sure a file is fully processed before uploading the next file, either via checking the upload status (in a 3s loop, as suggested in the docs) or the 'wait_for_processing' parameter in the PUT request.
The .nxs files in question are converted from one of our laboratory instruments and all have 1.3MB. Even 200 such are much smaller than the 8GB RAM and 4GB swap memory we work with on the server. We do hit the main memory limit before utilizing the swap memory, but it never resets and eventually no swap memory is left and the freeze happens. At this point, we were not able to do anything but manually restart the computer via the power button, it does not respond to anything.
We are running Oasis version 1.3.13 and pynxtools version 0.9.3
I did all my testing via python script and requests. I can test drag-and-drop tomorrow and get back to you. So far I used drag and drop only to check whether the parser is running successful or not. Our goal is to build an automated data pipline from the instrument(s) to our Oasis.
After a freeze an upload is usually stuck in the processing state, which i then reset via the admin console. The server is responsive in this state.
I tried to use the ellipsometry example to exclude problems with our NeXus files, although the parser always matches and runs successfully for our files. I now see that there are a bunch warnings (40+) in the logs after the parser has run both for our nexus files as well as for the ellipsometry file. 'WARNING: pynxtools.nomad.entrypoints:nexus_parser | error while setting field' (or attribute). However, no errors.
...

[rettigl]

I observed similar behavior. Parsing 140 Nexus files raised the swap memory to ~12 GBytes, reparsing them egain to ~22 GB. This memory seems to be stuck in the appworker processes. This was with 8 cores/appworkers and 8GB Ram.