Orchard Core 2.0.2 - Error after some calls to the API endpoints: Status:BadRequest Error: {\"status\":\"Error: - The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.\",\"item\":null}

[MarGraz]

Hi everyone,

in my Books microservice, which is a headless Orchard Core 2.0.2 app (I need to update it to the latest), I have some API endpoints used to retrieve books. However, after a few calls from the frontend, Orchard Core stops working and throws this error:

Status:BadRequest Error: {\"status\":\"Error: - The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.\",\"item\":null}

To make it working again I need to recreate the Docker container.

After searching online, it seems this issue is caused by the reloadOnChanges option for AppSettings being set to true (see this explanation). Additionally, it appears to primarily occur in non-Windows environments (see this). I'm using a Linux Docker container to host my Books microservice.

Does anyone know how I can set reloadOnChanges to false in my Program.cs?
@Piedone @MikeAlhayek, any suggestions? 🙏

Thank you

Below is my Program.cs where I retrieve the configuration to pass it to my handler:

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddOrchardCms()
    .ConfigureServices(services =>
    {
        // ------- Here I get the appsettings configuration because I need it in the Handler part
        ConfigurationModel config = builder.Configuration.Get<ConfigurationModel>();

        if (config == null)
        {
            throw new Exception("Was impossible to retrieve the appsettings.json configuration, that are necessary to instantiate the Handler classes. Please check the appsettings file.");
        }

        services.AddHttpClient();

        #region Add Handlers

        // Register BookHandler with dependencies injected
        services.AddScoped<IBookHandler>(sp =>
        {
            return new BookHandler(
                config.ExternalServices.UsersMicroserviceBaseUrl,
                config.ExternalServices.ErpService,
                sp.GetRequiredService<ILogger<BookHandler>>());
        });

        #endregion
    });

var app = builder.Build();

if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseOrchardCore();

// Remove in the API response the header "X-Powered-By: OrchardCore"
app.UsePoweredByOrchardCore(enabled: false);

app.Run();

[MarGraz]

I’m not sure if overwriting the appsettings and reloading them in Program.cs would be a proper solution.
Might Orchard Core overwrite the reloadOnChanges parameter again afterward?

var builder = WebApplication.CreateBuilder(args);

// builder.Configuration.Sources.Clear(); // Remove current configuration sources - REMOVED because Docker was returning error with this
builder.Configuration
    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: false) // Add the appsettings.json file without reloadOnChange
    .AddJsonFile($"appsettings.{builder.Environment.EnvironmentName}.json", optional: true, reloadOnChange: false) // As last configuration load the one based on the Environment
    .AddEnvironmentVariables();

builder.Services
    .AddOrchardCms()
    .ConfigureServices(services =>
    {
        ConfigurationModel config = builder.Configuration.Get<ConfigurationModel>();

        if (config == null)
        {
            throw new Exception("Was impossible to retrieve the appsettings.json configuration, that are necessary to instantiate the Handler classes. Please check the appsettings file.");
        }

[...]
 

[MikeAlhayek]

You could try increasing the limit of the container so it's higher than 128. Are you running lots of tenants on the server where you encounter this problem? Does your application cause lots of tenant reloads? You should write the application logs to a mounted volume so they are not lost during a crash and then check out the app logs for more clues to see why there are lots of file change notifications are happening. Does this happen with a specific tenant or all tenants?

[MarGraz]

Hi @MikeAlhayek, thank you for your reply.

I will respond to your points one by one:

1. I want to avoid increasing the 128 limit because I read that it consumes more memory. First, I want to understand the root cause that leads to this limit being reached;
2. Our Orchard Core Books microservice has only one tenant, which is the default one;
3. By Application Logs, do you mean the logs visible from the Docker console?

Thank you

[weirdyang]

have you tried using IOptions instead?

   builder.Services.Configure<ConfigurationModel >(
    builder.Configuration.GetSection("ConfigurationModel"));

   // your service
   BookHandler(IOptions<BookHandler> config)

[MarGraz]

Dear @weirdyang, I still get the same error despite I have applied this mod to avoid reloading the appsettings everytime.

As asked by @MikeAlhayek I checked the Docker logs, and the only error line I see is this one: Status:BadRequest Error: {\"status\":\"Error: - The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.\",\"item\":null}. Maybe is better if I share with you a larger part of the logs?

Also I was not able to increment the user limit (I think this is not the best solution).

Do you have any idea?

Thank you

[weirdyang]

hmm sorry to hear that. just curious, do you use any of the default modules? do you see this issue with just the default modules enabled e.g email? because it does set the default value by reading from the shell configuration too, so it shoudl cause the same issue.

https://github.com/OrchardCMS/OrchardCore/blob/main/src/OrchardCore.Modules/OrchardCore.Email.Smtp/Startup.cs

also the fix recommended by msft is to set the env variable to disable reload on change:
https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/?view=aspnetcore-8.0#systemioioexception-the-configured-user-limit-128-on-the-number-of-inotify-instances-has-been-reached

[MarGraz]

@weirdyang, thank you for your reply.

No, I don't use any default modules. This is an Orchard Core microservice that just re-maps (wraps) some third-party APIs. It runs on Docker Linux, and what I noticed is that when I get that error, only the APIs stop working, but the OC backoffice continues to function properly.

In the Program.cs, I'm using an IHttpClientFactory to instantiate different kinds of HttpClient instances and call various external services. I asked ChatGPT about this 😅, and it suggested the following:

The error strongly suggests that the issue is due to the Linux limit on inotify instances or watches, which monitor file system changes. This can cause problems when the limit is too low for your application’s needs, particularly with frameworks like Orchard Core that use file watching extensively (e.g., for themes, content, or configuration changes).

The usage of HttpClientFactory itself does not directly involve inotify, but it can contribute to the problem indirectly by increasing the number of open file descriptors if not used correctly. This is because HttpClientFactory manages connections and resources like sockets, which are file descriptors in Linux. If these resources are not properly reused or cleaned up, it can lead to exceeding the limits on file descriptors, which can then impact inotify.

What do you think about this?

Thank you

[MarGraz]

Hi @weirdyang, it seems that the problem was related to my implementation of the HttpClientFactory class, where it probably wasn't able to dispose the old sockets.

Now, instead of using HttpClientFactory, I'm using a single HttpClient, and it seems to be working correctly 🤞
I will update this post if the problem reappears.

Thank you all for your support 😊

[weirdyang]

oh wow, glad it's resolved. wouldn't have expected this to be the issue!

[MikeAlhayek]

Be careful with using a custom implementation of HttpClientFactory with multi tenancy. In fact, I highly discourage doing that due to memory leaks. And it sounds like you already experienced an issue with it.

[Skrypt]

As suggested here:

https://stackoverflow.com/questions/43469400/asp-net-core-the-configured-user-limit-128-on-the-number-of-inotify-instance

You should increase the fs.inotify.max_user_instances in /etc/sysctl.conf by running this command:

echo fs.inotify.max_user_instances=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

I could repro the issue under latest Ubuntu 24.something.

This also can be fixed by using environment vars. See stackoverflow.

This is not related to Orchard Core but to ASP.NET itself.

[MarGraz]

Thank you @Skrypt for your reply.

This could be a good strategy, but first I want to understand why it happens. Here I have updated the discussion.