eloop error in old version (3.13.3) of rebar3

[tothlac]

We have an older version of rebar3 (3.13.3) . We can't upgrade easily to a newer version, and we need to release old versions of our products, so the only option on those repositories is to use this specific version of rebar3.

This problem started appearing in our Jenkins builds last week:

[2023-06-09T12:19:11.922Z] ===> error (eloop) accessing file 
/var/jenkins/workspace/Upgrade_Release/our_app/_build/prod/lib/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts

There is a soft link in trails repository pointing back to the root folder of the repo, but still we did not have any problems with these rebar3 / trails versions. Probably the problem is also related to some changes on the Jenkins server. Still I think this version of rebar3 should be also able to handle a soft link.

As discussed it with https://github.com/elbrujohalcon last week, he mentions "OTOH… Your have a different issue: I don't know why it's doing that, but rebar3/relx should not include the examples folder in the release." in this ticket: inaka/cowboy-trails#106
What do you think, is he right about that?

[tsloughter]

Pls run rebar3 with DIAGNOSTIC=1 or DEBUG=1 (I don't remember which version added DIAGNOSTIC) so we can see what step exactly it is in when this happens.

Does this only happen in tar and not release? It could be an issue in systools:make_tar. What version of OTP are you on?

[tothlac]

I think DEBUG=1 still works in 3.18.0, but in 3.13.3 version it should be DEBUG=1. I will ask the devops guys to reproduce this error with DEBUG=1 added to the command.

The problem happens with both. In the last run it happened when executing rebar3 as prod tar command, but sometimes it does not fail on the base version, then it tries to build the target version, and there the same problem occurs when running rebar3 as prod release .

Versions used:

14:17:00  + rebar3 --version
14:17:01  ===> Rebar3 detected a lock file from a newer version. It will be loaded in compatibility mode, but important information may be missing or lost. It is recommended to upgrade Rebar3.
14:17:02  rebar 3.13.3 on Erlang/OTP 23 Erts 11.1.7

I will send the output using DEBUG=1 one later.

[tothlac]

After compile it calls relx, and this is the only thing after the compile phase we have in the logs:

[2023-06-12T11:56:15.384Z] ===> Starting relx build process ...
[2023-06-12T11:56:15.384Z] ===> state(api):
[2023-06-12T11:56:15.384Z]           log: (3:debug),
[2023-06-12T11:56:15.384Z]           config file: 
[2023-06-12T11:56:15.384Z]           goals: 
[2023-06-12T11:56:15.384Z]           output_dir: /var/jenkins/workspace/Upgrade_Release/our_app/_rel
[2023-06-12T11:56:15.384Z]           lib_dirs: 
[2023-06-12T11:56:15.384Z]           providers: 
[2023-06-12T11:56:15.384Z]                tar,
[2023-06-12T11:56:15.384Z]                relup,
[2023-06-12T11:56:15.384Z]                release,
[2023-06-12T11:56:15.384Z]                resolve_release,
[2023-06-12T11:56:15.384Z]                overlay,
[2023-06-12T11:56:15.384Z]                rel_discover,
[2023-06-12T11:56:15.384Z]                app_discover,
[2023-06-12T11:56:15.384Z]           provider config values: 
[2023-06-12T11:56:15.384Z]                {overlay_vars,[]},
[2023-06-12T11:56:15.384Z]                {system_libs,true},
[2023-06-12T11:56:15.384Z]                {default_libs,true},
[2023-06-12T11:56:15.384Z] 
[2023-06-12T11:56:15.384Z] ===> Running provider app_discover
[2023-06-12T11:56:15.384Z] ===> Resolving OTP Applications from directories:
[2023-06-12T11:56:15.384Z]           /var/jenkins/workspace/Upgrade_Release/our_app/_build/prod/lib
[2023-06-12T11:56:15.384Z]           /var/jenkins/workspace/Upgrade_Release/our_app/apps
[2023-06-12T11:56:15.384Z]           /home/****/otp/23.2.3/lib
[2023-06-12T11:56:15.384Z]           /home/****/elixir/lib/elixir/lib
[2023-06-12T11:56:15.384Z]           /var/jenkins/workspace/Upgrade_Release/our_app/_build/prod/rel
[2023-06-12T11:56:17.268Z] ===> Missing beam file elf_format /home/****/otp/23.2.3/lib/hipe-4.0.1/ebin/elf_format.beam
[2023-06-12T11:56:17.268Z] ===> Provider (app_discover) failed with: {error,
[2023-06-12T11:56:17.268Z]                                            {rlx_app_discovery,
[2023-06-12T11:56:17.268Z]                                             [{accessing,
[2023-06-12T11:56:17.268Z]                                               <<"/var/jenkins/workspace/Upgrade_Release/our_app/_build/prod/lib/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/src">>,
[2023-06-12T11:56:17.268Z]                                               eloop}]}}
[2023-06-12T11:56:17.268Z] ===> error (eloop) accessing file /var/jenkins/workspace/Upgrade_Release/our_app/_build/prod/lib/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/example/_checkouts/trails/src
[2023-06-12T11:56:17.268Z] 
script returned exit code 127

So probably the question is that, why does it want to include the example folder in the release.... ?

[tothlac]

Hi. Do you have any updates on this?

[ferd]

Not really. I was out all of last week. One question I have though is that if you can't bump the version, what can be backported as a fix? Or is it that you could deal with a 3.13.x patch release, just not a newer rebar3 version since it'd require a new Erlang?

[tothlac]

Our problem is that we use quite some plugins, and have lots of dependencies (around 250). These plugins and the lots of dependencies don't compile even with 3.14.0. On newer versions of the products I've modified lots of things so it works with 3.18.0, but we have these problems on older versions of our products where only 3.13.3 can be used. Because of this a patch release applied on 3.13.3 would be the only solution.

[ferd]

Yeah, since this is a 3+ years old version and a lot changed (the relx version used in there is closer to 4 years old now with 300+ commits since then and a major version bump), I'm not quite sure how to reproduce to even start tackling this. We could drop a new patch for it for you, but identifying the problem and fixing it will require assistance, because I don't necessarily have that sort of time to just try old versions to reproduce things by luck on versions out of our support scope.

A few more questions:

• does the path on disk with the infinite chain of _checkouts actually exist after the compile step or only shows up once relx executes? Could a hook be added to check and fail there to isolate that?
• rebar_file_utils:symlink_or_copy/2 is the abstraction file we use to try and run symlinks where possible, and which likely is involved here. Would it even be acceptable to make the builds much slower but always safe by just disabling symlinks?
• If not, do you think you could instrument the symlink calls to log them somewhere, such that when the run fails, we can see what sequences of calls led to this to backtrack elsewhere into the code?

My hunch is really that somewhere in there there's a relative path that's expected to be absolute (or both a relative and an absolute path) that get to create this circular reference. The question is where is it to hunt it down, and if so, whether porting a patch such as #2228 or #2276 could be back-ported or if something else is needed.

[tothlac]

"does the path on disk with the infinite chain of _checkouts actually exist after the compile step or only shows up once relx executes? Could a hook be added to check and fail there to isolate that?"

• unfortunately it is not easy to find it out, because it happens only on the Jenkins servers and only in some cases. I can't even access them.

Right now I'm not quite sure about the problem. It happens on the Jenkins servers then only question is now that, can we use a newer version of rebar3 on these repositories. If this is the case we can close this ticket, but we have to investigate this question first.

[ferd]

Alright. If it's hard for you to even know if the paths are real, I think you'll understand it will be extremely challenging for us to do anything about it while being even further removed from any of the problem repos.

I'm fine keeping this issue open until we get more data, but I'll tag it as awaiting further updates from your part. I don't think we the maintainers will realistically do much to chase this down if we don't even know it still happens on a newer build nor the shape of the original repo causing this sort of stuff, but if we have a better scoped-down failure mode, I'd be okay cutting a new patch release of 3.13 on an older branch if it fixes it for you.