fix(core): exponential backoff retry on cache put fail

[mike-fam]

This PR fixes the same issue as #7349, but implements a more resilient and less aggressive retry strategy. I regularly get cache errors on my pipeline, because I share the same local cache (i.e. EFS) between multiple EC2 agents. This is hard to reproduce since it's a race condition on my custom set-up. Given that this change won't affect the normal performance (since it's just an edge case), I think this change would be a nice-to-have.

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 41a4009. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 6 targets

• nx affected --targets=e2e,e2e-ci --base=512786f62c04cd9db22b4f0a66277b6716831f09 --head=41a4009f3d6a769ab305375ec2a481d87e4e55eb --parallel=1
• nx affected --targets=lint,test,build --base=512786f62c04cd9db22b4f0a66277b6716831f09 --head=41a4009f3d6a769ab305375ec2a481d87e4e55eb --parallel=3
• nx run-many -t check-imports check-commit check-lock-files check-codeowners documentation --parallel=1 --no-dte
• nx-cloud record -- nx format:check --base=512786f62c04cd9db22b4f0a66277b6716831f09 --head=41a4009f3d6a769ab305375ec2a481d87e4e55eb
• nx documentation --no-dte
• nx affected -t e2e-macos --parallel=1 --base=512786f62c04cd9db22b4f0a66277b6716831f09 --head=41a4009f3d6a769ab305375ec2a481d87e4e55eb

---

Sent with 💌 from NxCloud.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
nx-dev	⬜️ Ignored (Inspect)	Visit Preview	Mar 22, 2024 8:26pm

[mike-fam]

Hi @FrozenPandaz, could you please let me know if you have time to review this change, or if there's something else I need to do before getting this PR reviewed? Thanks.

[FrozenPandaz]

The changes look good. Currently, it would go a very long time before actually giving up. I decreased the amount of maximum time it will wait.

[github-actions]

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.