Race condition in concurrent pub git clone #3404

acoutts · 2022-04-27T23:03:56Z

Environment

pub version or flutter pub version:

Flutter 2.13.0-0.0.pre.366 • channel master • https://github.com/flutter/flutter.git
Framework • revision 0b97874895 (4 weeks ago) • 2022-04-01 21:20:01 -0400
Engine • revision 801d8c530f
Tools • Dart 2.17.0 (build 2.17.0-265.0.dev) • DevTools 2.12.1

OS version: macOS 12.2.1
Are you using the Chinese community mirror or a corporate firewall? No

Problem

I have a monorepo with several packages, using melos. I upgraded to melos 2.0 which introduced concurrent pub getting. The problem is several of my packages share the same external git dependency and the very first time I run melos bootstrap in a fresh repository it throws the following error:

Could not find a file named "pubspec.yaml" in "/builds/M5hecXxM/3/bottle/pay/app/.pub-cache/git/mockito-aa841a3c3762afab2c31486677015ff0a9abc2cf".
BootstrapException: Failed to install.: common at /builds/M5hecXxM/3/bottle/pay/app/packages/common.

When i run melos bootstrap a second time, it finishes without an error. It seems like there is some kind of race condition with the pub command related to this shared git dependency which causes the pub gets to fail the first time but work the second time.

Expected behavior

Should be able to pub get in parallel for git dependencies.

Actual behavior

I have to do it twice to succeed.

--trace output

<copy/paste the full output>

The text was updated successfully, but these errors were encountered:

jonasfj · 2022-05-06T14:24:43Z

It's not entirely clear what is going wrong here. If someone wants to create a minimal example using only dart pub get and a clean PUB_CACHE folder that would be preferable. Or even better make a test case for pub that reproduces the error with some confidence (yeah, it's probably somewhat intermittent).

If we know for sure what bits fail, we can try to make a solution.

One option is cloning to a temporary folder and moving atomically, but this has a tendency to cause problems for users on Windows anti-virus software installed.
So if it's possible that there is some locking mechanism in git we're not using correctly, then we could try to use that.

It also possible we should make a temporary folder + move and copy if moving fails.

Gustl22 · 2022-05-17T21:52:51Z

FYI @jonasfj don't know if it helps but I can reliably reproduce it with this scenario on this example. But only with flutter pub get, not with dart pub get.

sigurdm · 2022-06-09T12:27:12Z

Think I tracked it down to two issues:

#3455 (dart run does not refresh dependencies if the PUB_CACHE has changed)
flutter/flutter#105697 (bin/flutter and bin/dart has different pub caches).

ivan-dubovenko-m10 · 2023-05-04T04:45:05Z

flutter clean
pub cache clean
dart pub global activate melos

command:
  bootstrap:
    runPubGetInParallel: false

Does not resolve the issue.

Could not find a file named "pubspec.yaml" in ".../mobile-flutter/packages/features/request_funds".

After pulling repo, the package /features/request_funds was deleted from the project. But melos/flutter/dart still looking for it.

melos --version
3.0.1

dart --version
Dart SDK version: 2.19.5 (stable)

flutter --version
Flutter 3.7.8 • channel stable

flutter pub get fails:

flutter pub get
Running "flutter pub get" in mobile-flutter...
Warning: pubspec.yaml has overrides from pubspec_overrides.yaml
Resolving dependencies... 
Could not find a file named "pubspec.yaml" in "...mobile-flutter/packages/features/request_funds".
pub get failed

ivan-dubovenko-m10 · 2023-05-04T04:56:56Z

#3404 (comment)

Deleting pubspec_overrides in project root resolve it.

samdeane · 2023-10-31T11:11:35Z

I am seeing what appears to be a similar race condition. We have one external git dependency, used by several packages.

When we run melos bootstrap it fails on CI with one of a number of errors, all of which appear to be git related, and likely caused by simultaneous access. Sometimes it's a failure to get a lock, sometimes a cache issue.

Example Failure

Versions

> flutter --version                                                                           (deploy/internal/client)
Flutter 3.13.6 • channel [user-branch] • unknown source
Framework • revision ead455963c (5 weeks ago) • 2023-09-26 18:28:17 -0700
Engine • revision a794cf2681
Tools • Dart 3.1.3 • DevTools 2.25.0

> dart run melos --version                                                                    (deploy/internal/client)
3.2.0

samdeane · 2023-10-31T11:38:20Z

I hadn't discovered the runPubGetInParallel setting - that might do the trick. I will report back...

sigurdm · 2023-11-06T14:57:24Z

It's not entirely clear what is going wrong here.

I think it is pretty clear what is going on. Two pub-get processes discover that the same git-dependency is not cloned into the cache, and the both initiate the cloning process, and one steps the other on the toes.

I believe the following script reproduces very reliably:

#!/bin/bash
tmpdir=$(mktemp -d 2>/dev/null || mktemp -d -t 'exp')
mkdir "$tmpdir"/cache

# trap 'rm -rf $tmpdir' EXIT
for i in $(seq 1 2 3)
do
   mkdir "$tmpdir"/project"$i"
   echo "{'name':'p$i','environment':{'sdk':'^3.0.0'},'dependencies':{'p':{'git':'../dep'}}}" > "$tmpdir"/project"$i"/pubspec.yaml
done

mkdir "$tmpdir"/dep
git -C "$tmpdir"/dep init
echo "{'name':'p','environment':{'sdk':'^3.0.0'}}" > "$tmpdir/dep/pubspec.yaml"
git -C "$tmpdir"/dep add .
git -C "$tmpdir"/dep commit -am "pubspec.yaml"

for i in $(seq 1 2 3)
do
   PUB_CACHE=$"$tmpdir"/cache dart pub2 -C"$tmpdir"/project"$i" get &
done

wait

…tion When running the command "flutter pub get," the plugins are stored in the pub-cache directory along with their sha256sum hashes. The default location of the pub-cache directory is current $(HOST_DIR)/share/flutter/sdk/.pub-cache, which is not an acceptable choice by default because every plugin is re-downloaded during every build of a flutter application either during a new build or when building with the per-package-directory option enabled. Furthermore, keeping the pub-cache in its current location prevents users from committing the pub-cache directory to git for faster rebuilds of a Buildroot-based system, as users cannot store the pub-cache for later use. To fix the above issue completely, the following two changes must occur: - Change the hard-coded Flutter pub-cache location to $(DL_DIR)/br-flutter-pub-cache. - Remove the `rm -rf $(HOST_FLUTTER_SDK_BIN_SDK)/.pub-cache` and the associated comment about why the build system removes the .pub-cache directory. After further research, the help text of the precache command reads, "Populate the Flutter tool's cache of binary artifacts." The current reasoning listed in the comments is not accurate for a the following reasons: 1. We do not want to remove their directory if users already have a pub cache they have symlinked to. 2. If the flutter-sdk-bin package previously set up the pub-cache, then the pub-cache directory is set up with the options we want, and there is no reason to remove the pub-cache directory. Note that upstream considers it safe to have multiple instances of readers/writers to the pub cache concurently, which is a situation that can happen when two flutter-based pacakges are going to be built in parallel. There have been reports upstream [0] [1] [2] where concurrency was an issue, and they have always been fixed [3] [4] (or considered fixed already). So we can assune that, if the conncurrent ccess to the shared pub-cache causes issues, that will be an upstream bug that will get solved. If that turns out to be an unsolvable problem, we'll still have the option to run the pub-get commands under flock. [0] dart-lang/pub#1178 [1] dart-lang/pub#3404 [2] dart-lang/pub#3420 [3] dart-lang/pub#1178 (comment) [4] dart-lang/pub#1178 (comment) Signed-off-by: Adam Duskett <[email protected]> [[email protected]: add blurb about concurrent access] Signed-off-by: Yann E. MORIN <[email protected]>

…tion When running the command "flutter pub get," the plugins are stored in the pub-cache directory along with their sha256sum hashes. The default location of the pub-cache directory is current $(HOST_DIR)/share/flutter/sdk/.pub-cache, which is not an acceptable choice by default because every plugin is re-downloaded during every build of a flutter application either during a new build or when building with the per-package-directory option enabled. Furthermore, keeping the pub-cache in its current location prevents users from committing the pub-cache directory to git for faster rebuilds of a Buildroot-based system, as users cannot store the pub-cache for later use. To fix the above issue completely, the following two changes must occur: - Change the hard-coded Flutter pub-cache location to $(DL_DIR)/br-flutter-pub-cache. - Remove the `rm -rf $(HOST_FLUTTER_SDK_BIN_SDK)/.pub-cache` and the associated comment about why the build system removes the .pub-cache directory. After further research, the help text of the precache command reads, "Populate the Flutter tool's cache of binary artifacts." The current reasoning listed in the comments is not accurate for a the following reasons: 1. We do not want to remove their directory if users already have a pub cache they have symlinked to. 2. If the flutter-sdk-bin package previously set up the pub-cache, then the pub-cache directory is set up with the options we want, and there is no reason to remove the pub-cache directory. Note that upstream considers it safe to have multiple instances of readers/writers to the pub cache concurently, which is a situation that can happen when two flutter-based pacakges are going to be built in parallel. There have been reports upstream [0] [1] [2] where concurrency was an issue, and they have always been fixed [3] [4] (or considered fixed already). So we can assune that, if the conncurrent ccess to the shared pub-cache causes issues, that will be an upstream bug that will get solved. If that turns out to be an unsolvable problem, we'll still have the option to run the pub-get commands under flock. [0] dart-lang/pub#1178 [1] dart-lang/pub#3404 [2] dart-lang/pub#3420 [3] dart-lang/pub#1178 (comment) [4] dart-lang/pub#1178 (comment) Signed-off-by: Adam Duskett <[email protected]> [[email protected]: add blurb about concurrent access] Signed-off-by: Yann E. MORIN <[email protected]> (cherry picked from commit 3780925) Signed-off-by: Peter Korsgaard <[email protected]>

acoutts mentioned this issue Apr 27, 2022

feat: add support for Dart/Flutter SDK at custom sdkPath invertase/melos#284

Closed

7 tasks

blaugold mentioned this issue Apr 28, 2022

feat: add command/bootstrap/runPubGetInParallel melos.yaml option invertase/melos#285

Merged

7 tasks

sigurdm added the type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) label Apr 29, 2022

blaugold mentioned this issue May 4, 2022

fix: Could not find a file named "pubspec.yaml" invertase/melos#289

Closed

1 task

Gustl22 mentioned this issue May 17, 2022

[windows][macOS][git] Calling flutter pub get fails on CI with multiple projects with same dependency flutter/flutter#103981

Closed

thipokch mentioned this issue Jan 17, 2023

Build Issues thipokch/unself#1

Open

11 tasks

sigurdm changed the title ~~Race condition in concurrent pub requests in Dart 2.17~~ Race condition in concurrent pub git clone Jan 22, 2024

sigurdm mentioned this issue Jan 22, 2024

Use atomic moves when cloning git repos #4120

Merged

sigurdm closed this as completed in #4120 Jan 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Race condition in concurrent pub git clone #3404

Race condition in concurrent pub git clone #3404

acoutts commented Apr 27, 2022

jonasfj commented May 6, 2022

Gustl22 commented May 17, 2022 •

edited

Loading

sigurdm commented Jun 9, 2022

ivan-dubovenko-m10 commented May 4, 2023 •

edited

Loading

ivan-dubovenko-m10 commented May 4, 2023

samdeane commented Oct 31, 2023 •

edited

Loading

samdeane commented Oct 31, 2023

sigurdm commented Nov 6, 2023 •

edited

Loading

Race condition in concurrent pub git clone #3404

Race condition in concurrent pub git clone #3404

Comments

acoutts commented Apr 27, 2022

Environment

Problem

Expected behavior

Actual behavior

--trace output

jonasfj commented May 6, 2022

Gustl22 commented May 17, 2022 • edited Loading

sigurdm commented Jun 9, 2022

ivan-dubovenko-m10 commented May 4, 2023 • edited Loading

ivan-dubovenko-m10 commented May 4, 2023

samdeane commented Oct 31, 2023 • edited Loading

Example Failure

Versions

samdeane commented Oct 31, 2023

sigurdm commented Nov 6, 2023 • edited Loading

Gustl22 commented May 17, 2022 •

edited

Loading

ivan-dubovenko-m10 commented May 4, 2023 •

edited

Loading

samdeane commented Oct 31, 2023 •

edited

Loading

sigurdm commented Nov 6, 2023 •

edited

Loading