CI: Free more disk space with free-disk-space script

[whiteio]

Attempting to move the free-disk-space github action into the repo as it isn't currently maintained. This involves moving the script within free-disk-space into the repo.

I'm opening this as a draft to verify that the saved space is the same as the original action. Once that is verified, I'll look into integrating these two PRs to improve it:

• perf: speed up deletings jlumbroso/free-disk-space#26
• Remove snapd and microsoft-edge-stable jlumbroso/free-disk-space#24

Changes so far:

• Took the script from the free-disk-space repo and moved it into src/ci/scripts/
• Removed unnecessary if statements which were used to check for github action inputs which we don't need to check
• Swapped to using this script within the rust repo, instead of using the jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be GitHub action

See this issue for more info: rust-lang/infra-team#183

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Mark-Simulacrum (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[whiteio]

Confirmed that the script behaves the same when moved to the repo, still trying to figure out if jlumbroso/free-disk-space#26 makes a tangible difference.

Currently, free disk space can clean up 36GB with the changes up to this point, so a 10GB improvement so far.

[larziwau]

e447a4f (#134151)
does this commit really make a difference?

[whiteio]

e447a4f (#134151) does this commit really make a difference?

Nope, will be reverting that one, jlumbroso/free-disk-space#26 also doesn't seem to make a noticeable improvement, especially compared to the example in that PR

[marcoieni]

Hey @whiteio is there a milestone that you reached that we can merge?

E.g. getting rid of the GH Action and moving the scripts here is already something we are willing to merge. We can do the optimizations in other PRs 👍

[whiteio]

Hey @marcoieni sorry about the delay, I think it's in a good place now. In regards to the mingw-check-tidy failure due to line length greater than 100 chars, what do you think would be the correct approach to resolve these?

E.g. lines such as this:

sudo apt-get remove -y '^dotnet-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^dotnet-.*' --fix-missing] failed to complete successfully. Proceeding..."

[rust-log-analyzer]

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
ERROR: failed to download llvm from ci

    HELP: There could be two reasons behind this:
        1) The host triple is not supported for `download-ci-llvm`.
        2) Old builds get deleted after a certain time.
    HELP: In either case, disable `download-ci-llvm` in your config.toml:
    [llvm]
    download-ci-llvm = false
    
Build completed unsuccessfully in 0:00:22

[marcoieni]

Suggested change

	sudo apt-get remove -y '^aspnetcore-.*' || echo "::warning::The command [sudo apt-get remove -y '^aspnetcore-.*'] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y '^dotnet-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^dotnet-.*' --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y '^llvm-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^llvm-.*' --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y 'php.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y 'php.*' --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y '^mongodb-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mongodb-.*' --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y '^mysql-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing || echo "::warning::The command [sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y google-cloud-sdk --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-sdk --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y google-cloud-cli --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-cli --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y microsoft-edge-stable --fix-missing || echo "::debug::The command [sudo apt-get remove -y microsoft-edge-stable --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get remove -y snapd --fix-missing || echo "::debug::The command [sudo apt-get remove -y snapd --fix-missing] failed to complete successfully. Proceeding..."
	sudo apt-get autoremove -y || echo "::warning::The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding..."
	sudo apt-get clean || echo "::warning::The command [sudo apt-get clean] failed to complete successfully. Proceeding..."
	PACKAGES_TO_REMOVE=(
	'^aspnetcore-.*'
	'^dotnet-.*'
	'^llvm-.*'
	'php.*'
	'^mongodb-.*'
	'^mysql-.*'
	'azure-cli'
	'google-chrome-stable'
	'firefox'
	'powershell'
	'mono-devel'
	'libgl1-mesa-dri'
	'google-cloud-sdk'
	'google-cloud-cli'
	'microsoft-edge-stable'
	'snapd'
	)
	for pkg in "${PACKAGES_TO_REMOVE[@]}"; do
	sudo apt-get remove -y "$pkg" --fix-missing || echo "::warning::Failed to remove $pkg"
	done
	sudo apt-get autoremove -y || echo "::warning::The command [sudo apt-get autoremove -y] failed"
	sudo apt-get clean || echo "::warning::The command [sudo apt-get clean] failed failed"

We can simplify the code like this. What do you think?
The difference is that '^aspnetcore-.*' didn't have --fix-missing but I'm not sure why, so we can try to add it if you agree 👍

[marcoieni]

Let's write at the beginning of the file something like script is inspired by https://github.com/jlumbroso/free-disk-space, to give the author the credit they deserve 👍

[marcoieni]

why do you do this also on line 186?

[marcoieni]

this pattern is repeated over and over.

Can we extract it into a function?

Something like this:

removeDir() {
    dir=${1}

    BEFORE=$(getAvailableSpace)

    sudo rm -rf "$dir" || true

    AFTER=$(getAvailableSpace)
    SAVED=$((AFTER-BEFORE))
    printSavedSpace $SAVED "$dir"
}

so in the code you only do removeDir /usr/local/share/powershell

Consider also using an array as I suggested for the packages if you think it would look nice!

[marcoieni]

Suggested change

	# Remove tool cache
	# REF: https://github.com/actions/virtual-environments/issues/2875#issuecomment-1163392159
	BEFORE=$(getAvailableSpace)
	sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
	AFTER=$(getAvailableSpace)
	SAVED=$((AFTER-BEFORE))
	printSavedSpace $SAVED "Tool cache"

removing this was false by default, so I wouldn't do this here. We can evaluate it later and remove it in another PR if we can 👍

[marcoieni]

dh is a typo. See https://github.com/jlumbroso/free-disk-space/pull/37/files

[marcoieni]

Suggested change

	echo "$ dh -a /"
	echo ""
	df -a /
	echo "$ dh -a"
	echo ""
	df -a

imo dh -h / is enough.

[marcoieni]

Hey, unfortunately this is becoming blocking for things like rust-lang/infra-team#189 and #135774

Sorry if we couldn't wait longer 🙏

I'll create a new PR based on your work, thank you so much for taking the time to kick start this.

I'll indicate you as a co-author in my commit

[bors]

☔ The latest upstream changes (presumably #135959) made this pull request unmergeable. Please resolve the merge conflicts.

[whiteio]

@marcoieni Sorry about all the delays. Thank you for taking it over. I'll close this as you're change is merged. Sorry again!