Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: worker: Convert DC_[SectorSize]_[ResourceRestriction] if set #11224

Merged
merged 2 commits into from
Sep 5, 2023

Conversation

rjan90
Copy link
Contributor

@rjan90 rjan90 commented Aug 30, 2023

Related Issues

Closes #11223

Proposed Changes

Check if DC_[SectorSize]_[ResourceRestriction] enviroment variable is set and convert it to DC_[ResourceRestriction] to make sure its enforced as DC is a sector-size independent job.

Testing

  • export DC_512MiB_MAX_CONCURRENT=1
  • export DC_512MiB_MAX_PARALLELISM=1

On startup of the worker, check that the env-variables are being converted:

lotus-worker run --data-cid=true --addpiece=false --precommit1=false --unseal=false --precommit2=false --commit=false --replica-update=false --prove-replica-update2=false --regen-sector-key=false --sector-download=false 
2023-08-30T05:18:50.118-0400    INFO    main    lotus-worker/main.go:297        Starting lotus worker
2023-08-30T05:18:50.118-0400    WARN    main    lotus-worker/main.go:343        Converted DC_512MiB_MAX_PARALLELISM to DC_MAX_PARALLELISM, because DC is a sector-size independent job
2023-08-30T05:18:50.118-0400    WARN    main    lotus-worker/main.go:343        Converted DC_512MiB_MAX_CONCURRENT to DC_MAX_CONCURRENT, because DC is a sector-size independent job
2023-08-30T05:18:50.121-0400    INFO    main    lotus-worker/main.go:385        Remote version 1.23.3-rc2+butterflynet+git.1eec81622.dirty+api1.5.0

Scheduled a couple of DC-tasks with lotus-miner sealing data-cid 10gb-filecoin-payload.bin and checked that DC-limit is being enforced:

lotus-miner sealing workers
Worker 5d9c766a-abf0-4815-a4f4-c42cc3d38896, host Ubuntu-2004-focal-amd64-base
        TASK: UNK(1) DC(1/1) 
        CPU:  [|||||                                                           ] 1/12 core(s) in use
        RAM:  [|||||                                                           ] 8% 9.09 GiB/62.78 GiB
        VMEM: [|||||                                                           ] 8% 5.09 GiB/62.78 GiB

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

Check if DC_[SectorSize]_[ResourceRestriction] is set and convert to DC_[ResourceRestriction] to make sure its enforced by the scheduler.
@rjan90 rjan90 requested a review from a team as a code owner August 30, 2023 09:20
Make lint-happy
@rjan90 rjan90 changed the title fix: worker: Check if DC_[SectorSize]_[ResourceRestriction] is set fix: worker: Convert DC_[SectorSize]_[ResourceRestriction] if set Aug 30, 2023
Copy link
Contributor

@fridrik01 fridrik01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, and nice test plan!

@rjan90 rjan90 merged commit 37b8afd into master Sep 5, 2023
@rjan90 rjan90 deleted the fix/DataCid-env-check branch September 5, 2023 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DataCID (DC) resource restrictions ignored due to sector size independence
2 participants