Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛✨EC2 machine creation: try to create at least 1 machine #5408

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Mar 4, 2024

What do these changes do?

BEFORE: The autoscaling would try to create up to EC2_INSTANCES_MAX_INSTANCES instances and fail to do so if for example the EC2 number of machines quota is exceeded.
AFTER: The autoscaling will try to create between 1 and EC2_INSTANCES_MAX_INSTANCES instances. so if the quota is exceeded, EC2 will try to create between 1 and the asked number of instances.

Related issue/s

How to test

Dev Checklist

DevOps Checklist

@sanderegg sanderegg added a:autoscaling autoscaling service in simcore's stack a:clusters-keeper labels Mar 4, 2024
@sanderegg sanderegg added this to the Schoggilebe milestone Mar 4, 2024
@sanderegg sanderegg self-assigned this Mar 4, 2024
Copy link
Contributor

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 💯

@mguidon
Copy link
Member

mguidon commented Mar 4, 2024

Is there any useful log message reported back (through the API server) if we run out of quota?

@sanderegg
Copy link
Member Author

sanderegg commented Mar 4, 2024

Is there any useful log message reported back (through the API server) if we run out of quota?

@mguidon :

Unexpected error happened when starting EC2 instance: An error occurred (VcpuLimitExceeded) when calling the RunInstances operation: You have requested more vCPU capacity than your current vCPU limit of 1000 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.

Not through the api-server though

Copy link

codecov bot commented Mar 4, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.5%. Comparing base (2aa73cc) to head (62d1eb9).
Report is 8 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #5408      +/-   ##
=========================================
  Coverage    87.5%   87.5%              
=========================================
  Files        1322     959     -363     
  Lines       54234   42418   -11816     
  Branches     1175     228     -947     
=========================================
- Hits        47479   37146   -10333     
+ Misses       6506    5223    -1283     
+ Partials      249      49     -200     
Flag Coverage Δ
integrationtests 64.1% <ø> (-1.0%) ⬇️
unittests 85.0% <100.0%> (-0.5%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
packages/aws-library/src/aws_library/ec2/client.py 100.0% <100.0%> (ø)
...e_service_autoscaling/modules/auto_scaling_core.py 94.3% <100.0%> (ø)
...imcore_service_clusters_keeper/modules/clusters.py 100.0% <ø> (ø)

... and 380 files with indirect coverage changes

@sanderegg sanderegg force-pushed the autoscaling/bugfix/reduce-number-of-machines-if-cannot-start-them branch from aeadf1d to eeb6fce Compare March 4, 2024 16:11
@sanderegg sanderegg force-pushed the autoscaling/bugfix/reduce-number-of-machines-if-cannot-start-them branch from 5add2e3 to 7e99519 Compare March 5, 2024 07:38
@sanderegg sanderegg force-pushed the autoscaling/bugfix/reduce-number-of-machines-if-cannot-start-them branch from 7e99519 to 62d1eb9 Compare March 5, 2024 09:10
Copy link

sonarqubecloud bot commented Mar 5, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@sanderegg sanderegg enabled auto-merge (squash) March 5, 2024 09:30
@sanderegg sanderegg merged commit 4197044 into ITISFoundation:master Mar 5, 2024
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:autoscaling autoscaling service in simcore's stack a:clusters-keeper
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Computational clusters: when the EC2 vCPU quota is reached the clusters does not start processing
5 participants