Use initialized EBS storage instead of buffer machines

[sanderegg]

Concept

Instead of keeping running EC2 instances as buffer machine, we would only keep their respective EBS volumes up.

Needed changes

AMI:

• current boot script automatically uses the largest disk it finds to mount the docker folder, needs to change to only target EBS
• needs to set an EBS disk
• currently large EC2s have a larger disk (up to 3.4TB which is free of charge and available to the users), need an equivalent EBS? when yes we need to parametrize this and define what sizes are needed

Autoscaling:

• when an EC2 is started it received the "UserData" script that "pre-pulls" docker images such as s4l that is large,
• instead of keeping X running buffer machines, it would stop them instead and terminate the ones that are above the buffer number,
  • it must wait until the pre-pulling took effect before stopping the machines and I think there is no way but SSH in order to know that and that is not very nice, investigate some other way (maybe start with a hard-coded delay)
  • it must handle the disks data (images will accumulate over time and fill the disk), how?
  • if we only stop instances, then we need to book keep the available stopped instances. what is the advantage over shutting them down?
  • we need to book keep the created volumes and ensure they do not accumulate

Eisbock

Preview Give feedback

1. Autoscaling: computational autoscaling does not get the SSM settings and fails #6227
   High Priority a:autoscaling bug +3
   
2. Autoscaling: AWS EC2 tag value cannot exceed 256 characters: issue with keepeing pre-pulled images #6230
   a:autoscaling bug +2
   
3. Autoscaling: Instrumentation is broken #6238
   a:autoscaling +1
   
4. Upgrade autoscaling AMI generation code and upgrade to latest ubuntu 22.04 #6242
   a:infra+ops +1
   
5. Autoscaling: buffer are created 2x #6250
   a:autoscaling bug +2
   
6. Autoscaling: add metrics for buffer pools #6251
   a:autoscaling +1
   
7. Autoscaling: add metrics for scaling timings (time to start machine, time to stop, time to create buffer, etc...) #6252
   a:autoscaling +1
   
8. 🐛Autoscaling: SSM client key error fix #6299
   a:autoscaling +1
   
9. Upgrade OPS script to generate EC2_INSTANCES_ALLOWED_TYPES with new buffer system #6254
   a:infra+ops +1
   
10. 🐛Autoscaling: Buffer pools metrics always showing 0 #6314
    a:autoscaling +1
    

Tasks

Preview Give feedback

1. Modify AMI boot script to optionally skip instance storages
2. Have only 1 AMI with 500GB additional EBS disk?
3. ✨AWS-library: Added interface to AWS SSM #6032
   a:aws-library +1
   
4. ✨🚨Autoscaling: Prepare Warmed EBS-backed volumes to use as buffer for machines (⚠️ Devops) 🚨 #5923
   a:autoscaling +1
   
5. ✨Autoscaling: EBS-backed buffer, label EC2 machines with prepulled images list #6097
   a:autoscaling +1
   
6. Autoscaling: Use EBS-backed buffers #6045
   a:autoscaling +1
   
7. Tune up disks (throughput + IOPS) on root drive and docker drive --> using maxed out GP3s

[matusdrobuliak66]

https://depot.dev/blog/faster-ec2-boot-time

[sanderegg]

regarding autoscaling, I see currently 2 option:

pre-create complete EC2s:

1. start buffer machine of cheap type (such as t3.medium or so)
2. ensure startup is complete (such as pre-pulling) - using SSH, AWS SSM or hard-coded time
3. stop machine (only the EBS disks are left to pay (8GB root + 500GB docker))
4. when there is need for a new machine, first check if any stopped buffer machine is available, if yes set the correct type and start it
5. when the machine is not needed anymore, instead of shutting it down, it can be passed to the buffer handler
6. we might need to ensure the disk is cleaned between runs
7. we need to monitor the EBS volumes/stopped machine and possibly remove them

only keep initialized EBS volumes

1. start buffer machine of cheap type
2. ensure startup is complete
3. shutdown machine but keep EBS volume
4. when there is need for a new machine, first check if we have free EBS volumes around, if yes use them
5. monitor EBs volumes
6. handle cleanup of volumes

[sanderegg]

User story

• Prepare background on how this new system works
• Show measurements, and how is it changing
• Show costs and how it is changing

[sanderegg]

Create a graph of responsiveness vs costs for:

• current buffer system
• new EBS buffer system