Startup of dynamic sidecar

[sanderegg]

The dynamic sidecar in autoscaled nodes appears to take 10-15 seconds to start.
We want to understand why and if we can change that.

[YuryHrytsuk]

Hot buffers on AWS Master trial #1

dy-sidecar was healthy after 8 seconds
The container was created in 1-2 seconds, and rest 6 seconds it took for healthcheck to pass (container was in running state)

[YuryHrytsuk]

War buffer on aws master trial 1

node ready at 15:46

container created at 16:16
container started at 16:17

container is running / healthy at 16:25

[YuryHrytsuk]

Issue similar to what we have once we undrain the node and occasionally it is stuck doing nothing for ~30 seconds

• Swarm drain: Bulk sync to node timed out moby/moby#41524
• Docker swarm hangs 30 seconds on bulk sync to node XXX timed out (no container pops up on the node during this delay) moby/moby#47396

[sanderegg]

Here is a link to something that could be linked to it:
moby/moby#39518

Interesting link to the check_config.sh script: https://blog.hypriot.com/post/verify-kernel-container-compatibility/
As proposed in the linked issue, I ran

wget https://github.com/moby/moby/raw/master/contrib/check-config.sh
chmod +x check-config.sh
sudo ./check-config.sh

This returned everything OK but:

- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)

this is optional, so probably ok

and

 - "zfs":
    - /dev/zfs: present
    - zfs command: missing
    - zpool command: missing

[sanderegg]

Changes to the AWS AMI were undertaken to follow moby/moby#39518, the AMIs are deployed and the error messages disappeared. Whether this has an influence on the Bulk Sync error is not clear yet

[sanderegg]

considered as done until further notice or the issue with bulk error arises again.