(autoscaling): maxInstanceLifetime can't be left as is.

[bekir-ozturk]

Describe the bug

We use maxIinstanceLifetime property on our AutoScalingGroups to rotate old instances.

aws-cdk/packages/aws-cdk-lib/aws-autoscaling/lib/auto-scaling-group.ts

Line 247 in 36fd79d

readonly maxInstanceLifetime?: Duration;

We also want to use the same property to temporarily stop rotations during the day by setting it to zero. The problem we are running into today is that if a CDK deployment kicks in while the maxInstanceLifetime value was temporarily modified, the deployment will set the value back to the original that was configured in the CFN stack.

The initial idea was to leave the property 'unspecified' on the CDK side. The documentation mentions that setting the max instance lifetime to zero will clear the old value; so we hoped that having it 'unspecified' would keep the previous value. After some testing, it appears that leaving it unset will behave the same as setting it to zero.

We currently have no way of keeping the old value during a CDK deployment.

Expected Behavior

If maxInstanceLifetime is not specified in the CDK project, the property in the ASG should retain its old value after a CDK deployment.

Current Behavior

Even after leaving maxInstanceLifetime undefined in the CDK project, the deployment sets the property on the ASG to zero.

Reproduction Steps

• Create an ASG with CDK without specifying maxInstanceLifetime property. Deploy.
• Login to AWS console and update the maxInstanceLifetime value to 20 days.
• Without making any code changes, re-deploy the same CDK project.
• Login to AWS console and observe that the maxInstanceLifetime value is removed and is no longer 20 days.

Possible Solution

Solution 1

Change how an unspecified value is interpreted when the changes to a CloudFormation stack is being applied. If the value is unspecified, keep the old max instance lifetime as is.
This is probably not a reasonable solution since it would silently change the behavior of all the ASGs that are deployed through CDK. If the users are relying on the old behavior, their systems may fail.

Solution 2

Update the type of maxInstanceLifetime from (undefined | Duration) to (undefined | Unchanged | Duration).
This means that anyone updating to the new version will continue to get the same behavior, but it will be possible to use the new value to keep the value unchanged.

enum MaxInstanceLifetime {
    Unchanged
}

interface CommonAutoScalingGroupProps {
    ...
    readonly maxInstanceLifetime: undefined | MaxInstanceLifetime | Duration;
    ...
}

Additional Information/Context

No response

CDK CLI Version

2.131.0 (build 92b912d)

Framework Version

No response

Node.js Version

18

OS

AL2

Language

TypeScript

Language Version

No response

Other information

No response

[khushail]

Hi @bekir-ozturk , thanks for reaching out. I am not able to comprehend the "unspecified" value in the docs as its clearly mentioned in the cloudformation docs -

The default is null. If specified, the value must be either 0 or a number equal to or greater than 86,400 seconds (1 day).

Also I see - "You must specify a value of at least 86,400 seconds (one day). To clear a previously set value, specify a new value of 0. "

Apologies for my oversightedness but It does not state anywhere that keeping it "unspecified" will retain the previous value. There are some limitations mentioned as well.

With that being said, I assume it would be helpful to clarify the what would be the case of "unspecified" value mean in the docs, as proposed but would need considerations from the team.

Please feel free to clarify/share more insights if I have misinterpreted docs here!

[bekir-ozturk]

Hi @khushail,
Thanks for the swift reply!

It does not state anywhere that keeping it "unspecified" will retain the previous value.

This is true and I'm not claiming that the documents are misleading here.

My question is: there doesn't seem to be support for retaining the old value for maxInstanceLifetime during a CDK deployment. Can it be added?

[nmussy]

I don't think there's a way to change this behavior without a breaking change, either by changing the property type or the @default value interpretation.

The alternative would be to add a new property along the lines of

/**
 * If false, the {@link maxInstanceLifetime} duration will not be overwritten
 * when updating the AutoScalingGroup.
 *
 * @default true
 */
readonly maxInstanceLifetimeOverwrite?: boolean;

[pahud]

This is my initial deployment:

    new autoscaling.AutoScalingGroup(this, 'ASG', {
      vpc: getDefaultVpc(this),  
      machineImage: ec2.MachineImage.latestAmazonLinux2023(),
      instanceType: new ec2.InstanceType('t3.large'),
    })

Now if I update the maxInstanceLifetime value to 20 days from the console, this will cause the drift and CFN would have no idea about the change, which is generally not recommended.

Now if you run cdk diff, nothing will return and no cloudformation change set will be created because CFN doesn't think the stack has changed and if you run cdk deploy, nothing will change.

It won't be possible for CDK or CFN to persist the value you update from the console and this is generally an anti-pattern. You are recommended to manage any values with CDK in a config file or parameter store.

[pahud]

My question is: there doesn't seem to be support for retaining the old value for maxInstanceLifetime during a CDK deployment. Can it be added?

It depends on how you specify the old value.

If you specify that in your initial CDK deployment and update that in your 2nd deployment. You can always fall back that value by modifying the prop value back. However, if you specify that initially from the console, CDN/CFN has no idea about that change at all as it doesn't think the value from the template has changed hence nothing will happen.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[bekir-ozturk]

Hi @pahud,
I understand how this is a bad practice now. I think I'll use a lambda to listen for deployment events and reset the value back to the desired after it is overwritten by the cdk deployment.

Thanks!

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.