(aws-lambda-nodejs): NodejsFunction cannot remove VPC configuration from a function - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS

[a-h]

I created a project that adds a Lambda function to a VPC, so I set the VPC property of the configuration and deployed the function.

I then decided to stop the function from being part of the VPC, because the function's use case changed. I removed the vpc property, and re-deployed.

On this change, the CloudFormation template dropped into UPDATE_COMPLETE_CLEANUP_IN_PROGRESS and I couldn't make any more deployments. On looking in the event history, I could see the following:

resource sg-04ea2fedd8b4ff23a has a dependent object (Service: AmazonEC2; Status Code: 400; Error Code: 
DependencyViolation; Request ID: 38b9789c-12a6-4642-8100-6d03fcdf40a8; Proxy: null)

To resolve it, I had to manually delete the network interfaces related to the security group within the VPC. On this deletion, the CloudFormation stack became unstuck.

Reproduction Steps

    const vpc = new ec2.Vpc(this, "shared-vpc", {
      cidr: "10.0.0.0/16",
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: "public",
          subnetType: ec2.SubnetType.PUBLIC,
        },
        {
          cidrMask: 24,
          name: "private",
          subnetType: ec2.SubnetType.PRIVATE,
        },
      ],
    });

    const wildcardHandler = new lambdaNode.NodejsFunction(
      this,
      "wildcardHandler",
      {
        runtime: lambda.Runtime.NODEJS_12_X,
        entry: path.join(__dirname, "../handlers/http/wildcard.ts"),
        handler: "handler",
        memorySize: 1024,
        vpc: vpc, //TODO: First set it, then remove it.
      }
    );

What did you expect to happen?

For the function deployment to succeed completely.

What actually happened?

Subsequent deployments failed to complete with:

BackendStack failed: Error [ValidationError]: Stack:arn:aws:cloudformation:eu-west-2:xxxxxxxxxxxxxxxx:stack/BackendStack/3091e410-653d-11eb-adbd-0297c9045a12 is 
in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state and can not be updated.

Environment

• CDK CLI Version : 1.87.1 (build 9eeaa93)
• Node.js Version: v14.15.1
• OS : MacOS Bug Sur
• Language (Version): TypeScript 3.9.7

Other

I think the solution is to ensure that those network interfaces are deleted before the security group.

---

This is 🐛 Bug Report

[rubfergor]

I'm experiencing the same bug when trying to remove the VPC. The only solution to make the stack usable again is searching for the security group and remove it associated ENIs.

[eladb]

Thanks for reporting!

[hedrall]

I don't know if this will be helpful, but I did a little detailed research.

First, the ENI in problem is called Requester managed network interface , that lambda service automatically creates for each combination of conditions such as security group and lambda runtimes.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requester-managed-eni.html

The condition for this ENI to be removed is described like,,

You can't use the Amazon EC2 console to detach a network interface that is attached to a resource from another service, such as an Elastic Load Balancing load balancer, a Lambda function, a WorkSpace, or a NAT gateway. The network interfaces for those resources are deleted when the resource is deleted.

https://docs.amazonaws.cn/en_us/AWSEC2/latest/WindowsGuide/using-eni.html

In fact, when I tried it, the ENI automatically disappeared when I deleted the lambda VPC settings or deleted the Lambda function itself from the aws gui console.

The stack update completes the update first and then cleans up any items no longer needed.
So, since the VPC settings should disappear from lambda when the update is completed, the ENI is supposed to disappear automatically as described above, but in fact the ENI did not disappear during 'UPDATE_COMPLETE_CLEANUP_IN_PROGRESS'.

[corymhall]

@a-h it looks like this issue is partially fixed. When I tested today I still go the error message about the SecurityGroup failing to get deleted, but the stack did not get stuck in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS. I was able to make more updates the the stack successfully.

I think the remaining issue is with CloudFormation (it needs to successfully delete the SecurityGroup). I'll keep this issue open for tracking purposes.

[peterwoodworth]

I suggest to anyone running into this issue to report this to the CloudFormation Coverage Roadmap if it hasn't been reported there already.