Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): pass CloudFormation parameters to "cdk deploy" #6385

Merged
merged 16 commits into from Mar 4, 2020
Merged

feat(cli): pass CloudFormation parameters to "cdk deploy" #6385

merged 16 commits into from Mar 4, 2020

Conversation

ghost
Copy link

@ghost ghost commented Feb 20, 2020

This commit closes #1237

I have added the command line option --parameters, -P which allows for
adding CloudFormation parameters at deploy time.

The functionality for deploying with parameters was already present,
but I have just added a way for the command line to pass it to the
deployment code.

The motivation for adding this functionality has been that we have
a setup where we share an account with access to CloudFormation, which
is why we do not want to have some fields in plain text, while they do
not have access to other services, which did not support secrets within
the fields we want to keep secret. This allows us to have parameters
that can be kept secret from the templates, while also having parameters
with NoEcho.

I have verified this functionality by both deploying a cdk project
with a CfnParameter from @aws-cdk/core, and an original CloudFormation
template with parameters.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

This commit closes #1237

I have added the command line option --parameters, -P which allows for
adding CloudFormation parameters at deploy time.

The functionality for deploying with parameters was already present,
but I have just added a way for the command line to pass it to the
deployment code.

The motivation for adding this functionality has been that we have
a setup where we share an account with access to CloudFormation, which
is why we do not want to have some fields in plain text, while they do
not have access to other services, which did not support secrets within
the fields we want to keep secret. This allows us to have parameters
that can be kept secret from the templates, while also having parameters
with NoEcho.

I have verified this functionality by both deploying a cdk project
with a CfnParameter from @aws-cdk/core, and an original CloudFormation
template with parameters.
@ghost ghost requested review from skinny85 and SomayaB February 20, 2020 13:34
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 3223926
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: b195674
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for starting this PR - I'd also gotten started on a PR to implement this functionality, but I'm happy to collaborate on this one to sort out the remaining items.

  • how does this work with wild card deploys (i.e. cdk deploy "*") - would that result in every stack having duplicated parameters?
  • does cdk diff account for parameters? / any considerations there?
  • are there any additional considerations for nested stacks
  • We should update the README for the module
  • We should add integration tests

@shivlaks shivlaks changed the title feat(aws-cdk): pass cloudformation parameters to "cdk deploy" feat(cli): pass cloudformation parameters to "cdk deploy" Feb 20, 2020
@shivlaks shivlaks changed the title feat(cli): pass cloudformation parameters to "cdk deploy" feat(cli): pass CloudFormation parameters to "cdk deploy" Feb 20, 2020
@shivlaks shivlaks self-assigned this Feb 20, 2020
@ghost
Copy link
Author

ghost commented Feb 21, 2020

Thank you for your feedback!

  • Currently looking at the cdk diff, it does not appear to account for parameters.

  • I have not tried nested stacks yet, as I do not have a working example. (Still pretty new to cdk)

  • When looking at the README for the cli module I did not find any section where other cli parameters are mentioned, which is why I have omitted it for now. But if you know where the documentation would be fitting, I would love to write some.

  • I did not know you could do wild card deploy, but after testing it, it looks like this implementation does not work, as it tries to duplicate the parameters. Do you have any preference to the syntax for choosing the stack that a parameter belongs to? My suggestion would be --parameters "StackName:Key=Value"

@mergify mergify bot dismissed shivlaks’s stale review February 21, 2020 08:26

Pull request has been modified.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: d6c89af
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

This changes the syntax of the --parameter parameter, so that it is now
possible to specify which stack a parameter is meant for, i.e. of you
want a parameter to apply to `MyStack`, then you would pass it as
--parameters MyStack:key=value
While still being able to pass parameters to all stacks by omitting the
stack name, i.e. --paramters key=value

This is being added to support wild card deployments, and deployments
with multiple stacks that do not neccessary contain the same parameters.

I have verified these changes by deploying a multistack cdk project,
where the stacks contains different parameters, and some shared
parameters.
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 88658d7
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

I have added some integration tests of both wildcard and single
deployments with parameters. While also fixing a bug found while
testing.
@ghost
Copy link
Author

ghost commented Feb 21, 2020

I have added some tests and implemented the syntax for specifying which stack a parameter belongs to.
Please let me know what would be the next step for this feature, as I will continue working on this tomorrow.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: dc5a0f3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@john-tipper
Copy link

What’s needed to get this over the line please? It looks like we’re pretty close, no? We’re extremely keen to see this merged as it’ll open up using CDK for us as this particular issue is blocking us. Please let me know if I can help.

@ghost
Copy link
Author

ghost commented Feb 26, 2020

Thank you for your interest @john-tipper currently we need to check whether this works with nested stacks, and we need to consider if this should impact cdk diff or not. If it should, then that needs some work.
I am currently looking into making it work with diff, as when you try to deploy an identical stack with new parameters it still shows up as not changed, and needs a --force in order to be redeployed.

@shivlaks
Copy link
Contributor

Thank you for your interest @john-tipper currently we need to check whether this works with nested stacks, and we need to consider if this should impact cdk diff or not. If it should, then that needs some work.
I am currently looking into making it work with diff, as when you try to deploy an identical stack with new parameters it still shows up as not changed, and needs a --force in order to be redeployed.

  • I think it's okay to park cdk diff support for now unless we feel there's going to be unintended consequences from using parameters.
  • For wild card support, I think your suggestion is reasonable... might also be simpler to only support parameters when a stack is specified and not with wild-card deploys (i.e. add the capability to specify stack as an incremental feature)
  • Did you get a chance to see how this behaves with Nested Stacks yet?

Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would also be good to add a small snippet / sample into the README for posterity (and to please the linter)

@ghost
Copy link
Author

ghost commented Feb 27, 2020

I have implemented the version of the parameter parsing that I mentioned above, so if that is fine, then I think we could keep that.
I will now switch from working on the diff "issue", as this seems to be a bit more work, to testing the nested stack thing. Also just add a few lines to the readme.

Viphor and others added 2 commits February 27, 2020 10:31
Adding integration test for nested stacks.

Adding section in README.
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: df63489
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 237e296
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 73b783b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@shivlaks
Copy link
Contributor

shivlaks commented Mar 2, 2020

update: I spent some time testing it out

  • I think the convention of prefixing with --parameters "StackName:Key=Value" seems reasonable to me
  • Nested stacks are fine and can consume parameters through the parent stack
  • cdk diff doesn't account for parameters - which may be reasonable for now

to resolve:

  • how do we remove parameter from a stack?
  • since diff won't show parameters, will a change in parameters clobber previously existing parameters without warning? (should it?)

thoughts @john-tipper @dkanhakl

@joehillen
Copy link
Contributor

will a change in parameters clobber previously existing parameters without warning? (should it?)

I was just about to ask if this would happen. I don't think it should. I'm dealing with this right now with a stack that has a default parameter that is updated by an external event. I only want the default used for the initial stack creation. Whenever the stack template updates, I want it to use the existing non-default value.

@shivlaks
Copy link
Contributor

shivlaks commented Mar 3, 2020

will a change in parameters clobber previously existing parameters without warning? (should it?)

I was just about to ask if this would happen. I don't think it should. I'm dealing with this right now with a stack that has a default parameter that is updated by an external event. I only want the default used for the initial stack creation. Whenever the stack template updates, I want it to use the existing non-default value.

I agree that anything which is surprising to a customer without warning is not ideal (unless they're using the --force flag. I'm going to give this a quick test in a couple of hours and see what we can do about that. I'd love to resolve this one, but I see this is the primary remaining thing for getting this first commit in. I do think there should be a mechanism to remove a parameter as well (maybe it's as simple as providing an empty value to an existing key?)

@ghost
Copy link
Author

ghost commented Mar 3, 2020

Thank you for the feedback and interest. I need to understand what you mean by removing a parameter.

First of all, if I understand it correctly, regarding default parameters. If you have a default and apply a parameter from the cli, then it should follow the behaviour of cloudformation and override the default. (which is what should be implemented already)

But currently if you do not change anything in the template, then it will not detect a redeploy with different parameters as a change, and does need a --force parameter to actually install the new parameters.

@ghost
Copy link
Author

ghost commented Mar 3, 2020

will a change in parameters clobber previously existing parameters without warning? (should it?)

I was just about to ask if this would happen. I don't think it should. I'm dealing with this right now with a stack that has a default parameter that is updated by an external event. I only want the default used for the initial stack creation. Whenever the stack template updates, I want it to use the existing non-default value.

I have not tested the scenario that you mention here. Since I am still new to AWS, could you please elaborate how cloudformation handles this, or point me to some documentation of this behaviour?

The current implementation has only added that the ability to pass parameters to the deployment command, and not changed anything to the actual deployment itself. Does anyone have any knowledge of how this has been handled with the cloudformation parameters that cdk already uses?

Adding an example of using parameters to the README.md
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 2024a54
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 98b655a
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 65e3cc2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@joehillen
Copy link
Contributor

I have not tested the scenario that you mention here. Since I am still new to AWS, could you please elaborate how cloudformation handles this, or point me to some documentation of this behaviour?

The current implementation has only added that the ability to pass parameters to the deployment command, and not changed anything to the actual deployment itself. Does anyone have any knowledge of how this has been handled with the cloudformation parameters that cdk already uses?

It seems to be expected behavior for CF even though it's surprising and frustrating (typical for CF). If update-stack is sent to CF without parameters, then the default is used even if the parameters were set in a previous update-stack call. I think the fix would be to lookup parameters that were set in the stack and then pass them again with the next stack update.

It's probably outside the scope of this PR. I'll open another PR after this one is merged.

@joehillen
Copy link
Contributor

For some more context, this has been a long standing feature request for the AWS CLI aws/aws-cli#2589

@shivlaks
Copy link
Contributor

shivlaks commented Mar 3, 2020

It seems to be expected behavior for CF even though it's surprising and frustrating (typical for CF). If update-stack is sent to CF without parameters, then the default is used even if the parameters were set in a previous update-stack call. I think the fix would be to lookup parameters that were set in the stack and then pass them again with the next stack update.

It's probably outside the scope of this PR. I'll open another PR after this one is merged.

@joehillen do you mind creating a feature-request issue for that functionality? I agree that it should be addressed in a different PR

@dkanhakl this is looking good, I'm running some final tests and will make some quick tweaks to the README, but otherwise I think we're looking good to move this one forward! .

@shivlaks
Copy link
Contributor

shivlaks commented Mar 3, 2020

will a change in parameters clobber previously existing parameters without warning? (should it?)

I was just about to ask if this would happen. I don't think it should. I'm dealing with this right now with a stack that has a default parameter that is updated by an external event. I only want the default used for the initial stack creation. Whenever the stack template updates, I want it to use the existing non-default value.

It doesn't currently provide a helpful error message indicating that you've tried to do this. I'd like to address that in a follow-on PR as I think it would be helpful to detect and provide a helpful message indicating that it can be replaced by using --force

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 775b1aa
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 82ce5cf
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ran all the integ tests. I think we're okay to proceed with this!

@mergify
Copy link
Contributor

mergify bot commented Mar 4, 2020

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 521c313
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Mar 4, 2020

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 6551124 into aws:master Mar 4, 2020
eladb pushed a commit that referenced this pull request Mar 9, 2020
* feat(aws-cdk): pass cloudformation parameters to "cdk deploy"

This commit closes #1237

I have added the command line option --parameters, -P which allows for
adding CloudFormation parameters at deploy time.

The functionality for deploying with parameters was already present,
but I have just added a way for the command line to pass it to the
deployment code.

The motivation for adding this functionality has been that we have
a setup where we share an account with access to CloudFormation, which
is why we do not want to have some fields in plain text, while they do
not have access to other services, which did not support secrets within
the fields we want to keep secret. This allows us to have parameters
that can be kept secret from the templates, while also having parameters
with NoEcho.

I have verified this functionality by both deploying a cdk project
with a CfnParameter from @aws-cdk/core, and an original CloudFormation
template with parameters.

* feat(cli): pass cloudformation parameter to "cdk deploy"

This changes the syntax of the --parameter parameter, so that it is now
possible to specify which stack a parameter is meant for, i.e. of you
want a parameter to apply to `MyStack`, then you would pass it as
--parameters MyStack:key=value
While still being able to pass parameters to all stacks by omitting the
stack name, i.e. --paramters key=value

This is being added to support wild card deployments, and deployments
with multiple stacks that do not neccessary contain the same parameters.

I have verified these changes by deploying a multistack cdk project,
where the stacks contains different parameters, and some shared
parameters.

* feat(cli): pass cloudformation parameters to "cdk deploy"

I have added some integration tests of both wildcard and single
deployments with parameters. While also fixing a bug found while
testing.

* feat(cli): pass cloudformation parameters to "cdk deploy"

Adding integration test for nested stacks.

Adding section in README.

* feat(cli): pass cloudformation parameters to "cdk deploy"

Adding an example of using parameters to the README.md

* update README

* more README cleanup

Co-authored-by: Elad Ben-Israel <[email protected]>
Co-authored-by: Shiv Lakshminarayan <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
horsmand pushed a commit to horsmand/aws-cdk that referenced this pull request Mar 9, 2020
* feat(aws-cdk): pass cloudformation parameters to "cdk deploy"

This commit closes aws#1237

I have added the command line option --parameters, -P which allows for
adding CloudFormation parameters at deploy time.

The functionality for deploying with parameters was already present,
but I have just added a way for the command line to pass it to the
deployment code.

The motivation for adding this functionality has been that we have
a setup where we share an account with access to CloudFormation, which
is why we do not want to have some fields in plain text, while they do
not have access to other services, which did not support secrets within
the fields we want to keep secret. This allows us to have parameters
that can be kept secret from the templates, while also having parameters
with NoEcho.

I have verified this functionality by both deploying a cdk project
with a CfnParameter from @aws-cdk/core, and an original CloudFormation
template with parameters.

* feat(cli): pass cloudformation parameter to "cdk deploy"

This changes the syntax of the --parameter parameter, so that it is now
possible to specify which stack a parameter is meant for, i.e. of you
want a parameter to apply to `MyStack`, then you would pass it as
--parameters MyStack:key=value
While still being able to pass parameters to all stacks by omitting the
stack name, i.e. --paramters key=value

This is being added to support wild card deployments, and deployments
with multiple stacks that do not neccessary contain the same parameters.

I have verified these changes by deploying a multistack cdk project,
where the stacks contains different parameters, and some shared
parameters.

* feat(cli): pass cloudformation parameters to "cdk deploy"

I have added some integration tests of both wildcard and single
deployments with parameters. While also fixing a bug found while
testing.

* feat(cli): pass cloudformation parameters to "cdk deploy"

Adding integration test for nested stacks.

Adding section in README.

* feat(cli): pass cloudformation parameters to "cdk deploy"

Adding an example of using parameters to the README.md

* update README

* more README cleanup

Co-authored-by: Elad Ben-Israel <[email protected]>
Co-authored-by: Shiv Lakshminarayan <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pass CloudFormation Parameters to "cdk deploy"
5 participants