More flexible ways to conditionally set nuget settings (like sources)

[natemcmaster]

Feature suggestion:
Make it easier to vary NuGet feeds by build/test environments.

Scenario:
Build agents use different feeds than local dev builds. To enable this, our build systems rewrite NuGet.config files to ensure the build gets the right package feeds.

Using environment variables in the package feed value (#1852) isn't quite powerful enough to our build scenarios.

Ideas:

• NuGet.${env}.config which sits next to a NuGet.config file and is used instead when an env variable NUGET_CONFIG_ENV is set. (e.g. NuGet.ci.config, NuGet.test.config)
• Add syntax to NuGet.config to support varying packageSources by environments.

  <add key="feed" value="../build-artifacts/" when="'$(USE_CI_FEEDS)'=='true'" />

Also, if I've missed a feature that already enables these scenarios, please advise.

[bbowman]

My team is in a similar situation but my plan is to have the build agent look at the same feeds as a local dev. Why do you need / want the dev experience to be different from the build agent there? Seems like a place to encourage differences from local and build agent state making harder to track down issues?

[natemcmaster]

We want the local dev experience to work for those inside and outside of microsoft and to be relatively stable. So our NuGet.config files point to public feeds that only have good builds of packages. Our CI agents, on the other hand, consume feeds that are internal-only or have untested, pre-release packages.

[bbowman]

My point is that a dev on your project is also probably interested in the pre-release packages (and probably producing them locally and well as triggering CI builds that produce them). The consumers of your package probably only care about the stable ones but that seem different than what a contributor (internal or external) may want?

[natemcmaster]

My point is that a dev on your project is also probably interested in the pre-release packages

Good point. Yes, this happens, and when it does the dev has to modify their NuGet.config to pull the right feeds, make sure not to accidentally commit the NuGet.config change to source (this has never happened, of course ), and, when done, clear all nuget caches to get back to stable-only packages.

[bbowman]

We are still devising a system for our team to help prevent these situations but it goes as follows. We are envisioning 3 "tiers" of package sources that will always be set up in a nuget.config.

1. nuget.org - this location only includes stable packages, ready to be consumed by external consumers.
   example: FooPackage.1.0.0.nupkg
2. CI build server - this location includes prerelease packages for "the next candidate version" of a nuget.org package. It is tagged with the build time and a small string based on the branch it was built from
   example: FooPackage.1.0.1-20161118145703.dev
3. Local build outputs - this location includes prerelease packages for "the next candidate version" of a nuget.org package. It is tagged with the build time and a small string based on the branch it was built from (pr for private or pull request, not straight develop)
   example: FooPackage.1.0.1-20161118145703.pr

This means that a local dev gets the latest (due to time string version ordering) locally built package by default. The CI build picks up what was built from a previous step automatically (due to local path or CI path if the previous step also publishes). "True Consumers" only pick up stable nuget.org

The local/CI consumers just need to make sure to use 1.0.1-* in their project.json and then it should be "easy" to ensure everyone is getting what they want based on where they are and what has been built.

Since your team is already up and running, do you see any potential pitfalls with this approach? Would it work for you guys?

[natemcmaster]

We have the concept of 'tiers' too. See https://github.com/aspnet/Home/wiki/NuGet-feeds.

Most common pitfall, the dash star. "1.0.1-*" is seems nice at first because you don't have to change your project for every nightly build, but give it time and dash star will hurt at some point. On numerous occasions, restore has pulled in a package version that was unexpected...and it can be hard to figure out where it came from.

due to local path or CI path if the previous step also publishes

Here's another pitfall, and basically the reason for requesting the concepts of environments. Chained build configurations on CI need an additional feeds to get unpublished packages from earlier build steps.

[emgarten]

@natemcmaster would it work for you to define the feeds and global packages folder in MSBuild by setting this in your project, or including a target that sets this and has conditions? From your example it looks like you are asking for nuget.config to support MSBuild conditionals.

[natemcmaster]

it looks like you are asking for nuget.config to support MSBuild conditionals.

That was just one idea. I actually like the NuGet.${env}.config idea much better.

I haven't tried it yet, but I see that NuGet 4 supports a RestoreSources property in MSBuild. How will this interact with NuGet.config?

That said, we frequently use and will continue to need NuGet clients that are not part of MSBuild but would still benefit from NuGet.${env}.config. FWIW before asking, I gave consideration to just using the --sources parameter on NuGet.exe and dotnet-restore, but have found that, in practice, this gets messy in projects that do end-to-end build-pack-install testing.

[emgarten]

but have found that, in practice, this gets messy in projects that do end-to-end build-pack-install testing.

Can you give some examples of this? It sounds like there may be other problems here.

restore should be the only component reading the package sources, and the output packages folder used for downloaded packages is written to the project.lock.json/project.assets.json file. Build should be reading the packages folder from the assets file. Is something else reading the sources?

[natemcmaster]

Here is one scenario: I work on tools such as "dotnet watch" and others. True end-to-end validation requires build+packing the dotnet-watch project, and then dotnet-restore its output into a new test project. This new test project needs to include the same package feeds used to build dotnet-watch. To ensure tests get the right packages, the build of dotnet-watch and tests (which shell execute a dotnet-restore) need additional CI feeds. To make this work, I have two hacks. (1) the CI build steps edit NuGet.config to add additional CI feeds. (2) the end-to-end tests copy he modified NuGet.config into test projects (to get the required CI feeds) and shell execute "dotnet restore --sources $artifactPath" where $artifactPath includes the dotnet-watch that was just built.

[rrelyea]

You can set $(RestoreSources) to replace the nuget.config provided sources via msbuild project or props files. Given that, you can use msbuild conditions to do what was originally desired in this issue.
Closing. Let us know if something key is still missing.

[natemcmaster]

Thanks @rrelyea. Yes, we've moved to MsBuild properties and conditions to solve this issue....for the most part. There is one limitation that we have to workaround: MSBuild properties can't configure feeds that require credentials. The discussion about this one is here: #6045