RestoreSources set via MSBuild properties cannot use credentials

[natemcmaster]

I'm attempting to use MSBuild properties to define the feeds a project can use to restore. This does not work if feeds require credentials. The credentials are not loaded from the global config file in $env:AppData\NuGet.

Details

NuGet: NuGet Command Line 4.5.0.2
dotnet.exe --version: 15.5.0-preview-007044
OS version: Win 10

Repro

1. Create a project with an authenticated feed. For example, you can do this on VSTS.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <RestoreSources Condition="'$(CI)' == 'true'">
      https://myvsts.pkgs.visualstudio.com/_packaging/myfeedname/nuget/v3/index.json;
    </RestoreSources>
    <RestoreSources>
      $(RestoreSources);
      https://api.nuget.org/v3/index.json;
    </RestoreSources>
  </PropertyGroup>
  <ItemGroup>
     <PackageReference Include="MySecretPackage" Version="1.2.3" />
  </ItemGroup>
</Project>

2. Add your credentials to the user-profile config file. See https://docs.microsoft.com/en-us/vsts/package/nuget/dotnet-exe

nuget.exe sources add -name {feed name} -source {feed URL} -username {username} -password {your PAT} -StorePasswordInClearText

$env:APPDATA\NuGet\NuGet.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
  <packageRestore>
    <add key="enabled" value="True" />
    <add key="automatic" value="False" />
  </packageRestore>
  <bindingRedirects>
    <add key="skip" value="False" />
  </bindingRedirects>
  <packageManagement>
    <add key="format" value="1" />
    <add key="disabled" value="False" />
  </packageManagement>
  <packageSourceCredentials>
    <mysecretfeed>
      <add key="Username" value="xxxxx" />
      <add key="ClearTextPassword" value="xxxxxxxxxxxxxxxxxxxxx" />
    </mysecretfeed>
  </packageSourceCredentials>
</configuration>

3. dotnet restore

Result

Restore fails with HTTP 401 Unauthorized.

C:\Users\namc\.dotnet\x64\sdk\15.5.0-preview-007044\NuGet.targets(103,5): error : Failed to retrieve information about 'xxxxxxxxx' from remote source 'https://xxxxxxxxxxx.pkgs.visualstudio.com/_packaging/xxxxxxxxxxxxx/nuget/v3/flat2/xxxxxxxxx/index.json'.
C:\Users\namc\.dotnet\x64\sdk\15.5.0-preview-007044\NuGet.targets(103,5): error :   Response status code does not indicate success: 401 (Unauthorized).

[natemcmaster]

Workaround

It appears you can workaround this by also adding the feed to a NuGet.config file in the project. But this seems to defeat the purpose of using MSBuild properties. We were trying to get away from manipulating our NuGet.config file through automation, as discussed in #3972

[jainaashish]

@natemcmaster where have you added your secret feed in user-profile nuger config file? in your example, I can only see credentials being added there but not the feed itself. And then you specify this feed via RestoreSources as MSBuild property but how would NuGet relate this feed with credentials provided in NuGet.config file without the feed name?

[natemcmaster]

I didn't add the feed to the user-profile nuget config file because I don't want the feed to apply to all projects on the machine. In this example, I want the feed scoped to a specific project.

I understand why the problem is happening - NuGet doesn't have a way to figure out which credentials belong with the feed. In MSBuild, the RestoreSources property is effectively string[]. There is no way to set the source name.

Would it be possible to support setting RestoreSources an itemgroup in addition to the RestoreSources property? We could then set the source name as metadata so NuGet can use this to find credentials.

Example:

<Project>
 <ItemGroup>
  <RestoreSources Include="https://mysecretfeed/v3/index.json" Condition="'$(CI)' == 'true'">
    <Name>mysecretfeed</Name>
  </RestoreSources>
  <RestoreSources Include="https://api.nuget.org/v3/index.json" />
 </ItemGroup>
</Project>

You could even support setting the username/password explicitly, or by using other MSBuild properties and MSBuild variables.

  <RestoreSources Include="https://mysecretfeed/v3/index.json">
    <Username>commonusername</Username>
    <Password>$(MyPasswordFromEnvVar)</Password>
  </RestoreSources>

Just an idea. I'm sure there are other ways to solve this.

[nkolev92]

I thought there was another issue where this was discussed, but a big thing to consider here is the fact that we don't support any of these properties in the UI yet.

I'd like us to figure out that out before we continue adding new features.

[natemcmaster]

I think that UI thread was #5321

[tmat]

@rrelyea Any progress on this? I don't think this work should be blocked on figuring out UI. Not being able to use authenticated feeds via RestoreSources is a big problem for our build infrastructure.

[rrelyea]

VS 15.8 and .NET SDK 2.1.400 plan to have support for authentication plugins. VSTS team is also on track for having their plugin available at the same time.
This will enable msbuild and dotnet restore authentication scenarios. (today nuget.exe and VS already have support).

@nkolev92 is driving this from the NuGet side.

[natemcmaster]

With these new auth plugins in 2.1.400, how do we configure a credential for one or many of the feeds listed in the RestoreSources property?

[nkolev92]

Providing the credentials will be an interactive experience, which should negate the need for plain text configuration of credentials.

You'd need to do dotnet restore --interactive and auth using device flow.
If the build is on prem, the plugin itself should be able to figure it out. //cc @aldoms

[aldoms]

If the build is on VSTS/TFS, the plugin will automatically provide credentials for feeds hosted in the same collection the build is running on. For external feeds, you would need to provide them as service endpoints.

[natemcmaster]

What about non interactive environments? Is there an xplat equivalent to nuget.exe sources add? Also, how will this work for CI that is not VSTS? Today, aspnet builds on TeamCity and Jenkins.

[TFTomSun]

I face the same issue and would like to see a pure MSBuild non interactive solution for that, too. The build shouldn't depend on a specific build environment. The proposed solution with username and password and getting the password from an environment variable is the way to go from my point of view, because that is supported by all CI/CD tools.

[NightOwl888]

I managed to get this working using a variation of this fix, except using MSBuild properties.

<Project>
  <PropertyGroup Label="RestoreSources">
    <RestoreSources>
      $(RestoreSources);
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\;
      https://api.nuget.org/v3/index.json;
      https://myvsts.pkgs.visualstudio.com/_packaging/myfeedname/nuget/v3/index.json;
    </RestoreSources>
  </PropertyGroup>
</Project>

For some reason, the magic path C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\ seems to fix the credential issue, but I haven't looked at the code to investigate why that is the case. Also, unlike the issues I am linking to, this solution seems to work without specifying a username, password, or PAT if you are using Azure Pipelines hosted agents with Azure Artifacts feeds.

See Nuget restore fails on Azure Devops with message “unable to load the service index for source” for more details.

[NightOwl888]

I managed to get this working using a variation of this fix, except using MSBuild properties.

<Project>
  <PropertyGroup Label="RestoreSources">
    <RestoreSources>
      $(RestoreSources);
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\;
      https://api.nuget.org/v3/index.json;
      https://myvsts.pkgs.visualstudio.com/_packaging/myfeedname/nuget/v3/index.json;
    </RestoreSources>
  </PropertyGroup>
</Project>

For some reason, the magic path C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\ seems to fix the credential issue, but I haven't looked at the code to investigate why that is the case. Also, unlike the issues I am linking to, this solution seems to work without specifying a username, password, or PAT if you are using Azure Pipelines hosted agents with Azure Artifacts feeds.

See Nuget restore fails on Azure Devops with message “unable to load the service index for source” for more details.

Unfortunately, that seems to only work because of a caching issue. I ran the restore using nuget restore, rather than dotnet restore to try it out. After that, dotnet restore worked. But on a new pipeline, it didn't work by itself because I didn't run nuget restore first.

Fortunately, this solution will work for me because I can just pass in the feed ID that I am using and get , but it doesn't help with running the restore using MSBuild properties at all.

[NightOwl888]

I managed to get this working using a variation of this fix, except using MSBuild properties.

<Project>
  <PropertyGroup Label="RestoreSources">
    <RestoreSources>
      $(RestoreSources);
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\;
      https://api.nuget.org/v3/index.json;
      https://myvsts.pkgs.visualstudio.com/_packaging/myfeedname/nuget/v3/index.json;
    </RestoreSources>
  </PropertyGroup>
</Project>

For some reason, the magic path C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\ seems to fix the credential issue, but I haven't looked at the code to investigate why that is the case. Also, unlike the issues I am linking to, this solution seems to work without specifying a username, password, or PAT if you are using Azure Pipelines hosted agents with Azure Artifacts feeds.
See Nuget restore fails on Azure Devops with message “unable to load the service index for source” for more details.

Unfortunately, that seems to only work because of a caching issue. I ran the restore using nuget restore, rather than dotnet restore to try it out. After that, dotnet restore worked. But on a new pipeline, it didn't work by itself because I didn't run nuget restore first.

Fortunately, this solution will work for me because I can just pass in the feed ID that I am using and get , but it doesn't help with running the restore using MSBuild properties at all.

Workaround:

Using at least NuGet 4.8, run nuget restore and either pass it the location of NuGet.config or pass the vstsFeed parameter (a GUID to your Azure Artifacts feed). This will cache the credentials, and will probably be enough if you only have one TargetFramework defined in your build.

However, if you are multi-targeting using TargetFrameworks, subsequently run either dotnet restore or dotnet build without the --no-restore parameter and it will restore all of the target frameworks. It will function at that point because the credentials will be cached by nuget restore (or at least that is the case in Azure Pipelines).

This works whether you use RestoreSources or NuGet.config to define your source feeds.

[nkolev92]

However, if you are multi-targeting using TargetFrameworks, subsequently run either dotnet restore or dotnet build without the --no-restore parameter and it will restore all of the target frameworks. It will function at that point because the credentials will be cached by nuget restore (or at least that is the case in Azure Pipelines).

nuget.exe can restore multi targeting projects just fine.

dotnet.exe vs nuget.exe is more about whether you only have SDK based projects, vs a mix of packages.config, PackageReference common csproj and/or SDK style.

nuget.exe can do everything restore wise that dotnet.exe can.

[NightOwl888]

@nkolev92

Thanks. The reason I stated that is because when I tried it with netstandard2.0;net461 the project failed to restore, and I made the assumption it was because there were multiple targets because the error message was:

C:\Program Files\dotnet\sdk\2.2.105\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(208,5): error NETSDK1005: Assets file 'd:\a\1\s\tests\TestLibrary.Tests\obj\project.assets.json' doesn't have a target for '.NETFramework,Version=v4.6.1'. Ensure that restore has run and that you have included 'net461' in the TargetFrameworks for your project. [d:\a\1\s\tests\TestLibrary.Tests\TestLibrary.Tests.csproj]

I do not have packages.config, and all of the projects use the new .csproj format with <TargetFrameworks>.

But, without running dotnet restore after running nuget restore, the solution didn't restore correctly. Do note that I am running the restore on the solution file with a glob pattern - **/*.sln.

Of course, dotnet restore would work by itself if it weren't for the credential issues. There are a few other ways to get the credentials to work, but I haven't gone through the options to see whether they will work with RestoreSources.

[nkolev92]

@NightOwl888

Of course, dotnet restore would work by itself if it weren't for the credential issues. There are a few other ways to get the credentials to work, but I haven't gone through the options to see whether they will work with RestoreSources.

I might have an easier answer for that.
There are 2 ways in which NuGet can get credentials.

1. Explicitly passing the PAT to NuGet (currently, only possible through config file, the ask is to enable it through msbuild).
2. Use credential providers from the service provider (Azure DevOps, MyGet, Artifactory...) (there are 2 versions of commandline providers, 1 being cross plat and supported in dotnet.exe. Visual Studio has its own extensibility model)

They are evaluated in that order.
So if you have RestoreSources you can definitely use the plugins.

[TFTomSun]

To all who don't want to modify their nuget.config. Here's my current approach, until we have full support in MSBuild for that. The feed address and the credentials come from environment variables. I wrote a little tool that allows me to toggle the environment variables if i want to change the current private feed. Works fine in development and unattended environments.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key="globalPackagesFolder" value=".nuget/packages" />
    <add key="repositoryPath" value=".nuget/installed" />
  </config>

  <packageSources>
    <clear />

    <add key="local" value=".nuget/repository" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="myget" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json"></add>
    <add key="PrivateFeed" value="%Nuget_PrivateFeedAddress%" />
  </packageSources>
  <disabledPackageSources>
    <add key="local" value="true" />
    <add key="myget" value="true" />
    <!--<add key="nuget.org" value="true" />-->
  </disabledPackageSources>

  <packageSourceCredentials>
    <PrivateFeed>
      <add key="Username" value="%Nuget_PrivateFeedUserName%" />
      <add key="ClearTextPassword" value="%Nuget_PrivateFeedApiKey%" />
    </PrivateFeed>
  </packageSourceCredentials>
  <apikeys>
     <add key="PrivateFeed" value="%Nuget_PrivateFeedEncryptedApiKey%" /> 
  </apikeys>
</configuration>

[mthalman]

Is this issue taking into account the requirements of #6243? That issue was closed as dupe in favor of this one so I want to make sure this "uber" issue is accounting for it in the design.

There needs to be a solution that can:

• Work in automated, non-interactive scenarios
• Not require artifact credential provider (i.e. can target custom NuGet feeds with plain old username/password)
• Can re-use the same nuget.config that is used on a dev box. Shouldn't require a second copy of nuget.config that is meant for automation scenarios and should not require workarounds like this:

   <packageSourceCredentials>
     <PrivateFeed>
       <add key="Username" value="%Nuget_PrivateFeedUserName%" />
       <add key="ClearTextPassword" value="%Nuget_PrivateFeedApiKey%" />
     </PrivateFeed>
   </packageSourceCredentials>
  

[richlander]

@mthalman -- with your requirements, is it OK if your nuget.config with your password is part of the build context and becomes part of an intermediate layer?

[mthalman]

@mthalman Matt Thalman FTE -- with your requirements, is it OK if your nuget.config with your password is part of the build context and becomes part of an intermediate layer?

@richlander - For the context of others, when you say "build context" you're referring to a Docker build. I would prefer that credentials not be required to be within the nuget.config at all whether it be inside of a Docker build or outside of it. Suggestions like those given in #6243 are appealing because the creds do not need to be stored on disk and can be passed as arguments.

In some respects there isn't a perfect solution when it comes to Docker. Your options are the following:

• Pass a secret as an ARG.
  Downside: Gets exposed in the layer as an environment variable. Care needs to be taken to use multi-stage build to avoid publishing the layer with the secret.
• Store secret in a file that gets passed to Docker through build context.
  Downside: Secret is exposed on disk on host and in the layer. Care needs to be taken to use multi-stage build to avoid publishing the layer with the secret.
• Pass secret via Docker BuildKit secrets.
  Downside: A bit more complexity in authoring the Dockerfile. Secret must be stored on disk on the Docker host during the build.

[richlander]

I like the first option the best because it is the simplest and has a close analogue with a related docker run use case (build source by volume mounting in an SDK container). I also think this model is harder to do the wrong thing. Anything that uses files as the currency is more subject to accidents (like commiting secrets to git).

[SeppPenner]

This is an option I am interested as well. Currently, the restore process breaks our build process (Run in Docker). I don't really want to add a NuGet.config file or something inside the docker container...

The preferred way would be something like this:

dotnet restore --source https://mycustomsource.com/nuget --username JohnDoe --password YouDontKnowIt

[ericsampson]

@nkolev92 since you closed #10178, I'd like to bring the following requests from it here, as the discussion so far in this thread has been focused mainly on MSBuild:

Support should be added to the upstream tooling to pass the credentials into MSBuild something like:

• dotnet msbuild -restore --property:NEWPROP=TOKEN --property:RestoreSources=blah
• dotnet restore --api-key TOKEN1 --source blah1 --api-key TOKEN2 --source blah2
• dotnet build (same)
• same behavior for nuget.exe and msbuild.exe

[nkolev92]

Thanks for pasting that @ericsampson

All feedback from the linked issues will be considered when looking into this issue! :)

A quick clarification:
--api-key are never used by restore. --api-key only applies to push.

[ericsampson]

@nkolev92 right, but that's talking about requested future functionality.

[nkolev92]

Hey @ericsampson, I just wanted to clarify that api keys is not something we want to support with restore at all at this point.

Restore works well with basic, negotiate, ntlm etc.
Api Keys are just a header that's used for push scenarios only.

Given that any potential changes like that required both client + server changes, preserving the status quo as far as auth mechanisms go is the approach I'd recommend at this point.

[ericsampson]

@nkolev92 ok I understand, I think I misunderstood what was going on behind the scenes.

So then it would look potentially like this?
dotnet restore --source https://mycustomsource.com/nuget --username DoesntMatter --password NUGET_PAT
where username is unused by Azure Artifacts / GitHub Packages, and the "password" would be the PAT

And so there'd need to be two new MSBuild properties as well, correspondingly

[TheXenocide]

I found this old issue while googling around and it seems there still isn't a solution implemented yet. As an alternative to providing all nuget.config settings in MSBuild (though I would actually prefer using MSBuild Items, mentioned above), is it/would it be possible to have RestoreSources allow specifying the names of sources defined in nuget.config? In this way, a single nuget.config can have many potential sources and then the build can detect/decide which ones to use for this build? This could then be used in lieu of automating modification of the nuget.config to disable sources and such; e.g.

<RestoreSources Condition=" '$(ContinuousIntegrationBuild)' == 'true' AND '$(Configuration)' == 'Debug' ">nuget.org;Private Debug Feed</RestoreSources>

We're hoping to differentiate internal feeds by both debug/release configuration as well as by environment for CI/local dev. Previously we used RestoreConfigFile and had 2 separate CI-specific configs (Debug/Release) and just require developers to manually change the disabled feeds in the xml to switch between debug/release (because setting RestoreConfigFile from MSBuild properties in a project file/Directory.Build.props is not supported in Visual Studio; this feature is really only intended for command-line MSBuild use), but with the addition of Package Source Mapping this has increased multiple maintenance/duplication overhead and it would be nice to automatically switch feeds for dev environments as well.

I think I like the long-term idea of some sort of ItemDefinition for sources so that metadata like name, credentials, etc. can be grouped/conditioned together. Any suggestions in the meantime?