AWS CodeArtifact NuGet endpoint intermittently returning 401 in Visual Studio when trying to browse packages

[wr-cdargis]

Describe the bug
I have had a working setup for a few years using this as a guide: https://docs.aws.amazon.com/codeartifact/latest/ug/nuget-visual-studio.html

We have a nuget source backed by AWS Code Artifact configured through Visual Studio. It works 80% of the time. However intermittently we receive 401 errors when trying to view packages from this source through Visual Studio. Once I get a 401 I'll see it consistently until the problem somehow resolves on its own. Sometimes this takes 5 minutes, sometimes it takes close to an hour. It doesn't seem like restarting VS or the system help at all. This all happens without a change to the AWS profile that is being used for credentials.

  Response status code does not indicate success: 401 (Unauthorized).
[wellright\wellright-repository] Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
  Response status code does not indicate success: 401 (Unauthorized).
[wellright\wellright-repository] Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
  Response status code does not indicate success: 401 (Unauthorized).
[wellright\wellright-repository] Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
  Response status code does not indicate success: 401 (Unauthorized).
[wellright\wellright-repository] Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
  Response status code does not indicate success: 401 (Unauthorized).
[wellright\wellright-repository] Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
  Response status code does not indicate success: 401 (Unauthorized).

To Reproduce

1. Setup CodeArtifact
2. Upload .NET packages
3. Configure AWS to use CodeArtifact as NuGet source. Use credentials from local profile
4. Try to use Visual Studio to browse the NuGet packages.
5. Result: 401 errors ~20% of the time. Once we see the 401 it's stuck like that for an undetermined amount of time. Sometimes it fixes itself in 5 minutes. Sometimes it takes an hour.

Expected behavior
No 401

Screenshots

Toolkit Logs

2024-01-12 14:17:33,678 [46] DEBUG Amazon.AWSToolkit.Lambda.Util.LambdaTesterInstaller - Attempting to install or update Lambda Tester dotnet tool
2024-01-12 14:17:41,302 [41] INFO  Amazon.AWSToolkit.Lambda.Util.LambdaTesterInstaller - dotnet tool update -g Amazon.Lambda.TestTool-6.0
2024-01-12 14:17:41,302 [41] INFO  Amazon.AWSToolkit.Lambda.Util.LambdaTesterInstaller - C:\Users\blah\AppData\Local\Temp\c60b0f00-6b6d-4a00-af27-59b81e651db3\restore.csproj : error NU1301: Unable to load the service index for source https://wellright-060498233243.d.codeartifact.us-east-1.amazonaws.com/nuget/wellright-repository/v3/index.json.
Tool 'amazon.lambda.testtool-6.0' failed to update due to the following:
The tool package could not be restored.

Development System (please complete the following information):

• Windows Version: 10 pro
• Visual Studio Version: 17.6.4
• AWS Toolkit for Visual Studio Version: 1.41.0.0

[detam-amazon]

Derek from the CodeArtifact team. Thanks for bringing this to our attention. I've created an internal issue for our team to prioritize fixing this problem.

[wr-cdargis]

@detam-amazon thanks! I wish I had a reliable way to reproduce the issue. It just seems to happen randomly and once it does it's difficult to get it working again. Please let me know if there is some additional logging I can dig into or something.

[wr-cdargis]

Bump. This is still very much an annoying issue

[JKalady]

Getting a very similar issue. need a fix. very annoying.

[jg-well]

This has been showing up for me since ~2 months ago on any new repo I pull.
If I don't resolve it every 12 hours my package builds fail. It's very disruptive.

[mchlkovalsky]

Bump as well, a temporary (12hr) workaround is to run:

aws codeartifact login --tool dotnet --domain <your-domain-here> --repository <your repository here> --profile Prod