Extra round trip due to 407 proxy authentication when using proxy with basic auth

[jgilbert2017]

Description

Not sure, but this may be the same as #24865 .

I'm doing https requests using a proxy that requires basic auth using HttpClient.

I've got an EventListener setup for http and am seeing that each request is doing a CONNECT to the proxy and getting nacked with 407 and then following up with another request that correclty includes the Proxy-Authorization.

Is there any way to force the initial request to include the authorization information (or can i include a custom header on the initial connect)?

This extra round trip is highly inefficient.

Reproduction Steps

HttpClient request via proxy that requires username/password

Expected behavior

include Proxy-Authorization on initial request if the configured WebProxy includes NetworkCredentials

Actual behavior

407s

Regression?

unknown

Known Workarounds

No response

Configuration

.net 6, windows x64

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Not sure, but this may be the same as #24865 .

I'm doing https requests using a proxy that requires basic auth using HttpClient.

I've got an EventListener setup for http and am seeing that each request is doing a CONNECT to the proxy and getting nacked with 407 and then following up with another request that correclty includes the Proxy-Authorization.

Is there any way to force the initial request to include the authorization information (or can i include a custom header on the initial connect)?

This extra round trip is highly inefficient.

Reproduction Steps

HttpClient request via proxy that requires username/password

Expected behavior

include Proxy-Authorization on initial request if the configured WebProxy includes NetworkCredentials

Actual behavior

407s

Regression?

unknown

Known Workarounds

No response

Configuration

.net 6, windows x64

Other information

No response

Author:	jgilbert2017
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[wfurt]

You can try to set HttpClientHandler.PreAuthenticate. I don't remember if that works for Proxy but the intent is exactly what you describing. Especially basic auth leaks passwords so the the HttpClient will not send them unless asked for but status code.

[jgilbert2017]

yep, tried PreAuthenticate. no dice.

IMO there should be a way to either force the header for a proxy or allow me to insert a custom header and achieve the same.

When explicitly providing an ip/host/user/password via new WebProxy(ip, port) { Credentials = } I dont see how sending the header can leak.

[wfurt]

Triage: we can look at it. It should probably be controlled via property. Basic auth is insecure and it probably should not be used with plain-text like connect to proxy.

[matteocontrini]

Hello, has there been any progress on this issue?

I've encountered this too, on a proxy that doesn't return 407 but 401 so HttpClient doesn't actually work at all with this proxy. Plus, with #50763 debugging was made harder since the proxy response cannot be accessed.

I believe this behavior should at least be documented somewhere. I don't think it's expected by anyone, for example both curl and Python's urllib send credentials immediately if you provide them.

Is passing the Proxy-Authorization header manually a viable workaround or does it have drawbacks?

[jgilbert2017]

@wfurt it's been 1 1/2 years and upon revisting your last response and I would like to make a few points:

• "Basic auth is insecure and it probably should not be used with plain-text like connect to proxy."

I think the precise statement here is that basic auth is inseure IF used via plaintext HTTP. It is not insecure if used over HTTPS.

• Its good advice to say don't use insecure protocols, however, the user very often has no control over how the endpoint he is connecting to is configured. What you are stating can also apply to a blanket statement of don't use HTTP at all. While I agree that is a worthy goal, HTTP only servers still exist on the Internet and need to be accessed by people.

• Additionally, you must look at the cost/benefit of the design decision. What are the chances that a library user is accidentally providing credentials to a proxy server that didn't need them (which is what the existing design theoretically protects against) vs the penalty imposed against all users that actually need to access a proxy server that uses basic auth, which I would argue would be the overwhelming majority of cases. And unfortunately from my perspective this penalty (doubling the number of round trips) is extremely large.

[wfurt]

we may look at it for 9. With #31113 done, there at least could be extra layer of protection. (or the auth targets random users & malware) I would expect that the cost is not as high as for example #33964 as the extra round would be amortized over all the requests over duration of the proxy connection, right?

What bothers me personally is that PreAuthenticate could/would be common for proxy and the actual sites so somebody setting it for the proxy may accidentally spill beans in cases they do not envision.

[matteocontrini]

the extra round would be amortized over all the requests over duration of the proxy connection, right?

True, but in some cases proxies are rotating proxies and they may require a new connection/session in order to be assigned to a new exit IP, so if you reconnect often the impact might not be negligible. The current behavior also makes it more likely to be blacklisted because of too many attempts without credentials (happened to me).

What bothers me personally is that PreAuthenticate could/would be common for proxy and the actual sites so somebody setting it for the proxy may accidentally spill beans in cases they do not envision.

I'm not entirely familiar with what PreAuthenticate does exactly, but how could the proxy credentials be leaked to the actual website? Aren't proxy credentials passed in the Proxy-Authorization header, which should be then removed by the proxy server?

Also, I'm still not sure I understand why credentials aren't sent directly on the first request. I don't see where the security issue is. If you're explicitly setting a proxy and its credentials, why should the credentials not be sent? That's how other frameworks work, it's a reasonable expectation.

Plus, RFC9110 says that:

Likewise, upon receipt of a request that omits proxy credentials or contains invalid or partial proxy credentials, a proxy that requires authentication SHOULD generate a 407 (Proxy Authentication Required) response that contains a Proxy-Authenticate header field with at least one (possibly new) challenge applicable to the proxy.

Given the SHOULD, a proxy server can be RFC-compliant and not return 407. In that case the proxy won't work at all with the .NET HttpClient.

[jgilbert2017]

@wfurt unfortunately my proxy requests are one and done due to either the proxy closing the connection or rotation. as you say, PreAuthenticate is the common case for proxy usage, so please don't penalize normal users for a hypothetical benefit to those who don't know what they are doing.

[jgilbert2017]

How about an API like client.AllowInsecurePlaintextAuthentication = true which tells the library not to try to save the user from themselves?

[wfurt]

On normal request, PreAuthenticate will cause HttpClient to send credentials without waiting for server's request @matteocontrini. The proposed AllowInsecurePlaintextAuthentication seems overcomplicating to me. It is not question of if but more when. The code would already send credentials in plaintext when asked to.

I can agree with argument that if DefaultProxyCredentials is set explicitly, there is clear desire to authenticate and there may not be benefit of holding back the authentication. It may be trickier if Basic authentication is not what the server wants but I guess it can still return 407 with supported methods and perhaps fall-back to Kerberos/Negotiate. I do feel the the manual Proxy-Authorization may be decent workaround - perhaps using HttpClient.DefaultRequestHeaders.

Now there were also references to rotating proxies. AFAIK HttpClient does not handle that very well IMHO and it may be interesting to dig deeper into it. .NET would pick proxy on application start and that would generally remain static

[jgilbert2017]

imo rotating proxies do not need to be handled internally and are easily managed by the user who may have more application specific analytics as to which proxy to choose.

whatever API you think is not overly complicated i'm fine with as long is there is some way to avoid sending initial requests without the authentication information.

[yowl]

Second this request. For me the problem is diagnosing 407 errors is hard enough, without having HttpClient generate them by default.

Edit. Also confirm setting httpClient.DefaultRequestHeaders.ProxyAuthorization is a usable workaround.

[wfurt]

I like it @yowl because it is explicit and fully in your control.