OAuth not working with prefixed app or behind a reverse proxy

[TheoMathurin]

Tested with Panel 0.13.0 and some older versions, Bokeh 2.4.2.

I've been using the OAuth authentication feature provided by Panel with a custom provider for some time. It's working very well and I appreciate how easy it is to set up.

Now, I'm currently working on a new production environment where the app is behind a reverse proxy (see recent topic on discourse). The server is launched with something like:

panel serve myapp/ --oauth-provider provider --oauth-key key --oauth-secret secret --cookie-secret cookie-secret --oauth-encryption-key encryption-key --oauth-redirect-uri http://host/myapp --allow-websocket-origin host

With a random main.py in myapp/:

import panel as pn

content = pn.Column('Hey')
content.servable()

While trying to make it work, I realized that the auth.py module has some limitations regarding the management of URLs and redirection. Namely, the process fails if the app is available at a specific path. This is typically the case in a reverse proxy setup, but also if you use a prefix.

First, if you look at the latest version of the file, line 752 there is a return statement that builds the wrong URL. Trying to access http://host/myapp, the URL will change to http://host/login?next=%2Fmyapp, i.e. the path is ignored. This yields a 404 Error.

I've found that this can be fixed with the following change to the login_url method:

def login_url(self):
    if config.oauth_redirect_uri is None:
        return '/login'
    else:
        return urlparse(config.oauth_redirect_uri).path + '/login'

This will get the user to the provider page. Then, the path is also ignored in the redirection (self.redirect method call line 269) so that after the user has been authorized, Tornado redirects to http://host/ instead of http://host/myapp. Passing urlparse(redirect_uri).path instead of "/" solves the issue.

Note that the same issue arises without reverse proxying if you use a prefix, although the fix I put in place does not seem to work for some reason. In other words, you won't be able to set up authentication in this configuration:

panel serve myapp/ --oauth-provider provider --oauth-key key --oauth-secret secret --cookie-secret cookie-secret --oauth-encryption-key encryption-key --oauth-redirect-uri http://host:5006/prefix/myapp --prefix prefix 

[AndrewGriffinGIT]

@TheoMathurin Hi Theo, I am also running nginx reverse proxy and having hard time with azure oauth. Keeps saying The redirect URI does not match. Any tips on what my redirect should be set to on azure and what i need to specify on the command line? My nginx listens on 443 and reroutes to upstream server 127.0.0.1:5100

[TheoMathurin]

Hi @AndrewGriffinGIT

Have you tried editing the auth.py module as suggested? The redirect URI given on the command line should be the address used to get to your app, e.g. https://your-server/your_app.

Then the auth module should actually redirect to this URL, including the path. This is currently not the case, which is what the suggested changes address.

[AndrewGriffinGIT]

@TheoMathurin hmm i tried what you suggested but that returns ERROR: 'module' object is not callable

@Property
def login_url(self):
if config.oauth_redirect_uri is None:
return '/login'
else:
return urlparse(config.oauth_redirect_uri).path + '/login'

[MarcSkovMadsen]

Can you try urlparse.urlparse instead of urlparse?

Or "from urlparse import urlparse" instead of "import urlparse".

[AndrewGriffinGIT]

Still cant get this thing to work properly. No longer complains about redirect mismatch but app does not come up. Registered the redirect on azure. When i login into the server the authentication seems to be in a loop 3 or 4 times and never passes. Just tryong to hit my app server, authenticate with azure ad, then redirect back to app server on 443, then have nginx route traffic to an available port.

Does the redirect need to be --oauth-redirect-uri http://host:5006/prefix/myapp... I would prefer if i could use https://myappserver.com:443 so my reverse proxy could load balance.

[TheoMathurin]

The --oauth-redirect-uri should be https://myappserver.com + any path that is set in the nginx config for your app. If there's no path then your issue is likely not related to the one I raised here.

What's the URL in your browser at the end of the process, after authentication?

[AndrewGriffinGIT]

http {
upstream frontends {
least_conn;
keepalive_timeout 180s;
keepalive_requests 100;
#keepalive 10;
server 127.0.0.1:5100;
server 127.0.0.1:5101;
server 127.0.0.1:5102;
server 127.0.0.1:5103;
server 127.0.0.1:5104;
}
sendfile on;
tcp_nopush on;
tcp_nodelay on;
#keepalive_timeout 900;
types_hash_max_size 2048;
#include C:/Users/smaurice/AppData/Local/Programs/Python/Python35/Lib/site-packages/bokeh/server/static/;
include C:/Users/myuser/AppData/Local/ESRI/conda/envs/arcgispro-py3-pyviz/lib/site-packages/bokeh/server/static;

default_type application/octet-stream;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/plain application/x-javascript text/xml text/css application/json;
open_file_cache max=10000 inactive=10m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
ignore_invalid_headers on;
client_max_body_size 8m;
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;
connection_pool_size 256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size 4k;
output_buffers 4 32k;
postpone_output 1460;
server {
#listen
listen 443 ssl;
ssl_certificate ./ssl/certificate.pem;
ssl_certificate_key ./ssl/private.key;
server_name _;

access_log ./logs/access.log;
error_log ./logs/error.log debug;

location / {
proxy_pass http://frontends;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
}
}

[AndrewGriffinGIT]

--oauth-redirect-uri=https://myappserver.com

[AndrewGriffinGIT]

502
https://myappserver.com/login?next=%2F%3Fcode......

[AndrewGriffinGIT]

@property
def login_url(self):
    if config.oauth_redirect_uri is None:
        return '/login'
    else:
        return urlparse.urlparse(config.oauth_redirect_uri).path + '/login'

[TheoMathurin]

Ok I woud advise you to try to set up authentication first with a very simple app and the most basic nginx config possible (see Bokeh docs).

[AndrewGriffinGIT]

azure would only need the web redirect url setup right? Json web tokens wont fly around here.

[TheoMathurin]

For me as of 0.14.1 the fix does not seem to work. Behind a reverse proxy I get the following URL with a 404 error: http://host/myapp/login?next=%2Fmyapp. And with a prefix I get http://host:5006/prefix/myapp/login?next=%2Fprefix%2Fmyapp

I should have tried earlier, I learned it the hard way today in a professional setting.

Can someone confirm? If that is correct, can we reopen this or should I file a separate issue?

[TheoMathurin]

@philippjfr, if you have an idea about what should be looked at in order to fix this, I'm willing to investigate.

As reported in my previous comment, the path is correctly copied over but you are not redirected to the provider login page. If you're already logged in though, you can access the app directly as an authenticated user.

[TheoMathurin]

Regarding the prefix case, I've found that you can make it work if the OAuth redirect URI only includes the prefix in the path and not the full path to the app (e.g. http://host:5006/prefix instead of http://host:5006/prefix/myapp). The redirect will then get you to the app if it's the only app under that prefix and you don't set --disable-index-redirect.

In fact in the login_url method, apparently only the prefix should be added before '/login'.

Still trying to make the reverse proxy setup work. It should be noted that with Panel 0.12.4 + the fix I mentioned when I filed the issue, it does work when I try to acces the URL for which I now have a 404 error (http://host/myapp/login?next=%2Fmyapp).

[philippjfr]

Thanks @TheoMathurin. If you want let's set up a joint debugging session to figure this out.

[TheoMathurin]

Sure, I can reach out to you on gitter or via e-mail, whichever suits you best.

[TheoMathurin]

Ok it does work behind a reverse proxy if you specify the URL path when adding the endpoints:

@property
def endpoints(self) -> List[Tuple[str, Type[RequestHandler]]]:
    ''' URL patterns for login/logout endpoints.

    '''
    endpoints: List[Tuple[str, Type[RequestHandler]]] = []
    if self.login_handler:
        assert self.login_url is not None
        # Change in line below
        endpoints.append((urlparse.urlparse(config.oauth_redirect_uri).path + '/login', self.login_handler))
    if self.logout_handler:
        assert self.logout_url is not None
        endpoints.append(('/logout', self.logout_handler))
    return endpoints

Edit: This fix will work as is if the URI path happens to be the name of your app (see comment below)

[TheoMathurin]

In fact with this fix, it does not work anymore with a prefixed app... I will not consider a PR since it breaks something while fixing another. It looks like something a bit more elaborate is needed. Most notably it may be necessary to check whether a prefix has been set to add the adequate endpoint.

As I'm mostly interested in apps behind a reverse proxy though, I will use this change for now.

[TheoMathurin]

It's actually a bit tricky even without considering prefixes.

First, the endpoint string needs to correspond to the name of your app. For instance if the web server proxies requests from host/app1 to host:$port/myapp then the endpoint string should be "/myapp/login" while the string returned by login_url should be "/app1/login".

Second, if you're not using the name of your app as the path, the next parameter in the URL is not set correctly, so that the redirect immediatly after authentication does not work. It will redirect to host/myapp instead of host/app1. I could confirm that the redirect works by trying to access the login URL directly with the right next parameter (accessing localhost/app1/login?next=%2Fapp1 instead of localhost/app1/login?next=%2Fmyapp). The next_url returned by get_argument method in OAuthLoginHandler, and in turn the get_state method, is wrong.

[aspect-ndujar]

I can confirm we are also observing this issue with Version: 1.3.8
Our setup uses Keycloak as the OAuth provider, running inside a Kubernetes cluster and an Nginx reverse proxy.
How we fixed it:

nginx:

- host: "yourdomain.com"
        paths:
          - path: /panel-apps/

For the env vars:

• PANEL_OAUTH_KEY: panel-app
• PANEL_OAUTH_AUTHORIZE_URL: "http://yourdomain.com/auth/realms/auth"
• PANEL_OAUTH_TOKEN_URL: "http://yourdomain.com/auth/realms/token"
• PANEL_OAUTH_USER_URL: "http://yourdomain.com/auth/realms/userinfo"
• PANEL_OAUTH_REDIRECT_URI: "http://yourdomain.com/panel-apps/"

Then serve as:

panel serve --oauth-provider=generic --prefix /panel-apps/ panel_app1.py panel_app2.py

NOTE:
After trying several solutions, the actual trick was to serve the panel app with the correct --prefix parameter.

[TheoMathurin]

Update to my comment from last year, from what I have understood after some tests with the code as of 1.3.8 behind Apache acting as a reverse proxy.

Setup: a Panel app called myapp running at 127.0.0.1:5006/myapp (with --address and --allow-websocket-origin both set to 127.0.0.1) with configuration for a given OAuth service. PANEL_OAUTH_REDIRECT_URI should be http://127.0.0.1/app. Apache has rules to redirect http and ws requests for /app to 127.0.0.1:5006/myapp(/ws).

The first problem raised is now different: the login endpoint is now hardcoded to "/login" so you must have a dedicated rule that redirects 127.0.0.1/login to 127.0.0.1:5006/login. AFAIK, this is not expected behaviour. The URL path as entered by the client in the browser should be taken into account (maybe inferred from the redirect URI) so that the generic rule also works for the login endpoint.

The second point is still an issue. After authentication you are still redirected to http://127.0.0.1/myapp instead of http://127.0.0.1/app. While the host is correctly extracted from the redirect URI, the path is ignored and somehow replaced by that of the Tornado URL (returned by the get_argument method in get_state). Once you are authenticated though, you can manually enter the right URL and access the app.

But even then there's now a third issue. I still can't get my application to work. The browser indicates a failure to establish a websocket connection (WebSocket connection to 'ws://127.0.0.1/app/ws' failed) while I do have a rule that redirects ws://127.0.0.1/app/ws to ws://127.0.0.1:5006/app/ws which works fine if authentication is disabled.

[philippjfr]

Many thanks for reporting back here.

The first problem raised is now different: the login endpoint is now hardcoded to "/login" so you must have a dedicated rule that redirects 127.0.0.1/login to 127.0.0.1:5006/login. AFAIK, this is not expected behaviour. The URL path as entered by the client in the browser should be taken into account (maybe inferred from the redirect URI) so that the generic rule also works for the login endpoint.

Not fully following this point. Note that the endpoint is configurable with --login-endpoint.

After authentication you are still redirected to http://127.0.0.1/myapp instead of http://127.0.0.1/app. While the host is correctly extracted from the redirect URI, the path is ignored and somehow replaced by that of the Tornado URL (returned by the get_argument method in get_state).

This can also be overridden by setting --oauth-redirect-uri. By default the redirect_uri is generated as:

redirect_uri = "{0}://{1}".format(
      self.request.protocol,
      self.request.host
  )

and the path should be taken from the original URL you visited (although the logic here is convoluted and I'd have to look at this again.

But even then there's now a third issue. I still can't get my application to work. The browser indicates a failure to establish a websocket connection (WebSocket connection to 'ws://127.0.0.1/app/ws' failed) while I do have a rule that redirects ws://127.0.0.1/app/ws to ws://127.0.0.1:5006/app/ws which works fine if authentication is disabled.

Without seeing the precise configuration here I can't help. What I do know is that we routinely run authenticated Panel apps behind reverse proxies (primarily nginx) without issue.

[TheoMathurin]

Many thanks @philippjfr for your reply.

Not fully following this point. Note that the endpoint is configurable with --login-endpoint.

I did not know about --login-endpoint, apparently it's not documented.

I'm having a hard time trying to use it though. The default is just /login, and setting it to /app/login or /myapp/login does no good. Again to me it looks like there is a confusion between URL paths handled by Tornado and those seen by the client. If Tornado redirects the browser to some URL, it will then in turn be processed by the reverse proxy.

So for instance if you set --login-endpoint to /myapp/login, Tornado will redirect the browser to /myapp/login which leads Apache to return 404 (the app is accessible at /app). If you set it to /app/login, the path of the request received by Tornado after being proxied is /myapp/login which also yields a 404 error (this time returned by Tornado).

This can also be overridden by setting --oauth-redirect-uri. By default the redirect_uri is generated as:

redirect_uri = "{0}://{1}".format(
     self.request.protocol,
      self.request.host
  )

and the path should be taken from the original URL you visited (although the logic here is convoluted and I'd have to look at this again.

I do provide a redirect URI through the PANEL_OAUTH_REDIRECT_URI environment variable and I'm saying that the path is not taken from this variable or the URL you visited (as seen in the client browser), but from the local URL (Tornado application). It will work only if the two paths are the same (see my comment last year).

Without seeing the precise configuration here I can't help. What I do know is that we routinely run authenticated Panel apps behind reverse proxies (primarily nginx) without issue.

I'm using the rules indicated in the Bokeh docs on reverse proxying with Apache, but I agree that knowledge of the full configuration is necessary to debug this.

[TheoMathurin]

Anyway the first two issues are manageable. It only involves adding a rule for the login endpoint in Apache and modifying get_state to yield the correct next_url in auth.py.

I've looked into the third (new) issue. In the Apache logs I got the following:

[Mon Mar 11 20:55:01.351381 2024] [core:debug] [pid 9084:tid 140114324985408] protocol.c(1118): (28)No space left on device: [client 127.0.0.1:37552] Failed to read request header line Sec-WebSocket-Protocol: bokeh, eyJzZXNzaW9uX2lkIjogIklIVzNuc1ZMSjhlUFJBM0hkbzdiUDM0anB1VTNtdG9mUE1sSjI0ZlNhSG1SIiwgInNlc3Npb25fZXhwaXJ5IjogMTcxMDE4NzIwMCwgIl9fYmtfX3psaWJfIjogImVOcUZtWDJQbzdpV3hyOUtxXy02dXpzMWczbEpWZnBxdEdvcW1JUUtUbUg4QXRaSVY0QkpFVENFU3FoS3d2Yjk3bnVxZS01cVp1LS1xTHVrQ0J4akg1X3puTjlEX3VOelV4ZTZQcDBfZl9uMEg1X1h4X01FSHo0ai1fNW5DXzZoeno5OS12eFlWRTE5OTNnY3B0UFJmTnp0aS10ZDhWTF9hbjNjVGV2cTdyRzU0OFhIbmQ4LWstUDBsNjlmX0ZNeDZOOC1fX1g5MTk4LTItNXZNTzYzejRfTjZkZ2YzdnJmTHlQYl91M3pueWE0aTRfbHdkUWY4X3o3bjZlLWV6YkZ0RC1lLWhfUDJCNkd0LXVQTF9QeDVRVEx2OXNNNTdwNk85VjN0SDU5cThfVC1mc3V2bzg0MTZlN3J5XzE4SDFqOFhFLUdGUDg0djFzZmZwTGh0QmZQMzJmN05QMVlmRzNoZnN2bjc2T282bGxYVDRkcGw4ODVfNW5aX0hwTDA5ckZtOV8tbVFPWGYwcHJLdnUtQy1mdnUtbF9nWDI4RDFNMXFlMDJCZW53LTlmLVhqczE2cXF4LS1Qbk9ycjlFc3o5ZWFuQWlZX1ZNVjBPQTZfWEQtdV9OdjF2MV90elY5ZmY3Vi1YdjUwNkNIQ3Z4VHZoXzN2SHk5MU9mN2o2amk4X1BTdnZfenI5NkVQZjVyZ2ZIZ1phbjFYWDZ1bUdGNXFpSFRwZkI5Ml80LUE0bnFxbXJ2ME1IMFA5SEFjNmpfZmlZXzZ4eDE0OWtzeF9iZTdIOUg4ZmtEb3o5ZFg5WV9FMGNmcXJmLUk5WC1GNEM0WXFxTS1EQzhmdDFfbXdfalRKMTN2NFR6cm56NlZwei1NMjhLQzMyQjdILVBxNGFkNnVPUHA3OEhZbjM1czlXUDBabjlIWU5GM2NRSFBfWkVQSlNwZEY5WDNEMlh4NENLblFKWjNYLW1IaF9MQmhoT3FiTmVwNFlPSDdqNGVfeU52c2p0OFBGMktrNFpnd2FkX3l2a18zdjhmaS1LUEE5TDY5UDRqTEVOMVY1elBkLS1IOGVlemZ2OTVfMzFfVURsRFhYMGN6OGVRcDdvZTc3NmF3M3Y5LWU5d3N6b2V1MFA5b19yZWZnLXVfUTE5c2I0aDZ3dTZSeFo2V0N3Yzc1djc1ZVB1TjJmeFJZZU5LV19lbEdmMHFMTm9MRHNmbGIyeHRfSTZsdUhWYkh0MS0tYlZEM3ZQM2JzUG5vM3FfZjNTOVJ5WVpXSFpDNnVzeW9XbDk2aTh0LXVpY0VzWVVuaDd0eTdkdlZYWHFDcmMtdDc3V0hjQjUzSS1fMjA2ZHZYd3Z5MEwyVl8tT095YmZlOC1mRkVXOWhNT2ZfM0ZpVHNTc3hrM1JiZDA5WHAwQmNPV2xDVFY3Y3RGR2lxU21VNFVrME1hSkxkcUptM1ZlYTlDakZmV1hlZWFSNzRXMFJNeFpCZExXaWpaV2NKYUV0M1Jrd3l1ZThJOVYxaFhKNUhuU2ZGbG53XzBVVG02aVRFZUdjdG5hcHJuWWhWUkhucXE3cTVuMnBxY3JYUkNBc056d1MxdDhhbkNVWl9pNWpFZF9GaUc1RzNMUFpZNnpWUmxpcVZoRk9YaTRTbzduTzlXX3BsMjRoQ0g4YXh4TUNldDRsV0hudXExdVltUVg4VlFPVEljMzRvVjhRUnJYbFByNWFyUTVscWlGMGV1NDFzbDhLQlpFOUtWZVZUcmtSVTM5RVRRR0twc3pJVVZlU3lJckRTd0hONDFUQjlReXBqZVpiTTVGSWpFbkhzbTUzaGdHUGVxclJ3ZGRraktEUklIWkt1WnBqdGhXTVcxUnpLMVNpVDNKQjlaWXBaSGJXRXBRbzhrM1RVdXVRNEpwb0d5NkViemhyQlF2QllPSHVvUXZ4VXRYY1ZaczZNYzBkUlc2em8wWnhxU1JkNTd1QWdlcGtwT2hSSk5ENXJlN0xqTzB4YkxyZVFYWVpvbTRZSUwxRHl4Z2N3NS0tcndJSFpaZl9WNVdGbWxwVS02Nnh3VzRsNGFrX0RXR05raFQzUVJVZUU0cWNHMzB5eUN0YWh6M1NtZkI1NmI4RWttc2tHcFExODF2LUxjSm9sdXhjUTVMcWd6dHZLQWlneHRFSmVCbTRybW1uT2FDRWYzVFBpTGFxV01zcThSR3lJX2tiUlRQSnA1MEJ3b2pzTEtNVjRWTm9pMHlaemEwVHRGYWlwWE5PZHd6ckZSVDN5OThYaHZwcHByVWdxQ0s3TU1XTEFjT1l1dmNvVUxXUGRBT0gxVFF5UVp3dHM4MDdTeXI0ZUVtWlNpWmlGc3hYUXdkdHdzSDZzVk9aYlN2TVUyTVdWTEZqcFFlekxUOTdoWE04ZUJGd2ZpTWViMFZObkxnSzZpY3ltdjV6Z2NzeXBRY05oVUZtTGMxbnc1SzdUYzVSMDU4eUF5RU4tSnRZbGRyTFhNQmIzUW1TYUs0VlJJTC1lU05pb2dGNTdCSEN1UjA3WGFwZjAxTEhxdmtjRzRJcGtSaVMxc1prZS1NbUlkZHlQa1pUZWxRUVByYkE0NlMydy1xNHlzOFZETWdSdWo1bGhKUkJMNEg0ZjRPUjd3bS01TVhvbm9QYmUwQnpWeUtycG9JUkNlQ29TWndJbTdZNmJMN2VreFJlWTVzeU1iemlzZ3ZSRlZFSGpKb0U2eXA0S3Z4Qk9aMVZzdW1rZHQ2VXpoWmtfYVpwMzAwNDQ2YXJzTEdzWTY3OHdQeUtPSXZPX1dMNjd1bTBoaU05ZURFVkRiajBXZ0dqMXZyTXBDRVpfOUdEU2cxZWpCSTFLZmM3Nmt1NHhrM1BFZHlIc2otZEVTWE1YVW50N0lpblM1MEQ2VDAyTmh4SlQzT0NHaDJSWGhNcVV5V3FrUTRtWXBSVGdSVWxLWjJ5UG9VdUNDeWhZbHctLXBuRjYxUXhkcENQa0IzNFA4RllLSlJXSFVXcXowMDQ1ZkhkVUtTd1JxbG9GNnBWYjBtdmNtNWpPMkdTT0o0c0ZVSXBGWEgzWGJtN21jYzF1MHBNbHQxeTdGMG1KaVk2VzJOLWdlN1pUQlhkNUdqeWt6TGxuUmt4SWoxQVFTVUdtLU5ucEhCZWxqaVpfRTJ2amxlaVN3VjRnSFJTVldLUmN4S2pOcXk2R2JjeWwycEQ5ZXNoa1BpYTJmcUNYYXVFdmMydmhiYWk4ZHNtcTh6TDVPbE9ORktYWEM1NmFIc3ozWFBkM0ZNNTV5dm5GWVMwOHBObmxwZWJHUXpUa05zTmhKR2xkQ2UxSVlrektSU1ZGTnREZnJnbE1sYkF1Vk4zUlJVcWNmbXBPd19DS1FlV1JHTWNJakpWWjBsZmJlTzJpR1l0YnluUWZJcXdMdEZnNng5UkNScEJWRjNVVWJlclBtaEhXWGRGMjVOS1FER1Npcm1UNUJQSXBhVmw3VkUxT0h6WmpOVU91MjZFV3ZjaDZpdDEyNDlPV2FQdFlkRXRSMEh6M21zcFVpNGRhMGdlQnNSZXZ2T1NPUWpXWXVMT1RXcTlncGdzMkZ6T0t0QmswaDBEMVYyTGphR21mLWlLSmNna1lJaUxxa1d6RXJranVWSjR3WVNyeDhVLXg3YnR4S0p0b2lWRnpZNUYwUDJzOEZhYWwxNVZ1Qjg4S012c1FxemkzS0lIWS1zeUtfbHNMS0hKcHdiczI3VUYybzRkTnU5V0xMX3ZxY0NEcEswRHVvdXljaHI3aUNYSWE0WUdXTkNWd1B0UmluWFFBYUREcTZDNllMOU5RaHRyV19ZOGwxSjZCaUJfOVVvdWFWVzZORlYtUUEyaUozbkZ4S1BxNW9xNW8wVkNIVS1UcHh4dGN5UldMSDFNaFhKaGVTUEJjYzZndVBuTTVpRlhPanFsWm5kQzNlOWRwQTRMcXA2cGN1RDlSUXJocGJDLXdJSEcyWW83cFlLTDlnZ1ZWSmNZUnpQNHItd1U1UjRFR09oSEdvajRVZFg5TTVza28wR3FqVE1WNVRMODdFUGprZ2Z5ZWdCc3p5bVhlNkwzRTNFNGRmcVVSajJpLVZ3R1RJWjdHby1pc3FiTXZlNGRnaWpxX2lucndsYlhSaGZiU3RtSjZxQVhvTU5fdFk2bmZSQVIzTi1yMGVCQzNuSmxhZGppZ2VwY0NObFZxb0lQWlZhRzY1SktBdEc5UTVuMkVQd2N0TkJPTkdTNU1uRGlHVTBYV1ZSUzZUNDVxYTVGcmFGNnNPb2pmTmhFZWNaZ1g2OEY1bFRTLTc1YVZnWW9iLXVJQy1tVU1fVXlVeS13U3lWV2F4cTFia1hEQ0ZFMXVoV05BWjlya3BzNDJsT2JDSkVFSGFSbHowa3lVTUhrRmJMbVhXU05JU1NsQ3ppVHZzbHM3WE9VWUoxSURhc1hXRFN0Q1FkSGl4UVJmMmRkYThDVXUzekdxZUtOZnZ2TmVucXJkdXJOVldCaG9jTy1PN1lGRlVpeWlPTFc1ek93cTJqSUItcVlPRXVDb25FblY0OUlwZUcybUNLNTBqcWZrWWl4VHRTeHNTVm9xR3lhdWJNdFdVUTI1VmZDbUtQbHBEemo5eXNVd1kwbV9wME54QTk5X0t0ZXBBZXpjY3VFS3VtaUR0ajNZaGoyNHU4Vm9odkpOY3pXb1ZveUpZTHFUODBOa0lGN1k3czg2b3VJMXMzZVZYWVpzOERvQlZadF9UWnZsRVpjTUtpYmR4UzFQUmE1czRTdWtNRkxBYlhUVjNOMjJNUzN2cUE5TklhamRQT3NzdDBFUWhCWFNZbTJVbmZiTUhUdXlLNE9WU3NZaVd2ZXBvUUY4Wjc5eTROeDF4UU1ZR2NvbEZCUFJqamltUG5oUHBfdnJ0X3FIYU80WG5lbUFpRnU1eWdlcGk3OTU3R21TcXNHcDNhVHVMRXUzM3RhNXI1OEZkdXZiZTNpTzlLQXJQY3g2YzJ2MGc1NFAtdjZuNTRjc19Sbnl6N2NVX0EzTzgxam5EMVZSWS1xMHlaRXhNODFZY1VGd05CTWxIbENVemZwTDlFUUdvSVJVZ0h0c0drNkRaQVZBZXluNUNxUlhmZEw5RWxSbEhTSjRaeEl1U191eVZtVzR6V3g5U0d6LVRtVXo1RFNIeGlONlZkYkdWb0UtcFRZaVl4U3RuWnNOd2M0Tm1tUUFnN2FIZ0FPYXFheW00QzRWbXNkRDBUQ29fYnhXbXJhYXl4d0J3MXFVSXBvVmU4VXVKTkFSM3hQRUIwY3BwV29DVVl6VkVpUFhOWTJhYmt3eXhXX1g2VVE3LW9WeWJMYmV1YkpkaFhuTnFsU0ZLU0lmZWQ2SGxKSXlTSXB6R0dncU9TRzhTbHNwaXFmYUpqRFl4SjExc2lUanAwWlk0eHpsRnpWUTR3UzIzUFRzTnZTUEVaYTNYSXRvOXdqd3ljTFlzdXBVU1liMG03MGttRGtuclI3RmxSbWhXVHpGVGx3cWdNeG5HTG5VVVNweWp3dzBBb1VWSkJva0hJallJQy1icEFWYmsyQlZ0NE93eV8xQlo0MnNPeFZ1dENZc0RmR1JvOUluUURGTFFVTnZNdXlDd1JIaUZUSHR3Sld0dXdxSWo1OUd4Nks4SnNWMVVySVZELXh3TUVVbzB3QjViNFNDWDU1bjAwSTV3WjB1ajNpaFRKeDE0TWhIUXBFWFV3YlZ6YkVjM09KLUx6Qm9PY1Zoc3hZTmJENk9YT1J0M3QxYVdrSXB5ZTFwQncwaTNRdVJnZGJ5S2ZVVVZheHB5UV9zQzh6bnRhRnBodjRlY2VrNkZ1c1Vjb051YXp0cFFWMElocG1Gc0t5ZV9WVWJ2eVFwVHhWU3Z3U3lVMEZnVENYZmFydzd0a0ZWeWROWkJ4SE9yV1pmWjJHc0FRb0o4RDB6V09iTURMLTNHeXphRGs4cTR0ODJ3QUdCN1ptQ2M2QXB5MEFLVDBxTmp2QktyWEVaMk1ZeVUyR1lVSm5CS1FfM3FnTUFVTlFYa2FxUEV4Z053QTdCOXVWWU1taFU4UGJicFd6a1lWMW02QUlFd1d2aTdlcTFPUXRJLUZ1Tktya3hROXZSUlpnRWk5dEhUbWVGQ21pN3U4YmF3dlUwY2pINC00OWRDUG9DaDBrYy1OSFk2NEVjUkFpUS1vaTN0d1Y4WTlWeElBUTFhemJ1VkNhRTJJNTQxMEdnMlY3SG1seDJBVURvM3FleE1zZ3M1eWpQVGJETjZwWjFPWUE4QUtUR0NtbGJwcXNFc0JPTnFtcDYxWXFBclFhRGhBRU9DU05tUlNpd2FNYnNCVEZBSnhTb0J3eklMSVFhNUN1YXlqZHFQSElWY0Y5Q0hEMVJlQ3hKT0lHb1JaY0pRN1VCRGFURXZXUFJVSWh5U2Z2bVlpT2kxRHIxRk9uYzJBT1dSaEM5ejBwT0RaTWJoX1lzbi00YUpEMUEzcW1GSU9ZVXhRY29iRVl2OFZzejZFWnIzVmJmbXJDeTFvUllKU2tsbVBVQ2pHZ2lBdkw5TlVTUzNuSjVsZ045VnF5YmFVa3dkcmJSTjN3SEdRc3FiSTZ6S3l3ZFQxRmdCbUxtMjRtZExDSi1rWXNtcWNOeFZKcl9sanNZN3lHTW1BTllDQllDMHVaWHJxT2RadEUtbGVvV2V0NFB2elBIYXh4b29Fc3pacXpaUmw5bGdGSnlqbFNKNktKSGkybTRXVmViZnFremJpVzNPRlg3dzhybGh0Ri0tcGoxQXFBV3Nhc3lSaDhaTDJtWlZ5Z29nVEZobGwxeGgzWDNsa0Z2WnVSNE5tcmlDUEtsTnRPY2NHUkZhQUFLRzcxWTQwMUp2NDdVcU1nczVZZ1lVRFdEZnRsQzA5U25VS0FBSGZRUmJzMkJPWkFxT3oxdEJkZ0F3NzN6dFJ3cjBBSlRobHNKYU0zdTVUVHZ6bXM3SlJOZVVzazZwT291ZUdkZTdEd01HME5nckVkbFM2Qk9ZaDVNT3h5N3ZwZzZhOWdWZzdRem1aQ2h1aUNlWnVrS05YZmc2bjRtRVBBdVdSUXo2RFJyM0NHY2kwcTZaU3k1ZTQ2RHhXYURKTHFCdmFRc0dmVTFUS2FKOTZaQ0dHeF9BMm4tbDB0dDgxQWpKOEszQUFMTnJnUW9UQldCMnBHVHFxRmZSVGtBVmlaWHZnRDR3cU8tOHhKaHMtY1l0eEZKa0RoaXdma1JiaVFHQXpJMWs5REh0d1h6aDhUVjlSQ2xaNjdEc1JjSkVwQWpMcjFCRFQxV0tiRFowTl9WUms5QURhcTcyYVRCdWRaRGNFdFE0SHdCRnd4ZXZXQnN3SC1PbERLWTJtLU9yQXRNQno4LUt1WUZucTVtaHhpOUJzMmhMTExBS2hScDBWNFNKRjY4QnF1RmM0THdMenFjellRQm1VdGwxb0J1WTZ3Z2FPMEZIbWFyZ2VnWURzNmY4ZXFvUTZHemZIRVJ3ZmYzb1UyVkFPaEpNRTBPbUJYMERVNDFhM2pjT21QMEw3ZkJyemMyMlpnTFJnZHBnaHQ5cmpIT0NOU0l0ZnVKSS1MRlpxclJWMjBvMFhzbWJqRG5KeEZzU2dNbGV4eWFmNjBDb3F0TVg2TFZZQlRRaXBwcGpDOV9BVzlteGVQaklsek5nN3lBNE9YRzdzWGxJYmhKSFUyVWpRUUQtVTNsZFZjNFJwUkl0cXM0RGcwbHdrVFg3d21uc0REVTdFWm96QWI1STdZMmJDT1BIMWhLWEs5OUxzcVl0Qmg4VnEtaFZDQ0laQjVNNjBJWmhvZ1RrNlpaM0ZuUEdMTFB4SHVEeXNKWElZUm50Q1BSV0FNa25NQWdZWVBoOXR5TG
[Mon Mar 11 20:55:01.351460 2024] [core:info] [pid 9084:tid 140114324985408] [client 127.0.0.1:37552] AH00561: Request header exceeds LimitRequestFieldSize: Sec-WebSocket-Protocol
[Mon Mar 11 20:55:01.351465 2024] [core:debug] [pid 9084:tid 140114324985408] protocol.c(1481): [client 127.0.0.1:37552] AH00567: request failed: error reading the headers
[Mon Mar 11 20:55:52.292463 2024] [reqtimeout:info] [pid 9084:tid 140114333378112] [client 127.0.0.1:37548] AH01382: Request header read timeout

Note the unexpectedly long string sent with the Sec-WebSocket-Protocol header which breaks the size limit.

Looking at bokehjs, I think the relevant code that opens the websocket connection is here. We can see that the long string would correspond to this.token and maybe the authentication process results in passing something inappropriate. But since it's in bokehjs it's a bit tricky to debug.

[philippjfr]

That's right. Bokeh includes the token in the Websocket header on purpose. It's an inelegant solution but allows the original request and the Websocket connection to be handled by different replicas of the server. In nginx this sometimes requires setting

large_client_header_buffers 4 16k;

which should be documented in bokeh but we probably have to do a better job of making that visible. I'd hope there's an equivalent config option for Apache.

[TheoMathurin]

Yes there is! It's the LimitRequestFieldSize option mentioned in the logs and setting it to 16 ko fixes it.

Indeed this should definitely be mentioned in the Bokeh docs. I may submit a PR to that end.

Thanks for the tip @philippjfr

[philippjfr]

Tbh I think we probably also want a how-to set up a reverse proxy guide in Panel.

[TheoMathurin]

Ok. I've contributed to the Apache section in the Bokeh docs so I'm willing to help if needed.

[philippjfr]

If you just make a PR with a few bullet points I'd be happy to flesh it out.

[TheoMathurin]

Sure. However, in what way would this guide add information relative to the Bokeh guide? Are there elements that are specific to Panel in this regard?