Web Push as a way to ingest messages #765
Comments
|
Given that this requires some work to get done, I'd happily help out with a PoC to show the intended direction. But before I do that, it would be nice to understand if it's at all of interest within the intended scope of Gotify. |
|
As an aside, Web Push would lay the foundation for supporting end-to-end encrypted content in the future. Right now, my idea would be for gotify to act as the message relayer and the recipient, meaning it would terminate the encryption used in Web Push. |
|
Thanks for the thorough analysis. This would need some good amount work and require a lot of restructuring. I am good with standardization. Nomenclature mapping as I understand:
A couple question from me to make sure it is worth the effort:
Related but not the same thing: #344. Edit: Over-interpreted the scope a little. |
|
@AlexGustafsson Expanding a little on how I imagine it can work and potentially make it useful as an independent push service: Gotify impersonates as a WebPush UA and Service at the same time, and supports one or both workflows to subscribe to a Web-Push capable application:
Did I understand it correctly? Feel free to correct me if I have misunderstood the intended use of this. |
Yes, this is the gist of it. Act as a UA and service. Don't involve browser settings. One additional clarification; In the typical flow, the web app receives the subscription via the UA, then sends it to the application server. I'm not aware of any standard means used by application servers to handle this request (i.e. actually creating the subscription in the application server). As such, my current assumption is that the user would have to create a subscription in the Gotify Web UI, get the "PushSubscription object" and then provide it to their application server as they see fit. The win here would be that Gotify could act as a drop-in target for any service supporting web push, "shoehorning" its way in to existing flows. Just a quick flow example through code, based on MDN's example: https://developer.mozilla.org/en-US/docs/Web/API/PushSubscription#example -// Get a PushSubscription object from the browser
-const pushSubscription = (await serviceWorkerRegistration.pushManager.subscribe()).toJSON();
+// The user got the same JSON object from Gotify via its Web UI
// Create an object containing the information needed by the app server
const subscriptionObject = {
...pushSubscription,
encoding: PushManager.supportedContentEncodings,
/* other app-specific data, such as user identity */
};
// Stringify the object an post to the app server
fetch(`https://example.com/push/`, {
method: "post",
body: JSON.stringify(subscriptionObject);
}); |
|
Awesome, I love this idea. Just one more thing: If all it does is do a "virtual Web-Push subscription", why not build it as a dedicated library and in turn a gotify plugin or standalone server that just subscribes to the application server and transform all incoming requests into arbitrary actions? I assume it is possible to just encrypt all the app-specific state required in the subscription URL itself and this service can be fully stateless to the point of deployable as a cloud function. Basically:
I am open to collaboration regardless how you want to do this after I work on the OIDC issue, but if we are doing it here I have to be sure it is the right decision to put this logic in Gotify itself. |
|
I've started playing around with a plugin just to get a sense of how it could be achieved and understanding how Gotify works, using I think that we could get away with creating an application in Gotify and achieve something stateless if we do what you suggest. But as I understand it, it's out of the realms of what's possible as a plugin today? There are no means to create / list applications? I agree with your points. But in the manual case, how would the user end up with a shared secret and a ECDH public which Gotify associates with the subscription (in order to decrypt messages it receives)? Normally the browsers would get the endpoint from the server, but generate the keys itself. In our case, with Gotify acting as both the service and UA, I would assume the subscribe API could give the user the public key and shared key as extension headers in response to a call to Or we borrow User requests a subscription: Gotify generates keys and returns a ”fat" endpoint containing all necessary information to statelessly "route" and decrypt any message it receives. It also returns the parts typically generated by the UA so that the user can easily relay them to their application of choice. For a plugin to work, with just |
Regarding CouplingYou are free to do it either way, my personal suggestion is bottom up (get this working independently as a library first then connect with gotify) this for the reason of:
If you have a prototype and are comfortable feel free to share it here or invite me privately for a review. Regarding PluginsGotify plugins are pretty much just executable code we pull in, trust and pass in a couple handles so you can register arbitrary routes, it is more of a "plug board" to wire internal features up (like register your own arbitrary route and translate messages back to format). The primary reason I designed this plugin system is for the "last-mile" problem where every service has a custom webhook format or authentication and a plugin is more manageable than a custom server for the sole purpose of adopting a specific webhook service to the gotify format. I didn't really expect it to contain any highly complex logic. Your case is probably possible with plugin but I suggest try standalone first and then think about whether it fits, it doesn't have to be in my opinion. Brain storm for what you think a user wants not what the API wants. Regarding EncryptionI will have to think about this later when I have more time, but generally speaking I am not too interested in making this truly "E2E" for the same reason I didn't want gotify to be E2E (see #63). If you are dedicated to the gotify model of clients own other clients+applications don't spend time designing a system that provides more security than the current gotify model caps at, which in this specific case means I am leaning towards simplifying some elements of crypto in web-push in favor of a stateless deployment since we don't really have the concern of multiple consumers or a malicious push service here. I have some background in auditing crypto systems but not exactly the web-push model so I have to learn more specifics before giving a reply. Deliberate weakening (altering characteristics) of an existing crypto system is a slippery slope and I want to get it right. I will get back to you, but it will likely involve some deterministic keying based on application server identity and have a security characteristic similar to gotify itself (no separation between clients but applications each are fully isolated). I am looking at this technical overview and seeing how to adapt it: |
Regarding user workflowYour current approach looks good. Let's keep our options open by making My general vision for the benefit of this is we could potentially wire the browser PushManager API to you as well (and thus your notification will be delivered to gotify as opposed to that specific browser over their own push service). So a user can just visit a website, click "allow notifications", the website JS use the PushManager API to tell us the subscription details, we instead of requesting a subscription from the browser push service, request a virtual subscription from gotify instead and pass it to the website. I think this can drastically improve the usefulness of this feature by making most web push websites support gotify automatically, but it will likely require significant development effort because AFAIK Firefox is the only browser that has a semi-official way to use a custom push service, and on other browser we need to replace the PushManager object which isn't easy in third party service worker context. By making |
|
Thanks for all the good input. I'll be looking at playing with it in a separate setting then, bringing gotify into the mix once some progress is made and it's clearer what the right direction is. Mozilla doesn't seem to use the Web Push standard to subscribe to events from browsers. They're using something they call So to act as a drop-in replacement for Firefox users, we'd need to support that WebSocket and message format as well. Edit: The below parts are left for completeness, leading up to me finding the above info. I've done some tests with Firefox and from what I can tell, it's not really adhering to the spec? The setting you've referred to, I'm guessing is By mitming the endpoint, I'm seeing messages like these (keys and ids exchanged for random but similar values): // Outgoing, from the UA
{"channelID":"d9593f0f-cee5-4a59-8bea-309d8f70c812","messageType":"register","key":"hv+95/B6vaCXpk1gjZcQQnM7Q15J5QOu2//vqLiZU10="}
// Incoming, from the service
{"messageType":"register","channelID":"d9593f0f-cee5-4a59-8bea-309d8f70c812","status":200,"pushEndpoint":"https://updates.push.service
s.mozilla.com/wpush/v2/RwfvgYDn1Bt1ZIqb2iNQYB2pBE4xiexdmhx6h9UD01/Gn8rslSALHNur4/ZGXgaa6XkSKigTGiqCWVUsFxTNHKt1HW4wlXlawt0pSaeuziGchalliV1AmPrITm0leyJFxFkbd9kXz2c"}The endpoint in the incoming message is exactly the one I'm seeing from the So I'm guessing there are two things here to solve for the browser/Firefox use case - acting as a spec-compliant service AND work as a drop-in replacement for Firefox' WebSocket, mapping the requests to the compliant service API (which for all we know could be what Mozilla is doing). Worth noting is that the same WebSocket is also used for getting the actual events and acknowledging them. The message content seems to follow the same standards as the RFCs we've found. After going down a bit of a rabbit hole (I'm not accustomed to Mozilla's stuff) I found that there's a long history of push services and that they're using https://github.com/mozilla-services/autopush-rs these days. Of interest is probably their docs on the Web Push standard parts: |
|
When I think about it, though, it will probably be much easier implementing support for Firefox in parallel anyway - to improve tooling and testing. By using Mozillas' services, one would be testing towards a correct service at all times, piggybacking from their "known-good" implementation, rather than just reading specs. So something like a step 1 where you act as a client WebSocket using the same APIs Firefox is to create subscriptions and receive messages, then as a step 2 you create a compliant application server to send messages that can be received by the client, still using Mozillas' backend and then, as a step 3, you replace Mozilla's backend with a compliant server. Then it's "just" a matter of packaging the components to fit the use case we're trying to solve. Be it a Gotify (existing subscriptions) -> Firefox push, a Firefox (new subscriptions) -> Gotify push or a "standard" Web Push -> Gotify push. |
|
From a planning perspective just to reiterate I think the major ROI here is (1) When you can just take an off the shelf Web-Push library to write your application server without explicit gotify support. (2) When a user can use the PushManager API to subscribe to a WP web app without explicit support for gotify I have to look more into this but basically I think we only need to replicate the browser requests to the push service and application-side protocol. I agree with using a known good implementation as a reference or testing target but I encourage you to think out of the box for the internal implementation (how to encode the URL, etc) because our scale of deployment and privacy requirements are different. Regarding the browser requests: yes I think that will require browser specific effort although I still think it is likely the resistance for Firefox will be lowest. Let's focus on ROI (1) first and deal with how to get the subscription information from the browser to gotify later. |
|
I've written some code to try things out. It's a set of libraries and basic tools. It's all just a PoC and fairly hacky. There's basically zero dependencies, to learn and to not hide the scope of the required implementation. https://github.com/AlexGustafsson/web-push-poc Besides the basic Web Push functionality of handling encryption and VAPID, it basically contains means to:
The last tool shows how a stateless service could be written to act as a user agent and a push service. It essentially proves that our first ROI point could be achieved rather easily. Granted, there are a lot of todos, but they're mostly for a production-grade implementation, nothing that I think is required in the scope of a quick and dirty PoC. I've played around with a stateless endpoint token like we've discussed. It contains a version, a subscription id, the application server's public key and the user agent's private key. It should probably contain the authentication secret as well. All fields but the version are encrypted, the version is included in the additional data. See https://github.com/AlexGustafsson/web-push-poc/blob/main/cmd/agent/token.go. An example push endpoint looks like this: The token size is within 10B of Mozilla's tokens. They seem to be doing some sharding and sorting improvements and store the subscription (channel) id in a database. The application server's public key is included for quick verification of the sender. In order to PoC ROI 2, I've just taken a quick look at Firefox' extension APIs. Like you said, we'd effectively need to replace the |
|
Thanks for the update. I will get back to you next week , sorry I have been busy this week. |
|
@AlexGustafsson Sorry for the delay, I see you made some good progress on the push service to sender side . I am able to make some progress on the browser side, I can now reliably get a web push application to callback to my server and I can verify their VAPID JWT now (So basically we have an opaque webhook where we can authenticate to an origin). My POC is here: https://github.com/eternal-flame-AD/autopush-proxy-poc Do you have any real world sites or open source server that use DeclarativeWebPush you mentioned? |
|
My thought about potentially automating this process (to make it just work on browser and 3rd party site) have two alternatives, one is to find a way to replace the I think either are possible, the first is complex (potentially detectable by fingerprinters/JS challenges) but reliable the second is easy but less reliable. I think a major blocker for this (and the idea of ingesting DeclarativeWebpush in general) is still low adoption, I am having a lot of trouble finding test cases. |
|
I think the encrypted Web Push format is only worth adopting if we can close the loop all the way around from the browser to the web app, and it is likely too big to maintain here. However I am open to an architecture of making gotify support the plaintext DeclarativeWebhook format and a separate library to handle the encryption part (probably not owned by the Gotify org but I am open to collaborating with you outside). I can see the benefit of a standardized and not overly complicated format even if the adoption is currently low. |
Web Push provides standardized means not only for web clients to subscribe to and receive messages, but for servers to send them.
Previously, the subscription mechanism and the authentication mechanism servers would use were well-defined. Recent developments have also resulted in the actual message format moving to a standardized format, pushed by WebKit. The feature is available in the latest version of Safari Technology Preview.
I propose supporting Web Push in gotify - as a way of ingesting messages in a standardized way, making gotify compatible with existing and future Web Push-aware servers, meaning users of said applications could use gotify without dedicated support in the applications themselves.
In the normal Web Push flow, a user would request a subscription via their browser vendor's infrastructure. In this case, my suggestion is for gotify to act as that vendor. By letting users create a Web Push subscription via gotify, they could get the same reply from gotify as they would from their browser, letting them set up their applications to send messages to gotify.
Gotify would then receive messages from the services and respond like a Web Push server, supporting the subset of features that is mandated and makes sense in gotify's context.
As for the message format, I propose to use the format specified by WebKit, moving towards standardization: https://github.com/WebKit/explainers/blob/main/DeclarativeWebPush/README.md. W3C discussion
Useful article for an overview of Web Push: https://web.dev/articles/push-notifications-web-push-protocol.
Relevant RFCs and articles
Message field mapping
Below is a rough field mapping between Gotify's current message format and the message format proposed for Web Push. Note that there are additional fields in the proposed Web Push format that could be used for features like the image URLs, android actions, but they would likely not be supported by Web Push-compatible servers without additional work on their end.
{ "title": "My title", "message": "My message", "extras": { "client::display": { "contentType": "text/plain" }, "client::notification": { "click": { "url": "https://gotify.net" }, "bigImageUrl": "https://placekitten.com/400/300", }, "android::action": { "onReceive": { "intentUrl": "https://gotify.net" } } } }{ "title": "My title", "options": { "body": "My message" }, "default_action_url": "https://gotify.net" }The text was updated successfully, but these errors were encountered: