Add Windows precision touchpad (PTP) absolute position support (proof of concept)

[AlexGuo1998]

Summary

Add absolute position support for Windows precision touchpads, make them behave like a drawing tablet.

Details

• Add interface IHasTouchpadInput : IWindow for windows supporting PTP data reporting.
  • Currently only SDL2WindowsWindow and SDL3WindowsWindow, should also be possible on macOS once Source aspect ratio information missing in TouchFingerEvent generated by trackpads libsdl-org/SDL#12391 is fixed.
  • This interface may be removed, or the methods merged into the base class if implemented by most windows.
• Refactor Windows raw input related functions into WindowsRawInputManager.
  • On SDL2, Windows raw input is used for raw mouse. After this refactor, raw mouse register to WindowsRawInputManager.RawMouse event when enabled.
• Implemented HID touchpad reading according to Microsoft specs in WindowsTouchpadReader, using HidP_ Windows API functions.
  • Windows open touchpads in exclusive mode, so you are unable to "open the device" and read/write HID reports directly (like with normal drawing tablets). The only option is to subscribe to raw input events.
  • HidSharpCore is not used, because to extract data with it we need a HID descriptor in bytes, but raw input only provides "preparsed data". It's possible to reconstruct the descriptor from ppd (and this is what HidSharpCore is doing, hiding in an internal class), but the logic is complicated and I don't feel like duplicating it.
  • This is translated from another C++ code (written by myself).
• Add TouchpadHandler to post-process the data from IHasTouchpadInput windows, emits MousePositionAbsoluteInput.
  • The first touch point is used as the mouse pointer. (not ideal)
  • Sensitivity is supported (reused with raw mouse, not ideal). Lowest sensitivity (1) maps the whole touchpad to the whole window while respecting aspect ratio, higher ones (n) maps from a fraction (1/n) of the touchpad.

TODOs

• Possibly remove IHasTouchpadInput.
• Raw input device management.

  TODO: If we enable more raw input devices later, it can be possible to run this code every time when reports for another device is received.
  For the best result, extract the device manager code into a new class (and cache all connected devices)

• Implement for macOS. (See the SDL issue above)
• Suppress mouse click after touchpad actions. (Tapping the touchpad can generate a mouse click, based on Windows settings)
• Editable touchpad origin. (i.e. which point maps to the center of the window)
• Only enable TouchpadHandler in the actual game, not in menus. (Possibly in the main osu repo)
• If we want to emulate a mouse, we should figure out how to deal with the situation when multiple fingers are down.

  We just use the first reported point (For PoC). This might not be the first finger touched.

[AlexGuo1998]

TODO: catch possible ReadExceptions here

[Susko3]

This actually plays pretty well. But I've found that there is little to no difference if I carefully use what's currently available in osu!.

By enabling high precision mouse, any windows precision touchpad acceleration is ignored. If I position the in-game cursor in the middle of the screen and my finger in the middle of the touchpad, it's functionally identical to this PR. It feels like absolute input and is just as precise. With the added benefit of being able to use the touchpad like normal outside of gameplay.

I've played the same map with this PR and with release osu!(lazer), and it plays the same. There's some fiddling with mouse sensitivity to get the proportions right, but it's simple to adjust, and only needs to be done once.

From what I've tried, absolute touchpad input saves you from having to re-calibrate the centre of screen and centre of touchpad. (Side though: is it possible to do a one-shot with this API, sync positions at the start of the map, and let relative mode handle the rest?) With relative input, you have to calibrate yourself, but this also allows adjusting your posture mid-map.

Have you tried playing with high precision mouse like I described? Give it a try and report back your thoughts.

[AlexGuo1998]

I experimented with both versions for a while.

My setup

Touchpad: Magic Trackpad 3 with the driver from imbushuo, connected using Bluetooth.
(This touchpad have a reported resolution of 7612x5065, impressive.)

System: Windows 10 22H2, touchpad cursor speed = 7, mouse cursor speed = 10.

osu! release: 1024x768, cursor sensitivity = 2.00x

osu! development: 1024x768, cursor sensitivity = 3.80x (i.e. 1/3.8 of the height of the touchpad is used)

I do have another laptop with built-in PTP (Surface Laptop Studio 1st gen), but it's not around now. Will try it later.

They do feel similar, but there are some small differences:

• For tiny movements, dev is much smother than release. (Because in release, cursor can only move in the multiple of 2 pixels, while in dev moves in sub-pixel.)
  • Basically, OS is doing the rounding here. (Will it drift because of this rounding? Need more experiments.)
  • As a result, when I put the finger on the touchpad and trying to stay still, the cursor stays still more in dev than release. (See the video)
• Fast cursor movement in dev seems to be a little smoother than release. (Take with a grain of salt. I should log the data from both APIs and do a post-analysis.)

Video

The tiny square is the position read with raw input, and the white transparent rectangle is the mapped area (in dev).

2025-02-26.19-35-47.mp4

By enabling high precision mouse ... With the added benefit of being able to use the touchpad like normal outside of gameplay.

We can enable TouchpadHandler only during gameplay, in the main osu! project. Possibly with a toggle "Enable only during gameplay on/off" so users can have a choice.

From what I've tried, absolute touchpad input saves you from having to re-calibrate the centre of screen and centre of touchpad.

I think for users with a smaller touchpad, or using a larger portion of it, this auto re-calibration is important. Otherwise, they will risk hitting the border and unable to move further, and that can be very frustrating.

(Side though: is it possible to do a one-shot with this API, sync positions at the start of the map, and let relative mode handle the rest?)

Should be possible, but if one accidentally lift the finger (or on purpose, during rest segments), it's out of sync.

[AlexGuo1998]

After further testing, I confirmed that there is some drift when the finger remains almost still.

I plotted the results from both APIs. The X and Y values are displayed side by side. Raw mouse data points are in orange, and raw HID data points are in blue.

There is no noticeable drift when the finger moves quickly:

However, if the finger moves slowly, Windows tends to filter small movements, causing drift:

Additionally, the first raw mouse input is slightly delayed (~300ms) after touching the touchpad. I believe this is to making clicking more stable. This shouldn't be a problem unless the finger is lifted very often.

[Susko3]

Could you render the graphs again, but use the same X and Y scales for the different graphs. The second one is zoomed in so it may exagarate the drift in comparison to the first graphs. So either zoom in the first graphs, or zoom out the second.

I guess you could also graphs the deltas.

Anyways, after seeing the drift in the second image, I'm already more motivated to include this PR in some form. I'm inclined to have this in SDL, as it can benefit from raw input gathering running in a dedicated thread.