Adding an explainer for real-world hit testing #493
Conversation
|
About "Combining virtual and real-world hit testing", the intent appears to be that WebXR API takes care of real-world understanding, while the application is responsible for virtual objects. For consistency, should real-world information provided by VR headsets also be handled by the hit test API? I've created issue #494 "Using hit test API for floor/bounds in VR mode" with some more thoughts on this, and would welcome feedback there. |
| let inputSourcePose = frame.getPose(inputSource.source, xrReferenceSpace); | ||
| if (inputSourcePose) { | ||
| var virtualHitTestResult = scene.virtualHitTest(new XRRay(inputSource.transform)); | ||
| return { result:virtualHitTestResult, virtualTarget:virtualHitTestResult.target } |
There was a problem hiding this comment.
Is this simple logic ever enough? Presumably it's not too much more code to demonstrate taking the nearest hit among the two, which is likely what apps would actually do, to avoid preferring some distant virtual object 10m away in the next room relative to a 1m nearby wall.
There was a problem hiding this comment.
You mentioned at the F2F that this is due to not having an occlusion solution, and so virtual content will always be visible anyway. However, it doesn't sound like AR apps will produce intuitive experiences for users if things always end up available through walls.
Does this suggest that perhaps a simple on/off occlusion solution is needed to meet the credible "AR min-bar" for WebXR?
There was a problem hiding this comment.
I go back and forth on that question and haven't come to a solid conclusion. Perhaps you'd like to file an issue making that case? Either way I don't think it blocks merging this PR.
There was a problem hiding this comment.
Filed as immersive-web/proposals#45.
There was a problem hiding this comment.
I commented over there, but I'll put the comment here too: if we are providing one simple example, I would think @thetuvix's first suggestion of returning the closer one is correct. An app developer would only call this hypothetical method if they wanted to test both "as if" they were shooting a ray through the combined geometry, so it should return the closer one. If a dev wants to preferentially test some subset (e.g., "I'm interacting with a virtual object that's marked as SELECTED, and if it goes behind physical stuff, I still want to interact with it"), they can fairly obviously implement that. But THIS sample method makes no sense except in the "behave as if the geometry is combined" sense.
| } | ||
| ``` | ||
|
|
||
| On occasion, developers may want hit test results for the current frame even if they have not already created an `XRHitTestSource` to subscribe to the results. For example, when a virtual object needs to bounce off a real-world surface, a single hit-test result can be requested. The results will be delivered asynchronously, though they will be accurate for the frame on which the request was made. Otherwise, `requestAsyncHitTestResults()` shares the behavior of `getHitTestResults()` as described above. |
There was a problem hiding this comment.
Is the behavior of "request now, receive results later, but results are based on data from previous frame" what apps will actually want? Do we have a concrete scenario for wanting that?
I see two coherent paths for the following scenario:
At time A, the user presses a button. After some amount of wireless latency, the UA receives the press and sets controller state, available to the app for an
XRFramewith forward-predicted time B. At the frame for forward-predicted time C, the app gets results that match either...
- Time A, when the press actually occurred. This would be useful for hit-testing what a controller was pointing at back when the press occurred, (available through
XRInputSourcesChangeEvent.getHitTestSource()) or - Time C, for the forward-predicted time at which the app will target its current frame. This would be useful for a view-relative reticle. (available through
XRFrame.getHitTestResults(hitTestSource))
Time B seems arbitrary relative to the user's actions, and so going out of our way to return data for Time B but not Time A seems of low value. This makes me question if there is a strong value now for the arbitrary hit-test. (XRFrame.requestAsyncHitTestResults)
There was a problem hiding this comment.
A relevant nuance is that what XR APIs generally allow time-indexing of is dynamic data, for things like heads and controllers that are actually moving. In contrast, for static and semi-static data, such as planes and meshes, APIs generally only expose the latest snapshot of their poses and geometry.
Unless a plane itself is being posed smoothly every frame in a forward-predicted manner, the following may be a reasonable balance that gets apps what they need while imposing a minimal constraint on UA architecture:
- Raycast from the pose of the
XRSpaceat the exact historical or future time requested in thehitTestcall. - Raycast against the world geometry that is current at the moment when the data is delivered.
There was a problem hiding this comment.
As decided at the F2F, we're going to go ahead and merge this PR knowing this specific topic is unresolved. I'll file an issue detailing the problem and will copy these comments over there.
NellWaliczek
force-pushed
the
input-explainer-cleanup
branch
from
64c4ce5 to
15a0899
Compare
NellWaliczek
force-pushed
the
hit-testing-explainer
branch
from
eeb899d to
3c3ca70
Compare
| ## Real-world hit testing | ||
| A key challenge with enabling real-world hit testing in WebXR is that computing real-world hit test results can be performance-impacting and dependant on secondary threads in many of the underlying implementations. However from a developer perspective, out-of-date asynchronous hit test results are often, though not always, less than useful. | ||
|
|
||
| WebXR addresses this challenge through the use of the `XRHitTestSource` type which acts somewhat like a subscription mechanism. The presence of an `XRHitTestSource` signals to the user agent that the developer intends to query `XRHitTestResult`s in subsequent `XRFrame`s. The user agent can then precompute `XRHitTestResult`s based on the `XRHitTestSource` properties such that each `XRFrame` will be bundled with all "subscribed" hit test results. When the last reference to the `XRHitTestSource` has been released, the user agent is free to stop computing `XRHitTestResult`s for future frames. |
There was a problem hiding this comment.
FWIW (sorry for being absent last month), I think how this turned out @NellWaliczek
This PR contains an explainer with a proposal for adding real-world hit testing to WebXR. It builds strongly on the great work done by @lincolnfrog, @judax, @blairmacinyre, and others to clearly define the problem space. While it is expected that hit-testing will often be used to create anchors, this PR does not include a proposal for an anchors API; that will come in a subsequent PR but still needs a bit more design work.
Rather than a long-winded PR description, I'm hoping that the explainer stands on its own. Please don't hesitate to call out areas that are unclear.
For clarity's sake, the base for this PR is the input-explainer-cleanup branch in PR #492