Proposal: Exposing input events to worker threads

Problem

Today on the Web Platform only the main thread has access to the DOM elements and receives their related input events. In particular, Web Workers and Worklets do not have access to these. If we allow workers to receive input events for a given DOM node we enable many latency sensitive applications to leverage workers. As a result if developers want to do some event dependent logic like drawing on an off-screen canvas or using fetch in their worker thread they need to listen to those events on the main thread and postMessage them across to the worker thread. Here the worker is not only going to be blocked by the main thread to handle the events first but also suffers from the latency introduced by the extra thread hop. This is particularly unfortunate for the input events where the latency is important.

Proposal

We propose a delegation mechanism on the main thread. This way a target will be delegated to a worker (or multiple workers). Then the worker will get the EventTarget and can add any event listener it wants to it. Note that to keep the main thread sanity of event propagation logic we propose to make these listeners on the worker side always passive as well as not letting the worker to preventDefault or stopPropagation on the received event. In other words both main thread and worker can get the events at almost the same time (conditioned on the forking point of the event path and the target that is delegated).

Here is the full explainer for the feature and the exact proposed API.

Use Cases

• Low-Latency Drawing: Offscreen Canvas now provides workers with an off-main-thread drawing context. However any input driven drawing is still blocked by the main thread getting the user events/inputs and forwarding them to the worker to be drawn on the off screen canvas. With this latency paper-like drawing cannot be achieved on the web pages via workers.
• Gaming and XR: Similar to the drawing use case, low latency input handling and drawing is important for these applications and they can benefit from access to both input and canvas in worker threads.
• Page Analytics: Analytic scripts that process events without needing to access DOM. Some analytics scripts want to send user input over the network and only need access to the raw coordinates and state of the pointer including the buttons. In this case accessing DOM or knowing whether the default actions were prevented is not important.
• Interactive Animations: Animation Worklet enables scripted animation off-main-thread. By providing input events to Animation Worklet we enable rich interactive animation driven by the user input to be off-main thread to ensure smoothness and responsiveness. For many such effects passive access to pointer input events is sufficient.
• Low-Latency Interactive Audio: Audio Worklet provides a rich off-main-thread audio context which is being used to enable high-fidelity rich audio applications on the web. If audio worklet were able to process user input (including messages from Web MIDI API) directly it may help reduce the latency from user input to audio output which is important for interactive audio applications

@RickByers @majidvp @BoCupp-Microsoft @smaug @travisleithead

This looks useful for web game engines using OffscreenCanvas to host the engine in a worker. Currently we support this in Construct by forwarding all input events with postMessage, and this would simplify that and cut latency. This use case is very simple, since with this proposal there would be no handlers on the DOM, input events would only be used in the worker.

I have a few questions:

1. Currently we attach input event handlers to window to just forward everything. Will we be able to do the same with this API? i.e. worker.addEventTarget(window)
2. This looks like it covers pointer events, but we also forward keyboard events, mouse events, and device orientation/motion events. Is the intention to also cover these?
3. We also forward inputs from the Gamepad API. This is more awkward because as it stands, it does not fire events - so we just poll it and postMessage every frame. Should the Gamepad API itself be exposed in a worker, or should it be modified to fire events and work with this proposal?

It looks like it’d also be worth covering the “pointerrawmove” proposal with this.

So is the idea that hit-testing for a node which has worker-eventtargets attached would happen on-non-mainthread? Or would the forking happen after event path creation… I guess the text (and looks like the explainer too) seems to hint about latter, but that requires main thread hit-testing and what not, means that busy main thread will block event handling and performance might not get improved at all. This doesn’t feel like the right approach to me.

@AshleyScirra 1: Sure we can bake that in too. Can you file the GitHub issue for this s well. But out of curiosity. Any particular reason you are doing that on the window instead of the document element? Is there anything particular you get on the window that you don’t on the document? 2, 3: For the starter we wanted to focus on the pointerevents (including the pointerrawupdate) to make sure we get it right. But we will definitely expand to keyboard and gamepad and whatnot in the near future.

@smaug The explainer does indeed hint at off main thread hittesting and targeting to be able to get the performance benefit. That may potentially cause the hittest to be done on the last state of the DOM tree if the main thread was busy changing the DOM at that very moment. But we thought that is not going to be an issue as the page doesn’t control the timing of the input anyway and the input might have happened before that DOM change as well. So it shouldn’t break the pages in any way.

@smaug The forking model is exactly one of the design choices that I hope we get more feedback on. There is interesting trade-offs in terms of the performance and richness related to participating in event propagation on main thread. We have tried to examine a few options here.

Option (a) forks the event at root. This provides maximum performance isolation since hit-testing can happen without involving on main thread since it does not case about the event propagation path to be computed and earlier handler running and not preventing propagation.

Other options give more ability to the main thread to prevent worker from getting events say by registering a handler that prevents propagation. While we can do pre-computation and optimize the case when there is no handler registered but it is definitely more complex and a performance footgun which the other model does not have.

A sensible approach may be to start with (a) to ensure good default performance and then provide ability to opt-into other forking models if needed. I believe a majority of usecases will be fine with this forking model so starting here seems fine.

An alternative approach maybe to start by allowing event delegation only at root level (window/document) where there is no event propagation path to worry about and leave the question of forking model to the future.

Forking models can be implemented in JS, so I don’t see strong reasons to add such APIs to the platform. So, I think it would be better investigate (a) some more. But it isn’t really (a), since that is “Basically fork and delegate the event at the root of propagation path.”. We need to fork sooner. Before we know anything about propagation path in DOM.

I looked in to how the Construct engine handles input events with OffscreenCanvas a bit more. We can mix actual DOM elements with input events forwarded to OffscreenCanvas. For example there can be a <button> floating over the canvas; clicks to the button are handled separately to other clicks (to the document/window).

So I think actually we don’t want to forward all input events - otherwise inputs on other DOM elements could (in the most efficient forwarding model) be forwarded to the worker when they wouldn’t normally.

I think we can solve this by creating a viewport-sized element floating under everything, and use that as the target to forward input elements to the worker. Then if the user clicks a different DOM element, that element handles it; then the catch-all element behind it forwards everything else.

Would the browser be able to implement this efficiently? I suppose you only need to work out if the input position is over the target element at the earliest opportunity, and then decide whether to forward on that. Once an input has been forwarded to the worker there’s no opportunity to stop it so it’s effectively passive.

We also preventDefault() on the DOM in a couple of places to block default browser behavior (e.g. selection, pan-scrolling) for gaming purposes; but these could run in parallel to the input event being forwarded to the worker. In other words we only make a decision about preventDefault() on the DOM and the worker still uses the event in a passive manner.

So for our case I think it would be sufficient to forward based on the event target only, allow parallel dispatch to the worker and DOM, but only respect the preventDefault() of the DOM. Alternatively if everything was forwarded regardless of the DOM element, we could probably live with that by tracking where DOM elements are located on the worker and ignoring input events that land on them… but it would be more complicated.

I suggest to move the spec proposal under WICG and iterate over that. Feel free to file issues against the spec and send patches. @yoavweiss could you move the repo under WICG?

At

Worker thread:

// Made up library that given pointer event sequence can compute an active gesture similar to Hammer.js 
import {Recognizer} from 'gesture_recognizer.js'

at the Explainer, are static import statements possible outside of a JavaScript module; i.e., in a Worker?

Dynamic import import("./gesture_recognizer.js") https://v8.dev/features/dynamic-import is possible within a Worker.

Repo moved to https://github.com/WICG/input-for-workers

Happy incubatin’!