Additional use cases for InputDeviceCapabilities

In this thread I discussed a simple extension to UIEvent to provide access to an InputDeviceCapabilities object describing properties of the underlying input device. We shipped this API in Chrome 47 but it was really focused on solving one very specific use cases.

I’ve heard lots of other example of information developer want here, but it’s not clear to me what the key concrete use cases are, and what’s of most value to web developers? Maybe we can brainstorm here a little?

Here’s some of the main ideas I’ve heard:

1. When implementing custom focus rings (rather than relying on the browser’s built-in behavior), deciding whether a focus event is due to a device (like a keyboard) that needs focus rings. Related to (perhaps the underlying primitive explaining) this discussion.

2. Detecting whether a device sending ‘wheel’ events can support full 2D drag (and possibly zoom). Eg. in Google Maps, wheel events normally zoom. This makes the most sense for a physical mouse wheel. But with a high-precision touchpad that supports full 2d-panning and pinch-zoom (via ctrl-wheel events), it would be better to pan the map on most wheel events (like this demo page).

3. Detecting whether the device introduces a significant delay before firing click events. Eg. libraries like FastClick try to guess if they should synthesize their own click from TouchEvents. But in many scenarios/browsers this is unnecessary.

4. Determining whether the device sends mouse events (eg. mousemove, mouseenter and mouseleave when hovering). For example this is true for stylus and mouse devices on Android, but not for touch screen devices.

5. Are wheel events generated by the device precise and/or high-frequency? Eg. if the user is using a physical mouse wheel (that moves in ticks, each sending a single wheel event), then perhaps the page should animate positions between each event. But if wheel events are generated with precise delta values at a high rate, then it shouldn’t.

6. Determine whether an input device is implemented by currently taking up space on the screen. Eg. to tell if an on-screen keyboard is currently displayed so that the UI can be made more compact.

We discussed this a little on IRC in Sydney - I like this idea a lot and it might indeed provide an answer that is enough for solving event handing on the focus ring cases mentioned in the other thread just as well (maybe better) than a new event, but it’s not a capability. Would you change the name or shoe it in somehow?

Yeah that’s a good point. The point of using the name ‘capability’ (I think - it was a late-breaking change, I orignally just called it InputDevice and sourceDevice) was to force us to think in terms that will stand the test of time - how the device is used, not what it is.

So maybe the capability that matters here is something like directTargeting. A mouse can directly target elements, but a keyboard requires indirect navigation with the help of focus rings to indicate the current target. WDYT?

I definitely like this idea, though I’m not clear on how it would be exposed in the event.

One wrinkle is that at least focus events can be indirectly triggered by user interactions, like a button press which triggers a menu or modal dialog. We’d need to either be clever about deducing when this was the case (and, as we discovered, it won’t always be in the same control flow) or (more likely?) allow authors to specify this when explicitly moving focus.

Every UIEvent would have a sourceCapabilities.directTargeting boolean property (true if the event was triggered via a pointer, false if triggered via a keyboard / DPAD).

Interesting. When dispatching a UIEvent directly from JavaScript we allow authors to supply their own InputDeviceCapabilities instance. It seems reasonable to me that Element.focus() should optionally take an InputDeviceCapabilities too. At it’s most primitive this feature is really just plumbing passing some extra state around to help developers reason about causality. We had to do a bunch of plumbing internally to pass around source information for focus, so extending that out to focus() seems natural to me. Not sure how hard of a sell that will be though.

Oh and by the way, in case it’s not clear, I’m not in any way opposed to your media-query based proposal either, I just don’t understand the situation well enough yet to have an opinion.

After discussing with the WG there was kind of a split. Some implementers were worried because a MQ causes whole sheets to be added or removed requiring a lot of extra work and MQs are pretty carefully crafted to make sure they don’t happen very often - for example, a user might resize like crazy but we only pass a breakpoint once. Since focus is a thing that happens with great regularity (especially when using sequential navigation). There are some ways around this, but they’re technically complicated and therefore unlikely to convince people to do…

Given this, and known use cases it was thought that a pseudo-class is nearly as good and Mozilla has had an experimental implementation of almost precisely this for a while so it’s possible to slot it into existing selectors specs and move pretty quickly. The major concern this leaves me with is the DOM side - letting script participate intelligently without re-inventing it all. Personally I don’t relish the idea of adding another event, figuring out the lifecycle, going through another process, etc – so when I saw this I thought – something like this feels like a simpler solution to that end of it.

It would be useful to read advanced Wacom stylus input: pressure, tilt, and rotation. Pointer Events will be implemented in Chrome eventually, but the specification only supports pressure, tilt, and is far-future. All 3 controls are useful while simulating traditional media in drawing and painting applications.

It could be useful to access 3D Navigator device input. These can be used to navigate game environments or rotate 3d models.

I agree having this data would be useful, but unfortunately InputDeviceCapabilities isn’t the right place for it. InputDeviceCapabilities are about properties of the device that don’t change while it’s being used. We might want to consider things like supportsPressure thought.

The right way to get this input data exposed to JS is via one of the existing event APIs. For getting rotation into pointer events I’d encourage you to file an issue here, perhaps pointing at an existing use case for rotation. We’re hoping to ship pointer events in Chrome within the next year, so in terms of web standards that’s not THAT “far future”. But I’d also personally support adding stylus properties to other event APIs (TouchEvent on mobile, MouseEvent on desktop). But this is very contentious (with most people arguing you should just wait for pointer events). If you feel strongly, feel free to propose that to the relevant standards list.

Again that’s really about a richer input API. Whether or not a device is capable of doing 3d navigation is something that could make sense for InputDeviceCapabilities.

Hi, Iván from Leaflet here.

We’ve been struggling with MouseWheel events for the past few months, trying to detect the number of discrete ticks performed when a user uses the scrollwheel. As the MouseWheel event only reports scrolled pixels, but this amount varies wildly between browsers.

Let me quote from bug #4538 in Leaflet:

Chrome in Win7 indeed scrolls 100 pixels per wheel click, whereas in Linux it scrolls 50.

With the same hardware, Edge will scroll anything from 115px to 125px, firefox will scroll 48px, and Safari will scroll ~4px.

So exposing things like MacOSX’s hasPreciseScrollingDeltas is a very interesting use case. Furthermore, this is the perfect place to report the amount of pixels the browser is expected to scroll per discrete (or “low-res”) scroll tick, and end browser-detection kludges.

Let me quote from bug #4538 in Leaflet2:

Chrome in Win7 indeed scrolls 100 pixels per wheel click, whereas in Linux it scrolls 50.

So exposing things like MacOSX’s hasPreciseScrollingDeltas is a very interesting use case. Furthermore, this is the perfect place to report the amount of pixels the browser is expected to scroll per discrete (or “low-res”) scroll tick, and end browser-detection kludges.

Thanks for the scenario! If I understand correctly, you really want different behavior for discrete-wheel devices (one zoom level per tick) from devices with some sort of continuous zoom. I agree this scenario is important. There’s been some recent discussion of this on the Chrome team in the context of just adding ticks to WheelEvent. Rather than try to predict delta-per-tick, would it be better to expose ticks along with a hasPreciseScrollingDeltas bit on InputDeviceCapabilities?

I think the biggest issue here is that it’s not clear to us to what extent this API is implementable on Windows. We’ve struggled ourselves for our smooth scrolling feature to reliably determine when the user is using a discrete wheel vs. something with fine grained deltas. It seems we have to rely on a heuristic. But still, maybe better for the browser to have a heuristic that’s mostly right, then for every web developer to try to guess? /cc @bokan @garykac @teddink @jacobrossi

Yes - the idea is very aligned with adding ticks to WheelEvents. However, as s developer I wouldn’t know if the tick is due to a discrete-wheel device, or due to a threshold in a trackpad.

Also, I would like to know whether a WheelEvent is due to physical interaction on the device, or due to “inertia” scrolling after the user has lifted their fingers.

But still, maybe better for the browser to have a heuristic that’s mostly right, then for every web developer to try to guess?

Browsers have a much better chance at getting the heruistics right - I, as a web dev, would trust a proper implementation of ticks rather than the heruistics we use now.

So off the top of my head I’d suggest:

firesWheelEvents = true | false (keeping the current nomenclature of firesTouchEvents)

firesWheelTicks = true | false, the browser is sure enough about its heruistics/platform/etc to provide ticks in a consistent way; if false, web devs should rather rely on their own heruistics based on pixels/lines).

wheelSubTickResolution = true | false, whether the device can scroll by an amount smaller than one tick (wheeled mice vs trackpads), undefined when firesWheelEvents=false or unknown.

I’m not sure if InputDeviceCapabilities would be the right place for handling the mismatch between wheel deltas between browsers. For heruistics, it’d be great to know how much pixel delta is expected for a ““normal”” tick - something to tell the 42 pixels from Firefox in linux apart from the 125 pixels on Edge.

Totally understand the value here, but this is going to be challenging to get right on Windows for sure.

For example, intertia from wheels/trackpads is often implemented by the device’s proprietary driver via just injecting additional wheel messages with decreasing deltas over a curve. So the OS nor browser actually know that these are synthetic inertia events rather than actual physical user interaction. A heuristic here would be challenging.

Ticks would also be challenging with mice that can switch (e.g. many mice have a toggle between detented wheeling and smooth wheeling). Some mice that are smooth scrolling wheels still inject discrete, non-high-res wheel events for compatibility purposes.

Then there are all the differences between classic trackpads and “Precision Touchpads” on Windows.

It’s a mess sadly and not one easily fixed given the complex device and proprietary driver ecosystem out there.