Hi WICG,
I’d like to propose an API for web applications to utilize handwriting recognition services available on operating systems.
This would make it easier for developers to integrate handwriting recognition support in their applications, without having to use a third-party library or service.
Use cases
- Note taking web app. Users use a stylus to take notes, and the app converts the drawings to text in real time.
- Custom input fields. Developers can create custom handwriting input areas (e.g. Please sign here”), without using a pop-up virtual keyboard. This offers more immersive user experience.
Problem
We want to have an API that takes in a handwriting drawing (illustrated below), and tell us what characters were written, in real time:
Web apps can import handwriting recognition libraries via Web Assembly, but this would not allow the use of more advanced proprietary handwriting libraries (e.g. those available on the operating system). This topic of “Web Assembly vs Web API” was previously discussed in Shape Detection and Text-to-Speech APIs (to which the same constraints apply).
Given that handwriting recognition capability already exists in many OSes, we think web apps should have a way to use them:
- How can we expose this capability to web apps?
- How can we design a Web API that works well with different implementations?
- How can we design an extensible way for web apps to fine-tune the recognizer?
- What should we do if the operating system has no native handwriting recognition support?
Proposal
- Add classes necessary to represent handwriting drawings in the JavaScript world.
- Add a class so the app can instantiate and use handwriting recognizers provided by the OS or browser.
- Add a way for apps to query which features are supported on the platform (e.g. languages).