[Proposal] User Input of Mathematics

Introduction

Mathematics user input is an important topic. Mathematics user input facilitates scholarly and scientific communication scenarios as well as educational scenarios, providing new varieties of science, technology, engineering and mathematics exercises and activities.

To provide users with accessible mathematics user input, Web browser software applications could invoke operating system platform or third-party components and services. This resembles the way that touchscreen keyboards and handwriting recognition are provided on tablet computers.

Three varieties of components and services for mathematics user input are presently considered. Firstly, graphical user interface widgets with panels of buttons for entering mathematical notations. Secondly, mathematics handwriting recognition components and services. Thirdly, mathematics speech recognition components and services.

Input Elements for Mathematics

<input type="math" />

<matharea />

Hyperlinks

https://www.w3.org/community/exercises-and-activities/wiki/User_Input_of_Mathematics

There are other scientific notations that would benefit from dedicated input modes. Some of them are reasonable to combine, e. g. physics and chemistry formulae are often covered by the proposed math, but those sciences also have very specific requirements in some subfields. You should assert and document what kind of input methods already exist, e.g. in MS Office, and what would be required or even possible (re MathML, Unicode etc.).

By the way, I also see use cases for emoji and IPA input methods that are also independent of language.

@Crissov , thank you. I agree that, in addition to mathematics, sciences such as physics and chemistry would also benefit from new and standardized user input technology.

To your question, in Microsoft Office, the formats of UnicodeMath, LaTeX, OMML and MathML are supported. At some point upcoming, math braille or Nemeth braille will also be supported.

I’m presently inclined to utilize MathML.

What I’m hoping to do is to incubate this proposal so that it might become a new deliverable of the Web Platform Working Group.

It’s presently a good time to propose and to discuss new deliverables for the Web Platform Working Group [1].

I’m excited about this proposal. The user input of mathematical and scientific notations can advance scholarly and scientific communication as well as science, technology, engineering and mathematics educational exercises and activities.

I would like to request comments and feedback from implementers. Thank you.

References

[1] https://lists.w3.org/Archives/Public/public-webapps/2018JulSep/0025.html

Not against the math value. But the following issues should also be resolved:

1. First a clarification of what math is. More precisely, choose a format: MathML, LaTeX, AsciiMath, … Arguably, the most “standard” is MathML.

2. A “baseline” behaviour of the control should be specified in order to provide browser compatibility. For example a type=“text” has a very precise baseline behaviour: “a rectangle where text can be placed inside and when the text is bigger than the rectangle, bla bla bla” . Then, specific browses and devices improves the UX while keeping a common compatibility.

3. UI choice: equation (or formula) editors usually have a toolbar with the available math templates. Such toolbars are different in size from one solution to another. But the important decision is whether the UI will be implemented by the browser/device or be the Web page the responsible (that’s something like the HTML editors where the browser implements the raw edition capatility and a the toolbar menu needs a library like TinyMCE, CKEditor, etc.).

4. Is the target level relevant? primary, secondary, highschool, algebra, analysis, group theory, … Not clear that a single and generic UI for math is good for all. Browsers might have hard time to provide the solution that is best for the student without any explicit hint. For example, there is not point to offer a handwriting input method that is not able to input matrices if the exepected result is to write the inverse of a given matrix.

5. Being a litle bit picky myself, math is more than just equations and formulas. What about graphics?

@Daniel_Marques_WIRIS,

Mathematics Formats

First a clarification of what math is. More precisely, choose a format: MathML, LaTeX, AsciiMath, … Arguably, the most “standard” is MathML.

I’m inclined to first consider MathML. Interestingly, MathML can include other formats as annotation.

Scenarios

A “baseline” behaviour of the control should be specified in order to provide browser compatibility. For example a type=“text” has a very precise baseline behaviour: “a rectangle where text can be placed inside and when the text is bigger than the rectangle, bla bla bla” . Then, specific browses and devices improves the UX while keeping a common compatibility.

There are three scenarios to consider:

<input type="math" />

Some complexity arises as we consider the HTMLInputElement interface which utilizes strings for certain properties which we would like to hold MathML content. That is, the HTMLInputElement interface is presently defined with the type DOMString for the properties placeholder, defaultValue and value.

We could explore solutions including, but not limited to: (1) generalizing those properties’ types to the type object?, (2) adding properties resembling object? valueAsMath based upon the object? valueAsDate property.

<matharea />

Less complexity arises as we consider creating an entirely new interface, HTMLMathAreaElement based upon the HTMLTextAreaElement interface.

contenteditable

Discussion topics include a JavaScript API to invoke users’ mathematics user input components or services. When the user double-clicks on a mathematics expression in a contenteditable region, the browser can open a mathematics user input widget provided by the website, the browser, the platform or a third-party component or service. These widgets could also be invoked via JavaScript.

It is important to note that we could be discussing standardization of, for contenteditable scenarios, the double-click on mathematics expression opens user input widget behavior as opposed to user experiences where the caret or text cursor moves into and through mathematics expressions as we might observe in some computer algebra systems.

Input Events Level 2 utilizes a DataTransfer interface so we can transport, beyond text, MathML content in the input events system.

Interestingly, the Editing APIs are hosted by the Web Platform Working Group, specifically by The Editing Taskforce.

Implementing Mathematics User Input in Website JavaScript versus Utilizing Platform / Browser / Third-party Components or Services

UI choice: equation (or formula) editors usually have a toolbar with the available math templates. Such toolbars are different in size from one solution to another. But the important decision is whether the UI will be implemented by the browser/device or be the Web page the responsible (that’s something like the HTML editors where the browser implements the raw edition capatility and a the toolbar menu needs a library like TinyMCE, CKEditor, etc.).

With standardized mathematics user input document elements (e.g. <input type="math" />, <matharea />), operating systems could ship with mathematics user input widgets (resembling virtual keyboards), browsers could ship with default widgets, and there should also be third-party browser extensions as well as third-party native components and services utilizable by the browser applications. Reasons that users might prefer third-party components include accessibility features.

In addition to graphical user interface widgets, mathematics handwriting recognition and mathematics speech recognition solutions could be invoked when users double-click on mathematics expressions. Some components and services might provide multiple such input modalities.

To provide for website-based solutions for mathematics user input, we could allow browser settings which permit pages to override of the default behavior, e.g. with a callback function. If a user setting is such that overriding the default behavior is permitted, a page could provide a callback function to be called when the user double-clicks on a mathematics expression.

Providing Hints to Recognition Technology

Is the target level relevant? primary, secondary, highschool, algebra, analysis, group theory, … Not clear that a single and generic UI for math is good for all. Browsers might have hard time to provide the solution that is best for the student without any explicit hint. For example, there is not point to offer a handwriting input method that is not able to input matrices if the exepected result is to write the inverse of a given matrix.

I opened a proposal for document element metadata which is one approach to providing hints for mathematics recognition.

Presently, there are four types of mathematics recognition hints to consider for graphical user interface, handwriting and speech recognition scenarios. Firstly, there is the category or subject of input; this could be the variety of scientific notation expected, e.g. chemistry, or the variety of mathematics expected. Secondly, there are the symbols expected, e.g. “x” or “chi”, “w” or “omega”. Thirdly, entire expressions, expected user input content, can be provided as hints. Fourthly, templates can be provided, mathematical expressions with blanks, e.g. for fill-in-the-blank user experiences.

As interesting, there is also some content on these topics available at the wiki of the Educational Exercises and Activities Community Group.

MathJax Interoperability

We should consider MathJax interoperability with respect to the baseline behavior of the controls in the three scenarios aformementioned, <input type="math" />, <matharea /> and contenteditable.

Mathematics and Graphics

Being a litle bit picky myself, math is more than just equations and formulas. What about graphics?

Excellent idea. I wonder how this could be implemented? Should the drawing / graphical composition widget export InkML, SVG or graphics formats? We could also utilize the DataTransfer interface in a JavaScript API.

I believe a lot of the simpler use cases would be covered by setting the input language to be neutral, undefined or unknown with a subtag that would make the browser recognize or offer a specific subset of Unicode characters.

<input type="text" lang="x-math">
<input type="text" lang="und-ipa">
<input type="text" lang="zxx-particle-physics">

@Crissov,

Thank you. In addition to:

<input type="text" lang="x-math">
<input type="text" lang="und-ipa">
<input type="text" lang="zxx-particle-physics">

we could consider something like:

<input type="text" lang="x-latex">

and also possible is:

<input type="text" inputmode="math" />

however, the inputmode attribute is controversial.

AMS-LaTeX and MathJax

Some desirable features are possible with text-based mathematics input, e.g. AMS-LaTeX, libraries such as MathJax and perhaps a new JavaScript API for accessing external mathematics editors provided by operating systems, browsers and/or third-party vendors.

With a text-based input of ASCIIMath or AMS-LaTeX, libraries such as MathJax can produce displayable mathematics for users. This resembles the functionality shown on the MathJax front page.

Users could type AMS-LaTeX into text-based <input> elements and libraries such as MathJax could, upon completion of typing, render the resultant mathematics such that it were positioned where the <input> element would otherwise be. MathJax could also make it so that users could click on displayed mathematics to reveal text-based <input> elements with which to edit AMS-LaTeX content.

MathJax could also provide expected functionality for placeholder attributes.

JavaScript API

Students don’t know AMS-LaTeX – only expert users know AMS-LaTeX. Accordingly, we want to encapsulate the formats to be utilized by the underlying <input> elements and to provide users with desired user experiences.

In my opinion, desired user experiences are for users to select <input> elements to open and utilize external mathematics editors. Desired user experiences for such editors are to enter mathematics via mouse and keyboard, handwriting recognition and/or speech recognition. Desired user experiences include rendered mathematics appearing in documents confirming the processing of their user input. Desired user experiences include clicking on rendered mathematics expressions to edit them in an external mathematics editor.

With a JavaScript API for accessing external mathematics editors, libraries such as MathJax could utilize external mathematics editors to provide desired user experiences.

MathML

The desired user experiences entirely encapsulate the formats to be utilized. That is, end users needn’t encounter either AMS-LaTeX or MathML. Forms data, however, will either be text-based, e.g. AMS-LaTeX, or MathML data. For discussion purposes, I will argue that MathML is the way to go.

Mathematical Semantics

MathML includes presentational and semantic content. We can consider scenarios where forms data or contenteditable content are desired to include mathematics semantic content. This mathematics semantic content can be provided by external mathematics editors, perhaps utilizing contextual recognition hints.

Lurch is an example of a software which utilizes mathematics semantics. As an aside, we can envision, beyond a green / yellow / red light pattern on mathematics expressions, an information / warning / error pattern on mathematics expressions in mathematical proofs or perhaps on natural language sentences in argumentation.

MathML is Standard

Beyond being the most standard option, MathML is HTML5. There should be some means of inputting it in forms and some means of utilizing it with contenteditable.

Conclusion

In the event that there comes to be a Web Platform Working Group deliverable for mathematics user input, there will probably be AMS-LaTeX vs. MathML discussions and deliberations.

There will also probably be discussions and deliberations regarding a JavaScript API for opening external mathematics editors provided by operating systems, browsers or third-party vendors, providing them with contextual recognition hints, and obtaining the data, expressions and graphics, provided by them in multiple formats (e.g. DataTransfer).

Is there any interest in opening a GitHub repository, perhaps a WICG repository, and starting on some documents with respect to:

1. Mathematical and notational user input scenarios pertaining to forms and to contenteditable content

2. Hypertext document markup for providing users with mathematical and notational user input

3. A JavaScript API for utilization of and interoperation with mathematical and notational user input controls provided by websites, browsers, operating system platforms and third-party software vendors, providing such user input controls with contextual recognition hints and obtaining resultant data, expressions and graphics

Thanks for this proposal, Adam. I think I’d like to see three things before we open a WICG repo:

1. an implementation as a web component - and proof that web components are too limited to meet the use cases (i.e., further browser support is needed).
2. interest from at least one browser vendor.
3. demonstrated demand from developer community.

General questions…

Three varieties of components and services for mathematics user input are presently considered. Firstly, graphical user interface widgets with panels of buttons for entering mathematical notations.

Do OSs already provide these widget? Generally, browsers rely on OSs to provide these inputs natively.

Secondly, mathematics handwriting recognition components and services.

As above with regards to OSs providing these? - or could these be done inside the web app through an existing API.

Thirdly, mathematics speech recognition components and services.

As above.

Additionally, is there some relationship to MathML?

I would suggest starting a separate community group around this, and building up a community. Once you have a working implementation that is in wide use, you should bring the proposal back.