Share your biggest challenges with the broader community

Hey all! I wanted to let you know about a new collaboration between Microsoft, Google, and Mozilla (and hopefully others soon): The Web We Want. We using it as a listening tool to identify issues worthy of exploration. We’re not looking for solutions, but rather problem areas. If there’s something you’re throwing a lot of code at to work around a missing API or implement some design idea, please share it. If there’s a missing feature in DevTools or a missing piece in Web Driver, we’d like to know.

We’re also going to be running pitch sessions at web design & development conferences where folks can present their wants to the community in order to get direct feedback and support for their cause. The first one is in Washington DC at An Event Apart, but we’ll also be at Smashing Conference, Beyond Tellerrand, View Source, and others TBA.

Please share your wants, we’re listening!

PS - If you’re running an event and would like to host a Web We Want session, please reach out.

Memory-efficient APIs. Garbage-collection is great when you don’t care about when the memory is freed, and an intractable problem when you do.

In particular, getting ImageData from a MediaStream (local webcam, remote feed via WebRTC, captured from a video using HTMLMediaElement.captureStream(), etc) can generate a lot of garbage if you’re doing processing on each frame and don’t care about it once you’ve processed it. Trying to process 1080p@30 FPS will generate 230 MB/sec of discarded ImageData objects, which is a lot. 4K@30 FPS would be nearly 1 GB/sec.

Leaving it up to garbage collection leaves web developers at the mercy of the implementation. With low-spec devices this often isn’t up to snuff.

You probably know that Edge has that UserVoice page where developers can vote on features that they want to see in Edge. (Of course, now that Edge is Chromium based, that page is no longer relevant unless it’s updated.)

Well, how about that UserVoice (or similar) voting system but for the web platform instead. That way we could see which missing features are really most requested.

For example, I would vote for a standard API for event throttling, among other things.

Yep.

We are looking at rolling a feature like that into webwewant.fyi

Can you submit the event throttling idea?

This is an interesting idea. Can you submit it?

This is an interesting idea. Can you submit it?

Ok, I submitted it to The Web We Want. There’s also a topic on this forum (Efficiently Get ImageData from a Video MediaStreamTrack) and a (mostly) fully-formed proposal on GitHub with a link to patch implementations for both Chromium and Firefox, and performance test results showing the impact, but from what I’ve seen in the past, it’s most likely it will continue to be met with silence.

1. Similar to the concept of @dsanders11, an OfflineMediaContext (https://github.com/guest271314/OfflineMediaContext; https://github.com/whatwg/html/issues/2824; https://github.com/whatwg/html/issues/2981) or videoWorklet (https://github.com/w3c/css-houdini-drafts/issues/905) which, using the language of Web Audio API, " OfflineMediaContext is a particular type of method for rendering/mixing-down (potentially) faster than real-time. It does not render to the video hardware, but instead renders as quickly as possible, fulfilling the returned promise with the rendered result as an ImageBitmaps ."

2. Add replaceTrack() https://github.com/w3c/mediacapture-record/issues/167 and/or ability addTrack() and removeTrack() at MediaRecorder without the recorder instance stopping recording https://github.com/w3c/mediacapture-record/issues/147, or otherwise the ability to add multiple audio and video tracks (potentially having variable width and height) to a single recording https://github.com/w3c/mediacapture-record/issues/166 that is capable of playback at <video> element at Chromium (https://bugs.chromium.org/p/chromium/issues/detail?id=983777) and Firefox.

3. Fully implemented SSML parsing for Web Speech API, without any external services involved (https://lists.w3.org/Archives/Public/www-voice/2017OctDec/0000.html). A start https://github.com/guest271314/SpeechSynthesisSSMLParser.

4. Full implemented SpeechRecognition for Web Speech API, without any external services involved (https://github.com/w3c/speech-api/issues/56). At the very minimum the mandate that users of Web APIs should be asked permission, conspicuously, before the browser records their PII biometric data (voice) and sends the recording of their voice to an undisclosed, unknown remote web service, potentionally FOREVER, to do with what they will.

5. Since WebM is used by MediaRecorder, and though the author/maintainer is not interested in working on such a project themselves (https://gitlab.com/mbunkus/mkvtoolnix/issues/2576#note_185904126), a complete port of mkvmerge (https://mkvtoolnix.download/doc/mkvmerge.html) to JavaScript. In lieu of using Emscripten (https://github.com/emscripten-core/emsdk/issues/260) to port C++ (mkvmerge) to JavaScript, a workaround using Native Messaging and Native File System (“mkvmerge needs the whole source file to be present on disk.” https://gitlab.com/mbunkus/mkvtoolnix/issues/2576#note_185902903) https://github.com/guest271314/native-messaging-mkvmerge.

6. Not necessary, a browser API though challenging nonetheless (both to describe and achieve): Determine the exact multiple of 9 (OEIS A217626 https://oeis.org/A217626) which derives N permutations specifically without calculating all of the permutations (“I am offering a bounty of 500 for a solution that works out the elements directly without calculating the actual permutations.” https://codegolf.stackexchange.com/questions/175693/9erilous-9ermutations).

7. For the OS to not freeze when visiting this site. The OS (*nix, 32-bit) has now frozen twice when visiting this site and attempting to post a comment.

@AaronGustafson Should the list at Share your biggest challenges with the broader community be posted at the linked site as-is, or a post for each list item?

They should be posted individually to the webwewant.fyi site.

Submitted five proposals.

Is there a mechanism to publicly reference submitted proposals? Does The Web We Want send an email acknowledging receipt of the proposal?

@AaronGustafson https://github.com/WebWeWant/webwewant.fyi/issues/8

We send an email to submitters as they are added (assuming a real email is provided). We have a bit of a backlog right now.

Not following.

What does “a real email provided” mean?

If the email is included in the form submission a reply can be sent to that email address. What difference does it make if the provided email is “real” or not?

Submitted five proposals over 7 days ago.

The linked issue does not ask for The Web We Want to accept a submission or implement the idea described therein, the issue simply asks for a very basic reply email acknowledging that The Web We Want received the form submission at all.

Did the pricipal individuals/entities involved with conceiving and announcing The Web We Want not consider acknowledging receipt of submissions before launching the website, making public announcements, and asking for developers to share issues with code and ideas?

Are the principal individuals/entities involved with The Web We Want reading individual submissions before deciding whether or not to reply to the submission?

Has The Web We Want personnel really thought this project through? Or are very basic communication practices and procedures being considered as time moves along only when brought up by the individuals who submitted forms themselves?

Why cannot an email be sent to the email included in the form submission immediately upon receiving the submission to confirm receipt of the form submission?

BTW On its face the first paragraph of this thread

is a great effort in and of itself nonetheless.

FWIW Willing to help in that effort where am able.

Just noting I’ve gotten these up on the site.

What does “a real email provided” mean?

We’ve gotten some obviously fake email addresses that cannot be resolved.

(Back on topic) A common theme for me is getting more APIs to work in Web Workers. With OffscreenCanvas it’s now possible to do things like advanced WebGL rendering off the main thread. But support for audio, video, media streams, WebRTC and so on is still entirely missing from workers. This can be worked around but it makes it massively more complicated to get code that works on the DOM running in a worker.

AFAIK spec issues have all been filed etc., it’s just slow progress. I’d also like to see a commitment that all new APIs are made available in a worker too, unless there’s some specific reason why it can’t be. For example the recent Web Bluetooth is DOM-only in Chrome, so even new things are still adding to the pile of things not available in a worker.

Agreed. Could you submit that last want to the site? Something like “I want all new APIs (within reason) available to workers (including service worker)” or similar.

OK, I posted an entry for that.