WebMediaContainers Proposal

API that allows web applications to write, read, edit, extract, and merge audio and video to and from a media container.

MediaRecorder provides the capability to write real-time captured audio and video to a media container.

However MediaRecorder does not provide a means to write non-real-time audio and video to a media container.

WebCodecs proposal while having similar use cases, including

• Media file editing and transcoding

is not presently focused on development and standardization of writing the media output by that API to a media container API for containers? #24.

This propsal is very simple: an API focused on writing, reading, editing, extracting, muxing, merging and audio, video, subtitles, captions to and from a media container, both in real-time and post-production.

The proposal intends to specify WebMediaContainers API separate but concurrent to WebCodecs API, for the purpose of the two proposals to be developed at approximately the same rate, for the goal of interopability with while simultaneously capable of interaction with audio, video, text tracks APIs other than WebCodecs.

Explainer

Can you give some examples of what the API would look like?

Yes

videoWriter.addVideoFrame(videoFrame[i] /* image, Blob, etc. */, frameDuration /* , width, height */);
videoWiter.addAudioFrame(audioFrame[i] /* Float32Array, ArrayBuffer, etc. */, frameDuration, sampleRate, numberOfChannels);
// Matroska or WebM file
// completely FOSS where possible
// use openh264 if preferred or required
const file = await videoWriter.compile(); // file.mkv or file.webm or if required file.mp4

https://plnkr.co/edit/Inb676?p=info (store potentially variable video frames in an array, calculate frame duration, write a WebM file)

and/or

https://plnkr.co/edit/ThXd9MKYvEYq2kKyh8oc?p=preview (write to frames to a WebM file without storing input images in an array)

Unfortunately that implementation https://github.com/thenickdude/webm-writer-js does not support audio https://github.com/thenickdude/webm-writer-js/issues/1#issuecomment-533941846 nor does https://github.com/GoogleChromeLabs (https://github.com/GoogleChromeLabs/webm-wasm/issues/12), though this one does https://github.com/tseylerd/Webm-writer. Have been trying to encode audio using https://github.com/higuma/ogg-vorbis-encoder-js, however the browser (Chromium) invariably crashes (am currently at version 15 of testing).

The concept is very simple. Begin with a WebM writer that is independent of the browser implementation. Meaning the code is the same whether using Chromium or Firefox, which currently have, in some cases, wildly varying implementations of MediaStreamTrack, MediaRecorder and particularly MediaSource and HTML <video> element, where Chromium enforces an arbitrary output relevant to video frames at the <video> element.

A variable resolution video recorded at Chromium (deliberately) does not output variable resolution frames at HTML <video> element, though the same video does output variable resolution at Mozilla Forefox and Nightly implementation of HTML <video> element. Chromium crashes the tab when the resolution of the video track changes when HTMLVideoElement.captureStream() is called on the element. That is not acceptable and the technology currently exists to enable developers to avoid using MediaRecorder altogether since the implementations vary to such an extent.

@pthatcher Created repository https://github.com/guest271314/WebMediaContainers and used language that already posted in this proposal and the comments at WebCodecs proposal in Explainer. It is expected that the repository and Explainer will be edited.

Your input is very welcome. Kindly consider and suggest edits where necessary. The goal of the proposal is for absolute clarity in specification, and actual input and output by implementers.

Am at the front-end, testing what the source code of browsers actually output

von Braun believed in testing. I cannot emphasize that term enough. Test, test, test. Test to the point it breaks. - Ed Buckbee, NASA Public Affairs Officer

(https://github.com/guest271314/MediaFragmentRecorder/tree/chromium_crashes)

Know exactly how the API should (and should not) function after testing numerous means of creating video using existing APIs over and over again at https://github.com/guest271314/MediaFragmentRecorder/branches and reading the source code of Chromium and Mozilla browsers to determine what the capabilities of the code is. The proposal is possible using the current source code, generally, in the “media” directory of the code already implemented at the browsers, e.g., [MediaEncoder] Implement WebM 1.0 container writer. The only necessity would be to expose the decoders, encoders, muxers, and media container writers to the developer.

The basic concept is very simple.