OfflineMediaContext

guest271314 · 2019-08-13

OfflineMediaContext

Motivation:

Create a video collage (A Shared Culture; Jesse Dylan) in the browser, using only APIs shipped with the browser.
AT THE FRONTEND 2016 - Real time front-end alchemy, or: capturing, playing, altering and encoding video and audio streams, without servers or plugins! by Soledad Penadés

… so after a whole year of filing many bugs and crushing [sic] crashing the browser many many times, so like, all that, we are finally ready to talk about this kind of alchemy, the whole title is very long … so streams … it’s not like these streams you can’t join them … you can actually join them …

… you might be like, “how many streams can you have?” … MediaStreams are track containers and probably that doesn’t tell you anything, but Tracks can be audio or video, so if you imagine [sic] a DVD, a DVD can have several tracks of audio, and in theory, several tracks of video, you can change between the scenes, which never happens because producers just gave up on the idea [sic] … But that’s the whole idea of DVD’s, and like, in general, streams of … media that you can have various tracks of things. So, you can have any number of tracks, so you can have any number of tracks streams. That was the premise that we got sold DVD’s based on.

so we need people to have weird new ideas … we need more ideas to break it and make it better

Use it

Break it

File bugs

Request features

For now, two methods:

decodeVideoData()
merge()

decodeVideoData()

Get image data from any video that HTML <video> element can play, both the total duration of a single video file and specific time slices or media fragment identifiers of multiple video files, (potentially) faster than real-time (we can parse the term “real-time” here if necessary; for this case we can begin with the definition: faster than default playback rate at HTML <video> element). It does not render the video data to a display device, but instead renders as quickly as possible, fulfilling the returned promise with the rendered result (https://github.com/WebAudio/web-audio-api/issues/1527) as an Array of arrays of ImageData or ImageBitmap instances (see Efficiently Get ImageData from a Video MediaStreamTrack), or ReadableStream.

const omc = new OfflineMediaContext();
const merged = await omc.decodeVideoData([
                          , "data:video/webm;base64,<data>",
                          , <Blob>
                          , <Blob URL>
                          , <ArrayBuffer>
                          , <TypedArray>
                          , "file:///path/to/media/file.ogv#t=2,5"
                          , "https://path/to/media.file.mp4#t=5,7"
                          , "https://path/to/media.file.mkv#t=5,7"
                          ]); // Array of <ImageData|ImageBitmap> or ReadableStream

Possible solutions:

Video Worklet
Streams API
Write the code from scratch

merge()

Any video that is capable of being played at HTML <video> element at the browser used should be capable of being decoded and written to a WebM compliant file (https://bugs.chromium.org/p/chromium/issues/detail?id=980822), proven by MediaRecorder implementation. It should not be necessary to playback a previously determined list of full length videos and media fragments at HTML <video> to record using MediaRecorder and merge media fragments and entire videos into a single video (e.g., the experiments at https://github.com/guest271314/MediaFragmentRecorder). The procedure should be possible without using HTML <video> or the current implementation of MediaRecorder, which requires playback in “real time”.

const omc = new OfflineMediaContext();
const merged = await omc.merge([
                          , "data:video/webm;base64,<data>",
                          , <Blob>
                          , <Blob URL>
                          , <ArrayBuffer>
                          , <TypedArray>
                          , "file:///path/to/media/file.ogv#t=2,5"
                          , "https://path/to/media.file.mp4#t=5,7"
                          , "https://path/to/media.file.mkv#t=5,7"
                          ]); // WebM compliant file

Possible solutions:

Utilize the existing Media Decoder, Media Encoder, Web Media Player, WebM muxers already used in browser source code
Utilize the FOSS MKVToolNix
Write the code from scratch

Simiar proposals, issues: