[Proposal] video.requestAnimationFrame()

https://github.com/dalecurtis/video-animation-frame/blob/master/explainer.md

Summary

Today <video> elements have no means by which to signal when a video frame has been presented for composition nor any means to provide metadata about that frame.

We propose a new HTMLVideoElement.requestAnimationFrame() method with an associated VideoFrameRequestCallback to allow web authors to identify when and which frame has been presented for composition.

Example

  let video = document.createElement('video');
  let canvas = document.createElement('canvas');
  let canvasContext = canvas.getContext('2d');

  let frameInfoCallback = (_, metadata) => {
    console.log(
      `Presented frame ${metadata.presentationTimestamp}s ` +
      `(${metadata.width}x${metadata.height}) at ` +
      `${metadata.presentationTime}ms for display at ` +
      `${expectedPresentationTime}ms`);

    canvasContext.drawImage(video, 0, 0, metadata.width, metadata.height);
    video.requestAnimationFrame(frameInfoCallback);
  };

  video.requestAnimationFrame(frameInfoCallback);
  video.src = 'foo.mp4';

Output:

Presented frame 0s (1280x720) at 1000ms for display at 1016ms.
8 Likes

Would be great to have this functionality!

Any thoughts on generalizing the AnimationFrameProvider interface so it can support this usage?

https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider

Thanks Ken! I thought I had incorporated that feedback since you last mentioned it; hence the explainer says:

// Extends the AnimationFrameProvider mixin with the addition of an
// VideoFrameMetadata parameter on the FrameRequestCallback.

Can you elaborate on what you’re asking for? I must have misunderstood.

Ah, sorry, I hadn’t remembered to revisit the explainer.

We should get feedback from @fserb who added AnimationFrameProvider to the spec. Not all users of AnimationFrameProvider need the VideoFrameMetadata parameter, so maybe it makes sense for HTMLVideoElement’s requestAnimationFrame to be different from AnimationFrameProvider’s.

I think that this would be very useful to have.

Excellent proposal. I can see this becoming very handy for improving diagnostics like gitch detection. Support it fully.

Intent makes perfect sense and would be very valuable! Why is this API different from ontimeupdate? Is there a reason why the DOM event system is not sufficient? I’d expect this to be named something like onframeupdate which fires on every frame painted. Is this a question of timing? Can this question be addressed in the explainer?

Good questions… probably want clarity on these questions before proceeding.

I’ve been using the Chrome prototype to measure/control the latency of an application using MSE and have found it very useful.

Will video.requestAnimationFrame(frameInfoCallback) gurantee that every frame of an input video file or MediaStream is captured?