[Proposal] video.requestAnimationFrame()

#1

https://github.com/dalecurtis/video-animation-frame/blob/master/explainer.md

Summary

Today <video> elements have no means by which to signal when a video frame has been presented for composition nor any means to provide metadata about that frame.

We propose a new HTMLVideoElement.requestAnimationFrame() method with an associated VideoFrameRequestCallback to allow web authors to identify when and which frame has been presented for composition.

Example

  let video = document.createElement('video');
  let canvas = document.createElement('canvas');
  let canvasContext = canvas.getContext('2d');

  let frameInfoCallback = (_, metadata) => {
    console.log(
      `Presented frame ${metadata.presentationTimestamp}s ` +
      `(${metadata.width}x${metadata.height}) at ` +
      `${metadata.presentationTime}ms for display at ` +
      `${expectedPresentationTime}ms`);

    canvasContext.drawImage(video, 0, 0, metadata.width, metadata.height);
    video.requestAnimationFrame(frameInfoCallback);
  };

  video.requestAnimationFrame(frameInfoCallback);
  video.src = 'foo.mp4';

Output:

Presented frame 0s (1280x720) at 1000ms for display at 1016ms.
8 Likes
#2

Would be great to have this functionality!

Any thoughts on generalizing the AnimationFrameProvider interface so it can support this usage?

https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider

#3

Thanks Ken! I thought I had incorporated that feedback since you last mentioned it; hence the explainer says:

// Extends the AnimationFrameProvider mixin with the addition of an
// VideoFrameMetadata parameter on the FrameRequestCallback.

Can you elaborate on what you’re asking for? I must have misunderstood.

#4

Ah, sorry, I hadn’t remembered to revisit the explainer.

We should get feedback from @fserb who added AnimationFrameProvider to the spec. Not all users of AnimationFrameProvider need the VideoFrameMetadata parameter, so maybe it makes sense for HTMLVideoElement’s requestAnimationFrame to be different from AnimationFrameProvider’s.

#5

I think that this would be very useful to have.

#6

Excellent proposal. I can see this becoming very handy for improving diagnostics like gitch detection. Support it fully.

#7

Intent makes perfect sense and would be very valuable! Why is this API different from ontimeupdate? Is there a reason why the DOM event system is not sufficient? I’d expect this to be named something like onframeupdate which fires on every frame painted. Is this a question of timing? Can this question be addressed in the explainer?

#8

Good questions… probably want clarity on these questions before proceeding.

#9

I’ve been using the Chrome prototype to measure/control the latency of an application using MSE and have found it very useful.

#10

Will video.requestAnimationFrame(frameInfoCallback) gurantee that every frame of an input video file or MediaStream is captured?