[Proposal] video.requestAnimationFrame()

https://github.com/dalecurtis/video-animation-frame/blob/master/explainer.md

Summary

Today <video> elements have no means by which to signal when a video frame has been presented for composition nor any means to provide metadata about that frame.

We propose a new HTMLVideoElement.requestAnimationFrame() method with an associated VideoFrameRequestCallback to allow web authors to identify when and which frame has been presented for composition.

Example

  let video = document.createElement('video');
  let canvas = document.createElement('canvas');
  let canvasContext = canvas.getContext('2d');

  let frameInfoCallback = (_, metadata) => {
    console.log(
      `Presented frame ${metadata.presentationTimestamp}s ` +
      `(${metadata.width}x${metadata.height}) at ` +
      `${metadata.presentationTime}ms for display at ` +
      `${expectedPresentationTime}ms`);

    canvasContext.drawImage(video, 0, 0, metadata.width, metadata.height);
    video.requestAnimationFrame(frameInfoCallback);
  };

  video.requestAnimationFrame(frameInfoCallback);
  video.src = 'foo.mp4';

Output:

Presented frame 0s (1280x720) at 1000ms for display at 1016ms.
7 Likes

Would be great to have this functionality!

Any thoughts on generalizing the AnimationFrameProvider interface so it can support this usage?

https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider

Thanks Ken! I thought I had incorporated that feedback since you last mentioned it; hence the explainer says:

// Extends the AnimationFrameProvider mixin with the addition of an
// VideoFrameMetadata parameter on the FrameRequestCallback.

Can you elaborate on what you’re asking for? I must have misunderstood.

Ah, sorry, I hadn’t remembered to revisit the explainer.

We should get feedback from @fserb who added AnimationFrameProvider to the spec. Not all users of AnimationFrameProvider need the VideoFrameMetadata parameter, so maybe it makes sense for HTMLVideoElement’s requestAnimationFrame to be different from AnimationFrameProvider’s.

I think that this would be very useful to have.

Excellent proposal. I can see this becoming very handy for improving diagnostics like gitch detection. Support it fully.