Intent to Ship: Add support for video frame orientation metadata to WebCodecs

Some cameras and media will immediately begin exposing orientation information. In uncommon cases this orientation may change. If orientation changes during encoding through VideoEncoder frames with different orientation than the first are currently encoded with incorrect size and orientation. However the operation completes without a hard error. During discussions in the media working group it was decided that these cases should instead throw a non-fatal exception during encode(). There is already precedent for exception handling during encode() so authors should already be handling this case.

On Tue, Apr 1, 2025 at 7:16 PM Vladimir Levin <vmp...@chromium.org> wrote:

Some cameras and media will immediately begin exposing orientation information. In uncommon cases this orientation may change. If orientation changes during encoding through VideoEncoder frames with different orientation than the first are currently encoded with incorrect size and orientation. However the operation completes without a hard error. During discussions in the media working group it was decided that these cases should instead throw a non-fatal exception during encode(). There is already precedent for exception handling during encode() so authors should already be handling this case.

Can you clarify if "uncommon" here means that some cameras are known to change the orientation and those cameras are typically rare, or that most cameras will change the orientation if say the user changes the orientation, but that user behavior is rare? :)

I meant most cases will have a single fixed orientation for the duration of the session. Users changing their orientation is uncommon. It will almost always happen with users on mobile / hand-held devices accidentally or intentionally reorienting the device itself.

This does seem to introduce a new exception reason that seems almost expected to happen in some cases. Can you comment on whether the other exceptions that the author should already be happening are likewise "common"? There's a bit of a concern that authors that don't handle these exceptions will start breaking on orientation changing cases where previously they did not.

Anecdotally, I'd say exceptions are uncommon for encode(). Exceptions will be thrown if a frame is already closed or the codec has been closed due to an error or reclaimed by system pressure (an automatic process which is not uncommon, but also generates an error signal).

We discussed this concern in the working group. Keep in mind, encoding is already broken in these cases. It's just that instead of an exception the frame is either squished (perpendicular orientation) or encoded in mirror. As such the working group preferred a clear error signal to the silent issues that exist today.

Additionally, given low WebCodecs VideoEncoder usage on mobile the actual risk of any issues seems unlikely:

https://chromestatus.com/metrics/feature/timeline/popularity/3458

On Wed, Apr 2, 2025 at 8:37 AM Vladimir Levin <vmp...@chromium.org> wrote:

On Wednesday, April 2, 2025 at 12:05:53 AM UTC-4 Dale Curtis wrote:

On Tue, Apr 1, 2025 at 7:16 PM Vladimir Levin <vmp...@chromium.org> wrote:

Some cameras and media will immediately begin exposing orientation information. In uncommon cases this orientation may change. If orientation changes during encoding through VideoEncoder frames with different orientation than the first are currently encoded with incorrect size and orientation. However the operation completes without a hard error. During discussions in the media working group it was decided that these cases should instead throw a non-fatal exception during encode(). There is already precedent for exception handling during encode() so authors should already be handling this case.

Can you clarify if "uncommon" here means that some cameras are known to change the orientation and those cameras are typically rare, or that most cameras will change the orientation if say the user changes the orientation, but that user behavior is rare? :)

I meant most cases will have a single fixed orientation for the duration of the session. Users changing their orientation is uncommon. It will almost always happen with users on mobile / hand-held devices accidentally or intentionally reorienting the device itself.

I'm not sure how to reason about common-ness of the user device orientation changes, so I'm not convinced by this specific argument. 

Sorry, I forgot to cite my source: https://uma.googleplex.com/p/chrome/timeline_v2?sid=9db1427b181520ab8c02f338ff6bb809 (internal only unfortunately), but ~99% of streams never see more than the initial transform.

 

This does seem to introduce a new exception reason that seems almost expected to happen in some cases. Can you comment on whether the other exceptions that the author should already be happening are likewise "common"? There's a bit of a concern that authors that don't handle these exceptions will start breaking on orientation changing cases where previously they did not.

Anecdotally, I'd say exceptions are uncommon for encode(). Exceptions will be thrown if a frame is already closed or the codec has been closed due to an error or reclaimed by system pressure (an automatic process which is not uncommon, but also generates an error signal).

We discussed this concern in the working group. Keep in mind, encoding is already broken in these cases. It's just that instead of an exception the frame is either squished (perpendicular orientation) or encoded in mirror. As such the working group preferred a clear error signal to the silent issues that exist today.

Additionally, given low WebCodecs VideoEncoder usage on mobile the actual risk of any issues seems unlikely:

https://chromestatus.com/metrics/feature/timeline/popularity/3458

That seems a more compelling argument: the current state would be broken in a different way, and the usage is very low. So this makes me think the change is pretty safe. Nevertheless, can you comment on the type of website that could potentially be impacted? Ie what are the common usages of this API?

The most common of those affected would be some semi-RTC based site doing its own camera/capture work on Android. E.g., Zoom who we're working with on this.