• Interest from Web developers is evident on this Chromium bug.
• During TPAC 2022, this was presented in a breakout session. (slides, minutes)

It could also help for tools like https://www.chromatic.com/ for visual regression testing !

We use the browser to create videos for Storrito.com and Audiocado.com, the described feature would have saved me many headaches.

I would prefer access to an element's raw pixel buffer. Converting this to JPG, PNG, or compressed video frame beforehand would significantly slow down all video rendering efforts.

I am also very interested in this API and second that it will be important to get the frame in an uncompressed way.
For Remotion, we will be able to provide in-browser video rendering if this API ships and generally, this would open the door for web-based offline video editors to become more powerful (the status quo is that only canvas-based graphics can be exported as images).

This is going to be a big thing. Because there are many websites which are trying to capture elements for generating images like I am doing https://startupholic.com/post-thumbnail-maker or can be used to create images (For now, I have been capturing image by some work around using canvas.)

The proposed API would also work very well together with my pixel-density property proposal:
#74

This would allow an element to be captured as a raw pixel buffer, but also having the ability to scale the resolution up or down.
(by setting the pixel density before capturing the element)
So an element that has only 200x500 screen pixels could be captured at a much higher resolution (e.g. 600x3000) for much higher quality exports.

This would also work well with designers workflow of exporting assets at different resolutions for different DPI environments (usually 0.5x, 1x, 2x, 4x)

A repo is now live at https://github.com/WICG/element-capture

Happy incubating!!

It's a small detail, but this should probably be a global function (ex. getElementCapture) or have a much less generic (and longer name). Adding a property to HTMLElement prevents adding a reflected attribute named capture to any element forever. In general adding things high up in the class hierarchy is not great for long term evolution of the web.

My suggestion is to extend getDisplayMedia to accept an element option. Capturing an element or the screen/tab is pretty much the same thing security wise. Since this will prompt anyway, that's a small surface area change.

I'm a fan of extending getDisplayMedia from a DX and UX perspective. Users are familiar with it from a permissions flow. I have use cases where I want to constrain the users choice to only share a specific DOM tree, providing a reference to an element to getDisplayMedia feels very natural.

It's a small detail, but this should probably be a global function

One possibility I would like would be navigator.mediaDevices.getElementMedia(), to keep it close to getDisplayMedia.

I'm a fan of extending getDisplayMedia from a DX and UX perspective. Users are familiar with it from a permissions flow. I have use cases where I want to constrain the users choice to only share a specific DOM tree, providing a reference to an element to getDisplayMedia feels very natural.

Communicating to the user what kind of permissions they would be giving is going to be quite tricky whichever way we choose, because the nature and contents of an Element can change; think the contents of a div or iframe, for instance. For that reason, I am mostly thinking about prompting the user for permission to capture the entire tab, and then extrapolating from that permission a permission to capture occluded elements. The delta in permission seems manage-able to me. I intend to do a write-up on that. If we do end up going with that approach, then after prompting the user once to share their entire tab, you'd be able to capture any and all elements.

Why would this API even need permissions in the first place?

Unless it is for like a 3rd party iframe / cross origin resource.

Since I could just as well replicate the functionality with a library like html2canvas and get the underlying pixel data without having to ask for permissions.

I understand it for if you share your Desktop etc that goes beyond just the current browser window, but since all HTML data etc is already accessible, why would getting the pixel data of an element suddenly be an elevated security risk?

For a lot of use cases like capturing a preview or something, it would be very strange for a user to be suddenly prompted with a security warning, which might induce FUD for nor reason.

Just to clarify further, one of the main use cases for the API would be to generate previews of content
(think Google Docs, Figma File etc.)
This currently is only possible server side.

If this API should truly replace that need for doing it server side, then permissions are the wrong approach.
Since what if a user denies it and this breaks you workflow?
Then you'd still have to implement it server side, which defeats the whole point of an API

To summarize:
Permissions only make sense for cross origin content.
Since all DOM data is already accessible to the developer and could be used in a custom rasterizer (such as html2canvas), adding permissions to this makes no sense, because a developer could circumvent them anyway going the latter approach, albeit with more effort.

I could just as well replicate the functionality with a library like html2canvas and get the underlying pixel data

For anyone not aware it's worth mentioning html2canvas only attempts to re-draw the DOM using it's own heuristics and drawing tools available in the canvas. It has no way of asking the browser for an image for any portion of the page. It is imperfect for many edge cases and requires work-arounds if the content it's drawing has anything cross-origin anywhere in the tree it's trying to draw (you can taint the canvas very easily).

From the users perspective, this feature adds the ability to create a live video stream of any part of the page and stream it out to anywhere. As a user, if I visit a webpage, I don't expect a livestream of what I'm looking at to start happening without my awareness.

So for me the crux of it is that

• No solution currently exists to perfectly capture any portion of the page, even with permission. Only approximations can be made, and only static images realistically
• This feature would break most casual users expectation of what is possible without their awareness; a live video stream capturing what they are looking at

With the prevalence of 3rd party scripts being included on websites these days, including the trend of serving these from 1st party domains to avoid blockers, giving all of these scripts the ability to flip on a livestream without intervention feels like a step too far.

A script could just as easily record snapshots of document.body.innerHTML and "replay" the session on a backend if it had malicious intent, without the user ever knowing.

I really fail to see any added security benefit from asking for permissions.

Any data point that would go into creating a screenshot / video is already available to the browser (via parsing .getBoundingClientRect and computed style, as well as capturing event Listeners).

The modern internet is already a nightmare in terms of forced cookie banners on every site that require user consent.
Adding on top of that more popups / prompts for security permissions should only be done if it opens up capabilities otherwise sandboxed by the browser and not for things that are already available / could be easily replicated.

Understand where you're coming from, I think it's a very good point. From a pure security perspective this doesn't expose data not already exposed in some other fashion to the code running on the page (I think?). In that sense, I'm on the same page.

For me it comes down to the user being aware of what is happening / understanding what the browser is doing on their behalf (facilitating the video capture). That could be a permission flow, it could also be a box around the streamed area or a recording icon in the tab.

Without any indication that it's happening I would personally just disable the feature. I realise I might be an outlier here though 😛

If the tab indicates that a capture is happening (especially for longer time frames such as a video stream), then that could be a good non-invasive compromise without disrupting user / developer flow.

Similar to how its done for Tabs that play audio / capture webcam etc.

In terms of API, the other main important thing is compression.

It will be crucial to expose an API that is able to capture an uncompressed screenshot (png / uint8Array).
In terms of video, it becomes more difficult, since holding long uncompressed videos in memory can quickly overflow system memory (e.g. Blender exports all video frames as png to disk in a first step before "assembling" them to a video with whatever level of compression chosen.

Forcing compression on the video stream would be a solution, but then would break its use cases for exporting high quality video exports (think video editor etc.)

Regarding the security model:
Preventions need to be put into place that a website cannot for example load a third party <iframe> into the DOM and then capture a screenshot from the perspective of the user.

For the canvas element, as soon as a third party resource gets used to draw on a canvas (for example loading a network image without CORS), the canvas gets "tainted" and it is not possible to grab an image from it anymore.

If it's hard for browsers to handle these security concerns, then a permission prompt should be introduced, if the browser already has a notion of whether the DOM is tainted by third party content, then it would be cool to use it and allow capture of locally generated content without permission prompt.

@Awendel:

• The presence of cross-origin content is quite common, and must be accounted for. Furthermore, cross-origin content can be loaded after the capture starts.
• You can capture a single frame by plugging the track into a MediaStreamTrackProcessor.

@JonnyBurger, the proposal assumes (i) cross-origin isolation and (2) an opt-in header, likely in the form of a Required Document Policy. At the moment, this is briefly covered in the original message in this thread. I will expand on this later.

To ensure good velocity, I am starting out with the assumption of a prompt.

Having thought about this in the meantime, another approach occurs to me, which is relevant if the user has already initiated capture using pre-existing means such as getDisplayMedia(). In that case, the app already holds a video track, and that track can be mutated.

So where we already have Region Capture mutate an existing video track:

track.cropTo(cropTarget);  // Region Caputre

We could now have:

track.restrictCaptureTo(cropTarget);  // Element Capture

The resulting track would behave the same as previously, but:

1. Adoption by applications will be much easier, since no new requirements would be introduced (no requirement of cross-origin isolation, etc.)
2. Clear path for transitioning a given capture between different targets without reprompting the user.
3. Element Capture of an element in another tab now possible.
4. Audio initially outscoped, but the path forward is clear - mutate the audio track in similar fashion. This path has more flexible, as the video and audio don't have to be "focused" on the same target.

This path has more flexible, as the video and audio don't have to be "focused" on the same target.

This bit is very interesting. I'm imagining capturing an element that represents a video game being played, say a canvas, but I need to define where the audio should be captured from. That could be a seperate DOM tree that contains some <audio> elements or it could be "capture audio from the whole tab" and I'll play sounds programatically with JS.

Maybe audio source can be the second argument? track.restrictCaptureTo(cropTarget, audioSource); with audioSource being either a reference to an element, (an existing AudioTrack?) or the window (or whatever makes more sense to say "everything").

Maybe audio source can be the second argument? track.restrictCaptureTo(cropTarget, audioSource); with audioSource being either a reference to an element, (an existing AudioTrack?) or the window (or whatever makes more sense to say "everything").

There are separate tracks for audio and video, so I think it's enough to audioTrack.restrictCaptureTo(x) and videoTrack.restrictCaptureTo(y), with either distinct or identical x and y as the need arises. (Implementation-wise, I foresee Chrome focusing on video first.)

Ah of course, I forgot for a sec we are operating on the _video_Track

@eladalon1983

track.restrictCaptureTo(cropTarget); // Element Capture

It's interesting but with this approach it will be difficult to capture multiple elements.

There is draft of getDisplayMediaSet for capturing multiple display surfaces but it's about capturing multiple tabs or windows not multiple cropTargets in current tab

It's interesting but with this approach it will be difficult to capture multiple elements.

You could:

const track2 = track1.clone();
track1.restrictCaptureTo(cropTarget1);
track2.restrictCaptureTo(cropTarget2);

Implementation-wise, this will require some work in Chrome to support. But in terms of API shape, this should work.

There's also a more general requirement to be able to rasterize DOM elements without the output format being video.
By being able to rasterize to eg ImageBitmap first, one can still feed the image to a MediaStream using other existing APIs to achieve the same effect as this proposal.

I realise semantics and use cases can be different as the rasterized dom elements can be updated at any time which is automatically handled by the MediaStream but I guarantee that offering just a MediaStream API it will be (ab)used for the [snapshot] rasterizing use-case given the plethora of hacky solutions that exist:

• http://cburgmer.github.io/rasterizeHTML.js/
• https://github.com/PixelsCommander/HTML-GL
  • https://gist.github.com/PixelsCommander/a0b5882139cbb8a1781c#file-proposal-md
  • http://pixelscommander.com/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/

Something to think about?

There's also a more general requirement to be able to rasterize DOM elements without the output format being video. By being able to rasterize to eg ImageBitmap first, one can still feed the image to a MediaStream using other existing APIs to achieve the same effect as this proposal.

I realise semantics and use cases can be different as the rasterized dom elements can be updated at any time which is automatically handled by the MediaStream but I guarantee that offering just a MediaStream API it will be (ab)used for the [snapshot] rasterizing use-case given the plethora of hacky solutions that exist:

• http://cburgmer.github.io/rasterizeHTML.js/

• https://github.com/PixelsCommander/HTML-GL

  • https://gist.github.com/PixelsCommander/a0b5882139cbb8a1781c#file-proposal-md
  • http://pixelscommander.com/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/

Something to think about?

Definitely screenshot-production is an important use-case that interests many (me included). I think we should keep in mind that:

1. The currently proposed API shape relies on pre-existing means for obtaining the user's permission to capture cross-origin content. Moving away from this pre-existing means would require coming up with something new. The complexity involved should not be underestimated.
2. There is nothing "abusive" in grabbing individual frames out of a video stream. It's a legitimate use of the API.
3. It is much easier to grab single frames out of a video stream, than to compose multiple images into a video stream. Think of the individual timestamps, a/v sync, and the requirement to obtain the user's permission 30 times per second... :-P

There's also a more general requirement to be able to rasterize DOM elements without the output format being video. By being able to rasterize to eg ImageBitmap first, one can still feed the image to a MediaStream using other existing APIs to achieve the same effect as this proposal.
I realise semantics and use cases can be different as the rasterized dom elements can be updated at any time which is automatically handled by the MediaStream but I guarantee that offering just a MediaStream API it will be (ab)used for the [snapshot] rasterizing use-case given the plethora of hacky solutions that exist:

• http://cburgmer.github.io/rasterizeHTML.js/

• https://github.com/PixelsCommander/HTML-GL

  • https://gist.github.com/PixelsCommander/a0b5882139cbb8a1781c#file-proposal-md
  • http://pixelscommander.com/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/

Something to think about?

Definitely screenshot-production is an important use-case that interests many (me included). I think we should keep in mind that:

1. The currently proposed API shape relies on pre-existing means for obtaining the user's permission to capture cross-origin content. Moving away from this pre-existing means would require coming up with something new. The complexity involved should not be underestimated.
2. There is nothing "abusive" in grabbing individual frames out of a video stream. It's a legitimate use of the API.
3. It is much easier to grab single frames out of a video stream, than to compose multiple images into a video stream. Think of the individual timestamps, a/v sync, and the requirement to obtain the user's permission 30 times per second... :-P

1 is a valid point, however for 2 & 3:
It would be a waste of CPU and slow as well to encode a frame x times/second, just to immediately decode just a single frame, and with visual loss. You'd also lose the alpha channel because video codecs don't support this. That's the 'abusive' part of this, you'd be using something for something it was never designed to do.

Perhaps the option of a 'raw' video stream with raw rgba pixel 'encoding' and a single frame (0fps options) would allow for greater freedom of use here? It would adhere to the spec proposal, would be element capture api compatible and solve the rasterization use-case.

It would be a waste of CPU and slow as well to encode a frame x times/second

Indeed. And that's why I had previously proposed a dedicated API for screenshots. And such an API could be extended to support capturing an element. Until such a screenshot-grabbing API is introduced, the current Element Capture API can be used to polyfill, but it's of course imperfect when tackling missions other than that to which it is tailored (video).

and with visual loss

Why would there be visual loss? There should be no lossy encoding involved.

You'd also lose the alpha channel because video codecs don't support this.

That's an interesting consideration. Thanks for raising it. I'll keep it in mind.

That's the 'abusive' part of this, you'd be using something for something it was never designed to do.

I don't think the word "abusive" applies here. I would associate it more with the idea that the way the API is used is harmful to the user, which implies the misuse should somehow be blocked.