Future of onAcceleratedPaint?

Do not post support requests, bug reports or feature requests. Discuss CEF here. Non-CEF related discussion goes in General Discussion!

Future of onAcceleratedPaint?

Postby gbooker » Wed Mar 15, 2023 8:45 am

Greetings all,

I am a developer at Plex, and we are considering CEF for one of our apps. The current version uses a different mechanism to render a web interface over video playback, and are looking at replacing this with CEF. GPU rendering of the UI (web layer) without CPU copy-back is highly preferred so as to maintain high frame rates in the UI.

In our usage, we need the web interface to be rendered with a transparent background, since it is displayed on top of the video. As I understand it, the current state is as follows (please correct me if I got anything wrong):

  • Rendering with transparency requires OSR.
  • onAcceleratedPaint works on Windows with a shared D3D11 texture.
    • This is not called on other platforms; instead, onPaint is called, requiring a copy-back of damage rects
  • This PR contains patches to bring onAcceleratedPaint to Mac and Linux platforms. Further work which was updated to newer Chromium versions is available in Richard Jackson’s repo and OBS’s Github repo.
    • This work cannot be rebased beyond version 5060 due to architectural changes in Chromium

In light of the above, I’m wondering: what is the future of this use-case? We would like to know if changing to use CEF would require either sticking with 5060 or giving up texture sharing on Mac/Linux, or if another method will be available down the road. Could 3263, 3293, and/or 3294 yield the desired result?

For reference, our current implementation composites the UI and video via:
  • Windows: Rendering the UI and video in a separate child HWNDs letting Windows do the compositing
  • Mac: Same idea but using NSViews
  • Linux: Blitting the video and UI into a shared framebuffer and displaying the result.

Thanks for any info.
gbooker
Newbie
 
Posts: 3
Joined: Wed Mar 15, 2023 8:26 am

Re: Future of onAcceleratedPaint?

Postby magreenblatt » Wed Mar 15, 2023 10:56 am

onAcceleratedPaint works on Windows with a shared D3D11 texture

OnAcceleratedPaint currently requires external changes on all platforms.

This work cannot be rebased beyond version 5060 due to architectural changes in Chromium

Where did you find this information?

what is the future of this use-case?

The preferred (supported by Chromium) approach would be using the Ozone layer (#3263). Unfortunately Ozone is currently only supported on Linux.

For reference, our current implementation composites the UI and video

The web platform supports layering of UI and video elements. Have you considered UI and video in the same browser window, instead of performing the compositing yourself?
magreenblatt
Site Admin
 
Posts: 12744
Joined: Fri May 29, 2009 6:57 pm

Re: Future of onAcceleratedPaint?

Postby gbooker » Wed Mar 15, 2023 1:18 pm

magreenblatt wrote:OnAcceleratedPaint currently requires external changes on all platforms.

But as I understand it these changes do currently work on Windows (and not other platforms), correct?

magreenblatt wrote:
This work cannot be rebased beyond version 5060 due to architectural changes in Chromium


Where did you find this information?

There is a comment in that PR stating:
Going forward, it seems chromium 102 or 103 will be the last versions which work with this PR.

Indeed this recent chromium commit will prevent the disabling of skia :
https://github.com/chromium/chromium/co ... dc9bb99635

So it appears the means by which texture sharing was working on Mac/Linux in this PR has been removed in subsequent Chromium versions.

magreenblatt wrote:The preferred (supported by Chromium) approach would be using the Ozone layer (#3263). Unfortunately Ozone is currently only supported on Linux.

Would this be expanded to other platforms in the future? This really is the crux of our question and the reason for this thread: Would there be a future method of rendering a web page with transparent background via a shared texture or similar mechanism on Windows, Mac, and Linux?

magreenblatt wrote:The web platform supports layering of UI and video elements. Have you considered UI and video in the same browser window, instead of performing the compositing yourself?

The media playback stack in the browser isn't as good as MPV (which we currently use). Not only can MPV play far more containers and codecs than a web's media stack, it also does audio passthrough (to an AVR) and shader-based scaling / post-processing just to name the start of the feature differences our users would notice. Using MPV is the biggest difference in using this app instead of just playing media in a browser.
gbooker
Newbie
 
Posts: 3
Joined: Wed Mar 15, 2023 8:26 am

Re: Future of onAcceleratedPaint?

Postby magreenblatt » Wed Mar 15, 2023 2:52 pm

gbooker wrote:But as I understand it these changes do currently work on Windows (and not other platforms), correct?

I have not tested the PR changes, so unfortunately I can't add any clarification to the existing PR comments.

gbooker wrote:There is a comment in that PR stating:

Thanks, I missed that one.

gbooker wrote:
magreenblatt wrote:The preferred (supported by Chromium) approach would be using the Ozone layer (#3263). Unfortunately Ozone is currently only supported on Linux.

Would this be expanded to other platforms in the future? This really is the crux of our question and the reason for this thread: Would there be a future method of rendering a web page with transparent background via a shared texture or similar mechanism on Windows, Mac, and Linux?

That would be completely up to Google, and the response so far has been "maybe in the future, but not currently planned or staffed".

gbooker wrote:The media playback stack in the browser can't even begin to compete with MPV. Not only can MPV play far more containers and codecs than a web's media stack, it also does audio passthrough (to an AVR) and shader-based scaling / post-processing just to name the start of the feature differences.

Chromium currently supports a number of video backends, including platform APIs and ffmpeg for decoding and playback. You could potentially add a new backend for your preferred technology. That, at least, would be using an existing/supported cross-platform interface in Chromium, and you would have the full benefit of Chromium's existing compositor pipeline for rendering to a native (browser) window.
magreenblatt
Site Admin
 
Posts: 12744
Joined: Fri May 29, 2009 6:57 pm

Re: Future of onAcceleratedPaint?

Postby gbooker » Wed Mar 22, 2023 1:05 pm

Thanks for the info. You have given us some good information to consider.

magreenblatt wrote:Chromium currently supports a number of video backends, including platform APIs and ffmpeg for decoding and playback. You could potentially add a new backend for your preferred technology. That, at least, would be using an existing/supported cross-platform interface in Chromium, and you would have the full benefit of Chromium's existing compositor pipeline for rendering to a native (browser) window.

This does seem to also carry the danger that Chromium could break it at any time unless it is upstreamed. We noticed similar occurred to GStreamer.
gbooker
Newbie
 
Posts: 3
Joined: Wed Mar 15, 2023 8:26 am

Re: Future of onAcceleratedPaint?

Postby rjxray » Tue Jun 13, 2023 4:22 am

For the Microsoft case we too have the need for high speed CEF rendering to a layered canvas and have been using the shared texture patch up to CEF5060 / Chromium 103. I can confirm that this works well on Windows.

Once it became clear that the patch could not be further updated we needed an alternative and have now successfully implemented using the Windows Graphics Capture API to get the texture from an external CEF window. The CEF window is rendered directly by Chromium and drawn off the visible screen and then it's texture is captured and drawn within our app. A reference to the browser can obtained when the CEF window is created and it can then be treated as a normal OSR browser.

An example of using the WGI interface can be found at https://github.com/robmikh/Win32CaptureSample. I don't know whether there is anything similar for Linux or Mac
rjxray
Expert
 
Posts: 117
Joined: Wed Jun 07, 2017 4:31 am


Return to CEF Discussion

Who is online

Users browsing this forum: No registered users and 2 guests