Fast resizing failure

[Freak123456]

Hi, I was just send here to report a bug(?) that arose only a few month ago using WasResized().
https://github.com/cefsharp/CefSharp/issues/2547

CEF Version: 3.3578.1860.g36610bd (x64) (http://opensource.spotify.com/cefbuilds/index.html)
C++ 17 x64 via MSVC 2017
Win10 1809 x64

It was first reported in October 2018 and I found the issue discussion in search for the same bug in C++ only a few days ago.
I call WasResized() in the applications main thread whenever a Size message from windows via Win32 arrives for the window I show the off screen rendered html content in I get via "OnPaint()".
CEFs MessageLoop runs in another thread and the command line arguments are always and only:
"multi-threaded-message-loop" "off-screen-rendering-enabled" "enable-gpu" "cache-path=cache"
Data arrives via OnPaint() just fine even when resizing quite fast and rough until the "Check failed: ..." error occures.
It seems there is some kind of mutex/spinlock missing whichs guards the data while "AsyncLayerTreeFrameSink::SubmitCompositorFrame()" runs over it at the same time.
The same error occures when calling "WasResized()" from the CEF UI Thread via "CefPostTask()".
NO other CEF function is triggered by the application other than "WasResized()" after the page has been loaded and some Objects, Functions and a custom protocol have been registered.

[2019.01.14 20:49:08.217] OnWindowResolutionChanged(): (1316 886)
[2019.01.14 20:49:08.227] OnWindowResolutionChanged(): (1311 883)
[0114/204908.239:FATAL:async_layer_tree_frame_sink.cc(190)] Check failed: last_submitted_size_in_pixels_.height() == frame.size_in_pixels().height() (887 vs. 886)
[2019.01.14 20:49:08.241] OnWindowResolutionChanged(): (1298 876)
[2019.01.14 20:49:08.253] OnWindowResolutionChanged(): (1291 872)
...
[2019.01.14 20:49:08.729] OnWindowResolutionChanged(): (619 470)
[0114/204908.736:ERROR:gles2_cmd_decoder.cc(5642)] GLES2DecoderImpl: Context lost because resize failed.
[0114/204908.736:ERROR:gles2_cmd_decoder.cc(5772)] Error: 5 for Command kResizeCHROMIUM
[0114/204908.737:ERROR:gpu_channel_manager.cc(215)] Exiting GPU process because some drivers cannot recover from problems.

I and some other people from the other than C++ world would love to see this problem being solved quickly by a fix or a "correct usage hint".
Thanks for your time.

[magreenblatt]

Does the problem reproduce with the cefclient sample app?

[amaitland]

I've had limited success reproducing the issue myself, had quite a few user reports though. The one time I did manage to reproduce the issue with the CefSharp MinimalExample it disappeared after a reboot.

Users started reporting the issue after upgrading to the 3497 branch, no code changes were made to the CefRenderHandler implementation.

I don't believe @Freak123456 is actually using CefSharp, so it's either that both our CefRenderHandler implementations are incorrect or there is a bug.

Anyone wishing to test with cefclient should use the following args

Code: Select all

cefclient --multi-threaded-message-loop --off-screen-rendering-enabled --enable-gpu --cache-path=cache


It's difficult for me to debug as I cannot reliably reproduce the problem.

@Freak123456 if you can test with cefclient and report back that would be helpful.

[ebahar]

Hi,
We also encounter this issue in our application (rarely) stating branch 3497.
we notice is happen more often on high DPI (125 \ 15)..

[Freak123456]

Solved, BUT:

I tested various scenarios. (Can't even remember them all now. ^^)
To clarify:
- using C++
- libcef_dll_wrapper cmaked for MS Visual Studio 2017 Win64 AND set CEF_RUNTIME_LIBRARY_FLAG="/MD" (!!!, My library needs this later on or the linker complains.)
- My Project - A: dynamic library for CEF handling
- My Project - B: the most basic executable to run CEF in subprocess
- My Project - C: using A and B in executable which shows the described error in the first post

Example, as requested:
cefclient (--multi-threaded-message-loop --off-screen-rendering-enabled --enable-gpu --cache-path=cache):
In Debug/Release without command line arguments: no problems
In Debug with command line arguments: [0116/225949.188:FATAL:lock_table.cc(18)] Check failed: file->path().IsAbsolute(). -> crash within 2s, not showing google website, but toolbar
In Release with command line arguments: MASSIVE lag when resizing, ~5-10s (Intel Core i7, GTX 1070, And my application is at least 100x faster. O.o)

I wrote the above and then had the enlightenment. I made a simple mistake. Maybe some other people too.
"Set CefSettings.multi_threaded_message_loop = true (Windows only). This will cause CEF to run the browser UI thread on a separate thread from the main application thread. With this approach neither CefDoMessageLoopWork() nor CefRunMessageLoop() need to be called. CefInitialize() and CefShutdown() should still be called on the main application thread. You will need to provide your own mechanism for communicating with the main application thread (see for example the message window usage in cefclient_win.cpp). You can test this mode in cefclient on Windows by running with the “--multi-threaded-message-loop” command-line flag."

"Solved" (in my library 'A'):
CefSettings cefSettings = {};
cefSettings.windowless_rendering_enabled = true;
cefSettings.multi_threaded_message_loop = true; // !!!!!!!!!!!!!!!!!!!!!!!!!! not "false", as before
cefSettings.external_message_pump = false;

The reason is obviously enabled thread protection. -.-

"BUT":
"Set CefSettings.multi_threaded_message_loop = true (Windows only)."
What do I have to do in Linux when I have to set 'false'?


Thanks for the quick responses.

[magreenblatt]

Linux support was added in https://bitbucket.org/chromiumembedded/cef/issues/2512

[amaitland]

The reason is obviously enabled thread protection. -.-

Calling multi threaded message loop `thread protection` isn't exactly accurate. Regardless, CefSharp uses multi threaded message loop by default. So the many reports I've seen are already using this.

We also encounter this issue in our application (rarely) stating branch 3497.

@ebahar can you provide more details about your application.

[ebahar]

It's .net application (using CEFglue) OSR (WPF).

i was able also to reproduce it with cefclient :
cefclient --multi-threaded-message-loop --off-screen-rendering-enabled --disable-gpu --disable-gpu-composition
as u said it's not easy to reproduce itת but on high dpi it's happen more frequent.

i have a patch that seems to fix it (but i don't thing it's best the solution):

Code: Select all

void RenderWidgetHostImpl::SubmitCompositorFrame(
    const viz::LocalSurfaceId& local_surface_id,
    viz::CompositorFrame frame,
    base::Optional<viz::HitTestRegionList> hit_test_region_list,
    uint64_t submit_time) {
  // Ensure there are no CopyOutputRequests stowed-away in the CompositorFrame.
  // For security/privacy reasons, renderers are not allowed to make copy
  // requests because they could use this to gain access to content from another
  // domain (e.g., in a child frame).
  if (frame.HasCopyOutputRequests()) {
    bad_message::ReceivedBadMessage(GetProcess(),
                                    bad_message::RWH_COPY_REQUEST_ATTEMPT);
    return;
  }

  auto new_surface_properties =
      RenderWidgetSurfaceProperties::FromCompositorFrame(frame);


  /***************************************************************/
  if (local_surface_id == last_local_surface_id_ &&
      new_surface_properties != last_surface_properties_) {
    // PATCH: this happen on resizing OSR window
    // and do not print error
    bool handled = false;
    if (view_) {
      handled = view_->OnDidNotProduceFrameSurfaceMismatch(frame.metadata.begin_frame_ack);
     
      if (handled) {
        std::string error = base::StringPrintf(
          "[OOPIF? %d] %s\n", view_ && view_->IsRenderWidgetHostViewChildFrame(),
          new_surface_properties.ToDiffString(last_surface_properties_).c_str());
        LOG(WARNING) << "Surface invariants violation: " << error;

        if (view_) {
          frame.metadata.begin_frame_ack.has_damage = false;
          view_->OnDidNotProduceFrame(frame.metadata.begin_frame_ack);
        }
        std::vector<viz::ReturnedResource> resources =
          viz::TransferableResource::ReturnResources(frame.resource_list);
        renderer_compositor_frame_sink_->DidReceiveCompositorFrameAck(resources);

        return;
      }
    }
     =/***********************end pathc********************************/
    std::string error = base::StringPrintf(
        "[OOPIF? %d] %s\n", view_ && view_->IsRenderWidgetHostViewChildFrame(),
        new_surface_properties.ToDiffString(last_surface_properties_).c_str());
    LOG(ERROR) << "Surface invariants violation: " << error;

    static int invariants_violation_count = 0;
    ++invariants_violation_count;
    UMA_HISTOGRAM_COUNTS_1000("Compositing.SurfaceInvariantsViolations",
                              invariants_violation_count);

    if (features::IsSurfaceInvariantsViolationLoggingEnabled()) {
      static auto* crash_key = base::debug::AllocateCrashKeyString(
          "surface-invariants-violation", base::debug::CrashKeySize::Size256);
      base::debug::ScopedCrashKeyString key_value(crash_key, error);
      base::debug::DumpWithoutCrashing();
    }

    if (view_) {
      frame.metadata.begin_frame_ack.has_damage = false;
      view_->OnDidNotProduceFrame(frame.metadata.begin_frame_ack);
    }
    std::vector<viz::ReturnedResource> resources =
        viz::TransferableResource::ReturnResources(frame.resource_list);
    renderer_compositor_frame_sink_->DidReceiveCompositorFrameAck(resources);

    return;
  }

  last_local_surface_id_ = local_surface_id;
  last_surface_properties_ = new_surface_properties;

  last_received_content_source_id_ = frame.metadata.content_source_id;

  // |has_damage| is not transmitted.
  frame.metadata.begin_frame_ack.has_damage = true;

  last_frame_metadata_ = frame.metadata.Clone();

  if (enable_surface_synchronization_) {
    if (view_) {
      // If Surface Synchronization is on, then |new_content_rendering_timeout_|
      // is stopped in DidReceiveFirstFrameAfterNavigation.
      view_->SubmitCompositorFrame(local_surface_id, std::move(frame),
                                   std::move(hit_test_region_list));
      view_->DidReceiveRendererFrame();
    } else {
      std::vector<viz::ReturnedResource> resources =
          viz::TransferableResource::ReturnResources(frame.resource_list);
      renderer_compositor_frame_sink_->DidReceiveCompositorFrameAck(resources);
    }
  } else {
    // Ignore this frame if its content has already been unloaded. Source ID
    // is always zero for an OOPIF because we are only concerned with displaying
    // stale graphics on top-level frames. We accept frames that have a source
    // ID greater than |current_content_source_id_| because in some cases the
    // first compositor frame can arrive before the navigation commit message
    // that updates that value.
    if (view_ &&
        frame.metadata.content_source_id >= current_content_source_id_) {
      view_->SubmitCompositorFrame(local_surface_id, std::move(frame),
                                   std::move(hit_test_region_list));
      view_->DidReceiveRendererFrame();
    } else {
      if (view_) {
        frame.metadata.begin_frame_ack.has_damage = false;
        view_->OnDidNotProduceFrame(frame.metadata.begin_frame_ack);
      }
      std::vector<viz::ReturnedResource> resources =
          viz::TransferableResource::ReturnResources(frame.resource_list);
      renderer_compositor_frame_sink_->DidReceiveCompositorFrameAck(resources);
    }

    // After navigation, if a frame belonging to the new page is received, stop
    // the timer that triggers clearing the graphics of the last page.
    if (last_received_content_source_id_ >= current_content_source_id_) {
      did_receive_first_frame_after_navigation_ = true;
      if (new_content_rendering_timeout_ &&
          new_content_rendering_timeout_->IsRunning()) {
        new_content_rendering_timeout_->Stop();
      }
    }
  }
}


[Freak123456]

I'm sorry, but I just got the same error again even with my "solution" in the last post but it takes more (not much) effort to reproduce with my application.
And since 'ebahar' got the same results with cefclient it seems like a real bug.

I hope you guys find a quick solution for that.

[rljohn]

I've encountering the same issue Freak123456 describes, with a similar setup: C++17 x64 VS2017 on Win7 SP1.

We're using off-screen rendering and an implementation that calls CefDoMessageLoopWork() within our own loop.

The issue reproduced in the following revisions as soon as WasResized() was called after a WM_SIZE event
CEF 3.3578.1863.gbf8cff2
CEF 3.3538.1851.g5622787
CEF 3.3497.1821.g532ce6b

I've rolled back to CEF 3.3440.1806.g65046b7 which fixes the issue. I was unable to reproduce in cefclient.