[MacOS] - UpdateScrollbarTheme Mojo message leads to crash

[azathioprine]

Hello everyone! I am using CEF 120.2.5+g618ea46 with Chromium 120.0.6099.216 inside an Unreal Engine plugin aiming to support both the Windows and MacOs Arm64 platforms.

CEF is doing OSR and I'm initializing it with no_sandbox = true, external_message_pump = true and multi_threaded_message_loop = false.

I got to the point where pages render OK on both platforms. I can load websites, navigate around, mouse and keyboard events get passed, all seems right. The Windows side implementation currently functions without crashes.

However, the MacOS Arm64 implementation frequently crashes with this back trace:

Code: Select all

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=1, subcode=0x3b35d6ef4)

  * frame #0: 0x00000003b35d6ef4 Chromium Embedded Framework`mojo::InterfaceEndpointClient::Accept(mojo::Message*) [inlined] base::ImmediateCrash() at immediate_crash.h:146:3 [opt]

    frame #1: 0x00000003b35d6ef4 Chromium Embedded Framework`mojo::InterfaceEndpointClient::Accept(mojo::Message*) [inlined] logging::CheckFailure() at check.h:192:3 [opt]

    frame #2: 0x00000003b35d6ef4 Chromium Embedded Framework`mojo::InterfaceEndpointClient::Accept(mojo::Message*) [inlined] mojo::InterfaceEndpointClient::SendMessage(this=0x00000004dd53f580, message=0x000000016f75c3e0, is_control_message=false) at interface_endpoint_client.cc:578:3 [opt]

    frame #3: 0x00000003b35d6eb4 Chromium Embedded Framework`mojo::InterfaceEndpointClient::Accept(this=0x00000004dd53f580, message=0x000000016f75c3e0) at interface_endpoint_client.cc:565:10 [opt]

    frame #4: 0x00000003b35f107c Chromium Embedded Framework`mojo::internal::SendMojoMessage(receiver=0x00000004dd53f580, message=0x000000016f75c3e0) at send_message_helper.cc:23:26 [opt]

    frame #5: 0x00000003b1b83118 Chromium Embedded Framework`content::mojom::RendererProxy::UpdateScrollbarTheme(this=0x00006000039a5f70, in_params=<unavailable>) at renderer.mojom.cc:937:3 [opt]

    frame #6: 0x00000003b2479474 Chromium Embedded Framework`-[SystemThemeObserver notifyPrefsChangedWithRedraw:](self=<unavailable>, _cmd=<unavailable>, redraw=<unavailable>) at theme_helper_mac.mm:231:43 [opt]

    frame #7: 0x0000000184ba2b1c CoreFoundation`__CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 148

    frame #8: 0x0000000184c36db8 CoreFoundation`___CFXRegistrationPost_block_invoke + 88

    frame #9: 0x0000000184c36d00 CoreFoundation`_CFXRegistrationPost + 440

    frame #10: 0x0000000184b71648 CoreFoundation`_CFXNotificationPost + 768

    frame #11: 0x0000000185c8d464 Foundation`-[NSNotificationCenter postNotificationName:object:userInfo:] + 88

    frame #12: 0x0000000189117f74 AppKit`___NSMainRunLoopPerformBlockInModes_block_invoke + 44

    frame #13: 0x0000000184bada48 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 28

    frame #14: 0x0000000184bad95c CoreFoundation`__CFRunLoopDoBlocks + 356

    frame #15: 0x0000000184bacdec CoreFoundation`__CFRunLoopRun + 2440

    frame #16: 0x0000000184babe0c CoreFoundation`CFRunLoopRunSpecific + 608

    frame #17: 0x000000010335ffb8 UnrealEditor-Core.dylib`PerformBlockOnThread(FCocoaRunLoopSource&, void () block_pointer, NSArray*, NSString*, bool) [inlined] FCocoaRunLoopSource::RunInMode(this=0x000000013c5fffc0, WaitMode=string=Empty) at CocoaThread.cpp:109:3 [opt]

    frame #18: 0x000000010335ffa8 UnrealEditor-Core.dylib`PerformBlockOnThread(ThreadSource=0x000000013c5fffc0, Block=0x0000600002e88e40, SendModes=5 elements, WaitMode="kCFRunLoopDefaultMode", bWait=<unavailable>) at CocoaThread.cpp:423:17 [opt]

    frame #19: 0x00000001033600c8 UnrealEditor-Core.dylib`GameThreadCall(Block=<unavailable>, SendModes=<unavailable>, bWait=<unavailable>) at CocoaThread.cpp:456:3 [opt] [artificial]

    frame #20: 0x0000000101a53788 UnrealEditor-ApplicationCore.dylib`-[FCocoaWindow windowDidResignMain:](self=0x000000014c0e1a10, _cmd=<unavailable>, Notification=<unavailable>) at CocoaWindow.cpp:303:3 [opt]

    frame #21: 0x0000000184ba2b1c CoreFoundation`__CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 148

    frame #22: 0x0000000184c36db8 CoreFoundation`___CFXRegistrationPost_block_invoke + 88

    frame #23: 0x0000000184c36d00 CoreFoundation`_CFXRegistrationPost + 440

    frame #24: 0x0000000184b71648 CoreFoundation`_CFXNotificationPost + 768

    frame #25: 0x0000000185c8d464 Foundation`-[NSNotificationCenter postNotificationName:object:userInfo:] + 88

    frame #26: 0x000000018855ef64 AppKit`_NXEndKeyAndMain + 144

    frame #27: 0x000000018855e05c AppKit`-[NSApplication _handleDeactivateEvent:] + 720

    frame #28: 0x0000000188bf5cf4 AppKit`-[NSApplication(NSEventRouting) sendEvent:] + 1224

    frame #29: 0x000000035b7d59b8 UnrealEditor-CEF3Utils.dylib`-[NSApplication(self=<unavailable>, _cmd=<unavailable>, event=<unavailable>) cef3UtilsSendEvent:] at MacCefApplicationCategory.cpp:54:2 [opt]

    frame #30: 0x00000001888438cc AppKit`-[NSApplication _handleEvent:] + 60

    frame #31: 0x00000001883f7cdc AppKit`-[NSApplication run] + 512

    frame #32: 0x00000001006ddbe4 UnrealEditor`tchar_main(ArgC=<unavailable>, ArgV=u"⠀㱯\U00000001") at LaunchMac.cpp:456:2 [opt]

    frame #33: 0x00000001006dd874 UnrealEditor`main(ArgC=2, Utf8ArgV=0x000000016f75f388) at LaunchMac.cpp:418:1 [opt]

    frame #34: 0x00000001847460e0 dyld`start + 2360

So far, I have spent days trying to debug and understand exactly what's going wrong, but I'm currently unable to identify the issue.

In certain random instances, I can use the browser normally. In other instances, the crash happens. The render helper processes seem to spawn and continue running in the background in both situations, as I can see log messages related to them continuing to appear in the the console even after the main UE process is stopped by the debugger at the frame in the call stack above. This made me wonder if the issue could be caused by some form of race condition that appears once the main Browser thread is trying to exchange messages with the newly spawned render helpers.

This crash obviously doesn't appear when running the provided cefsimple.app with "--off-screen-rendering-enabled", but it does appear when running my implementation with either my own render helper or with cefsimple as a render helper - so I assume it's not caused by something I'm doing in my implementation of the render process helper - but by something I'm doing wrong on the "Browser" thread end of things.

I don't know if this is relevant, but since my implementation is not following the default CEF folder structure, I'm initializing CEF with custom absolute paths for "framework_dir_path", "main_bundle_path" or "browser_subprocess_path" (since things sometimes work without a crash I'm assuming this isn't the cause of the issue).

I tried taking a look inside the relevant Chromium code indicated by the back trace - particularly inside the "InterfaceEndpointClient::SendMessage" call from inside "mojo/public/cpp/bindings/lib/interface_endpoint_client.cc" - as the crash appears to be triggered by one of the checks present there. However, these checks seem to target conditions/behaviors that are deeply embedded into Chromium's IPC system. So I'm not sure what sort of clues to extract from them regarding to what I'm doing wrong during CEF setup/initialization to cause a certain message to end up on the wrong sequence or with the wrong flags.

I also looked at what the main "CrBrowserMain" thread is doing before the crash (separate from the thread that crashes, this is the actual thread in which I initialize CEF and run the message loop) - it seems that the crash is being triggered after "CefDoMessageLoopWork()" runs for the first or second time after the render processes spawn. I can see some other messages being exchanged between this thread and the render processes here and then the crash is triggered on the app's main thread.

I am currently out of ideas regarding where to look to debug this issue. If anyone has any tips that could point me in the right direction, I would really appreciate your time and your input. Thank you in advance!

[azathioprine]

I have some more updates on this issue.

Turns out the crash isn't really intermittent - it just felt like it was so to me, because of my entangled setup (I use a Bluetooth keyboard to quickly switch between PC/Mac, sometimes I use the Mac remotely from my PC, etc.). Since my last post above, I was able to identify a way to reliably trigger it, and it's quite bizarre.

It turns out the crash will trigger on the MacOS implementation any time I have an external mouse or keyboard connected to the MacBook I'm using for development!

If I'm just using the built-in keyboard/touchpad, everything is running fine.

If I connect a mouse/keyboard through Bluetooth/USB while my implementation is running, it seems like some OS level events to which Chromium subscribes (probably through that "SystemThemeObserver" class) fire and trigger certain calls which end up crashing my CEF implementation with the call stack above.

If I already have my mouse/keyboard connected when I run my CEF implementation, the OS level calls related to the "SystemThemeObserver" class are triggered right after CEF initializes.

Now I'm even more confused. I guess I'm doing something wrong that creates the proper conditions for this bizarre chain of events which results in that Mojo message being sent on the wrong thread, eventually causing Chromium to crash.

Again, if anyone has any ideas why this could happen or has ever seen something like this before, I would really appreciate your input.

Will further update the post when/if I find out more interesting details. Thank you!

[magreenblatt]

This crash obviously doesn't appear when running the provided cefsimple.app with "--off-screen-rendering-enabled"

Please test with the cefclient app as the cefsimple app doesn't implement off-screen rendering. You can pass "--off-screen-rendering-enabled --external-message-pump" command-line flags to try and match your setup.