Heap allocations are leaked

Having problems with building or using the CefGlue .NET/Mono binding? Ask your questions here.

Moderator: fddima

Heap allocations are leaked

Postby scidoner » Fri Nov 29, 2013 8:55 pm

OK I'll start with some basics:

Client OS: Windows 8.1 64-bit
CEF version: 3.1650.1503
Xilium.CefGlue downloaded revision: e4777a271575

Part of testing for my product, I run it under the application verifier that comes with the Windows SDK. I enable all of the basic checks in the application verifier (exceptions, handles, leaks, locks, etc). In order to view the output of the verifier, I use the global flags utility to start my application running under WinDbg.

My application runs fine under the verifier, no problem. CEF is working fine - no problem. No issues from the app verifier while the application is running and loading webpages in CEF. Excellent.

Closing the application though, is where much of the app verifier results will show. So I close my application - BOOM. There are a bazillion heap leaks, all with the same/similar stack trace. No heap leaks if you just load the CEF run-time in the process. They only happen once you start actually loading websites using CEF.

To make sure my application was not the problem, I have tested using the client app from CefGlue and the WinForms demo app from CefGlue - the same heap leaks occur under the same app verifier settings. Now, I also tested the native CEF demo application (which completely eliminates CefGlue - no heap leaks with that one!!).

Here is an example verifier stop message about a heap leak:

Code: Select all
=======================================
VERIFIER STOP 00000900: pid 0x458: A heap allocation was leaked.

   1E650D18 : Address of the leaked allocation. Run !heap -p -a <address> to get additional information about the allocation.
   03FA1C14 : Address to the allocation stack trace. Run dps <address> to view the allocation stack.
   2319AFE8 : Address of the owner dll name. Run du <address> to read the dll name.
   5B470000 : Base of the owner dll. Run .reload <dll_name> = <address> to reload the owner dll. Use 'lm' to get more information about the loaded and unloaded modules.


=======================================
This verifier stop is continuable.
After debugging it use `go' to continue.

=======================================


Now, here is the allocation stack for one of them (they are all the same - and you can keep pressing F5 over and over, there's a bazillion):

Code: Select all
0:000> dps 03FA1C14
03fa1c14  03f15a44
03fa1c18  0000f803
03fa1c1c  00200000
03fa1c20  5d3f8d9c verifier!AVrfDebugPageHeapAllocate+0x23c
03fa1c24  77993165 ntdll!RtlDebugAllocateHeap+0x32
03fa1c28  77930934 ntdll!RtlpAllocateHeap+0x411fc
03fa1c2c  778ef584 ntdll!RtlAllocateHeap+0x14c
03fa1c30  5d90a792 vrfcore!VfCoreRtlAllocateHeap+0x16
03fa1c34  5d460196 vfbasics!AVrfpRtlAllocateHeap+0xe2
03fa1c38  7723141d KERNELBASE!LocalAlloc+0x88
03fa1c3c  5d460c4d vfbasics!AVrfpLocalAlloc+0xa8
03fa1c40  5b47e803 d3d9!D3D9CreateDirectDrawObject+0x5b
03fa1c44  5b47d25f d3d9!FetchDirectDrawData+0xeb
03fa1c48  5b47e70a d3d9!InternalDirectDrawCreate+0x150
03fa1c4c  5b47fd33 d3d9!CEnum::CEnum+0x232
03fa1c50  5b4801d7 d3d9!Direct3DCreate9Impl+0x80
03fa1c54  5b48027e d3d9!Direct3DCreate9Ex+0x27
03fa1c58  121c9e22 libcef!ui_surface_d3d9_utils::CreateDevice+0x52 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\ui\surface\d3d9_utils_win.cc @ 40]
03fa1c5c  121b9447 libcef!PresentThread::ResetDevice+0x187 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\ui\surface\accelerated_surface_win.cc @ 224]
03fa1c60  121b90fc libcef!PresentThread::InitDevice+0xcc [c:\cef\workspace\cef3-windows-1650\download\chromium\src\ui\surface\accelerated_surface_win.cc @ 197]
03fa1c64  121be169 libcef!AcceleratedPresenter::DoReleaseSurface+0x39 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\ui\surface\accelerated_surface_win.cc @ 921]
03fa1c68  121c618b libcef!base::internal::RunnableAdapter<void (__thiscall AcceleratedPresenter::*)(void)>::Run+0x1b [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\bind_internal.h @ 134]
03fa1c6c  121c5f2a libcef!base::internal::InvokeHelper<0,void,base::internal::RunnableAdapter<void (__thiscall AcceleratedPresenter::*)(void)>,void __cdecl(AcceleratedPresenter * const &)>::MakeItSo+0x1a [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\bind_internal.h @ 872]
03fa1c70  121c5b95 libcef!base::internal::Invoker<1,base::internal::BindState<base::internal::RunnableAdapter<void (__thiscall AcceleratedPresenter::*)(void)>,void __cdecl(AcceleratedPresenter *),void __cdecl(AcceleratedPresenter *)>,void __cdecl(AcceleratedPresenter *)>::Run+0x45 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\bind_internal.h @ 1169]
03fa1c74  10228aef libcef!base::Callback<void __cdecl(void)>::Run+0x2f [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\callback.h @ 396]
03fa1c78  102e1f43 libcef!base::MessageLoop::RunTask+0x2d3 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_loop.cc @ 493]
03fa1c7c  102e22c4 libcef!base::MessageLoop::DeferOrRunPendingTask+0x34 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_loop.cc @ 506]
03fa1c80  102e2810 libcef!base::MessageLoop::DoWork+0xe0 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_loop.cc @ 617]
03fa1c84  10382f7b libcef!base::MessagePumpDefault::Run+0xbb [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_pump_default.cc @ 32]
03fa1c88  102e1ad7 libcef!base::MessageLoop::RunInternal+0xf7 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_loop.cc @ 441]
03fa1c8c  102e18ae libcef!base::MessageLoop::RunHandler+0x2e [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\message_loop\message_loop.cc @ 414]
03fa1c90  103523c9 libcef!base::RunLoop::Run+0x29 [c:\cef\workspace\cef3-windows-1650\download\chromium\src\base\run_loop.cc @ 48]


So clearly there is a leak somewhere. There's nothing about CefGlue in the stack trace which is very weird, but it does not occur with the native CEF client. IS this the right place to post this? I could have posted it on the CEF web site, but they probably wouldn't be able to reproduce it (I certainly couldn't just with CEF and no CefGlue).
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby scidoner » Sun Dec 01, 2013 1:06 am

Let me see if I can work out the cause of this myself...
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby scidoner » Sun Dec 01, 2013 1:34 am

OK I lied. I can reproduce it with the native application (CefGlue, it is not your fault sir). Running in debug also has some output in the console log (seems to detect its own leaks):
6monthsinaleakyboat.jpg
6monthsinaleakyboat.jpg (12.58 KiB) Viewed 21021 times
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby magreenblatt » Sun Dec 01, 2013 10:52 am

LEAK messages come from the render process. For performance reasons Chromium intentionally terminates processes without freeing memory. It's faster to allow the OS to clean up process memory then to explicitly free all objects. If you want to identify unexpected leaks you should run valgrind or tsan with the Chromium suppression list: http://www.chromium.org/developers/how- ... g-valgrind
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: Heap allocations are leaked

Postby scidoner » Sun Dec 01, 2013 11:08 am

Then why is windbg and app verifier pulling heap leaks from the browser process??

I should have been clearer. I can reproduce the heap leaks detected with app verifier with the native client also. Same stack trace.

No problems testing Google chrome or chromium in the same fashion.
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby magreenblatt » Sun Dec 01, 2013 5:22 pm

What does "in the same fashion" mean? Are you testing a debug or release build? How are you running it? Did you try with Chromium content_shell and chrome built from source code at the same revision as CEF? Try building chrome/content_shell without aura and see if that makes a difference.

Also, are you loading the same web content in everything?
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: Heap allocations are leaked

Postby scidoner » Sun Dec 01, 2013 6:42 pm

I have tried both debug and release builds for Cef . I used a release build of chrome and chromium from the same revision. In the same fashion simply means under the app verifier with leak detection turned on.

I loaded up the Google home page in all scenarios. Sorry I don't have time to play much further and build source code :(. Otherwise I would.... :). I'm just concerned that using libcef leaks heap allocations in the host process. Try it yourself and see what you get, maybe its just my PC, never know.
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby magreenblatt » Sun Dec 01, 2013 10:23 pm

If the "leaks" only occur on shutdown then I don't consider it an issue, and it may even be intentional behavior in Chromium. If memory usage continues to grow in the browser process during usage such that you eventually reach OOM then that is potentially an issue. Are you experiencing that?
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: Heap allocations are leaked

Postby scidoner » Sun Dec 01, 2013 10:55 pm

magreenblatt wrote:If the "leaks" only occur on shutdown then I don't consider it an issue, and it may even be intentional behavior in Chromium. If memory usage continues to grow in the browser process during usage such that you eventually reach OOM then that is potentially an issue. Are you experiencing that?


I'll get a memory profiler onto it and let you know :) AFAIK, the app verifier doesn't detect heap leaks until the process is terminated (does it by hooking HeapAlloc, etc calls and then matching them with a HeapFree or equivalent call).
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm

Re: Heap allocations are leaked

Postby scidoner » Mon Dec 02, 2013 12:27 am

The only way to detect memory leaks reliably is to detect them when the process shuts down - so no, the fact that they are reared up at process shutdown does not make it a non-issue (for the host process, anyway - if the render process is terminating without freeing, I don't care - like you said, for performance). But the host process is long running, it is not allowed to do the same thing.

OK. Some simple test results: (I am using a .NET memory profiler, but it is able to profile unmanaged memory too). I am profiling only the host application, not the render processes. The profiler also performs a full generational garbage collection when taking memory snapshots - so assuming the managed side of things are set up properly (i.e. object finalizers just in case I missed a Dispose call), unmanaged resources should still be freed.

---------------------------------
Application start snapshot: 51.82MB unmanaged memory. - BASELINE

TEST 1:
After loading a webpage snapshot: 71.08MB unmanaged memory
After closing that webpage snapshot: 73.19MB unmanaged memory.

TEST 2:
After loading a webpage snapshot: 84.58MB unmanaged memory.
After closing that webpage snapshot: 84.55MB unmanaged memory.

TEST 3:
After loading a webpage snapshot: 94MB unmanaged memory.
After closing that webpage snapshot: 93.61MB unmanaged memory.

TEST 4:
After loading a webpage snapshot: 103.7MB unmanaged memory.
After closing that webpage snapshot: 103.2MB unmanaged memory.
---------------------------------

Memory usage has jumped nearly half, from opening and closing a Cef browser 4 times (I certainly call Dispose on all the CefGlue objects). My application hosts a CefBrowser inside a tab control. I am testing by opening and closing a tab. The tab when it is closed disposes of all its controls (one - the CefBrowser).

Although not all of this is accounted for by libcef, the CLR is allocating most of the memory for GC and other things (however the CLR is not leaking).

Which brings me to the next test...
Lets run the same test again but this time only showing memory usage by libcef and d3d9 itself (still running inside my application here):

---------------------------------
Application start snapshot: 50.5MB unmanaged memory. - BASELINE

TEST 1:
After loading a webpage snapshot: 69.89MB unmanaged memory (2.55MB used by d3d9 and 1.49MB used by libcef)
After closing that webpage snapshot: 71.38MB unmanaged memory (2.54MB used by d3d9 and 1.522MB used by libcef).

TEST 2:
After loading a webpage snapshot: 81.45MB unmanaged memory (4.86MB used by d3d9 and 1.56MB used by libcef).
After closing that webpage snapshot: 81.96MB unmanaged memory (4.86MB used by d3d9 and 1.39MB used by libcef).

TEST 3:
After loading a webpage snapshot: 90.71MB unmanaged memory (7.18MB used by d3d9 and 1.56MB used by libcef).
After closing that webpage snapshot: 91.39MB unmanaged memory (7.18MB used by d3d9 and 1.56MB used by libcef).

TEST 4:
After loading a webpage snapshot: 101.8MB unmanaged memory (9.5MB used by d3d9 and 1.59MB used by libcef).
After closing that webpage snapshot: 101.3MB unmanaged memory (9.49MB used by d3d9 and 1.59MB used by libcef).
---------------------------------

I would think that by destroying CefBrowser it would release all of that memory? Maybe I am wrong. The pattern looks very much like a leak to me. What happens if I open, say, 12 tabs with a CefBrowser, and then close them all down? Let's find out..

---------------------------------
Application start snapshot: 52.02MB unmanaged memory (20.3MB used by CLR). - BASELINE

TEST 1:
After loading 12 webpages (tabs) snapshot: 109.6MB unmanaged memory (24.9MB used by CLR, 9.545MB used by d3d9 and 4.95MB used by libcef)
After closing 12 webpages (tabs) snapshot: 110.7MB unmanaged memory (25MB used by CLR, 9.49MB used by d3d9 and 4.93MB used by libcef).

TEST 2:
After loading 12 webpages (tabs) snapshot: 116MB unmanaged memory (25.5MB used by CLR, 9.545MB used by d3d9 and 5.36MB used by libcef)
After closing 12 webpages (tabs) snapshot: 113MB unmanaged memory (25.4MB used by CLR, 9.49MB used by d3d9 and 5.08MB used by libcef).

TEST 3:
After loading 12 webpages (tabs) snapshot: 122MB unmanaged memory (25.5MB used by CLR, 9.545MB used by d3d9 and 5.48MB used by libcef)
After closing 12 webpages (tabs) snapshot: 115.5MB unmanaged memory (25.4MB used by CLR, 9.49MB used by d3d9 and 5.4MB used by libcef).

TEST 4:
After loading 12 webpages (tabs) snapshot: 123MB unmanaged memory (25.5MB used by CLR, 9.545MB used by d3d9 and 5.68MB used by libcef)
After closing 12 webpages (tabs) snapshot: 116MB unmanaged memory (25.5MB used by CLR, 9.49MB used by d3d9 and 5.6MB used by libcef).
---------------------------------
scidoner
Newbie
 
Posts: 7
Joined: Fri Nov 29, 2013 8:46 pm


Return to CefGlue Forum

Who is online

Users browsing this forum: No registered users and 15 guests