Linux/CEF1: tcmalloc problems

[Czarek]

Hi Marshall,

We're having this strange error, it appears when we change the order of imports (it's a python application),
when CEF library is imported before the wxWidgets library the app works fine, no errors. But when the
wxWidgets library is imported before CEF then we get errors like this:

Code: Select all

./third_party/tcmalloc/chromium/src/tcmalloc.cc:286] Attempt to free invalid pointer 0xf21c698
Segmentation fault


The stack trace in GDB:

Code: Select all

Program received signal SIGSEGV, Segmentation fault.
0xb347988b in tcmalloc::Abort() () from /usr/lib/python2.7/dist-packages/cefpython1/libcef.so
(gdb) bt
#0  0xb347988b in tcmalloc::Abort() () from /usr/lib/python2.7/dist-packages/cefpython1/libcef.so
#1  0xb3487cf7 in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) () from /usr/lib/python2.7/dist-packages/cefpython1/libcef.so
#2  0x0add71a0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)


We have removed any calls to CEF in the application (no CefInitialize, no browser), we're only making a python
import that loads the library using: "ctypes.CDLL(libcef_so, ctypes.RTLD_GLOBAL)" - an equivalent of dlopen()
with the RTLD_GLOBAL mode. There is only one line of CEF C++ code running, a global variable holding a CefRefPtr
to the CefClient instantiated class. The error appears later when there is no CEF code being called, importing CEF
works fine, the error appears X seconds later for no sensible reason.

I have spent some time debugging our problem today with pdb. I have located a section of code dealing
with initialization of component displaying animated gif that was causing the seg fault. Commenting it allowed
the program to run further but it eventually seg faulting on initialization of wx glcanvas (which we use to display
our OpenGL based network view). The last failing statement is this one:

Code: Select all

> /lib/python2.7/site-packages/wx-2.8-gtk2-unicode/wx/glcanvas.py(136)GLCanvasWithContext()
-> val = _glcanvas.new_GLCanvasWithContext(*args, **kwargs)
--Call--
> /lib/python2.7/encodings/utf_8.py(15)decode()
-> def decode(input, errors='strict'):
> /lib/python2.7/encodings/utf_8.py(16)decode()
-> return codecs.utf_8_decode(input, errors, True)
--Return--
> /lib/python2.7/encodings/utf_8.py(16)decode()->(u'MtnmViewer', 10)
-> return codecs.utf_8_decode(input, errors, True)
./third_party/tcmalloc/chromium/src/tcmalloc.cc:286] Attempt to free invalid pointer 0xf21c698
Segmentation fault


So it is somewhat bizarre. Why the chromium/src/tcmalloc.cc gets involved? Is it possible that somehow rather
than whatever standard tcmalloc is, we have replaced it with chromium's version by virtue of importing it in
the particular order ? Or perhaps there is some memory/stack corruption and this chromium related message is a
total fluke?

Using CEF 1 branch 1453 revision 1268, Chrome 27.0.1453.93 on Ubuntu 12.04 LTS 32-bit.
We will be updating to CEF 3 soon.

Best regards,
Czarek

[magreenblatt]

Why the chromium/src/tcmalloc.cc gets involved? Is it possible that somehow rather
than whatever standard tcmalloc is, we have replaced it with chromium's version by virtue of importing it in
the particular order ? Or perhaps there is some memory/stack corruption and this chromium related message is a
total fluke?

The tcmalloc functions are a likely place for crashes in the case of memory corruption. Initialization of tcmalloc on Windows installs hooks that affect the whole process. It's very possible that the same is occurring on Linux.

[Czarek]

I'm not yet sure about the conclusions, does that mean that the CEF tcmalloc is better at detecting memory corruptions than
the implementation that comes with wx/Gtk? Or does the tcmalloc hook break things up when used with wx/gtk?

[magreenblatt]

I'm not yet sure about the conclusions, does that mean that the CEF tcmalloc is better at detecting memory corruptions than
the implementation that comes with wx/Gtk? Or does the tcmalloc hook break things up when used with wx/gtk?

Does wx install its own memory allocator? You can try building CEF/Chromium without tcmalloc to see if it makes a difference.

[Czarek]

Btw. this application runs also on Windows and there are no errors, they appear only on Linux,
on both systems the wx library is used.

[Czarek]

It seems to me that if you try to free the memory that was allocated using normal alloc and then
try to free it after the tcmalloc hooking has taken place, it's gonna end bad.

From the tcmalloc documentation (Caveats section at the bottom):

http://goog-perftools.sourceforge.net/doc/tcmalloc.html

Don't try to load TCMalloc into a running binary (e.g., using JNI in Java programs). The binary will have allocated some objects using the system malloc, and may try to pass them to TCMalloc for deallocation. TCMalloc will not be able to handle such objects.

We've been importing wx library before the CEF library, so the memory allocation already took place
in the wx library using normal alloc, then the CEF tcmalloc hooks up, causing errors when trying to
free memory allocated using normal malloc.

But why does it work on Windows without problems? The problems seem to be only on Linux. Is it possible
that tcmalloc is used only on Linux? You said that it installs hooks that affects the whole process on Windows,
but are you sure of that? I've googled and found this page:

http://www.chromium.org/developers/test ... for-chrome

I see that it mentions "chrome.exe" - is it possible that tcmalloc is used only in the chrome.exe launcher
executable?

By default, chrome.exe now uses tcmalloc to allocate pages.

Also in the tcmalloc documentation I see that only Linux is officially supported:

For some systems, TCMalloc may not work correctly on with applications that aren't linked against libpthread.so (or the equivalent on your OS). It should work on Linux using glibc 2.3, but other OS/libc combinations have not been tested.

So it seems that integrating CEF into application requires the CEF library to be loaded the very first,
at least on Linux.

CEF is a library, one of many libraries that application uses, it shouldn't make such critical decisions as
changing the memory allocation implementation that affects the whole application - without asking?

[magreenblatt]

Your conclusions sound correct. To copy/paste from a recent chromium-dev post:

On windows, we don't compile against tcmalloc directly. Instead we compile against an allocator target which picks the allocator to use at runtime based on an environment variable very very early in process startup:
https://code.google.com/p/chromium/code ... e=cs&l=271

This has several implications, but one of which is you can't link directly against tcmalloc on windows like you can on linux. Instead we have this funky base/allocator/allocator_extension(_thunks).h system by which you can register a function pointer to call into allocator-specific logic from allocator-agnostic library code.

[Czarek]

How do I disable the tcmalloc / build CEF without tcmalloc on Linux? For Windows I see that I would have
to set the CHROME_ALLOCATOR environment variable to "winheap", but what are the instructions for Linux,
is it even possible? I see only four options in the allocator_shim.cc file:

Code: Select all

TCMALLOC,    // TCMalloc is the default allocator.
JEMALLOC,    // JEMalloc.
WINHEAP,  // Windows Heap (standard Windows allocator).
WINLFH


Is it a good idea to disable it on Linux? Is performance going to suffer much because of it? Or should users
be informed that the CEF library must loaded the very first? But still that doesn't sound right to me, it's a
third party library, it shouldn't do such thing by default.

[magreenblatt]

How do I disable the tcmalloc / build CEF without tcmalloc on Linux?

Add linux_use_tcmalloc=0 in your ~/.gyp/include.gypi file. See here for a general example of the technique.

Is it a good idea to disable it on Linux? Is performance going to suffer much because of it? Or should users
be informed that the CEF library must loaded the very first?

I don't currently have an answer to these questions. Perhaps you can tell us after you're done experimenting .

[magreenblatt]

Is it a good idea to disable it on Linux? Is performance going to suffer much because of it? Or should users
be informed that the CEF library must loaded the very first?

I don't currently have an answer to these questions. Perhaps you can tell us after you're done experimenting .

How did this experiment turn out for you? I'm thinking of disabling tcmalloc by default in 2171+ to avoid problems like https://code.google.com/p/javachromiume ... ail?id=137.