Addressing Crash - OnNoMemory and DoDecodeImage

[gbaltare]

Hello,

I'm diagnosing an crash issue when using CEF 3.3396.1779.g36f9eab / Chromium 67.0.3396.79 on windows 10.

I've collected a dmp of the application at the time of the exception and it appears that the CEF render process crashes due after OnNoMemory is called further on the stack after DoDecodeImage.
I am unable to reproduce this exact issue in my environment but can reproduce different out of memory crashes.
I've been testing the flag --max-old-space-size which I understand increases the amount of available JS heap size. I have observed that by specifying this flag that some crashes are alleviated (e.g. allocating many sizable arrays) but others are not (e.g. allocating very large strings).

So I am wondering if anyone knows whether specifying this flag would increase the amount of available memory for DoDecodeImage and help alleviate the crash?

If not, perhaps there is some other suggestion on how this crash can be avoided?

Crash callstack:
KERNELBASE.dll!_RaiseException@16 () Unknown
libcef.dll!base::`anonymous namespace'::OnNoMemory(unsigned int size) Line 55 C++
[External Code]
libcef.dll!discardable_memory::ClientDiscardableSharedMemoryManager::AllocateLockedDiscardableSharedMemory(unsigned int size, int id) Line 374 C++
libcef.dll!discardable_memory::ClientDiscardableSharedMemoryManager::AllocateLockedDiscardableMemory(unsigned int size) Line 219 C++
libcef.dll!cc::`anonymous namespace'::AllocateDiscardable(const SkImageInfo & info) Line 52 C++
libcef.dll!cc::SoftwareImageDecodeCacheUtils::DoDecodeImage(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, SkColorType color_type) Line 68 C++
libcef.dll!cc::SoftwareImageDecodeCache::DecodeImageIfNecessary(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, cc::SoftwareImageDecodeCacheUtils::CacheEntry * entry) Line 384 C++
libcef.dll!cc::SoftwareImageDecodeCache::GetDecodedImageForDrawInternal(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image) Line 527 C++
libcef.dll!cc::SoftwareImageDecodeCache::DecodeImageIfNecessary(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, cc::SoftwareImageDecodeCacheUtils::CacheEntry * entry) Line 452 C++
libcef.dll!cc::SoftwareImageDecodeCache::DecodeImageInTask(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, cc::SoftwareImageDecodeCache::DecodeTaskType task_type) Line 348 C++
libcef.dll!cc::`anonymous namespace'::SoftwareImageDecodeTaskImpl::RunOnWorkerThread() Line 95 C++
libcef.dll!content::CategorizedWorkerPool::RunTaskInCategoryWithLockAcquired(cc::TaskCategory category) Line 362 C++
libcef.dll!content::CategorizedWorkerPool::RunTaskWithLockAcquired(const std::vector<cc::TaskCategory,std::allocator<cc::TaskCategory> > & categories) Line 340 C++
libcef.dll!content::CategorizedWorkerPool::Run(const std::vector<cc::TaskCategory,std::allocator<cc::TaskCategory> > & categories, base::ConditionVariable * has_ready_to_run_tasks_cv) Line 232 C++
libcef.dll!content::`anonymous namespace'::CategorizedWorkerPoolThread::Run() Line 35 C++
libcef.dll!base::SimpleThread::ThreadMain() Line 69 C++
libcef.dll!base::`anonymous namespace'::ThreadFunc(void * params) Line 94 C++

Many thanks,
Grant

[magreenblatt]

Are you using a 32-bit build? If so, make sure to set /LARGEADDRESSAWARE. If that doesn’t help you should upgrade to a supported CEF version: https://bitbucket.org/chromiumembedded/ ... -supported

[ndesktop]

This is something I constantly see on both 32 and 64 bit builds (from crash dumps submitted by customers).
/LARGEADDRESSAWARE is enabled, and I saw this from 20 versions ago up to v76 (latest version currently deployed in our product).

PNG and JPEG both triggers this (I think PNG decoder fires this more than JPG one).
I have seen AllocateLockedDiscardableMemory(unsigned int size) with size from 20 MB to 800 MB.
Sadly, I do not have info about available memory (typically these are laptops or tables with 4-8 GB RAM).

I'm not suggesting a solution here, but if someone comes up with some fallback (like replacing with a placeholder small image, possibly invoking a "low memory" callback) I would be very interested.

A typical callstack:

Code: Select all

STACK_TEXT: 
WARNING: Stack unwind information not available. Following frames may be wrong.
0653f298 10a8397d e0000008 00000001 00000001 KERNELBASE+0x845d
0653f2b4 112f9681 00d68000 0653f2d8 10ad6530 libcef!base::`anonymous namespace'::OnNoMemory+0x1d
0653f2c0 10ad6530 00d68000 05b4f0e0 05b61360 libcef!base::allocator::WinCallNewHandler+0x11
0653f2d8 1392d263 00d68000 0653f2fc 123e37ed libcef!malloc+0x30
0653f2e4 123e37ed 00d68000 06840418 05b61360 libcef!operator new+0x1a
0653f2fc 123e32d9 00d68000 00000095 00000000 libcef!blink::PNGImageReader::CreateInterlaceBuffer+0x11
0653f354 12b4f7e6 630ed0c0 00000000 00000000 libcef!blink::PNGImageDecoder::RowAvailable+0x141
0653f374 0f408097 06840418 630ed0c0 00000000 libcef!`anonymous namespace'::pngRowAvailable+0x24
0653f3a8 0f407e5f 06840418 06840418 00002000 libcef!cr_png_push_process_row+0x198
0653f3c0 0f407c9b 06840418 71325eae 00002000 libcef!cr_png_process_IDAT_data+0xbf
0653f3ec 0f407584 06840418 62d2c8f0 00000000 libcef!cr_png_push_read_IDAT+0x119
0653f404 12b4fae2 06840418 62d2c8f0 71325ea6 libcef!cr_png_process_data+0x40
0653f434 12b4f9d5 0653f44c 00000036 00000000 libcef!blink::PNGImageReader::ProcessData+0x5a
0653f46c 123e2b69 6d4b0f18 00000000 05b613f8 libcef!blink::PNGImageReader::Decode+0x125
0653f49c 119d42d8 00000000 13d11abd 00000000 libcef!blink::PNGImageDecoder::Decode+0x91
0653f504 12b536f5 00000000 003e8017 62e42900 libcef!blink::ImageDecoder::DecodeFrameBufferAtIndex+0x6e
0653f58c 123e775c 00000000 0653f5cc 0653f5cb libcef!blink::ImageDecoderWrapper::Decode+0x12d
0653f6a8 119d618f 6d4b0f18 00000000 00000000 libcef!blink::ImageFrameGenerator::DecodeAndScale+0x132
0653f7a0 10c9e3ba 0653f86c 7d8e1000 000023c0 libcef!blink::DecodingImageGenerator::GetPixels+0x319
0653f800 10c9e2d3 7d8e1000 0653f86c 00000000 libcef!cc::PaintImage::DecodeFromGenerator+0xaa
0653f82c 12bb70a5 7d8e1000 0653f86c 00000000 libcef!cc::PaintImage::Decode+0x53
0653f8a0 1250503b 0653f8f0 0653fc24 069f8fe0 libcef!cc::SoftwareImageDecodeCacheUtils::DoDecodeImage+0xc5
0653fa4c 1250582e 0653fc24 069f8fe0 62eb1f90 libcef!cc::SoftwareImageDecodeCache::DecodeImageIfNecessary+0x1cb
0653fadc 1250539d 0653fb20 0653fc24 069f8fe0 libcef!cc::SoftwareImageDecodeCache::GetDecodedImageForDrawInternal+0xbe
0653fc7c 12504cec 069f8f9c 069f8fe0 62eb2ae8 libcef!cc::SoftwareImageDecodeCache::DecodeImageIfNecessary+0x52d
0653fcf4 12506682 069f8f9c 069f8fe0 00000000 libcef!cc::SoftwareImageDecodeCache::DecodeImageInTask+0x6c
0653fd58 12655e67 00000000 13d0e4a8 04d44fa0 libcef!cc::`anonymous namespace'::SoftwareImageDecodeTaskImpl::RunOnWorkerThread+0x62
0653fdc0 126558f3 00000001 04d44fa0 04ded3e0 libcef!content::CategorizedWorkerPool::RunTaskInCategoryWithLockAcquired+0x5d
0653fdd8 12655893 04ded3e0 04d44fb8 04ded388 libcef!content::CategorizedWorkerPool::RunTaskWithLockAcquired+0x39
0653fdf4 12656154 04ded3e0 04d45008 0653fe58 libcef!content::CategorizedWorkerPool::Run+0x2b
0653fe04 10aaa25a 04ded388 0000001a 00000000 libcef!content::`anonymous namespace'::CategorizedWorkerPoolThread::Run+0x14
0653fe58 10aa9445 04ded388 00000430 00000430 libcef!base::SimpleThread::ThreadMain+0x1ca
0653fe7c 7768ef3c 04d4d4c8 0653fec8 778d3618 libcef!base::`anonymous namespace'::ThreadFunc+0x95
0653fe88 778d3618 04d4d4c8 700cb131 00000000 kernel32+0x4ef3c
0653fec8 778d35eb 10aa93b0 04d4d4c8 00000000 ntdll+0x63618
0653fee0 00000000 10aa93b0 04d4d4c8 00000000 ntdll+0x635eb

FOLLOWUP_IP:
libcef!base::`anonymous namespace'::OnNoMemory+1d
10a8397d 68080000e0      push    0E0000008h


[gbaltare]

I've since enabled the LARGEADDRESSAWARE flag and a report of a crash with the following call stack. It also appears to be around running out of memory but is not around DoDecodeImage this time. Instead it seems to be around onRequestBitmap.

I'm just wondering if there are any additional thoughts give this latest crash. Especially regarding whether upgrading the CEF version being used is likely to resolve this issue.

KERNELBASE.dll!_RaiseException@16() Unknown
libcef.dll!base::`anonymous namespace'::OnNoMemory(unsigned int size) Line 55 C++
[External Code]
> libcef.dll!discardable_memory::ClientDiscardableSharedMemoryManager::AllocateLockedDiscardableSharedMemory(unsigned int size, int id) Line 374 C++
libcef.dll!discardable_memory::ClientDiscardableSharedMemoryManager::AllocateLockedDiscardableMemory(unsigned int size) Line 219 C++
libcef.dll!SkDiscardableMemory::Create(unsigned int bytes) Line 40 C++
libcef.dll!SkMipMap::Build(const SkPixmap & src, SkDestinationSurfaceColorMode colorMode, SkDiscardableMemory *(*)(unsigned int) fact) Line 592 C++
libcef.dll!SkMipMap::Build(const SkBitmap & src, SkDestinationSurfaceColorMode colorMode, SkDiscardableMemory *(*)(unsigned int) fact) Line 787 C++
libcef.dll!SkMipMapCache::AddAndRef(const SkBitmap & src, SkDestinationSurfaceColorMode colorMode, SkResourceCache * localCache) Line 411 C++
libcef.dll!SkDefaultBitmapControllerState::processMediumRequest(const SkBitmapProvider & provider) Line 106 C++
libcef.dll!SkDefaultBitmapControllerState::SkDefaultBitmapControllerState(const SkBitmapProvider & provider, const SkMatrix & inv, SkFilterQuality qual) Line 138 C++
[Inline Frame] libcef.dll!SkInPlaceNewCheck(void * storage, unsigned int size, const SkBitmapProvider & a1, const SkMatrix & a2, const SkFilterQuality &) Line 434 C++
libcef.dll!SkDefaultBitmapController::onRequestBitmap(const SkBitmapProvider & bm, const SkMatrix & inverse, SkFilterQuality quality, void * storage, unsigned int size) Line 153 C++
libcef.dll!SkBitmapController::requestBitmap(const SkBitmapProvider & provider, const SkMatrix & inv, SkFilterQuality quality, void * storage, unsigned int storageSize) Line 22 C++
libcef.dll!SkBitmapProcInfo::init(const SkMatrix & inv, const SkPaint & paint) Line 89 C++
[Inline Frame] libcef.dll!SkBitmapProcState::setup(const SkMatrix & inv, const SkPaint &) Line 61 C++
libcef.dll!SkBitmapProcLegacyShader::MakeContext(const SkShaderBase & shader, SkShader::TileMode tmx, SkShader::TileMode tmy, const SkBitmapProvider & provider, const SkShaderBase::ContextRec & rec, SkArenaAlloc * alloc) Line 107 C++
libcef.dll!SkImageShader::onMakeContext(const SkShaderBase::ContextRec & rec, SkArenaAlloc * alloc) Line 119 C++
libcef.dll!SkShaderBase::makeContext(const SkShaderBase::ContextRec & rec, SkArenaAlloc * alloc) Line 113 C++
libcef.dll!SkBlitter::Choose(const SkPixmap & device, const SkMatrix & matrix, const SkPaint & origPaint, SkArenaAlloc * alloc, bool drawCoverage) Line 1041 C++
[Inline Frame] libcef.dll!SkAutoBlitterChoose::SkAutoBlitterChoose(const SkPixmap & dst, const SkMatrix & matrix, const SkPaint & paint, bool drawCoverage) Line 25 C++
libcef.dll!SkDraw::drawPaint(const SkPaint & paint) Line 223 C++
libcef.dll!SkBitmapDevice::drawPaint(const SkPaint & paint) Line 298 C++
libcef.dll!SkCanvas::internalDrawPaint(const SkPaint & paint) Line 1894 C++
libcef.dll!SkCanvas::drawPaint(const SkPaint & paint) Line 1631 C++
libcef.dll!SkPixmap::scalePixels(const SkPixmap & actualDst, SkFilterQuality quality) Line 323 C++
libcef.dll!cc::SoftwareImageDecodeCacheUtils::GenerateCacheEntryFromCandidate(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::DecodedDrawImage & candidate_image, bool needs_extract_subset, SkColorType color_type) Line 139 C++
libcef.dll!cc::SoftwareImageDecodeCache::DecodeImageIfNecessary(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, cc::SoftwareImageDecodeCacheUtils::CacheEntry * entry) Line 460 C++
libcef.dll!cc::SoftwareImageDecodeCache::DecodeImageInTask(const cc::SoftwareImageDecodeCacheUtils::CacheKey & key, const cc::PaintImage & paint_image, cc::SoftwareImageDecodeCache::DecodeTaskType task_type) Line 348 C++
libcef.dll!cc::`anonymous namespace'::SoftwareImageDecodeTaskImpl::RunOnWorkerThread() Line 95 C++
libcef.dll!content::CategorizedWorkerPool::RunTaskInCategoryWithLockAcquired(cc::TaskCategory category) Line 362 C++
libcef.dll!content::CategorizedWorkerPool::RunTaskWithLockAcquired(const std::vector<cc::TaskCategory,std::allocator<cc::TaskCategory> > & categories) Line 340 C++
libcef.dll!content::CategorizedWorkerPool::Run(const std::vector<cc::TaskCategory,std::allocator<cc::TaskCategory> > & categories, base::ConditionVariable * has_ready_to_run_tasks_cv) Line 232 C++
libcef.dll!content::`anonymous namespace'::CategorizedWorkerPoolThread::Run() Line 35 C++
libcef.dll!base::SimpleThread::ThreadMain() Line 69 C++
libcef.dll!base::`anonymous namespace'::ThreadFunc(void * params) Line 94 C++
[External Code]

Thank you,
Grant

[HarmlessDave]

I have seen JavaScript code trigger out of memory crashes when the renderer is trying to repaint the browser window.

I've also seen crashes even with /LARGEADDRESSAWARE when zooming in on very high resolution images like a 13K x 10K JPG. Chromium needs to decode bitmaps as uncompressed 32-bits per pixel so that JPEG requires at least one 546 MB buffer if it can't be decoded at reduced scale. Before Chromium 65 the image was always decoded at full resolution even if the display size was set to 100x100 pixels. ( https://bugs.chromium.org/p/chromium/is ... ?id=558070 )

So that's two things you might investigate:
- What script code was running, and does removing it (for testing purposes) get rid of the crash?
- What images were being loaded?