Ninja crashes on full build

[Phylanx]

Hi!

I made changes on the 2704 branch and wanted to do a recompile of the whole directory.
For this I delete the out folder in chromium/src directory and exeucte my compile batch (which I have attached).
The batch does the following:
It sets some environment variables, writes all output into a build logfile (with redirected syserr) and finally executes a statement like this:

Code: Select all

call python D:\Git\cef\automate\automate-git.py --download-dir=D:\Git\cef\2704 --branch=2704 --depot-tools-dir=D:\Git\cef\2704\depot_tools --force-build --force-distrib >> "CEFBuild_2704_20171211_08133864.log" 2>&1


Everything worked fine until last week.
After hours of compiling I got a MessageBox "ninja.exe stopped working" which meant it had an access violation somewhere in its binary.
The build failed and the binaries weren't created.
Viewing the logfile, I saw that this happened on the end of the build (logfile size ca 800 MB which was a normal full build logfile size), so I restarted the compile, 10 minutes later it finished and the binaries were created.
But I don't trusted them, so I restarted a build over the weekend. Then ninja.exe again crashed, this time somewhere in the middle (400MB of logfile size).
I didn't find a hs_err*.log or something like that, can I find it somewhere else to attach it here?
What I did find was an EventLog entry for ninja.exe, did zip and attach it.

I also checked the ninja.exe in depot_tools and saw that it was modified on 16th of Nov 2017. Was a new ninja.exe published that results in this problems?
Can I somehow sync back to the old (for me working) ninja.exe?


//EDIT:
Here is a snippet of the last lines of the crashed compile (compressed whole logfile is 23MB big) :

Code: Select all

[15421/15955] CXX obj\third_party\WebKit\Source\modules\mediasource\modules.URLMediaSource.obj
FAILED: obj/third_party/WebKit/Source/modules/mediasource/modules.URLMediaSource.obj
ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\WebKit\Source\modules\mediasource\modules.URLMediaSource.obj.rsp /c ..\..\third_party\WebKit\Source\modules\mediasource\URLMediaSource.cpp /Foobj\third_party\WebKit\Source\modules\mediasource\modules.URLMediaSource.obj /Fdobj\third_party\WebKit\Source\modules\modules.cc.pdb
[15422/15955] CXX obj\third_party\WebKit\Source\core\css\cssom\webcore_remaining.PositionValue.obj
FAILED: obj/third_party/WebKit/Source/core/css/cssom/webcore_remaining.PositionValue.obj
ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\WebKit\Source\core\css\cssom\webcore_remaining.PositionValue.obj.rsp /c ..\..\third_party\WebKit\Source\core\css\cssom\PositionValue.cpp /Foobj\third_party\WebKit\Source\core\css\cssom\webcore_remaining.PositionValue.obj /Fdobj\third_party\WebKit\Source\core\webcore_remaining.cc.pdb
ninja: build stopped: subcommand failed.
--> Download Directory: D:\Git\cef\2704
--> Depot Tools Directory: D:\Git\cef\2704\depot_tools
--> Updating depot_tools
-------- Running "update_depot_tools.bat" in "D:\Git\cef\2704\depot_tools"...
-------- Running "D:\Git\cef\2704\depot_tools\git.bat config --get remote.origin.url" in "D:\Git\cef\2704\cef"...
--> CEF Branch: 2704
--> CEF URL: https://bitbucket.org/chromiumembedded/cef.git
--> CEF Source Directory: D:\Git\cef\2704\cef
-------- Running "D:\Git\cef\2704\depot_tools\git.bat rev-parse HEAD" in "D:\Git\cef\2704\cef"...
-------- Running "D:\Git\cef\2704\depot_tools\git.bat fetch" in "D:\Git\cef\2704\cef"...
-------- Running "D:\Git\cef\2704\depot_tools\git.bat rev-parse origin/2704" in "D:\Git\cef\2704\cef"...
--> CEF Current Checkout: ec3e9ed7fecc0d9f37a96951ba1bb26fd4f64dc7
--> CEF Desired Checkout: ec3e9ed7fecc0d9f37a96951ba1bb26fd4f64dc7 (origin/2704)
--> CEF Output Directory: D:\Git\cef\2704\out_2704
-------- Running "D:\Git\cef\2704\depot_tools\git.bat config --get remote.origin.url" in "D:\Git\cef\2704\chromium\src"...
--> Chromium URL: https://chromium.googlesource.com/chromium/src.git
-------- Running "D:\Git\cef\2704\depot_tools\git.bat rev-parse HEAD" in "D:\Git\cef\2704\chromium\src"...
-------- Running "D:\Git\cef\2704\depot_tools\git.bat rev-parse refs/tags/51.0.2704.103" in "D:\Git\cef\2704\chromium\src"...
--> Chromium Current Checkout: c4febd5ae385dee28312cfb1ce0c653b0919044a
--> Chromium Desired Checkout: c4febd5ae385dee28312cfb1ce0c653b0919044a (refs/tags/51.0.2704.103)
--> GYP_MSVS_VERSION=2015
--> GYP_GENERATORS=ninja
--> DEPOT_TOOLS_WIN_TOOLCHAIN=0
--> GYP_MSVS_OVERRIDE_PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0
-------- Running "D:\Git\cef\2704\chromium\src\cef\cef_create_projects.bat" in "D:\Git\cef\2704\chromium\src\cef"...
-------- Running "ninja -C out\Debug cefclient" in "D:\Git\cef\2704\chromium\src"...
-------- Running "ninja -C out\Release cefclient" in "D:\Git\cef\2704\chromium\src"...
Traceback (most recent call last):
  File "D:\Git\cef\automate\automate-git.py", line 992, in <module>
    if options.buildlogfile else None)
  File "D:\Git\cef\automate\automate-git.py", line 55, in run
    shell=(sys.platform == 'win32'))
  File "D:\Git\cef\2704\depot_tools\win_tools-2_7_6_bin\python\bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', '-C', 'out\\Release', 'cefclient']' returned non-zero exit status 1
Done 11.12.2017 11:04:09,86


[Phylanx]

I had a backup of the whole depot_tools folder (modDate of ninja.exe is 13th of january 2017), so I tried to use that instead of the original:
1. delete depot_tools
2. copy depot_tools into original folder
3. use "--no-depot-tools-update" switch to prevent ninja.exe from being replaced again (was not replaced).
4. delete chromium/src/out folder
5. start compile

Result:
Ninja.exe crashed again, at about the half of the compilation (400MB logfile size)

I'll now ask a colleague that didn't compile CEF for some time, maybe she doesn't have this problem now.

[magreenblatt]

2704 branch is quite old already. You can try manually syncing depot_tools to a version that existed back in April 2016 and set the `DEPOT_TOOLS_UPDATE=0` environment variable.

[Phylanx]

I stepped back in depot_tools with following command (in depot_tools folder) :

Code: Select all

D:\Git\cef\2704\depot_tools>git checkout 28a1ff42d9477f548a37afc89813012b673b406c
Previous HEAD position was ffb3c43... Add various improvements to my_activity.py
HEAD is now at 28a1ff4... Updata ninja to v1.7.2.


Afterwards I started the build, now the yasm.exe crashes ("yasm.exe funktioniert nicht mehr") :

Code: Select all

Name der fehlerhaften Anwendung: yasm.exe, Version: 0.0.0.0, Zeitstempel: 0x4ce6c46f
Name des fehlerhaften Moduls: unknown, Version: 0.0.0.0, Zeitstempel: 0x00000000
Ausnahmecode: 0xc0000005
Fehleroffset: 0x643b83c4
ID des fehlerhaften Prozesses: 0x5f78
Startzeit der fehlerhaften Anwendung: 0x01d3733235257f50
Pfad der fehlerhaften Anwendung: D:\Git\cef\2704\chromium\src\third_party\yasm\binaries\win\yasm.exe
Pfad des fehlerhaften Moduls: unknown
Berichtskennung: 33a1a0dd-0529-4b48-b0b0-d40fb34c08ce
Vollständiger Name des fehlerhaften Pakets:
Anwendungs-ID, die relativ zum fehlerhaften Paket ist:


I searched and found compile errors on different places (non reproducable on that specific places). see:

Code: Select all

Hinweis: Einlesen der Datei: d:\git\cef\2704\chromium\src\third_party\icu\source\common\ustr_imp.h
[2413/15955] CXX obj\third_party\icu\source\common\icuuc.ucnvbocu.obj
FAILED: obj/third_party/icu/source/common/icuuc.ucnvbocu.obj
ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\icu\source\common\icuuc.ucnvbocu.obj.rsp /c ..\..\third_party\icu\source\common\ucnvbocu.cpp /Foobj\third_party\icu\source\common\icuuc.ucnvbocu.obj /Fdobj\third_party\icu\icuuc.cc.pdb
[2414/15955] CXX obj\third_party\icu\source\common\icuuc.unistr_case.obj
Hinweis: Einlesen der Datei: d:\git\cef\2704\chromium\src\third_party\icu\source\common\unicode/utypes.h


Code: Select all

Hinweis: Einlesen der Datei:  D:\Git\cef\2704\chromium\src\mojo/public/cpp/bindings/strong_binding.h
Hinweis: Einlesen der Datei: D:\Git\cef\2704\chromium\src\components/profile_service/user_id_map.h
[13391/15955] LIB obj\components\leveldb\leveldb_lib.lib
[13392/15955] LIB obj\components\profile_service\profile_service_lib.lib
[13393/15955] CXX obj\third_party\skia\src\core\skia_library.SkUnPreMultiply.obj
FAILED: obj/third_party/skia/src/core/skia_library.SkUnPreMultiply.obj
ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\skia\src\core\skia_library.SkUnPreMultiply.obj.rsp /c ..\..\third_party\skia\src\core\SkUnPreMultiply.cpp /Foobj\third_party\skia\src\core\skia_library.SkUnPreMultiply.obj /Fdobj\skia\skia_library.cc.pdb
ninja: build stopped: subcommand failed.


Code: Select all

Hinweis: Einlesen der Datei: d:\git\cef\2704\chromium\src\third_party\icu\source\common\uassert.h
Hinweis: Einlesen der Datei:  C:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt\assert.h
[2401/15955] RULE Building jcgrass2.obj
FAILED: gen/third_party/libjpeg_turbo/jcgrass2.obj
D:\Git\cef\2704\depot_tools\python276_bin\python.exe gyp-win-tool action-wrapper environment.x86 libjpeg_target_assemble_ef6bd96b250026f7f1223e39360a997e.18441dbd79e80e513eb5373cc1808998.rsp ..\..\third_party\libjpeg_turbo
[2402/15955] CXX obj\third_party\icu\source\common\icuuc.resbund_cnv.obj
Hinweis: Einlesen der Datei: d:\git\cef\2704\chromium\src\third_party\icu\source\common\unicode/utypes.h


Any suggestions what this problem could be?

[magreenblatt]

Do you have the most recent VS2015 update installed?

[Phylanx]

Hi!
For version 2704 we need a VS 2015u2 installed.

But I think I found a solution, my over night build finished without problems yesterday.
My only changes between working/not working:

Tell TrendMicro OfficeScan that these specific "ninja.exe" and "yaml.exe" are trusted programs and to add them in the exclusion list for behaviour monitoring.
I'll try a build again and if it again works, I guess the AV was the problem.
I'll let you know the results.
Thanks so far and sorry for the noise.

[Phylanx]

It was the anti virus, hope this helps other people with similar problems.