Resolve issue #798 - investigate Qt build performance

[ghost]

Fixing issue #798

Summary:
Qt build investigated, negligible performance impact gained from excluding non-critical code

Process:
Qt is built via a SConscript kicking off the build to patch and then "configure" the Qt build with a bunch of specific parameters, followed by qmake being run on pro files to generate makefiles. Some of these Makefiles are then executed, depending on the configuration options as well as the SUBDIRs listed in the pro files.

Code:
A quick search of the code for Qt components yields a wide range of them being used in the GEE build, but also large sections that upon deeper inspection that may not be utilized at all - such as SQL, Network, or Embedded. However, removing one or more of these components from the Qt build results in a failure to build Qt Designer in the tools section. Qt Designer is needed for its UIC or user-interface compiler. Thusly, we really can't cut out any large Qt components and still complete the GEE build.

This means that all we can really remove from the build are the examples/tutorials/demos/docs components, as mentioned in the issue.

Build:
The current Qt build process finds 241 pro files in the Qt directory, displays a message stating that it's creating Makefiles, and then proceeds to generate a makefile for each of the pro files.
This can be a little confusing though, because despite a Makefile being created for things like the example projects, whether or not the project is actually built is determined by other parts of the Qt build hierarchy. These parts are currently disabled in the GEE build process, meaning that we do not actually build the examples.
It is possible to skip the generation of some of these Makefiles with a paramater in the configuration process with "-fast"
By adding "-fast" qt will only generate makefiles for library and subdirectory targets (defined by the SUBDIRS, as mentioned above).
This means that we can now cut down the makefile generation, and eliminate some confusing output in the build process. This is not a silver bullet though, as it still generates makefiles as "wrappers, which will in turn run qmake" and the files still appear in the build just with "(fast)" after them.
This brings the pro files found (and subsequent makefiles generated) from 241 to 84

Further options can be injected into the configuration to improve the build:
"-nomake examples -nomake demos -nomake tests -nomake docs"
Will prevent a few extra makefiles from being made, bringing the count down from 84 to 80 project files. This injection can happen in the custom SConscript located at /src/third_party/qt/SConscript, preventing the need to mess with the qt source tarball.

Another option is the outright removal of the directories in the Qt tarball - examples, tutorials, doc
This requires a change in the Makefile, removing these directories and subsequent make calls as targets which requires some cascading changes (or phony targets in the existing Makefiles), and the overall time-save for this added complexity (and changing of source tarball) is not even 1s compared to the above nomake options.

Conclusion:
In total, adding these nomake config parameters resulted in roughly a 0.2% increase in build performance (5 seconds saved on a 54 minute, single core build). Though this is an increase, some users have noted that the "-nomake ..." parameters only work on X11 environments. A 0.2% speedup may not be worth the potential issue later on, and since the actual examples/tutorials themselves are still skipped, even if their pro files aren't. I have added the change to the SConscript for further testing and evaluation purposes, but this may end up being an issue closed with no action.

time scons -j1 release=1 build 2>&1 | tee build.log

Patched:
real 54m52.113s
user 49m0.080s
sys 6m37.518s

Master:
real 54m57.707s
user 49m6.637s
sys 6m39.950s

[tst-nfarah-zz]

Were all the .pro looked at or only the examples listed in the issue?

[ghost]

Yes, all pro files in the hierarchy and subsequent makefiles were investigated - including the ones mentioned in the issue. Furthermore, an easy determination of what actually ends up being built can be made by recursively finding all executables in the src/NATIVE-REL-x86_64/third_party/qt/qt-x11-free-3.3.8b directory with
find . -type f -perm /u=x,g=x,o=x -exec ls -l {} ;
This will further show that none of the aforementioned tutorials are compiled in either build - patched or master (except for the 3 in tools/linguist/tutorials are not excluded, even with -nomake tutorials, but their compilation time is only ~2 seconds for all three and compilation is executed via the tools/tools.pro inclusion of linguist SUBDIR in the build).

[tst-ccamp]

@tst-recruit1 Did you determine that all the .pro files that are built are required to be built, or did you determine that it's difficult to prevent them from being built?

[ghost]

I've determined that of the 241 pro files found by the build, it's possible to drop the count down to 80 with just the change to the SConscript's "configure" of the qt build that is incorporated in this commit. Of the remaining 80, there are 17 unnecessary tutorial pro files that still generate Makefiles - to clarify though, these are not built into programs aside from the exception mentioned above. Adding the "-fast" command at the beginning of this commit's change to the configure params makes it so that these 17 pro files only generate wrappers instead - defined by https://doc.qt.io/archives/qt-4.8/configure-options.html
Preventing the wrappers from even being created would require removal of the directories (which in turn causes the main Makefile to fail) - to fix this, it's possible to instead change the Makefile inside of the tutorial directory to a phony Makefile and empty out the tutorials directory. This allows the build to complete, but requires either modification of the source tarball, or a patch. Creation of a makefile, such as for tutorial/t1, takes
"real 0m0.030s" or 0m0.5s for all 17. Creation of the wrappers (-fast) is even faster.

Preventing the aforementioned 3 tutorials from compiling inside of tools/linguist/tutorials that amount to a 2 second speedup would require patching /tools/linguist/linguist.pro to not include the tutorials.

All three avenues are possible speedups. The committed one amounts to a ~6 second gain. If desired, I can do the patches, but it will be a few patch files for a <0.5s speedup, and another patchfile for a ~2 second speedup.

All numbers are relative to the 54 minute build - across 1 core. Speedup diminishes linearly with job count (-j 8 for instance would mean the 8.5s possible speedup becomes a 1 second speedup).

[tst-nfarah-zz]

@tst-recruit1 let's have this PR include removal of all .pro that were deemed not required.

[tst-ccamp]

Ah OK. I reran the build and did see that the pro files finished compiling rather quickly. The reason this ticket exists is that in the past these files would take multiple seconds per file to build. Perhaps this was initially caused by a bug in the compiler or maybe a later optimization. Oh well, it doesn't look like it's a problem anymore. Thank you for running this to ground.

[tst-ccamp]

Looks good to me

[ghost]

Thanks, that's good to hear that it's approved! I had fun with it, and got a great handle on the build process while working on it. Looking forward to more!

[tst-dsugar]

This makes sense to do, as they are not needed to build fusion.