Bjarne Stroustrup made the impression of a down-to-earth guy with a very healthy attitude about C++ features and their development. I was excited to learn that the C++0x draft standard had just been declared final the week before. That means the content of the standard is de-facto finished now, and it will be formalized as an ISO standard in about one year. There sure are quite a number of nice goodies in C++0x, I can’t wait to get my hands dirty on code using it.

For the C++ groupies among us, I took photographs of all presentation slides and of course also of Bjarne Stroustrup himself.

But today I happened onto another one of those little inanities which so often add up to a huge stinking pile. I decided to share this particular example, as I think it may help people to understand why I’m feeling so strongly about Qt.

A while ago, I filed Qt bug #5732, together with a patch to fix the problem. In addition to the fix for the central problem reported in the bug, I also provided a second patch to correct another problem I stumbled upon while I worked on the actual bug fix. The bug was acknowledged by a Qt developer and the code got fixed. They decided to fix it their own way instead of applying my patch, which is fair enough. I may not have agreed with the way it was implemented, but the problem I reported was fixed.

Today, I again looked at the Qt OpenGL code, and happened to stumble upon the OpenGL extension checks. Apparently, the extension checking had now finally been factored out into a separate class, with a more efficient implementation to avoid the numerous temporary memory allocations of the string splitting method that was used before. Of course I was curious and had a look at the code which implements the string parsing.

Here is how it looks like:

According to the log message the piece of code just shown went through internal code review. For comparison, here is the equivalent piece of code from my original patch, which was not considered good enough to be applied:

So, instead of just taking the code handed to them on a silver platter, the Qt developers decided to roll their own version. Which I wouldn’t even consider a problem if it weren’t for the fact that the code that went in is longer, less readable, and generally ugly. It’s also broken, because there is not a single word in the OpenGL specification about glGetString() always returning a list terminated by a space character, contrary to what the code comment claims about “all conformant platforms”.

Until recently, I used to have a long list of assignments to module_autogenargs['...'] in my ~/.jhbuildrc file to set up default configure options for various modules. This became unwieldy and there was always one module I forgot.

Autoconf comes to the rescue with its support for site defaults. For illustration, here is my ${prefix}/etc/config.site file:

The effect is as if I had passed --enable-silent-rules --disable-static --enable-warnings=fatal to every module’s ./autogen.sh, except that I don’t get any annoying warnings if a module doesn’t actually support a particular option. The test conditionals ensure that command-line arguments have precedence over the default values in the config.site file.

Read the manual for the gory details, if you want to. Otherwise, enjoy!

All this would not have been possible without the help of Frédéric Péters, who did the actual integration work in the library-web module and has been very responsive to our numerous questions and tweak requests.

Leading up to the migration, the build infrastructure of the GNOME C++ bindings was completely restructured. All the copy’n'paste of build files had degenerated into an unmaintainable mess. When it became clear that we lacked the infrastructure to properly support cross-references from one documentation module to another, I began to cook up something new to replace the mess. The result of this work is the new mm-common module, which provides the shared build support files for the C++ binding modules. As a novelty, it even comes with documentation!

It is quite a bit of work to convert an existing module to use mm-common, and the transition took more time than I had expected. Despite that, the feedback was very positive, and a number of module maintainers started to convert their modules to mm-common on their own. Many thanks to David King, Fabien Parent, Jonathon Jongsma, José Alburquerque, Krzesimir Nowak and Murray Cumming for your work and feedback!

For the background, I highly recommend to read the original paper by Peter Miller, Recursive Make Considered Harmful. Here’s an appetizer:

For large UNIX projects, the traditional method of building the project is to use recursive make. On some projects, this results in build times which are unacceptably large, when all you want to do is change one file. In examining the source of the overly long build times, it became evident that a number of apparently unrelated problems combine to produce the delay, but on analysis all have the same root cause.

First off, here are the results for the Glom build, measured before and after the switchover:

These results were obtained on a Lenovo desktop PC with an Intel Core 2 Duo E7200 at 2.53 GHz and 3 GiB RAM. The operating system is Ubuntu 9.04 (jaunty) for AMD64, installed on an ext4 volume of an encrypted LVM partition. At the time of the measurements, the full GNOME desktop interface was up and running, but otherwise idle. The entire measurement was repeated once with nearly identical results, and the timings presented here are the averages of the two runs.

As can be seen, the switch to non-recursive make reduced the time of a full Glom build by about 19 % on this machine, or more than one minute, which I think is quite decent. However, before you start to convert your own module, be aware that the performance gain you’ll see depends heavily on the specifics of your project. It’s practically guaranteed that you’ll see some improvement, but it could be a meager 0.1 % speedup. However, 40 % are just as possible. To help people gauge the possible effect on their project, I’ll try to outline the factors which I found to matter most in my experience.

Essentially, performance is improved because of two things: First, the time for make to get going and resolve dependencies includes a constant part, which introduces a significant delay even if there is nothing to rebuild. The second problem is that even with parallel make, subdirectories are normally entered in sequence, in order to prevent the number of parallel processes from increasing exponentially. A subdirectory cannot be left until all its targets have finished building. Unless the execution times of all parallel build processes overlap perfectly, one or more processor cores will idle. This effect will be more pronounced if the average number of source files per subdirectory is small. In the pathological case of one file per directory, there will be no parallelism at all. A classic example is the directory hierarchy of source code examples found in many library modules.

The programming language used also plays a role. On average, the idle periods due to imperfect overlap of parallel compilation processes will be longer if compilation is slow. Typically, a C++ source file will require substantially more time to compile than a C source file. The same holds for large source files versus small source files. The optimization level and other compiler settings also influence the compile time per object. With non-recursive make, the stalls due to imperfect overlap will only occur for targets which depend on multiple intermediate targets, such as when linking object files into a library. Even when make needs to wait for a dependency before it can continue with a target, it will still continue building any independent targets. Thus, the builds of multiple independent executables or libraries overlap, thereby hiding the stalls.

With all that mind, I think the 19 % speedup in the case of Glom should be considered above average. The directory hierarchy is deeply nested, with a small number of source files per subdirectory, although not as small as to be pathological. One important factor is that Glom used to build an intermediate static library in each subdirectory, which were later linked to produce the actual target binary. The new non-recursive build system omits these intermediate libraries and simply links all object files from all subdirectories in one go to produce the final target binary.

Automake has included support for non-recursive make for quite a while now, and it turns out to be surprisingly straightforward and easy to use. All you need to get going is AM_PROG_CC_C_O in your configure.ac for C compiler support, and AUTOMAKE_OPTIONS = subdir-objects in your Makefile.am. To refer to files in a subdirectory, just include the relative directory prefix. Automake variable names also include the directory prefix of the file names they are derived from. The *.am files in the Glom repository may be helpful as real-world examples.

Happy make -j∞ !