Separate Preprocess and Compile Performance

Unlike most other build systems, build2 performs an explicit header dependency extraction step (-M, /showIncludes, etc). The conventional approach is to perform this as part of the compilation itself. It is a clever trick but unfortunately it doesn't work for auto-generated headers. And in build2 we do support auto-generated headers.

One notable aspect of the dependency extraction step is that it is essentially a full preprocessor run. In fact, for /showIncludes, one has to either preprocess or compile so if you only need the header dependencies, you have to send the preprocessed output to /dev/null. Which is a waste, of course: why not save the preprocessed output and then compile that?

In fact, if we can pull this off, it opens some intriguing possibilities: For starters, we can ignore comment-only changes by hashing the preprocessed output. More importantly, we will be all set for distributed compilation and caching since we now have a self-contained translation unit that we can ship to a remote host. Finally, the build system can analyze the preprocessed output, for example, to extract C++ module dependencies, which, in many ways, are not unlike header dependencies, just being on the language rather than preprocessor level.

Note that the separate preprocess and compile setup is not without challenges. For details see these GCC mailing list, Clang mailing list, and /r/cpp discussions. But I believe we managed to pull it off in build2 with the separate preprocess and compile mode now being the default for the big three. GCC and Clang are pretty solid with VC and its broken preprocessor being the iffy one, so we will have to wait and see. So far we haven't seen any issues on our builds (which means standard libraries and system headers are all good). Plus, this can be disabled for specific translation units, project directories, and entire projects.

Ok, finally we are getting to the interesting part: the performance. While in build2 dependency extraction during compilation is not an option, it would be interesting to see how it compares. So what we did is one better: we disabled the dependency extraction altogether. So the first column is the time it takes for just compiling which is our base or 100%. We expect it to be the fastest.

We would expect the new mode, which compiles the preprocessed output (saved as the side effect of dependency extraction), to be the second fastest. This is our second column.

The old mode, which extracts dependencies without producing the preprocessed output and then compiles the original source, should be the slowest. This is our third column. Note that we don't have a measurement for VC since there is no way not to produce preprocessed output in this mode.

Finally, we can add a column for the "reprocess" mode: save the preprocessed output but force the use of the original source during compilation (this uses the disabling mechanism mentioned above). Compared to column two this will give us the cost of preprocessing our sources.

The test was done on a Linux machine with 4-core/8-thread i7-6820HQ 2.70GHz CPU, 64GB of RAM, and a Samsung 950 PRO NVMe SSD. It involved rebuilding from scratch the build2 build system and package manager without optimization. The numbers (time taken) are only comparable for the same compiler. Less is better.

compiler       src      dep+pre    dep+src    dep+pre+src
---------------------------------------------------------
GCC   5.4      100%      99.2%     104.8%     104.0%
Clang 3.8      100%     107.1%     107.9%     112.1%
VC    15u0     100%     122.3%       --       128.5%

A couple of observations: If we have to have a separate dependency extraction step then compiling the preprocessed output is faster than compiling the source. The cost of preprocessing the source is about 5% of the (non-optimized) build time for all three compilers.

Also, keep in mind that the first column is not the traditional way of extracting dependencies – it does not extract them at all, does not get their modification times, and so on. The "true" traditional time will be somewhere between the first and the second columns.