build2 0.3.0 Release Notes

These release notes provide a more detailed discussion of major new features in the build2 toolchain version 0.3.0, including motivation for implementing them and their usage examples. For the complete list of changes, see the Release Announcement or the NEWS files in the individual packages.

Build System

The build system now supports what we call High Fidelity Builds (HFB). Traditionally, changes to the overall build environment such as compile options or the compiler itself have to be tracked and handled manually, usually with a clean followed by an update. This is both burdensome and error-prone. In build2 the C++ compile and link rules now detect when the compiler, options, or input file set have changed and trigger the update of the affected targets.

This is implemented by storing corresponding values (or their checksums) in the auxiliary dependency (.d) files. These are the same files that store the extracted header dependencies (and that can be found in other build systems). In build2 they have been extended to store other kinds of out-of-band (i.e., those not explicitly captured in the buildfile) dependencies.

The C++ compiler changes are detected by storing a compiler sha256 checksum. This checksum is computed in a compiler type-specific way, normally the output of the -v/--version option, and will at least detect a compiler type change (e.g., from g++ to clang++) and version change (e.g., a compiler upgrade). For example, for GCC the checksum is based on the -v output which includes the compiler name, version, and its configure command line. So even if the exact same version of GCC gets rebuilt with a different configuration, build2 will detect this.

The compile/link options are detected by storing their checksum. Thus if we change -O2 to -O3 in build/config.build or add -I in a buildfile, the affected object files will be automatically updated.

Similarly, build2 stores the input file for the C++ compilation and the checksum of the input files for linking. Thus it will detect such subtle changes as replacing foo.cpp with foo.cxx (note that the object file name stays the same) or removing a no longer needed source file from the project.

One interesting application of the HFB support is the automatic removal of the -rpath directories from libraries and executables that are about to be installed. In build2 the install operation triggers the update pre-operation to make sure the target is up-to-date. The link rule detect this update for install condition and omits -rpath options from the command line. This in turn triggers the option checksum mismatch (provided the target wasn't previously updated for install) and the target is re-linked without rpaths.

Note also that while we only talked about the C++ compile/link rules, the same applies to the ar/ranlib-based static linking. So if you change ar to llvm-ar, build2 will detect this.

Another major build system improvement is the expanded command line variable override support. While we normally use persistent configurations (so that we don't have to repeat the configuration on every invocation), we often need to temporarily override certain parts of the configuration. For example, we may want to quickly recompile our project with clang++ instead of g++ or with the debug information instead of optimization. It is also useful to be able to control the scope of such overrides. For example, we may only want to recompile tests with the debug information.

In build2 a command line variable can be an outright override (=), prefix (=+), or suffix (+=), for example:

b config.cxx=clang++ config.cxx.coptions+=-g config.cxx.poptions=+-I/tmp

Here we override the C++ compiler, append -g to the compile options and prepend -I/tmp to the preprocess options (so that /tmp is searched in before any other directories). Note that the automatic rebuild of affected targets happens because of the HFB support discussed above.

Prefix/suffix overrides are applied at the outsets of values set in buildfiles, provided these values were set (in those buildfiles) using =+/+= and not an assignment with expansion, for example, given this command line:

b x=+P x+=S

And this buildfile:

x = y
print $x # P y S

x =+ p
x += s
print $x # P p y s S

But:

x = A $x B
print $x # A P p y s S B

By default an override applies to all the projects mentioned in the buildspec as well as to their subprojects. We can restrict an override to not apply to subprojects by prefixing it with %, for example:

b %config.cxx=clang++ configure

An override can also be made global (i.e., it applies to all the projects, including the imported ones) by prefixing it with !. As an example, compare these two command lines:

b config.cxx.coptions+=-g
b '!config.cxx.coptions+=-g'

In the first case only the current project and its subprojects will be recompiled with the debug information. In the second case, everything that the current project requires (e.g., all the imported libraries) will be rebuilt with the debug information.

Finally, we can also specify the scope from which an override should apply. For example, this is how we can rebuild the tests/ sub-directory with the debug information:

b tests/:config.cxx.coptions+=-g

build2 now also performs a much more thorough C++ compiler detection and information extraction (version, target platform). Each C++ compiler is now assigned an id which has the <type>[-<variant>] form. Currently recognized C++ compilers are:

gcc            GCC g++
lang           Vanilla Clang clang++
clang-apple    Apple Clang clang++ and the g++ "alias"
icc            Intel icpc
msvc           Microsoft cl.exe

Below is the sample C++ compiler information extracted by build2 when using g++-4.9:

g++-4.9:
  id         gcc
  major      4
  minor      9
  patch      2
  build      (Ubuntu 4.9.2-0ubuntu1~14.04)
  signature  gcc version 4.9.2 (Ubuntu 4.9.2-0ubuntu1~14.04)
  checksum   ddf7dc8aaee550a2aa991d[...]eb77da0beeb1a04659fcede
  target     x86_64-linux-gnu

Note that while we now do support the Intel C++ compiler on Linux, VC++ is still a work in progress.

Finally, command line options, variables, and buildspec can now be specified in any order. This is especially useful if you want to re-run the previous command in the verbose mode or add a forgotten configuration variable:

b test -v
b configure config.cxx=clang++

Package Manager

The pkg-build command will now offer to drop prerequisites that were automatically built but are no longer necessary. A common case where this condition can arise is the upgrade or downgrade of a package. For example, a new version of a package may have gotten rid of a dependency. As a result, after an upgrade this dependency may no longer be used by any other package. bpkg will now detect this situation and offer to clean things up. You can suppress this behaviour with the --keep-prerequisite option.

The pkg-build command now also updates all the packages at once (that is, with a single build system invocation) instead of sequentially one at a time. This should improve performance, especially once the build system supports parallelism.

Also, similar to the build system, we can now specify bpkg command options and arguments in any order. For example:

bpkg update -v
bpkg install libfoo config.install.root=/opt/foo

Repository Web Interface

We can now customize the web page logo and menu entries via the brep module configuration. It is also now possible to run several instances of the brep module on a single Apache server. The configuration can be specified at the Apache2 root, VistualHost, and Location levels. For example:

<VirtualHost *:80>

  LoadModule brep_module .../mod_brep.so

  <Location "/rep1">
    SetHandler brep
    brep-root /rep1
    brep-conf .../rep1.conf
    ...
  </Location>

  <Location "/rep2">
    SetHandler brep
    brep-root /rep2
    brep-conf .../rep2.conf
    ...
  </Location>

</VirtualHost>

The INSTALL file has been expanded with instructions on how to run the database loader via cron rather than systemd timers, how to enable Apache2 compression of the brep output, and how to optimize brep CSS with SASS.