build2 | 0.13.0 Release Notes

These notes provide a more detailed discussion of major new features, including the motivation for implementing them and their usage examples. For the complete list of changes, refer to the Release Announcement or the NEWS files in the individual packages. See also the discussion of these release notes on r/cpp/ and r/programming/.

The main focus of this release is build system support for project-specific configurations and ad hoc recipes. Another notable new feature is support for alternative (e.g., with include/src split) source code layouts by the bdep-new command. While the below notes discuss these and other features in detail, here is a one line TL;DR for each:

1Infrastructure
1.1New CI configurations (15 added, 57 in total)
2Toolchain
2.1SPDX as default license name
2.2Private library installation by default
3Build System
3.1Project-specific configurations
3.2Ad hoc recipes
3.3Project-local importation
3.4Ad hoc importation and "glue buildfiles"
4Project Dependency Manager
4.1Source code layout customizations
5Package Dependency Manager
5.1Package download proxies

A note on backwards compatibility: this release cannot be upgraded to from 0.12.0 and has to be installed from scratch.

1 Infrastructure

1.1 New CI configurations (15 added, 57 in total)

The following new build configurations have been added to the CI service:

linux_debian_9-gcc_8.4
linux_debian_10-gcc_9.3
linux_debian_10-gcc_10.1
linux_debian_10-clang_10.0[_libc++]

macos_10.15-clang_11.0        (Xcode 11.5 Clang 11.0.3)
macos_10.15-gcc_9.3_homebrew

windows_10-msvc_16.4
windows_10-msvc_16.5
windows_10-msvc_16.6
windows_10-clang_10.0_llvm_msvc_16.6[_lld]
windows_10-gcc_9.2_mingw_w64

In addition, there are now optimized and static optimized builds for the latest version of every configuration. All in all, there are now 57 build configurations (up from 42 in the previous release) that cover a wide range of versions for all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, and FreeBSD).

2 Toolchain

2.1 SPDX as default license name

The SPDX License Expression is now the default scheme for the license package manifest value as well as the bdep-new command's --type|-t,license sub-option. Auto-detected licenses now also result in the SPDX License ID in the license package manifest value.

In its simplest form, the SPDX license is just an ID, for example:

license: MPL-2.0

If the package is licensed under multiple licenses, then an SPDX license expression can be used to specify this, for example:

license: Apache-2.0 OR MIT
license: MIT AND BSD-2-Clause

See the license package manifest value documentation for details and the list of commonly used license IDs.

2.2 Private library installation by default

On UNIX-like operating systems the installation script now installs the toolchain libraries into a private subdirectory of lib/. This means that when installed into a shared location like /usr/local/, the toolchain implementation details can no longer clash with anything that is already installed or might be installed in the future into the same location by the user.

Note that this mechanism is also available to other projects that use build2. See install module for details.

3 Build System

3.1 Project-specific configurations

A project can now use the config directive to define project-specific config.<project>.* variables, similar to the build system core and modules. For example:

config [bool]   config.libhello.fancy    ?= false
config [string] config.libhello.greeting ?= 'Hello'

With these configuration points available, the user of the libhello library can configure it using the same mechanism as what's already used to configure the compiler, options, importation, etc. For example:

$ b configure: libhello/          \
    config.cxx=clang++            \
    config.cxx.coptions=-O3       \
    config.libhello.fancy=true    \
    config.libhello.greeting=Howdy

Inside libhello, these configuration variables can then be used in buildfiles and/or propagated to the source code using the command line, .in file substitution, etc. For example:

if $config.libhello.fancy
  cxx.poptions += -DLIBHELLO_FANCY

cxx.poptions += "-DLIBHELLO_GREETING=\"$config.libhello.greeting\""

Or using the .in file:

// libhello/config.hxx.in

#pragma once

#define LIBHELLO_FANCY    $config.libhello.fancy$
#define LIBHELLO_GREETING "$config.libhello.greeting$"

See the Project Configuration chapter in the build system manual for details and more examples (including using if constexpr for configuration).

3.2 Ad hoc recipes

With ad hoc recipes it is now possible to provide custom implementations of operations (update, test, etc) for certain targets directly in buildfiles.

Note that in this release support for ad hoc recipes is at the "technology preview" stage. In particular, there is no documentation (other than the examples below) and there might be some rough edges.

Let's look at a few examples. This is how we can pick a configuration header based on the platform:

tclass = $cxx.target.class

hxx{config}: hxx{config-linux}:   include = ($tclass == 'linux')
hxx{config}: hxx{config-windows}: include = ($tclass == 'windows')
hxx{config}: hxx{config-macos}:   include = ($tclass == 'macos')
hxx{config}:
{{
  cp $path($<) $path($>)
}}

While this may seem like a lot of "code" for something as simple as copying a file, doing this properly in a build system is trickier than it might seem. For example, you might be tempted to write something "simpler" like this:

hxx{config}: hxx{config-$cxx.target.class}
{{
  cp $path($<) $path($>)
}}

It sure looks elegant. And it will sort of work until we try to prepare a distribution of our project which will end up with only the input header corresponding to the platform on which we've prepared the distribution. See Conditions for details on this issue.

The recipe body (cp $path($<) $path($>)) is pretty simple: we use the cp utility to copy the file corresponding to the prerequisite (available to the recipe in the $< special variable) to the file corresponding to the target (available in the $> special variable).

Let's look at another, more elaborate, example that shows how to embed binary data into the source code with the help of the xxd(1) utility:

import! xxd = xxd%exe{xxd}

<{hxx cxx}{foo}>: file{foo.bin} $xxd
{{
  diag xxd ($<[0])

  i = $path($<[0]) # Input.
  h = $path($>[0]) # Output header.
  s = $path($>[1]) # Output source.
  n = $name($<[0]) # Array name.

  # Get the position of the last byte (in hex).
  #
  $xxd -s -1 -l 1 $i | sed -n -e 's/^([0-9]+):.*$/\1/p' - | set pos

  if ($empty($pos))
    exit "unable to extract input size from xxd output"
  end

  # Write header and source.
  #
  echo "#pragma once"                         >$h
  echo "extern const char $n[0x$pos + 1];"   >>$h
  echo "extern const char $n[0x$pos + 1]= {"  >$s
  $xxd -i <$i                                >>$s
  echo '};'                                  >>$s
}}

While explaining this example in detail is beyond the scope of these release notes, one aspect worth highlighting is the use of an ad hoc target group (<{hxx cxx}{foo}>) to indicate that the recipe will produce both files with a single execution.

Note that in both examples, the utilities (cp, echo, and sed) are builtins which means these recipes are portable and will work even on Windows. See the Testscript Manual for the list of available builtins.

If you are familiar with make, the notion of a recipe shouldn't be anything new to you. Note, however, that there is a lot going on underneath in build2 recipes compared to make recipes (which are just shell commands). In particular, besides portability, with build2 you get the corresponding clean operation implementation automatically as well as the output file cleanup on recipe failure. There is also a lot of change tracking performed in order to make sure the target is updated if the recipe itself changes, any variable that it references changes, or any tool that it uses changes.

Ad hoc recipes can also be used to customize a part of the update chain otherwise handled by rules. For example, in embedded systems development it is often required to perform a custom link step:

obje{foo}: cxx{foo}
obje{bar}: cxx{bar}

<exe{test} file{test.map}>: obje{foo bar}
{{
  diag ld ($>[0])
  $cxx.path $cc.loptions $cxx.loptions $cxx.mode -o $path($>[0]) \
    "-Wl,-Map=$path($>[1])" $path($<) $cxx.libs $cc.libs
}}

While the above examples are all for the update operation, ad hoc recipes can be used for other operations, such as test. For example:

exe{hello}: cxx{hello}
% test
{{
  diag test $>
  $> 'World' >>>?'Hello, World!'
}}

Here we have the recipe header (line starting with %) which specifies the operation(s) this recipe is for. If the header is omitted, then the update operation is assumed.

The above recipes are written in a shell-like language called Buildscript that has similar semantics to Testscript tests. Another language that can be used to write recipes is C++. For example:

./:
{{ c++ 1

  recipe
  apply (action, target& t) const override
  {
    text (recipe_loc) << "Hello, " << t;
    return noop_recipe;
  }
}}

A recipe written in C++ plugs directly into the build system rules machinery and can do pretty much anything a build system core or module can.

3.3 Project-local importation

An import without a project name is now recognized as importation from the same project. For example, given the libhello project that exports the lib{hello} target, a buildfile for an executable in the same project instead of doing something like this:

include ../libhello/
exe{hello}: ../libhello/lib{hello}

Can now do:

import lib = lib{hello}
exe{hello}: $lib

The main advantage of project-local importation over inclusion is the ability to move things around without having to adjust locations in multiple places (the only place we need to do it is the export stub). This advantage becomes noticeable in more complex projects with a large number of components.

See the Target Importation section in the manual for details.

3.4 Ad hoc importation and "glue buildfiles"

If the target being imported has no project name and is either absolute or is a relative directory, then this is treated as ad hoc importation. Semantically it is similar to normal importation but with the location of the project being imported hard-coded into the buildfile.

While doing so would generally be a bad idea, this type of import can be used to create a special "glue buildfile" that "pulls" together several projects, usually for convenience of development. One typical case that calls for such a glue buildfile is a multi-package project. To be able to invoke the build system directly in the project root, we can add a glue buildfile that imports and builds all the packages:

import pkgs = */

./: $pkgs

See the Target Importation section in the manual for details.

4 Project Dependency Manager

4.1 Source code layout customizations

The bdep-new command now supports a number of new source layout customization mechanisms. In particular, the split include/src layout is now supported out of the box:

$ bdep new -l c++,cpp -t lib,split libhello

$ tree libhello/
libhello/
├── include/
│   └── libhello/
│       └── hello.hpp
└── src/
    └── libhello/
        └── hello.cpp

Or, if you prefer the more traditional structure without the subdirectory in src/:

$ bdep new -l c++ -t lib,split,subdir=hello,no-subdir-source libhello

$ tree libhello/
libhello/
├── include/
│   └── hello/
│       └── hello.hxx
└── src/
    └── hello.cxx

See the SOURCE LAYOUT section in bdep-new(1) for details and a large number of examples.

5 Package Dependency Manager

5.1 Package download proxies

The new --pkg-proxy option can be used to specify the proxy server to use when fetching package manifests and archives from remote pkg repositories. This can be particularly useful when re-fetching the same packages over and over both to speed things up and to reduce bandwidth usage. For example, we use this mechanism in our CI infrastructure.

See the option documentation in bpkg-common-options(1) for details.