build2 | 0.16.0 Release Notes

These notes provide a more detailed discussion of major new features, including the motivation for implementing them and their usage examples. For the complete list of changes, refer to the Release Announcement or the NEWS files in the individual packages.

The two main areas of focus in this release are the system package manager integration and more advanced functionality for more complex projects. The system package manager support includes both consumption and production of binary distribution packages, including uploading them from the CI builds. This support required a large amount of ground work which produced functionality useful in its own right, such as relocatable installations, installation filtering, and installation manifests. The more advanced functionality includes built-in support for the Objective-C/C++ and Assembler with C Preprocessor (.S) compilation, dynamic target extraction in addition to prerequisites, and support for buildfile importation.

There are also quite a few smaller features in this release, such as new functions, commands, and builtins, as well as a large amount of maintenance work, including the low verbosity diagnostics overhaul and support for diagnostics buffering. We have also continued with our performance optimizations in the build system and the package manager which should especially help projects with a large number of dependencies.

The following sections discuss these and other new features in detail.

A note on backwards compatibility: this release cannot be upgraded to from build2 0.15.0 and has to be installed from scratch.

1.1New CI configurations
2Build System
2.1Objective-C/C++ compilation
2.2Assembler with C Preprocessor compilation
2.3Explicit target groups
2.4Dynamic target extraction
2.5buildfile importation
2.6Installation filtering
2.7New functions, commands, and builtins
3Package Dependency Manager
3.1System package manager integration
3.2Binary distribution package generation
3.3Package build configuration
3.4New package-description manifest values
4Project Dependency Manager
4.1New type, language manifest values

1 Infrastructure

1.1 New CI configurations

The following new build configurations have been added to the CI service:



macos_12-clang_14.0       (Xcode 14.2,   Apple Clang 14.0.0)
macos_13-clang_14.0       (Xcode 14.3.1, Apple Clang 14.0.3)
macos_13-clang_15.0       (Xcode 15.0,   Apple Clang 15.0.0)

windows_10-msvc_17.6                  (LTSC)

Note that the Linux configurations now support building for both x86_64 and aarch64 (ARM64).

There are also two new build classes: sys and bindist. The sys class contains configurations that support the system package manager integration for package consumption. They can be used to test against system-installed versions of packages or satisfy dependencies that are not yet available from source. The bindist class contains configurations that support the generation and upload of binary distribution packages. Currently, this class contains the following configurations:


All in all, there are now 89 build configurations that cover a wide version range of all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, FreeBSD as well as WebAssembly).

2 Build System

2.1 Objective-C/C++ compilation

While it may seem like Objective-C's days are numbered, a surprising number of C/C++ libraries still use it as "glue" to access system APIs on Mac OS. It is possible to handle this with ad hoc recipes and rules, however, things can get quite hairy, especially if we also need to pass header search paths for libraries that we use. To help with this, the c and cxx build system modules now support Objective-C/C++ compilation.

Specifically, the c and cxx modules now provide the c.objc and cxx.objcxx submodules which can be loaded in order to register the m{}/mm{} target types and enable Objective-C/C++ compilation in the c and cxx compile rules. Note that c.objc and cxx.objcxx must be loaded after the c and cxx modules, respectively, and while the m{}/mm{} target types are registered unconditionally, compilation is only enabled if the C/C++ compiler supports Objective-C/C++ for this target platform. Typical usage:

using cxx
using cxx.objcxx
# buildfile
lib{hello}: {hxx cxx}{*}
lib{hello}: mm{*}: include = ($ == 'macos')

See Objective-C Compilation and Objective-C++ Compilation for details.

2.2 Assembler with C Preprocessor compilation

Did you know that both GCC and Clang support compiling assembler files that make use of the C Preprocessor? Such files traditionally have the .S extension to distinguish them from the vanilla assembler files (which use .s). For example:

/* hello.S
#  define HELLO_RESULT 0


.global hello
  movq $HELLO_RESULT, %rax

The c module now provides the submodule which can be loaded in order to register the S{} target type and enable the Assembler with C Preprocessor compilation in the c compile rule. Typical usage:

using c
# buildfile
exe{hello}: {h c}{* -hello.c}

# Use the C implementation as a fallback if no assembler.
assembler = ($c.class == 'gcc' && $ == 'x86_64')

exe{hello}: S{hello}: include = $assembler
exe{hello}: c{hello}: include = (!$assembler)

See Assembler with C Preprocessor Compilation for details.

2.3 Explicit target groups

The underlying build2 build model supports two kinds of target groups (targets that have multiple member targets): explicit and ad hoc. And while build system modules can define either kind, in buildfiles we can only define ad hoc groups. To recap, here is an example of an ad hoc group (see Ad hoc recipes for background):

<{hxx cxx}{data}>: file{data.bin} $xxd

One notable limitation of ad hoc groups is that they cannot be "see-through", meaning that when the group is listed as a prerequisite of a target, the matching rule "sees" its members, rather than the group itself. If the set of ad hoc group members is static, then we can make do by listing the members explicitly. For example:

exe{hello}: cxx{hello} {hxx cxx}{data}

However, this does not work if the set of group members is discovered dynamically. To help with that, this release adds support for defining explicit groups in buildfiles.

Specifically, a user-defined explicit target group must be derived from the group{} base target type. If desired, it can be marked as "see-through" with an attribute. For example:

define [see_through] thrift_cxx: group

define thrift: file
thrift{*}: extension = thrift

exe{hello}: cxx{hello} thrift_cxx{data}
thrift_cxx{data}: thrift{data}

Notice how we are listing the thrift_cxx{data} target group as a prerequisite of exe{hello} expecting the C++ compile rule to instead see all its hxx{} and cxx{} members.

Explicit group members can be specified statically, injected by an ad hoc rule, or extracted dynamically by the depdb-dyndep builtin (see Dynamic target extraction). For example:

thrift_cxx{data}<{hxx cxx}{data_constants}>: thrift{data}        #Static

thrift_cxx{~'/(.+)/'}<{hxx cxx}{^'/\1_types/'}>: thrift{~'/\1/'} #Inject
  depdb dyndep --dyn-target ...                                 #Dynamic

See Dynamic target extraction for the continuation of the Thrift example.

2.4 Dynamic target extraction

The previous release added support for the dynamic prerequisite extraction in the form of the depdb dyndep builtin command (see Dynamic dependencies in ad hoc recipes for background). In this release we have extended this functionality to also support the dynamic target extraction which can be used to handle cases where the set of produced files cannot be predicted ahead of compilation, typically because it depends on the contents of the inputs.

One such case is the Apache Thrift compiler, which produces a variable set of C++ header and source files depending on the definitions in the .thrift file being compiled.

The dynamic target extraction is enabled with the new --dyn-target option of the depdb dyndep builtin command. For example:

thrift_cxx{~'/(.+)/'}: thrift{~'/\1/'}
  depdb dyndep --dyn-target ...

With this option specified, depdb dyndep will extract and enter as group members any additional targets mentioned in the provided dependency information. For example, if the dependency information is provided in the make format, then any files listed on the left hand side of : will be added as group members in addition to adding files listed on the right hand side as prerequisites.

Specifically, if the recipe target is an explicit group, then the dynamically extracted targets are added as its members. Otherwise, the listed targets are added as ad hoc group members. In both cases the dynamically extracted target is ignored if it is already specified as a static member or injected by a rule. Note also that this functionality is not available in the --byproduct mode of depdb dyndep.

For a complete example of handling Thrift compilation see the hello-thrift project.

2.5 buildfile importation

Currently, a project can define target types and implement ad hoc build rules in buildfiles for its own use. One can also write in C++ a build system module that defines target types and implements build rule to allow reusing them by many projects. While the latter approach provides maximum flexibility and power, with the addition of dynamic dependency extraction support (for both prerequisites and targets) and the ability to define explicit groups, implementing reusable rules in buildfiles using higher-level mechanisms is becoming a viable alternative. Such rules could be shipped together with the tools, such as source code generators, that they provide compilation support for or as separate add-on packages. The only missing bit of functionality to enable this is the ability to reuse buildfile fragments between projects. This feature aims to fill this gap.

A project can now export buildfiles that can then be imported by other projects. Specifically, a project can now place *.build files into its build/export/ subdirectory. Such files can then be imported by other projects as buildfile{} targets. For example:

import thrift%buildfile{thrift-cxx}

While for other target types the semantics of import is to load the project's export stub and return the exported target, for buildfile{} the semantics is to source the imported buildfile at the point of importation.

Note that care must be taken when authoring exported buildfiles since they will be sourced by other projects in unpredictable circumstances. In particular, the import directive by default does not prevent sourcing the same buildfile multiple times (neither in the same project nor in the same scope). As a result, if certain parts must only be sourced once per project (such as target type definitions), then they must be factored into a separate buildfile (also in build/export/) that is imported by the "main" exported buildfile with the once attribute. For example, the above may contain:

import [once] thrift%buildfile{thrift-cxx-target-type}

See also install Module for details on the exported buildfile installation.

2.6 Installation filtering

While project authors determine what gets installed at the buildfile level, the users of the project can now further filter the installation using the config.install.filter variable.

The value of this variable is a list of key-value pairs that specify the filesystem entries to include or exclude from the installation. For example, the following filters will omit installing headers and static libraries (notice the quoting of the wildcard):

$ b install config.install.filter='include/@false "*.a"@false'

See Installation Filtering for details.

2.7 New functions, commands, and builtins

This release adds a good number of new functions. Their signatures are listed below (refer to the NEWS file for their descriptions):


$regex.filter[_out]_{match,search}(<vals>, <pat>)

$find(<sequence>, <value>)
$find_index(<sequence>, <value>)

$integer_sequence(<begin>, <end>[, <step>])

$is_a(<name>, <target-type>),
$filter[_out](<names>, <target-types>)

The Buildscript recipes and Testscript can now make use of the for and while loop commands as well as the find builtin. For example:

find gen/ -type f -name '*.?xx' | for -n f
  echo $f >>$t

3 Package Dependency Manager

3.1 System package manager integration

The bpkg-pkg-build(1) command will now query (unless --sys-no-query is specified) the system package manager on Debian (or alike, such as Ubuntu) and Fedora (or alike, such as RHEL) for versions of packages that are specified as coming from the system (the sys: scheme). For example, if running the following command on one of these distributions:

$ bpkg build hello ?sys:libsqlite3

Then pkg-build will query the system package manager for the installed version of libsqlite3 and fail if none is present.

Additionally, if --sys-install is specified, pkg-build will attempt to install such packages if not present but available from the system package repository. For example, running the below command on Debian may produce the following output if libsqlite-dev is not already installed:

$ bpkg build --sys-install hello ?sys:libsqlite3
  sys-install libsqlite3-0/3.42.0-1 (required by sys:libsqlite3)
  configure sys:libsqlite3/3.42.0 (required by hello)
  new hello/1.0.0
continue? [Y/n] y
installing debian packages...
The following NEW packages will be installed:
The following packages will be upgraded:
  libsqlite3-0 sqlite3
Do you want to continue? [Y/n] y
Setting up libsqlite3-0:amd64 (3.42.0-1) ...
Setting up libsqlite3-dev:amd64 (3.42.0-1) ...
Setting up sqlite3 (3.42.0-1) ...

This support is also available in the project dependency manager (bdep), both in the init and sync commands. For example:

$ bdep init --sys-install -C @gcc cc -- ?sys:libsqlite3

For a more detailed walk-through of this functionality, see Using System-Installed Dependencies in the toolchain introduction.

3.2 Binary distribution package generation

The new bpkg-pkg-bindist(1) command can be used to automatically generate binary distribution packages from build2 packages for Debian (or alike, such as Ubuntu), Fedora (or alike, such as RHEL), and for other operating systems as installation archives. For Debian and Fedora, dependencies can be satisfied with system packages, build2 packages, or bundled.

For example, to generate a Debian package (running on Debian or alike):

$ bpkg bindist --recursive=auto --private -o /tmp/hello-deb/ hello
generated debian package for hello/1.0.0:

$ sudo apt-get install /tmp/hello-deb/hello_1.0.0-0~debian12_amd64.deb

And to generate a Fedora package (running on Fedora or alike):

$ bpkg bindist --recursive=auto --private hello
generated fedora package for hello/1.0.0:

$ sudo dnf install ~/rpmbuild/RPMS/x86_64/hello-1.0.0-1.fc38.x86_64.rpm

And to generate an installation archive (running on Windows in this example):

$ bpkg bindist --recursive=auto   ^
  --private                       ^
  --distribution=archive          ^
  -o C:\tmp\hello-zip\            ^
  config.install.relocatable=true ^
generated archive package for hello/1.0.0:

See the bpkg-pkg-bindist(1) command documentation for details and additional examples.

It is also possible to generate and upload binary distribution packages as part of the CI builds as discussed in the next section.

3.3 Package build configuration

A package can now customize in its manifest the build configuration used during CI. This includes specifying configuration variables, forcing specific versions of dependencies, satisfying dependencies with system packages, and enabling/disabling build steps.

These customizations are controlled by a new family of package manifest values (here <name> is the name given to a configuration):

<name>-builds        -- build class expression (same as builds)
<name>-build-include -- build include pattern (same as build-include)
<name>-build-exclude -- build exclude pattern (same as build-exclude)
<name>-build-config  -- custom build configuration

For example:

# Test with extra functionality enabled.
extras-build-config: config.libfoo.extras=true
# Test with system-installed libsqlite3.
system-builds: sys
system-build-config: ?sys:libsqlite3

This functionality can also be used to generate binary distribution packages as part of the CI builds (using build configurations form the bindist class mentioned earlier) and upload them as part of the build result. Such uploaded binary packages are then made available for download from the package version page on the repository web interface. For example:

# Enable Debian binary distribution package generation and upload.
bindist-debian-builds: bindist
bindist-debian-build-include: linux_debian*-**
bindist-debian-build-include: linux_ubuntu*-**
bindist-debian-build-exclude: **

See the *-build-config package manifest value documentation for details and more examples.

Note that the project dependency manager (bdep) ci command was extended with the new --target-config and --package-config options to match this new functionality. See bdep-ci(1) for details.

3.4 New package-description manifest values

When packaging a third-party project we often write a separate README file that provides information specific to the build2 functionality, such as the recommended usage, configuration variables, etc. This brings the question of which README file to specify in the description-file manifest value: upstream's or ours.

To resolve this question we have added the package-description-file (as well as the package-description and package-description-type) manifest values for providing the build2-specific description in addition to upstream's. The typical usage would be:

# manifest
# buildfile
./: ... doc{} ...

The recommended file name for package README is PACKAGE-README or README-PACKAGE. See the package-description package manifest value documentation for details.

4 Project Dependency Manager

4.1 New type, language manifest values

The bdep-new command now adds the type and language values to manifest of newly created projects. For example:

name: hello
type: lib
language: c++

Note that type is only added if the project type (library, executable) cannot be deduced from its name (for example, based on the lib prefix).

This information can help better describe the project and is used, for example, by bpkg-pkg-bindist to produce correct binary distribution packages. See type, language package manifest values documentation for details.