build2 | 0.16.0 Release Notes
These notes provide a more detailed discussion of major new features,
including the motivation for implementing them and their usage examples. For
the complete list of changes, refer to the Release
Announcement or the NEWS files in the individual
packages.
The two main areas of focus in this release are the system package
manager integration and more advanced functionality for more complex
projects. The system package manager support includes both consumption and
production of binary distribution packages, including uploading them from
the CI builds. This support required a large amount of ground work which
produced functionality useful in its own right, such as relocatable
installations, installation filtering, and installation manifests. The more
advanced functionality includes built-in support for the Objective-C/C++ and
Assembler with C Preprocessor (.S) compilation, dynamic target
extraction in addition to prerequisites, and support for buildfile
importation.
There are also quite a few smaller features in this release, such as new functions, commands, and builtins, as well as a large amount of maintenance work, including the low verbosity diagnostics overhaul and support for diagnostics buffering. We have also continued with our performance optimizations in the build system and the package manager which should especially help projects with a large number of dependencies.
The following sections discuss these and other new features in detail.
A note on backwards compatibility: this release cannot be upgraded to
from build2 0.15.0 and has to be installed from
scratch.
1 Infrastructure
1.1 New CI configurations
The following new build configurations have been added to the CI service:
freebsd_12.4-clang_13.0 freebsd_13.2-clang_14.0 linux_debian_12-gcc_13 linux_debian_12-clang_15[_libc++] linux_debian_12-clang_16[_libc++] macos_12-clang_14.0 (Xcode 14.2, Apple Clang 14.0.0) macos_13-clang_14.0 (Xcode 14.3.1, Apple Clang 14.0.3) macos_13-clang_15.0 (Xcode 15.0, Apple Clang 15.0.0) macos_13-gcc_13_homebrew windows_10-msvc_17.5 windows_10-msvc_17.6 (LTSC) windows_10-clang_15.0_msvc_msvc_17.6 windows_10-clang_16.0_llvm_msvc_17.6 windows_10-gcc_12.2_mingw_w64
Note that the Linux configurations now support building for both
x86_64 and aarch64 (ARM64).
There are also two new build classes: sys and bindist.
The sys class contains configurations that support the system package manager integration for package
consumption. They can be used to test against system-installed versions of
packages or satisfy dependencies that are not yet available from source. The
bindist class contains configurations that support the generation and upload of
binary distribution packages. Currently, this class contains the following
configurations:
linux_debian_11-gcc_10.2-bindist linux_debian_12-gcc_12-bindist linux_fedora_37-gcc_12.2-bindist linux_fedora_38-gcc_13-bindist
All in all, there are now 89 build configurations that cover a wide version range of all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, FreeBSD as well as WebAssembly).
2 Build System
2.1 Objective-C/C++ compilation
While it may seem like Objective-C's days are numbered, a surprising
number of C/C++ libraries still use it as "glue" to access system APIs on
Mac OS. It is possible to handle this with ad hoc recipes and rules, however, things can get quite
hairy, especially if we also need to pass header search paths for libraries
that we use. To help with this, the c and cxx
build system modules now support Objective-C/C++ compilation.
Specifically, the c and cxx modules now provide
the c.objc and cxx.objcxx submodules which can be
loaded in order to register the m{}/mm{} target
types and enable Objective-C/C++ compilation in the c and
cxx compile rules. Note that c.objc and
cxx.objcxx must be loaded after the c and
cxx modules, respectively, and while the
m{}/mm{} target types are registered
unconditionally, compilation is only enabled if the C/C++ compiler supports
Objective-C/C++ for this target platform. Typical usage:
# root.build # using cxx using cxx.objcxx
# buildfile
#
lib{hello}: {hxx cxx}{*}
lib{hello}: mm{*}: include = ($cxx.target.class == 'macos')
See Objective-C Compilation and Objective-C++ Compilation for details.
2.2 Assembler with C Preprocessor compilation
Did you know that both GCC and Clang support compiling assembler files
that make use of the C Preprocessor? Such files traditionally have the
.S extension to distinguish them from the vanilla assembler
files (which use .s). For example:
/* hello.S */ #ifndef HELLO_RESULT # define HELLO_RESULT 0 #endif text .global hello hello: movq $HELLO_RESULT, %rax ret
The c module now provides the c.as-cpp
submodule which can be loaded in order to register the S{}
target type and enable the Assembler with C Preprocessor compilation in the
c compile rule. Typical usage:
# root.build # using c using c.as-cpp
# buildfile
#
exe{hello}: {h c}{* -hello.c}
# Use the C implementation as a fallback if no assembler.
#
assembler = ($c.class == 'gcc' && $c.target.cpu == 'x86_64')
exe{hello}: S{hello}: include = $assembler
exe{hello}: c{hello}: include = (!$assembler)
See Assembler with C Preprocessor Compilation for details.
2.3 Explicit target groups
The underlying build2 build model supports two kinds of
target groups (targets that have multiple member targets): explicit and ad
hoc. And while build system modules can define either kind, in
buildfiles we can only define ad hoc groups. To recap, here is
an example of an ad hoc group (see Ad
hoc recipes for background):
<{hxx cxx}{data}>: file{data.bin} $xxd
{{
...
}}
One notable limitation of ad hoc groups is that they cannot be "see-through", meaning that when the group is listed as a prerequisite of a target, the matching rule "sees" its members, rather than the group itself. If the set of ad hoc group members is static, then we can make do by listing the members explicitly. For example:
exe{hello}: cxx{hello} {hxx cxx}{data}
However, this does not work if the set of group members is discovered dynamically. To help with that, this
release adds support for defining explicit groups in
buildfiles.
Specifically, a user-defined explicit target group must be derived from
the group{} base target type. If desired, it can be marked as
"see-through" with an attribute. For example:
define [see_through] thrift_cxx: group
define thrift: file
thrift{*}: extension = thrift
exe{hello}: cxx{hello} thrift_cxx{data}
thrift_cxx{data}: thrift{data}
Notice how we are listing the thrift_cxx{data} target group
as a prerequisite of exe{hello} expecting the C++ compile rule
to instead see all its hxx{} and cxx{}
members.
Explicit group members can be specified statically, injected by an ad hoc
rule, or extracted dynamically by the depdb-dyndep builtin (see
Dynamic target extraction). For example:
thrift_cxx{data}<{hxx cxx}{data_constants}>: thrift{data} #Static
thrift_cxx{~'/(.+)/'}<{hxx cxx}{^'/\1_types/'}>: thrift{~'/\1/'} #Inject
{{
depdb dyndep --dyn-target ... #Dynamic
}}
See Dynamic target extraction for the continuation of the Thrift example.
2.4 Dynamic target extraction
The previous release added support for the dynamic prerequisite
extraction in the form of the depdb dyndep builtin command (see
Dynamic dependencies in ad hoc recipes for
background). In this release we have extended this functionality to also
support the dynamic target extraction which can be used to handle cases
where the set of produced files cannot be predicted ahead of compilation,
typically because it depends on the contents of the inputs.
One such case is the Apache Thrift compiler, which produces a variable
set of C++ header and source files depending on the definitions in the
.thrift file being compiled.
The dynamic target extraction is enabled with the new
--dyn-target option of the depdb dyndep builtin
command. For example:
thrift_cxx{~'/(.+)/'}: thrift{~'/\1/'}
{{
depdb dyndep --dyn-target ...
}}
With this option specified, depdb dyndep will extract and
enter as group members any additional targets mentioned in the provided
dependency information. For example, if the dependency information is
provided in the make format, then any files listed on the left
hand side of : will be added as group members in addition to
adding files listed on the right hand side as prerequisites.
Specifically, if the recipe target is an explicit group, then the dynamically extracted
targets are added as its members. Otherwise, the listed targets are added as
ad hoc group members. In both cases the dynamically extracted target is
ignored if it is already specified as a static member or injected by a rule.
Note also that this functionality is not available in the
--byproduct mode of depdb dyndep.
For a complete example of handling Thrift compilation see the hello-thrift
project.
2.5 buildfile importation
Currently, a project can define target types and implement ad hoc build
rules in buildfiles for its own use. One can also write in C++
a build system module that defines target types and implements build rule to
allow reusing them by many projects. While the latter approach provides
maximum flexibility and power, with the addition of dynamic dependency
extraction support (for both prerequisites
and targets) and the ability to define explicit groups, implementing reusable rules in
buildfiles using higher-level mechanisms is becoming a viable
alternative. Such rules could be shipped together with the tools, such as
source code generators, that they provide compilation support for or as
separate add-on packages. The only missing bit of functionality to enable
this is the ability to reuse buildfile fragments between projects. This
feature aims to fill this gap.
A project can now export buildfiles that can then be
imported by other projects. Specifically, a project can now place
*.build files into its build/export/ subdirectory.
Such files can then be imported by other projects as
buildfile{} targets. For example:
import thrift%buildfile{thrift-cxx}
While for other target types the semantics of import is to
load the project's export stub and return the exported target, for
buildfile{} the semantics is to source the imported
buildfile at the point of importation.
Note that care must be taken when authoring exported
buildfiles since they will be sourced by other projects in
unpredictable circumstances. In particular, the import
directive by default does not prevent sourcing the same
buildfile multiple times (neither in the same project nor in
the same scope). As a result, if certain parts must only be sourced once per
project (such as target type definitions), then they must be factored into a
separate buildfile (also in build/export/) that is
imported by the "main" exported buildfile with the
once attribute. For example, the above
thrift-cxx.build may contain:
import [once] thrift%buildfile{thrift-cxx-target-type}
See also install
Module for details on the exported buildfile
installation.
2.6 Installation filtering
While project authors determine what gets installed at the
buildfile level, the users of the project can now further
filter the installation using the config.install.filter
variable.
The value of this variable is a list of key-value pairs that specify the filesystem entries to include or exclude from the installation. For example, the following filters will omit installing headers and static libraries (notice the quoting of the wildcard):
$ b install config.install.filter='include/@false "*.a"@false'
See Installation Filtering for details.
2.7 New functions, commands, and builtins
This release adds a good number of new functions. Their signatures are
listed below (refer to the NEWS
file for their descriptions):
$path.posix_string(<path>)
$path.posix_representation(<path>)
$regex.filter[_out]_{match,search}(<vals>, <pat>)
$find(<sequence>, <value>)
$find_index(<sequence>, <value>)
$integer_sequence(<begin>, <end>[, <step>])
$is_a(<name>, <target-type>),
$filter[_out](<names>, <target-types>)
The Buildscript recipes and Testscript can now make use of the for
and while
loop commands as well as the find
builtin. For example:
find gen/ -type f -name '*.?xx' | for -n f echo $f >>$t end
3 Package Dependency Manager
3.1 System package manager integration
The bpkg-pkg-build(1)
command will now query (unless --sys-no-query is specified) the
system package manager on Debian (or alike, such as Ubuntu) and Fedora (or
alike, such as RHEL) for versions of packages that are specified as coming
from the system (the sys: scheme). For example, if running the
following command on one of these distributions:
$ bpkg build hello ?sys:libsqlite3
Then pkg-build will query the system package manager for the
installed version of libsqlite3 and fail if none is
present.
Additionally, if --sys-install is specified,
pkg-build will attempt to install such packages if not present
but available from the system package repository. For example, running the
below command on Debian may produce the following output if
libsqlite-dev is not already installed:
$ bpkg build --sys-install hello ?sys:libsqlite3 sys-install libsqlite3-0/3.42.0-1 (required by sys:libsqlite3) configure sys:libsqlite3/3.42.0 (required by hello) new hello/1.0.0 continue? [Y/n] y installing debian packages... The following NEW packages will be installed: libsqlite3-dev The following packages will be upgraded: libsqlite3-0 sqlite3 Do you want to continue? [Y/n] y ... Setting up libsqlite3-0:amd64 (3.42.0-1) ... Setting up libsqlite3-dev:amd64 (3.42.0-1) ... Setting up sqlite3 (3.42.0-1) ... ...
This support is also available in the project dependency manager
(bdep), both in the init and sync
commands. For example:
$ bdep init --sys-install -C @gcc cc -- ?sys:libsqlite3
For a more detailed walk-through of this functionality, see Using System-Installed Dependencies in the toolchain introduction.
3.2 Binary distribution package generation
The new bpkg-pkg-bindist(1)
command can be used to automatically generate binary distribution packages
from build2 packages for Debian (or alike, such as Ubuntu),
Fedora (or alike, such as RHEL), and for other operating systems as
installation archives. For Debian and Fedora, dependencies can be satisfied
with system packages, build2 packages, or bundled.
For example, to generate a Debian package (running on Debian or alike):
$ bpkg bindist --recursive=auto --private -o /tmp/hello-deb/ hello ... generated debian package for hello/1.0.0: /tmp/hello-deb/hello_1.0.0-0~debian12_amd64.deb /tmp/hello-deb/hello-dbgsym_1.0.0-0~debian12_amd64.deb /tmp/hello-deb/hello_1.0.0-0~debian12_amd64.buildinfo /tmp/hello-deb/hello_1.0.0-0~debian12_amd64.changes $ sudo apt-get install /tmp/hello-deb/hello_1.0.0-0~debian12_amd64.deb
And to generate a Fedora package (running on Fedora or alike):
$ bpkg bindist --recursive=auto --private hello ... generated fedora package for hello/1.0.0: ~/rpmbuild/RPMS/x86_64/hello-1.0.0-1.fc38.x86_64.rpm ~/rpmbuild/RPMS/x86_64/hello-debuginfo-1.0.0-1.fc38.x86_64.rpm $ sudo dnf install ~/rpmbuild/RPMS/x86_64/hello-1.0.0-1.fc38.x86_64.rpm
And to generate an installation archive (running on Windows in this example):
$ bpkg bindist --recursive=auto ^ --private ^ --distribution=archive ^ -o C:\tmp\hello-zip\ ^ config.install.relocatable=true ^ hello ... generated archive package for hello/1.0.0: C:\tmp\hello-zip\hello-1.0.0-x86_64-windows10.zip
See the bpkg-pkg-bindist(1)
command documentation for details and additional examples.
It is also possible to generate and upload binary distribution packages as part of the CI builds as discussed in the next section.
3.3 Package build configuration
A package can now customize in its manifest the build
configuration used during CI. This includes specifying configuration
variables, forcing specific versions of dependencies, satisfying
dependencies with system packages, and enabling/disabling build steps.
These customizations are controlled by a new family of package
manifest values (here <name> is the name given
to a configuration):
<name>-builds -- build class expression (same as builds) <name>-build-include -- build include pattern (same as build-include) <name>-build-exclude -- build exclude pattern (same as build-exclude) <name>-build-config -- custom build configuration
For example:
# Test with extra functionality enabled. # extras-build-config: config.libfoo.extras=true
# Test with system-installed libsqlite3. # system-builds: sys system-build-config: ?sys:libsqlite3
This functionality can also be used to generate binary
distribution packages as part of the CI builds (using build
configurations form the bindist class mentioned earlier) and
upload them as part of the build result. Such uploaded binary packages are
then made available for download from the package version page on the
repository web interface. For example:
# Enable Debian binary distribution package generation and upload. # bindist-debian-builds: bindist bindist-debian-build-include: linux_debian*-** bindist-debian-build-include: linux_ubuntu*-** bindist-debian-build-exclude: ** bindist-debian-build-config: \ ?sys:libsqlite3 +bpkg.bindist.debian:--recursive=auto +bbot.bindist.upload: \
See the *-build-config
package manifest value documentation for details and more examples.
Note that the project dependency manager (bdep)
ci command was extended with the new
--target-config and --package-config options to
match this new functionality. See bdep-ci(1) for
details.
3.4 New package-description manifest
values
When packaging a third-party project we often write a separate
README file that provides information specific to the
build2 functionality, such as the recommended usage,
configuration variables, etc. This brings the question of which
README file to specify in the description-file
manifest value: upstream's or ours.
To resolve this question we have added the
package-description-file (as well as the
package-description and package-description-type)
manifest values for providing the build2-specific description
in addition to upstream's. The typical usage would be:
# manifest # ... description-file: README.md package-description-file: README-PACKAGE.md
# buildfile
#
./: ... doc{README.md README-PACKAGE.md} ...
The recommended file name for package README is
PACKAGE-README or README-PACKAGE. See the package-description
package manifest value documentation for details.
4 Project Dependency Manager
4.1 New type, language manifest
values
The bdep-new
command now adds the type and language values to
manifest of newly created projects. For example:
name: hello type: lib language: c++
Note that type is only added if the project type (library,
executable) cannot be deduced from its name (for example, based on the
lib prefix).
This information can help better describe the project and is used, for
example, by bpkg-pkg-bindist to produce correct binary distribution packages. See type,
language package manifest values documentation for
details.