build2 | 0.14.0 Release Notes
These notes provide a more detailed discussion of major new features,
including the motivation for implementing them and their usage examples. For
the complete list of changes, refer to the Release
Announcement or the NEWS files in the individual packages.
See also the discussion of this release on r/cpp/
and r/programming/.
The main focus of this release is support for build-time dependencies and the host/target configuration split that it necessitates. This support required a large amount of ground work which produced functionality useful in its own right, such as hermetic builds and configuration linking. Another notable new feature is ad hoc regex pattern rules. The following sections discuss these and other new features in detail.
A note on backwards compatibility: this release cannot be upgraded to from 0.13.0 and has to be installed from scratch.
1 Infrastructure
1.1 New CI configurations (15 new, 58 in total)
The following new build configurations have been added to the CI service:
freebsd_13.0-clang_11.0 linux_debian_10-gcc_10.2 linux_debian_10-gcc_11.2 linux_debian_10-clang_11.0[_libc++] linux_debian_10-clang_12.0[_libc++] linux_debian_11-clang_13.0[_libc++] macos_11-clang_12.0 (Xcode 12.5.1 Clang 12.0.5) macos_11-clang_13.0 (Xcode 13 Clang 13.0.0) macos_11-gcc_11.2_homebrew windows_10-msvc_16.9 windows_10-msvc_16.11 windows_10-clang_12.0_msvc_msvc_16.11 windows_10-clang_13.0_llvm_msvc_16.11[_lld] windows_10-gcc_10.2_mingw_w64 linux_debian_10-emcc_2.0.25 (Emscripten)
In addition, there are now optimized NDEBUG builds for
select configurations. All in all, there are now 58 build configurations that
cover a wide range of versions for all the major compilers (GCC, Clang, and
MSVC) on all the major platforms (Linux, Mac OS, Windows, FreeBSD as well as
WASM).
2 Toolchain
2.1 Standard pre-installed build system modules
While we can now build and load build system modules on the fly (and even
list them as build-time dependencies of our
projects), this will be inconvenient and inefficient for widely-used
modules. As a result, with this release we are introducing a notion of
standard pre-installed build system modules that the user can assume
are available out of the box in a regular build2 toolchain
installation. In particular, the official install scripts will build and
install such modules unless explicitly requested not to.
The first such build system module is libbuild2-kconfig which allows using the
Linux kernel configuration system (Kconfig) to configure
build2-based projects. See the earlier
announcement for details on standard pre-installed modules.
2.2 Performance optimizations
Our ongoing efforts to package the Boost libraries identified a number of performance issues when working with heavily inter-dependent libraries. This led to optimizations both in the build system and the package manager that should also benefit other projects.
Additionally, the build system now uses the LZ4 compression when storing intermediate build results (such as partially preprocessed C/C++ translation units) which leads to a substantial reduction in the disk space usage during the build.
Finally, a number of optimizations have been implemented for
git repository fetching in the package manager.
3 Build System
3.1 Hermetic build configurations
Hermetic build configurations save environment variables that affect the
project along with other project configuration in the
config.build file. These saved environment variables are then
used instead of the current environment when performing operations on the
project, thus making sure the project "sees" exactly the same environment as
during configuration. The built-in ~host and
~build2 configurations are now hermetic. As part of this work
we now also track changes to the environment in non-hermetic configurations
and automatically rebuild affected targets.
To create a hermetic configuration we use the
config.config.hermetic configuration variable. For example:
$ b configure config.config.hermetic=true
The two use-cases where hermetic configurations are especially useful are when we need to save an environment which is not generally available (for example, an environment of a Visual Studio development command prompt) or when our build results need to exactly match the specific configuration (for example, because parts of the overall result have already been built and installed, as is the case with build system modules). See Hermetic Build Configurations in the build system manual for details.
3.2 Ad hoc regex pattern rules
The previous release introduced support for ad hoc recipes which allow providing
custom implementations of operations (update,
test, etc) for a specific target directly in
buildfiles. In this release we are extending this functionality
with support for ad hoc rules which allow providing such custom
implementations of operations for multiple targets that match a pattern.
An ad hoc pattern rule consists of a pattern that mimics a dependency declaration followed by one or more recipes. For example:
exe{~'/(.*)/'}: cxx{~'/\1/'}
{{
$cxx.path -o $path($>) $path($<[0])
}}
If a pattern matches a dependency declaration of a target, then the recipe is used to perform the corresponding operation on this target (see ad hoc recipes for background). For example, the following dependency declaration matches the above pattern which means the rule's recipe will be used to update this target:
exe{hello}: cxx{hello}
While the following declarations do not match the above pattern:
exe{hello}: c{hello} # Type mismatch.
exe{hello}: cxx{howdy} # Name mismatch.
Note also that in build2 no intermediate targets are created
implicitly and ad hoc rules only match explicit dependency declarations. For
example, the above rule will not build exe{goodbye} given the
following buildfile even if cxx{goodbye}
exists:
./: exe{goodbye}
On the left hand side of : in the pattern we can have a
single target or an ad hoc target group. The single target or the first
(primary) ad hoc group member must be a regex pattern (~). The
rest of the ad hoc group members can be patterns or substitutions
(^). For example:
<exe{~'/(.*)/'} file{^'/\1.map/'}>: cxx{~'/\1/'}
{{
$cxx.path -o $path($>[0]) "-Wl,-Map=$path($>[1])" $path($<[0])
}}
On the right hand side of : in the pattern we have
prerequisites which can be patterns, substitutions, or non-patterns. For
example:
<exe{~'/(.*)/'} file{^'/\1.map/'}>: cxx{~'/\1/'} hxx{^'/\1/'} \
hxx{common}
{{
$cxx.path -o $path($>[0]) "-Wl,-Map=$path($>[1])" $path($<[0])
}}
Substitutions on the left hand side of : and substitutions
and non-patterns on the right hand side are added to the dependency
declaration. For example, given the above rule and dependency declaration,
the effective dependency is going to be:
<exe{hello} file{hello.map}>: cxx{hello} hxx{hello} hxx{common}
Similar to ad hoc recipes, ad hoc rules can be written in Buildscript or C++.
3.3 Complete C++20 modules support with GCC
The build2 build system now provides conforming and scalable
support for all the major C++20 Modules features when used with GCC. This
includes named modules, module partitions (both interface and
implementation), header unit importation, and include translation. All of
these features are also supported in libraries, including consumption of
installed libraries with information about modules and importable headers
conveyed in pkg-config files. As part of this effort we have
also created a collection of examples that
demonstrate C++20 Modules features that impact the build process. See the earlier announcement for
details on this support.
3.4 Warning suppression from external C/C++ libraries
The c and cxx modules now define a notion of a
project's internal scope. If specified, header search path options
(-I) exported by libraries that are outside of the internal
scope are automatically translated to appropriate "external header search
path" options (-isystem for GCC/Clang, /external:I
for MSVC 16.10 and later). This suppresses compiler warnings in such
external headers (/external:W0 is automatically added unless a
custom /external:Wn is specified).
In the future this functionality will be extended to side-building BMIs for external module interfaces and header units.
Note that this functionality is not without limitations and drawbacks and, if needed, should be enabled explicitly. See the Compilation Internal Scope in the build system manual for details.
3.5 Pre-defined config.<project>.develop
variable
This config.<project>.develop variable allows a project
to distinguish between development and consumption builds. While normally
there is no distinction, sometimes a project may need to provide additional
functionality during development. For example, a source code generator which
uses its own generated code in its implementation may need to provide a
bootstrap step from the pre-generated code. Normally, such a step is only
needed during development.
If used, this variable should be explicitly defined by the project with
the bool type and the false default value. For
example:
config [bool] config.hello.develop ?= false
The project manager initializes a project for development unless an alternative value is specified on the command line. For example:
$ bdep init @install config.hello.develop=false
To change the development mode of an already initialized project, use
bdep-sync:
$ bdep sync @install config.hello.develop=false
See Project Configuration in the build system manual for details.
3.6 Automatic DLL symbol exporting
It is now possible to automatically generate a .def file
that exports all symbols from a Windows DLL both for C and C++ libraries.
Here is a typical arrangement (using a C++ library as an example):
lib{foo}: libul{foo}: {hxx cxx}{**} ...
lib{foo}: def{foo}: include = ($cxx.target.system == 'win32-msvc')
def{foo}: libul{foo}
if ($cxx.target.system == 'mingw32')
cxx.loptions += -Wl,--export-all-symbols
See Automatic DLL Symbol Exporting in the build system manual for details.
3.7 Kconfig configuration support
The build2 toolchain now includes the libbuild2-kconfig
standard pre-installed build system module
that allows using the Linux kernel configuration system (Kconfig) to
configure build2-based projects.
Kconfig is the configuration system of the Linux kernel. Over the years it has evolved into a sophisticated variability modeling language and a toolset that are used by the Linux kernel to manage over ten thousand configuration options. It is also increasingly being used by other projects that need to manage complex configurations.
The build2 Kconfig module acts as a second-level
configuration mechanism to the builtin configuration support. Specifically,
it integrates the execution of one of the Kconfig configurators into the
configure meta-operation and loads the resulting configuration
file presenting its values as kconfig.* variables in
buildfiles. See the earlier announcement for details on
this functionality.
4 Project Dependency Manager
4.1 Build-time dependencies
Normally our projects will have libraries as dependencies. Such dependencies are called runtime, since our projects need them during execution. However, sometimes we may only wish to use a dependency during the build, typically a tool, such as a source code generator. This kind of dependency is called build-time.
Why do we need to distinguish between the two kinds of dependencies? The primary reason is cross-compilation: if we build a tool in the same (cross-compiling) build configuration as our project, then we will not be able to execute it during the build (since it's built for a different target than what we are running). But even if we are not planning to cross-compile, there are other good reasons: if we have multiple build configurations for our project, we may want to share a single build of the tool between them (why waste time building the same thing multiple times). And even if we only have a single build of our project, we may want to build the tool with different options (for example, optimized instead of debug).
In order to properly support build-time dependencies, we need to distinguish them from runtime and we need an ability to build them in a separate build configuration. The bulk of the time spent on this release went into making this happen automatically. For details and examples see Build-Time Dependencies and Linked Configurations in the toolchain introduction.
4.2 Build configuration linking
Building build-time dependencies in separate configurations is just one application of the more general configuration linking mechanism which allows us to build a package in one configuration while its dependencies – in one or more linked configurations. This, for example, can be used to create a "base" configuration with common dependencies that are shared between multiple configurations (sometimes also referred to as build configuration overlaying). For details and examples see Build-Time Dependencies and Linked Configurations in the toolchain introduction.
4.3 Configuration preservation during synchronization
Configuration of a project is now preserved during synchronization. To
reconfigure a project from scratch, use the new --disfigure
bdep-sync option. For example (see config.<project>.develop for
background):
$ bdep sync config.hello.develop=false # develop=false (new) $ bdep sync # develop=false (preserved) $ bdep sync --disfigure # develop=true (default)
5 Package Dependency Manager
5.1 Build-time dependencies and configuration linking
Support for build-time dependencies and configuration linking in the project manager is
built on top of the corresponding mechanism in the package manager, which
means this functionality (including automatic host and module configuration
creation) is available when using bpkg directly, for example,
for package consumption. See bpkg-cfg-create(1)
for details.