build2 | 0.12.0 Release Notes
These notes provide a more detailed discussion of major new features,
including the motivation for implementing them and their usage examples. For
the complete list of changes, refer to the Release
Announcement or the NEWS files in the individual packages.
See also the discussion of these release notes on r/cpp/
and r/programming/.
The main objective of this release was support for build system modules and it took a while to tie up all the loose ends in this area. As a result, this release has accumulated quite a large number of other major new features across all the tools in the toolchain (build system, package and project dependency managers, etc). While the below somewhat long notes discuss most of them in detail, here is a one line TL;DR and a link for each feature:
A note on backwards compatibility: in this release the build system saw a
major reorganization (discussed below). As a result, this release cannot be
upgraded to from 0.11.0 and has to be installed from
scratch.
In addition to the toolchain changes, the CI service now offers the following new build configurations:
linux_debian_10-gcc_9.2 linux_debian_10-clang_9.0 linux_debian_10-clang_9.0_libc++ macos_10.15-clang_11.0 macos_10.15-gcc_9.2_homebrew freebsd_12-clang_8.0 windows_10-msvc_16.3 windows_10-clang_8.0_msvc_16.3 windows_10-clang_9.0_msvc_16.3 windows_10-clang_9.0_msvc_16.3_lld
The total now stands at 40 build configurations and covers a wide range of versions for all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, and FreeBSD).
And in other news, we now have the Contributing section on our Community page as well as the Project Roadmap.
1 Toolchain
1.1 Local toolchain installation
By default the install scripts download source archives necessary to
bootstrap the toolchain (build2-toolchain plus, in case of
Windows, base utilities) and build the final installation from packages
obtained from the package repository (cppget.org). This allows
upgrading to a newer version of build2 directly from packages
and without having to go through the bootstrap process.
Now all the install scripts and batch files support the
--local option to skip building the final version from packages
and instead build it from the build2-toolchain archive
directly. This reduces the build time by about half and also allows
performing an offline installation by pre-downloading the necessary source
archives as described in How to
install offline? Note that in case of a local installation, an upgrade
to a newer version involves performing the same from-scratch
installation.
1.2 Default options files (aka tool config files)
All the tools in the build2 toolchain now support
configuration via pre-defined files. However, instead of having a separate
config file format, the build2 toolchain uses default
options files which contain the same options as what can be specified on
the command line.
The default options files are searched for in the .build2/
subdirectories of each outer directory beginning from the start
directory until reaching either the home directory or the filesystem
root and then in the .build2/ subdirectory of the home
directory and finally in the system directory (for example,
/etc/build2/) if configured.
Where exactly the start directory is as well as which files are searched for is defined for each tool. And, for some tools, like the project and package dependency managers – for each command (and, sometimes, even for each command's mode).
For example, most bdep commands start searching for default
options files in the project directory. This allows us to customize the
behavior of the bdep's commands on the per-project basis and
even commit them to the project's repository so that they can be shared
between everyone working on the project. For example, to automatically use
the .hpp/.cpp naming scheme for our source code we
could add the .build2/bdep-new.options file with the following
contents to our project's repository:
--lang c++,extension=?pp
See the b(1) (the
DEFAULT OPTIONS FILES section) as well as the bdep-default-options-files(1)
and bpkg-default-options-files(1)
man pages for details.
2 Build System
2.1 Dynamically-buildable/loadable build system modules
A major chunk of build system work went into support for build system
modules that are developed independently of the build system core. Such
modules can be used to extend the build system to support new languages,
source code generators, new operations, functions, target types, etc. In
fact, build2 was designed from the ground up to allow external
modules to customize pretty much every aspect of the build system
functionality. For example, the configuration support is provided by the
config module that is bundled with the build system core but if
you wanted to, you could write your own and use that instead.
In a nutshell, a build system module is a separately-packaged shared
library project with the libbuild2-<modname> name. When
build2 sees using <modname> in a
buildfile, it will build and load this library.
In this release we have the ability to dynamically update (if necessary) and load such modules. Plus, the CI service fully supports testing build system modules so that you can make sure modules that you develop work on all the major platforms and compilers. In the next release we plan to plumb this support through the package manager so that the whole process of downloading, building, and loading a build system module (as well as any tools, such as source code generators, that they may depend on) is completely seamless.
If you would like to start playing with build system modules, there is libbuild2-hello
to get you started. Plus some folks have already begun experimenting with
real modules: see the "Yacc
module?" mailing list thread (continues
here). Finally, see the Build
system reorganization mailing list post for technical details on modules
support.
2.2 Build system as a library and build contexts
Two other interesting features came from the work on build system modules: build system as a library and the notion of a build context.
Before this release the build system was all in the b
executable. Now it is split into libbuild2 (build system core),
a bunch of build system modules (for example, libbuild2-c,
libbuild2-cxx, etc; called bundled modules) and the
b executable (build system driver). With this arrangement,
someone else (for example, an IDE author) can write their own build system
driver with their own set of modules, build state loading logic, and so on.
In particular, this could be a great way to experiment with the
build-by-convention ideas, alternative build description formats,
etc. See the Build
system reorganization mailing list post for details.
And in order to support automatic building of build system modules we needed the ability to preempt the current build in order to build the module. To support this we have introduced the notion of a build context which contains all the non-const global state of a build system. As a result we can now have multiple independent builds at the same time.
2.3 Pattern matching (switch)
If you are not familiar with pattern matching from other languages, an example is probably the best way to illustrate the idea:
switch $cxx.target.class, $cxx.target.system
{
case 'windows', 'mingw32'
cxx.libs += -lrpcrt4
case 'windows'
cxx.libs += rpcrt4.lib
case 'macos'
cxx.libs += -framework CoreFoundation
}
One interesting feature of build2's switch is
the ability to specify the match function so that we can use other kinds of
patterns in addition to straight comparison (regex, shell wildcard patterns,
case-insensitive comparison, etc). For example:
switch $cxx.target.cpu: regex.match
{
case 'i[3-6]86'
...
case 'x86_64'
...
}
See Pattern
Matching (switch) for details.
2.4 Clang targeting MSVC and MSVC installation discovery
This release adds first-class support for Clang targeting the MSVC
runtime, including using LLD. In particular, this means the
build2 toolchain itself can now be built with Clang on Windows
(see build2-install-clang.bat)
and several Clang MSVC build
configurations are now part of our CI service.
We were pleasantly surprised by how stable Clang's support for targeting MSVC is. 8.0.1 that ships with Visual Studio has been rock solid and we ran into only one mis-compile in 9.0.0 which has already been fixed for 10 and 9.0.1. What's more, LLVM binutils (LLD, LLVM-lib, etc) work well as replacements for Microsoft's versions and we even get thin archive support with LLVM-lib. And now that the MSVC runtime itself is open source, this opens up a real possibility for cross-compiling to Windows.
Note also that build2 uses the "straight" Clang drivers and
not the clang-cl wrapper (but limited support for
clang-cl is also provided). See Clang
Compiler Toolchain for details.
As part of this work we have also implemented automatic installation discovery for MSVC 15 (2017) and later, similar to how Clang does it. In particular, this allows building outside the Visual Studio development command prompts. See MSVC Compiler Toolchain for details.
2.5 Configuration exporting and importing
The new config.config.save variable can be used to specify
the alternative file to write the configuration to as part of the
configure meta-operation. For example:
$ b configure: proj/ config.config.save=my-config.build
The new config.config.load variable specifies additional
configuration files to be loaded after the project's default
config.build, if any. For example:
$ b create: cfg/,cc config.config.load=my-config.build
Both config.config.load and config.config.save
recognize the special - file name as an instruction to
read/write from/to stdin/stdout, respectively. For example:
$ b configure: src-prj/ config.config.save=- | \ b configure: dst-prj/ config.config.load=-
The config.config.load also recognizes the
~host special configuration name. This is the default host
configuration that corresponds to how the build system itself was built.
For example:
$ b create: tools/,cc config.config.load=~host
2.6 Compiler mode options as part of
config.{c,cxx}
In this release we have extended the config.c and
config.cxx variables' semantics to allow specifying compile
mode options. Such options are not overridden by buildfiles and are
passed last (after cc.coptions and
{c,cxx}.coptions) in the resulting command lines. Note that
they are also cross-hinted between config.c and
config.cxx. For example:
$ b config.cxx="g++-9 -m32" # implies config.c="gcc-9 -m32"
But:
$ b config.cxx="clang++ -stdlib=libc++" config.c=clang
3 Project Dependency Manager
3.1 Creating new projects with existing files
The bdep-new
command now supports creating new projects with existing files. It also
recognizes and handles the following existing filesystem entries in the
project root:
.git -- assume VCS is already initialized and is Git LICENSE -- try to guess the manifest license from contents README.md -- try to extract the manifest summary line from contents
Overall, the idea is to streamline the workflow where we create a project
on one of the hosting services (GitHub, GitLab, etc) and then initialize it
with bdep-new.
Also, to this effect, specifying the project name is now optional and if omitted, the current working directory is assumed to be the project name.
Here is an example of this streamlined workflow:
$ # create project with LICENSE and README.md on a Git hosting service $ git clone .../libhello.git $ cd libhello $ bdep new -t lib
You can also see it in action in the CppCon 2019 Why is C++ so Slow? lightning talk.
4 Package Dependency Manager
4.1 Specifying full version constraint on the command line
The pkg-build
command now allows specifying the full package version constraint on the
command line. For example:
$ bpkg build "bar < 2.0.0"
See bpkg-pkg-build(1)
for details.