`build2` | `0.15.0` Release Notes

These notes provide a more detailed discussion of major new features, including the motivation for implementing them and their usage examples. For the complete list of changes, refer to the Release Announcement or the NEWS files in the individual packages. See also the discussion of this release on lobste.rs, r/cpp/, and r/programming/.

The overall theme of this release is more advanced functionality that is becoming necessary as we start to use build2 to handle more complex projects. Specifically, the package manager now supports a number of more advanced dependency declaration features, including conditional dependencies, dependency alternatives, and dependency configuration. On the build system side we now have rule hints, support for dynamic dependencies in ad hoc recipes, and the ability to save user metadata in C/C++ libraries (plus 23 other items mentioned in the NEWS file). The following sections discuss these and other new features in detail.

Another new development is the creation of the HOWTO repository with practical advice on using build2 to achieve common tasks. At the time of the release the repository contained 10 articles.

A note on backwards compatibility: this release cannot be upgraded to from 0.14.0 and has to be installed from scratch.

Infrastructure

1.1	New CI configurations

Toolchain

2.1	New standard pre-installed build system modules
2.2	JSON output
2.3	Performance optimizations

Package Dependency Manager

3.1	Dependency groups
3.2	Conditional dependencies
3.3	Dependency alternatives
3.4	Dependency configuration

Build System

4.1	Rule hints
4.2	User metadata in C/C++ libraries
4.3	Dynamic dependencies in ad hoc recipes

Project Dependency Manager

5.1	Limited support for packages with non-standard version

1 Infrastructure

1.1 New CI configurations

The following new build configurations have been added to the CI service:

freebsd_13-clang_13.0

linux_debian_11-gcc_12.1
linux_debian_11-clang_14.0[_libc++]

macos_12-clang_13.1          (Xcode 13.4 Clang 13.1.6)
macos_12-gcc_12.1_homebrew

windows_10-msvc_17.2
windows_10-clang_14.0_llvm_msvc_17.2[_lld]
windows_10-gcc_11.2_mingw_w64

linux_debian_11-emcc_3.1.6   (Emscripten)

In addition, all the Linux configurations in the default class now include system packages necessary for GUI development and run a mock X server via Xvfb.

All in all, there are now 66 build configurations that cover a wide range of versions for all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, FreeBSD as well as WebAssembly).

2 Toolchain

2.1 New standard pre-installed build system modules

This release adds one new standard pre-installed build system module: autoconf (see the earlier announcement for background on standard pre-installed modules).

The autoconf module provides GNU Autoconf emulation for processing config.h.in files (or their CMake/Meson variants). Similar to Autoconf, this module provides built-in support for a number of common HAVE_* configuration options. However, the values of these options are not discovered by dynamic probing, such as trying to compile a test program to check if the feature is present. Instead, they are set to static expected values based on the platform/compiler macro checks. Using this module simplifies porting projects that rely on this functionality in their existing build systems.

2.2 JSON output

All the information-querying commands in the package and project managers as well as the structured result output and the info meta-operation in the build system can now produce JSON output. This should make it easier to integrate build2 with other tools, IDEs, etc.

For details see the --stdout-format option in bpkg-pkg-status(1), bdep-status(1), and bdep-config-list(1) as well as the --structured-result option and info meta-operation in b(1).

2.3 Performance optimizations

This release includes a large number of performance optimizations in the build system which should especially help projects that make use of heavily inter-dependent libraries (such as Boost). For example, on some tests the up-to-date check times went down from 1.2s to 0.3s.

3 Package Dependency Manager

3.1 Dependency groups

It is now possible to specify multiple packages within a single depends value in the manifest and they can share a version constraint. For example:

depends: { libboost-any libboost-log libboost-uuid } ~1.77.0

See the depends package manifest value for details.

3.2 Conditional dependencies

Sometimes we may only need a library on a certain platform or if a certain feature of our project is enabled. This can now be achieved with conditional dependencies. For example:

depends: libposix-getopt ^1.0.0 ? ($cxx.target.class == 'windows')

depends: libcurl ^7.76.0 ? ($config.hello.network)

The condition expression is evaluated after loading the project's root.build. As a result, variable values set by build system modules (like cxx.target.class above) that are loaded in root.build as well as the project's configuration (like config.hello.network above) can be referenced in dependency conditions. However, there are implications of the package manager now acting as a special build system driver which are discussed in Package Build System Skeleton.

See the depends package manifest value for details.

3.3 Dependency alternatives

Sometimes our project could use several alternatives for a dependency. For example, our project could work with MySQL or MariaDB. Or we could use either version 5 or 6 of the Qt libraries. This can now be expressed with dependency alternatives. For example:

depends: libmysqlclient ^5.0.3 | libmariadb ^10.2.2

depends: libQt5Core ^5.15.3 | libQt6Core ^6.2.2

To communicate to the build system which dependency alternative was selected we can use reflected configuration variables. For example:

depends: libmysqlclient ^5.0.3  config.hello.db='mysql' | \
         libmariadb     ^10.2.2 config.hello.db='mariadb'

In the above example, our project will be configured with config.hello.db set to either mysql or mariadb depending on which alternative was selected by the package manager.

See the depends package manifest value for details.

3.4 Dependency configuration

Probably the most important new package manager feature is the ability to configure dependencies. Let's say libmariadb exposed three configuration variables: boolean config.libmariadb.cache (enable client-side caching support) and config.libmariadb.tls (enable secure connections) as well as integer config.libmariadb.buffer (the size of some hypothetical buffer). If all we need is to enable a few features (that is, set a few bool configuration variables to true), then we can use the require dependency clause. For example (notice that we have switched to the multi-line form of the depends value):

depends:
\
libmariadb ^10.2.2
{
  require
  {
    config.libmariadb.cache = true

    if ($cxx.target.class != 'windows')
      config.libmariadb.tls = true
  }
}
\

The contents of the require clause is a buildfile fragment that is expected to set one or more dependency configuration variables. As we can see from the above example, it can contain some elaborate logic, such as conditions based on values set by build system modules or the project's configuration (including reflected by previous depends values).

If we need to set any non-boolean configuration variables, then instead of require we use the prefer and accept clauses. For example:

depends:
\
libmariadb ^10.2.2
{
  # We prefer the cache but can work without it.
  # We need the buffer of at least 4KB.
  #
  prefer
  {
    config.libmariadb.cache = true

    config.libmariadb.buffer = ($config.libmariadb.buffer < 4096 \
                                ? 4096                           \
                                : $config.libmariadb.buffer)
  }

  accept ($config.libmariadb.buffer >= 4096)
}
\

As shown in the above example, prefer/accept allow us to express more complex dependency configuration semantics.

For details and more examples, see the depends package manifest value and Dependency Configuration Negotiation.

4 Build System

4.1 Rule hints

Rule hints can be used to resolve ambiguity when multiple rules match the same target as well as to override an unambiguous match. A rule hint is specified as a target attribute. For example, to link a C executable as C++:

[rule_hint=cxx] exe{hello}: c{hello}

The C/C++ link rule now supports matching libraries without any sources or headers with a hint. This seemingly strange arrangement can be useful for creating "metadata libraries" whose only purpose is to convey metadata (options to use and/or libraries to link).

4.2 User metadata in C/C++ libraries

Speaking of C/C++ libraries, this release adds support for conveying user metadata with such libraries, including in the generated pkg-config files. For example, we may need to pass the configuration information to the library's tests so that they know which features are enabled and therefore should be tested. Or we may need to let the users of our library know the location of its assets.

For details on how to achieve this see the How do I convey additional information (metadata) with executables and C/C++ libraries? HOWTO article (which, as the title suggests, also explains how to do the same for executables).

4.3 Dynamic dependencies in ad hoc recipes

The previous two releases added support for ad hoc recipes and pattern rules. However, one prominent feature that was still missing is support for dynamically discovered dependencies (as, for example, produced by the GCC/Clang -M option family). This release fills that gap.

Specifically, the depdb builtin now has the new dyndep command that can be used to extract dynamic dependencies from program output or a file. For example, from program output:

obje{hello.o}: cxx{hello}
{{
  s = $path($<[0])
  o = $path($>)

  poptions = $cxx.poptions $cc.poptions
  coptions = $cc.coptions $cxx.coptions

  depdb dyndep $poptions --what=header --default-type=h -- \
    $cxx.path $poptions $coptions $cxx.mode -M -MG $s

  diag c++ ($<[0])

  $cxx.path $poptions $coptions $cxx.mode -o $o -c $s
}}

Or, alternatively, from a file:

obje{hello.o}: cxx{hello}
{{
  s = $path($<[0])
  o = $path($>)
  t = $(o).t

  poptions = $cxx.poptions $cc.poptions
  coptions = $cc.coptions $cxx.coptions

  depdb dyndep $poptions --what=header --default-type=h --file $t -- \
    $cxx.path $poptions $coptions $cxx.mode -M -MG $s >$t

  diag c++ ($<[0])

  $cxx.path $poptions $coptions $cxx.mode -o $o -c $s
}}

The above depdb-dyndep commands will run the C++ compiler with the -M -MG options to extract the header dependency information, parse the resulting make dependency declaration (either from stdout or from file) and enter each header as a prerequisite of the obje{hello.o} target, as if they were listed explicitly. It will also save this list of headers in the auxiliary dependency database (hello.o.d file) in order to detect changes to these headers on subsequent updates. The --what option specifies what to call the dependencies being extracted in diagnostics. The --default-type option specifies the default target type to use for a dependency if its file name cannot be mapped to a target type.

The above depdb-dyndep variant extracts the dependencies ahead of the compilation proper and will handle auto-generated headers (see the -MG option semantics for details) provided we pass the header search paths where they could be generated with the -I options (passed as $poptions in the above examples).

If there can be no auto-generated dependencies or if they can all be listed explicitly as static prerequisites, then we can use a variant of the depdb-dyndep command that extracts the dependencies as a by-product of compilation. In this mode only the --file input is supported. For example (assuming hxx{config} is auto-generated):

obje{hello.o}: cxx{hello} hxx{config}
{{
  s = $path($<[0])
  o = $path($>)
  t = $(o).t

  poptions = $cxx.poptions $cc.poptions
  coptions = $cc.coptions $cxx.coptions

  depdb dyndep --byproduct --what=header --default-type=h --file $t

  diag c++ ($<[0])

  $cxx.path $poptions $coptions $cxx.mode -MD -MF $t -o $o -c $s
}}

Other options supported by the depdb-dyndep command:

--format <name>: Dependency format. Currently only the make dependency format is supported and is the default.
--cwd <dir>: Working directory used to complete relative dependency paths. This option is currently only valid in the --byproduct mode (in the normal mode relative paths indicate non-existent files).
--adhoc: Treat dynamically discovered prerequisites as ad hoc (so they don't end up in $<; only in the normal mode).
--drop-cycles: Drop prerequisites that are also targets. Only use this option if you are sure such cycles are harmless, that is, the output is not affected by such prerequisites' content.
--update-{include,exclude} <tgt>|<pat>: Prerequisite targets/patterns to include/exclude (from the static prerequisite set) for update during match (those excluded will be updated during execute). The order in which these options are specified is significant with the first target/pattern that matches determining the result. If only the --update-include options are specified, then only the explicitly included prerequisites will be updated. Otherwise, all prerequisites that are not explicitly excluded will be updated. If none of these options is specified, then all the static prerequisites are updated during match. Note also that these options do not apply to ad hoc prerequisites which are always updated during match.

The common use-case for the --update-exclude option is to omit updating a library which is only needed to extract exported C/C++ preprocessor options. Here is a typical pattern:

import libs = libhello%lib{hello}

libue{hello-meta}: $libs

obje{hello.o}: cxx{hello} libue{hello-meta}
{{
  s = $path($<[0])
  o = $path($>)

  lib_poptions = $cxx.lib_poptions(libue{hello-meta}, obje)
  depdb hash $lib_poptions

  poptions  = $cxx.poptions $cc.poptions $lib_poptions
  coptions  = $cc.coptions $cxx.coptions

  depdb dyndep $poptions --what=header --default-type=h \
    --update-exclude libue{hello-meta} -- \
      $cxx.path $poptions $coptions $cxx.mode -M -MG $s

  diag c++ ($<[0])

  $cxx.path $poptions $coptions $cxx.mode -o $o -c $s
}}

As another example, sometimes we need to extract the "common interface" preprocessor options that are independent of the library type (static or shared). For example, the Qt moc compiler needs to "see" the C/C++ preprocessor options from imported libraries if they could affect its input. Here is how we can implement this:

import libs = libhello%lib{hello}

libul{hello-meta}: $libs

cxx{hello-moc}: hxx{hello} libul{hello-meta} $moc
{{
  s = $path($<[0])
  o = $path($>[0])
  t = $(o).t

  lib_poptions = $cxx.lib_poptions(libul{hello-meta})
  depdb hash $lib_poptions

  depdb dyndep --byproduct --drop-cycles --what=header \
    --default-type=h --update-exclude libul{hello-meta} --file $t

  diag moc ($<[0])

  $moc $cc.poptions $cxx.poptions $lib_poptions \
    -f $leaf($s) --output-dep-file --dep-file-path $t -o $o $s
}}

Planned future improvements include support for the lines (list of files, one per line) input format in addition to make and support for dynamic targets in addition to prerequisites.

5 Project Dependency Manager

5.1 Limited support for packages with non-standard version

While bdep can only be used on packages that follow the standard versioning scheme (a subset of semantic versioning), the bdep-ci(1) and bdep-publish(1) commands can now be used on packages that utilize other versioning schemes (as long as they are valid package versions). This helps with testing and publishing of third-party projects that are being packaged for build2.

build2 | 0.15.0 Release Notes

1 Infrastructure

1.1 New CI configurations

2 Toolchain

2.1 New standard pre-installed build system modules

2.2 JSON output

2.3 Performance optimizations

3 Package Dependency Manager

3.1 Dependency groups

3.2 Conditional dependencies

3.3 Dependency alternatives

3.4 Dependency configuration

4 Build System

4.1 Rule hints

4.2 User metadata in C/C++ libraries

4.3 Dynamic dependencies in ad hoc recipes

5 Project Dependency Manager

5.1 Limited support for packages with non-standard version

`build2` | `0.15.0` Release Notes