Boost Libraries Packaged for `build2`

Posted on 02 Nov 2021 by Boris Kolpackov with comments on r/cpp/

We have added support for the build2 build system to 136 out of 145 Boost libraries and version 1.77.0 is now available from the cppget.org package repository. This means that you can now use Boost as dependencies in your projects with everything building using the same build system and in a single invocation (see benefits of this approach).

Note that while the Boost versioning scheme might look like semver, it is not, with breaking changes routinely introduced without incrementing the major version component. Also note that the dependencies between Boost libraries themselves use the == constraint which means only libraries from a single version of Boost can be used in any given build.

As a result, if you would like to be on the conservative side, use the ~ constraint to only allow changes in the patch version component:

depends: libboost-multi-index ~1.77.0

Otherwise, the ^ constraint is a reasonable choice, especially if you are working on a library that you plan to publish to cppget.org (if several libraries use ~, it may be impossible to use them together):

depends: libboost-multi-index ^1.77.0

See Package Version Constraint for background.

With this conversion, Boost libraries are now continuously built by our CI service. While things work reasonably well on established platform/compiler combinations, some newer and alternative build configurations have issues. Notable standouts with a substantial number of build errors are Clang targeting MSVC, Mac OS with GCC, and Emscripten (see all build errors; note that Mac OS with GCC and Emscripten builds are currently excluded as being unusable). We have reported these issues to the Boost developers.

There are also quite a few libraries with warnings and you may want to enable the Compilation Internal Scope functionality to suppress them in your projects.

The following Boost libraries are not packages as of version 1.77.0 (see reasons):

libboost-compatibility
libboost-compute
libboost-coroutine
libboost-graph-parallel
libboost-mpi
libboost-parameter-python
libboost-property-map-parallel
libboost-python
libboost-wave

Implementation Details

This section provides additional details on the conversion.

The conversion work is performed in the github.com/build2-packaging/boost repository. Initially we used the Boost superproject as a git submodule for the upstream source but that proved impractical during CI: it took longer to fetch all the recursive submodules than to compile and link. As a result, we now import a copy of the source files for each release.

Due to the large number of libraries as well as frequent additions of new libraries, a typical manual conversion was not an option. As a result, the process is automated with a script that performs the following main steps:

Levelize the libraries according to their dependencies.
This is done with the help of the boostdep tool and is made possible by the recent efforts of Boost developers to get rid of dependency cycles. While this levelization is not strictly necessary for the end result, it allowed us to convert and test the libraries incrementally, one level at a time.
Generate build system and package information with bdep-new.
With support for various source code layouts in bdep-new we were able to fairly easily re-create the layout used by the Boost libraries (add ,binless for a header-only library):
```
bdep new                                                    \
  --lang c++                                                \
  --type lib,split,subdir=boost,no-subdir-source,no-version \
  libboost-foo
```
The result required only automated tweaks for most libraries. For some libraries, however, we had to provide customized buildfiles as overrides (see the downstream/ directory in the repository).
Add dependency information to package manifest.
Again with the help of boostdep we automatically determine the list of direct dependencies for each library and add it to the package manifest file.
Copy information from meta/libraries.json to package manifest.
All the Boost libraries provide the meta/libraries.json file that contains machine-readable meta-information about the library. We copy some of this information, such as the library description, to the generated package manifest file.

One major area that has not yet been dealt with is tests. Currently, each library only includes a "smoke test" that is auto-generated for most libraries and is manually written for the more tricky ones (again, see the downstream/ directory in the repository).

The inter-dependence of Boost libraries is now the stuff of legends and it uncovered some performance corner cases in both the build2 build system and the package manager. These were largely addressed in the latest release of build2. However, to improve the performance even further, each Boost package also deduplicates its interface dependencies. For example:

import intf_libs  = libboost-assert%lib{boost_assert}
...
...
...
import intf_libs += libboost-utility%lib{boost_utility}

intf_libs = $cxx.deduplicate_export_libs($intf_libs)

Boost Libraries Packaged for build2

Implementation Details

Boost Libraries Packaged for `build2`