Using Symlinks in build2 Projects
TL;DR: Symlinks (short for symbolic links) is a powerful mechanism for organizing one's projects that, with a bit of effort, can be made to work even on Windows. As a result, we consider symlinks (to both files and directories) fair game for package development (as opposed to package consumption; see below for details).
At the time of this writing, popular git hosting services
such as GitHub and GitLab still showed symlinks in their web interfaces as
text files with the link target path as their contents. Hopefully this will
change in the future.
| 1 | Use-Cases |
|---|---|
| 2 | Symlinks and Windows |
| 3 | Symlinks and Package Consumption |
1 Use-Cases
In build2 projects symlinks are commonly used to make the
same files appear in multiple places or to non-invasively overlay an
existing project with buildfiles and, potentially, reorganize its structure
to better align with the supported source/output arrangements.
A typical example of the first case are files such as
README.md, LICENSE, AUTHORS, etc.,
that are shared by several packages in a multi-package project. For
example:
hello/ ├── .git/ ├── hello/ │ ├── ... │ ├── LICENSE -> ../LICENSE │ └── manifest ├── libhello/ │ ├── ... │ ├── LICENSE -> ../LICENSE │ └── manifest ├── LICENSE └── packages.manifest
See libbuild2-hello
for a concrete project that employs this technique.
The second situation normally arises when packaging third-party projects
for build2. If the project being packaged is hosted in a
git repository, then it can be made available inside the
package repository as a git submodule (customarily in a
directory called upstream/) with the relevant source files
picked and arranged in a desired filesystem structure using symlinks. For
example:
zlib/ ├── .git/ ├── libz/ │ ├── libz/ │ │ ├── zlib.h -> ../../upstream/zlib.h │ │ ├── zutil.c -> ../../upstream/zutil.c │ │ ├── ... │ │ └── buildfile │ ├── tests/ │ │ └── minigzip/ │ │ ├── minigzip.c -> ../../../upstream/test/minigzip.c │ │ └── buildfile │ └── manifest ├── upstream/ │ └── ... └── packages.manifest
The github.com/build2-packaging
organization contains many other (including more elaborate), examples.
2 Symlinks and Windows
Symlinks are widely used in the Unix world but until recently required running with administrative privileges on Windows, making them effectively unusable. However, starting with Windows 10 update 1703, enabling the Developer Mode removes this restriction. See Symlinks in Windows 10 for details, including on how to create symlinks on Windows.
With the Developer Mode enabled, there are several ways to instruct
git (specifically, the commonly used Git for Windows distribution), to use
real symlinks on Windows. You can check the "Enable symbolic links" checkbox
in the "Configure extra options" installer window to enable symlinks by
default. This effectively sets the global core.symlinks
git configuration value to true so the same can be
achieved post-installation with:
> git config --global core.symlinks true
Alternatively, we can enable symlinks on the per-repository basis before cloning:
> git clone -c core.symlinks=true ...
Note also that unlike POSIX, Windows distinguishes between file symlinks
and directory symlinks. However, git does not record the
symlink type and will always attempt to create a file symlink. As a result,
if a directory symlink is used by a project and such a project is developed
on Windows, then the type of this symlink needs to be explicitly specified
in the .gitattributes file (see the relevant git
commit
for details). For example, if src/ is a symlink that points to
upstream/src/, then we need to add the following entry to
.gitattributes:
src symlink=dir
3 Symlinks and Package Consumption
The use of symlinks is limited to package development because they
require the Developer Mode which not all Windows users may be able or
willing to enable. The build2 toolchain includes extra
mechanisms for making sure packages that use symlinks in their
git repositories can still be consumed on platforms that lack
the necessary support. Specifically, when preparing a package's archive
distribution, all symlinks are converted to copies (including those with
relative target paths pointing to entries within the package). And when
directly checking out a package from its git repository,
bpkg detects symlinks and if real symlinks are not available,
automatically converts them to hardlinks/junctions if possible and to copies
otherwise.
To make sure that all the published packages can be consumed without symlink support, Windows machines in the CI service are running with the Developer Mode disabled.