Using Symlinks in build2 Projects

TL;DR: Symlinks (short for symbolic links) is a powerful mechanism for organizing one's projects that, with a bit of effort, can be made to work even on Windows. As a result, we consider symlinks (to both files and directories) fair game for package development (as opposed to package consumption; see below for details).

At the time of this writing, popular git hosting services such as GitHub and GitLab still showed symlinks in their web interfaces as text files with the link target path as their contents. Hopefully this will change in the future.

2Symlinks and Windows
3Symlinks and Package Consumption

1 Use-Cases

In build2 projects symlinks are commonly used to make the same files appear in multiple places or to non-invasively overlay an existing project with buildfiles and, potentially, reorganize its structure to better align with the supported source/output arrangements.

A typical example of the first case are files such as, LICENSE, AUTHORS, etc., that are shared by several packages in a multi-package project. For example:

├── .git/
├── hello/
│   ├── ...
│   ├── LICENSE  -> ../LICENSE
│   └── manifest
├── libhello/
│   ├── ...
│   ├── LICENSE  -> ../LICENSE
│   └── manifest
└── packages.manifest

See libbuild2-hello for a concrete project that employs this technique.

The second situation normally arises when packaging third-party projects for build2. If the project being packaged is hosted in a git repository, then it can be made available inside the package repository as a git submodule (customarily in a directory called upstream/) with the relevant source files picked and arranged in a desired filesystem structure using symlinks. For example:

├── .git/
├── libz/
│   ├── libz/
│   │   ├── zlib.h   -> ../../upstream/zlib.h
│   │   ├── zutil.c  -> ../../upstream/zutil.c
│   │   ├── ...
│   │   └── buildfile
│   ├── tests/
│   │   └── minigzip/
│   │       ├── minigzip.c -> ../../../upstream/test/minigzip.c
│   │       └── buildfile
│   └── manifest
├── upstream/
│   └── ...
└── packages.manifest

The organization contains many other (including more elaborate), examples.

2 Symlinks and Windows

Symlinks are widely used in the Unix world but until recently required running with administrative privileges on Windows, making them effectively unusable. However, starting with Windows 10 update 1703, enabling the Developer Mode removes this restriction. See Symlinks in Windows 10 for details, including on how to create symlinks on Windows.

With the Developer Mode enabled, there are several ways to instruct git (specifically, the commonly used Git for Windows distribution), to use real symlinks on Windows. You can check the "Enable symbolic links" checkbox in the "Configure extra options" installer window to enable symlinks by default. This effectively sets the global core.symlinks git configuration value to true so the same can be achieved post-installation with:

> git config --global core.symlinks true

Alternatively, we can enable symlinks on the per-repository basis before cloning:

> git clone -c core.symlinks=true ...

Note also that unlike POSIX, Windows distinguishes between file symlinks and directory symlinks. However, git does not record the symlink type and will always attempt to create a file symlink. As a result, if a directory symlink is used by a project and such a project is developed on Windows, then the type of this symlink needs to be explicitly specified in the .gitattributes file (see the relevant git commit for details). For example, if src/ is a symlink that points to upstream/src/, then we need to add the following entry to .gitattributes:

src symlink=dir

3 Symlinks and Package Consumption

The use of symlinks is limited to package development because they require the Developer Mode which not all Windows users may be able or willing to enable. The build2 toolchain includes extra mechanisms for making sure packages that use symlinks in their git repositories can still be consumed on platforms that lack the necessary support. Specifically, when preparing a package's archive distribution, all symlinks are converted to copies (including those with relative target paths pointing to entries within the package). And when directly checking out a package from its git repository, bpkg detects symlinks and if real symlinks are not available, automatically converts them to hardlinks/junctions if possible and to copies otherwise.

To make sure that all the published packages can be consumed without symlink support, Windows machines in the CI service are running with the Developer Mode disabled.