NAME
bpkg-repository-types – repository types, structure, and URLs
DESCRIPTION
This help topic describes the repository types recognized by
bpkg, their structure, and the format of their URLs.
Currently three types of repositories are supported: archive-based
pkg, directory-based dir, and
version control-based git.
The repository location may specify the repository type as part of the
URL scheme component in the type+protocol
form. For example:
git+https://example.com/foo dir+file:///tmp/repo
Note that the explicit specification is only needed when the correct type
cannot be guessed from the URL. See bpkg-rep-add(1) for
details.
PKG REPOSITORIES
A pkg repository is archive-based. That is, it
contains a collection of various packages/versions as archive files. For
more information on the structure of pkg repositories
refer to The
build2 Package Manager Manual. The
pkg repository location can be a local directory path or
an http(s):// URL.
DIR REPOSITORIES
A dir repository is directory-based. That is,
it contains a collection of various packages as directories but only a
single version per package can be present in such a repository. The
dir repository location can be a local directory path or
a file:// URL.
A dir repository is expected to contain either the
manifest or packages.manifest file
in the root directory of the repository. If it only contains
manifest, then it is assumed to be a simple,
single-package repository with the manifest file being
its package manifest. Otherwise, the packages.manifest
file should list the locations of available packages as described in Package
List Manifest for dir Repositories.
A dir repository may also contain the
repositories.manifest file in the root directory of the
repository. This file can be used to describe the repository itself as well
as specify its prerequisite and complement repositories. See Repository
List Manifest for details on the format and semantics of this file.
GIT REPOSITORIES
A git repository is version control-based.
That is, it normally contains multiple versions of the same package (but can
also contain several, usually related, packages in the same repository).
A git repository has the same structure and manifest
files as the dir repository. See Package
List Manifest for dir Repositories and Repository
List Manifest for details on their format and semantics.
Theoretically, a git repository may contain as many
package versions as there are commits. Practically, however, we are normally
only interested in a small subset of them while fetching and processing the
necessary information for all of them could be prohibitively expensive. As
a result, by default, only advertised tags in the
refs/tags/v* form where the part after
v is also a valid standard
version are considered to be sources of useful package versions. These
commits normally correspond to released versions and are called the
default set. Note that only the latest revision of each such version
is considered.
Instead of the default set, it is possible to provide a custom set of
available versions by specifying one or more commit ids and/or references
and/or reference patterns in the repository URL fragment (see
git-ls-remote(1) for details on advertised references).
For example:
https://example.com/foo.git#v1.2.3 https://example.com/foo.git#master https://example.com/foo.git#af234f56 https://example.com/foo.git#tags/releases/* https://example.com/foo.git#HEAD,tags/v1.*.*,heads/feature-*
Furthermore, it is possible to expand (or narrow down) the default set
using the special ## fragment notation. For example:
https://example.com/foo.git##HEAD - default set plus HEAD https://example.com/foo.git##heads/* - default set plus branches https://example.com/foo.git##-v1.* - default set minus v1.*
A git repository URL fragment is a comma-separated
list of reference filters in the following form:
[refname][@commit]
Either refname, commit, or both
must be specified. If both are specified then refname is
only used to minimize the amount of data fetched and
commit is expected to belong to its history. For
example:
.../foo.git#master@48fba3625d65941bb85a39061bcf795d4949c778
The refname part can be an abbreviated commit id or
an advertised reference or reference pattern under
refs/. While commit must be the
complete, 40-characters SHA1 that need not be advertised. For convenience, a
40-characters filter that consists of only hexadecimal digits is assumed to
be commit even if not prefixed with
@. In an unlikely event this produces an incorrect
result, the @-form with omitted
commit can be used. For example:
.../foo.git#48fba3625d65941bb85a39061bcf795d4949c778 (commit id) .../foo.git#deadbeefdeadbeefdeadbeefdeadbeefdeadbeef@ (reference)
The refname part can use the *
and ? wildcard pattern characters with the standard
semantics as well as the ** character sequence which
matches in subdirectories, recursively. For example:
.../foo.git#tags/v* - tags/v1.2.3 but not tags/old/v0.1.0 .../foo.git#tags/v** - tags/v1.2.3 and tags/old/v0.1.0
A relative refname is searched for in
refs/, refs/tags/, and
refs/heads/ as well as among symbolic references like
HEAD. To anchor it to refs/ make it
absolute, for example:
.../foo.git#tags/v* - refs/tags/v1.2.3 but also refs/heads/tags/voo .../foo.git#/tags/v* - refs/tags/v1.2.3 only
While a refname pattern is allowed not match any
references, a non-pattern that doesn't resolve to a reference is
invalid.
If a refname starts with minus
(-) then it is treated as an exclusion filter –
any references that it matches are excluded from the set included by the
preceding filters (or the default set). For example:
.../foo.git#v*,-v1.* - exclude v1.* from v* .../foo.git##-v1.* - exclude v1.* from default set
To support specifying literal leading minus, a
refname that starts with plus (+) is
treated as an inclusion filter. For example:
.../foo.git#+x - include x .../foo.git#+-x - include -x .../foo.git#++x - include +x
Currently supported git protocols are
git://, ssh:// (but not
scp pseudo-URL syntax), http://, and
https:// for remote repositories and
file:// for local repositories. While
bpkg tries to minimize the amount of information
(history) fetched, it is not always possible for some protocols and/or
server configurations, as discussed next.
A git repository accessible via
http(s):// can use either dumb or smart
protocol (refer to the git documentation for details).
The dumb protocol provides only limited support for fetch minimization and
if this protocol is used, then bpkg has no choice but to
download a substantial amount of history.
The smart protocol allows fetching of minimal history for tags and
branches. Whether this is also possible for (all) commit ids depends on
whether the server is configured to allow fetching unadvertised commits. For
details, refer to the
uploadpack.allowReachableSHA1InWant and
uploadpack.allowAnySHA1InWant git
configuration values.
The git:// and ssh:// protocols
are similar to smart http:// in that they support
fetching minimal history for tags and branches and may or may not support
this for commit ids depending on the server configuration. Note, however,
that unlike for http(s)://, for these protocols
bpkg does not try to sense if fetching unadvertised
commits is allowed and always assumes that it is not. Also note that the
sensed or assumed protocol capabilities can be overridden for a
git repository URL prefix using the
--git-capabilities option (bpkg-common-options(1)).
Based on this information, to achieve optimal results the recommended
protocol for remote repositories is smart https://.
Additionally, if you are planning to refer to unadvertised commit ids, then
also consider configuring the server to allow fetching unadvertised
commits.
The file:// protocol has the same fetch minimization
support as git:// and is therefore treated the same.
BUGS
Send bug reports to the users@build2.org mailing list.