Changes to Package Review/Transition Policies

Posted on 14 Aug 2024 by Boris Kolpackov

TL;DR: We are inviting the build2 community to help with reviewing package submissions. We are also adjusting the package transition policies between queue.cppget.org and cppget.org as well as between sections within cppget.org. Specifically, migrating a package from queue.cppget.org to cppget.org no longer requires a review, only that the package builds in at least one configuration. As a result, the alpha, beta, and testing sections of cppget.org may now contain unreviewed packages. However, a package can only be migrated from testing to stable if it passes a review. Read below for details.

When we first set up cppget.org and queue.cppget.org, our plans were to allow the quality of the published packages to improve organically. Specifically, queue.cppget.org was meant to only filter out completely broken submissions that fail to build in any build configurations.

However, as the wider build2 community started publishing their packages (which were and still are dominated by packages of third-party projects converted to build2), we quickly realized that this organic policy is not going to work. The complexity and variability of C/C++ builds combined with the baggage and anti-patterns brought from upstream build systems led to a rather poor quality of the early packages. The fact that we were still working out the best practices of packaging third-party projects for build2 did not help either. As a result, we switched to a stricter policy of reviewing every package submission before it is moved to cppget.org. All these reviews were meant to be performed by the build2 core team members but practically they were all performed by me.

Several things have changed since those early days. The number of package submissions have increased substantially and I am no longer able to keep up even if I do nothing other than review packages. Bringing in other core team members will likely only help temporarily and will also slow down the build2 toolchain development. On the bright side, the quality of the submitted packages has increased substantially. Plus, we now have a detailed, step-by-step Packaging Guide as well as a sizable collection of HOWTOs. Based on this we feel that several policy adjustments in this area are now due.

Firstly, we will no longer require a review to migrate a package from queue.cppget.org to cppget.org. Instead, we are reverting to our original organic policy of only requiring one successful build. However, we will still require a successful review before we migrate a package from testing to stable within cppget.org. See Package Review for a more detailed description of package transitions and the associated policies.

Secondly, we are inviting the wider build2 community to help with reviewing package submissions. At the same time, we will limit the build2 core team reviews to foundational packages only (think, openssl, curl boost, etc) in order to free up time for the build2 toolchain development.

What are the key implications of these changes? Firstly, the alpha, beta, and testing sections of cppget.org will now contain unreviewed packages and this will likely lead to the decrease in the overall quality of these sections. Also note that while these sections were always volatile (unlike stable), the chances of packages being yanked from them are now higher. To mitigate this somewhat, we will aim not to publish foundational packages to the testing section without a review. On your side, you can switch to the stable section at the expense of potentially losing early access to new packages.

Secondly, if nobody reviews a package (or a new version of a package), then it will remain in the testing section, potentially indefinitely. So there is an incentive to review packages that you are using or would like to use if you would like for them to end up in the stable section.

Let's now discuss the package review in more detail. The process is described in the new Package Review chapter of the packaging guide and covers the initial package submission, new version, and new revision. The review is checklist-driven except for minor changes.

To support the review process there are also a number of new features in the web interface of cppget.org. Specifically, for each package version we now display the review information. For example, if you look at libsqlite3, you will see that each version has passed a review and if you click on the review number, you will see the review details, such as who performed it.

The second new web interface feature is the Advanced Package Search page that, in particular, allows you to easily get a list of unreviewed packages. We've also added the Unreviewed shortcut to the menu bar.

Putting this all together, you should now be able to easily find packages that need reviewing and follow detailed instructions to perform and submit a review.

Let's also discuss who should consider performing reviews. While anyone is welcome to submit a review with all the results recorded and displayed, when making a decision to migrate a package from testing to stable, we will only consider reviews from knowledgeable build2 users with experience packaging third-party software. The reason for this requirement is again the complexity and variability of C/C++ builds. Even with a checklist-driven review there are many nuances that require good judgment based on experience.

For background, this review process was heavily inspired by the Linux kernel development. Specifically, Linux developers review proposed changes and mark them with a number of tags, such as Reviewed-by. Below is the relevant quote from that document that gives a good description for what it means to offer a Reviewed-by tag and that matches many of the aspects of our review policy:

Reviewer’s statement of oversight

By offering my Reviewed-by tag, I state that:

  1. I have carried out a technical review of this patch to evaluate its appropriateness and readiness for inclusion into the mainline kernel.
  2. Any problems, concerns, or questions relating to the patch have been communicated back to the submitter. I am satisfied with the submitter’s response to my comments.
  3. While there may be things that could be improved with this submission, I believe that it is, at this time, (1) a worthwhile modification to the kernel, and (2) free of known issues which would argue against its inclusion.
  4. While I have reviewed the patch and believe it to be sound, I do not (unless explicitly stated elsewhere) make any warranties or guarantees that it will achieve its stated purpose or function properly in any given situation.

A Reviewed-by tag is a statement of opinion that the patch is an appropriate modification of the kernel without any remaining serious technical issues. Any interested reviewer (who has done the work) can offer a Reviewed-by tag for a patch. This tag serves to give credit to reviewers and to inform maintainers of the degree of review which has been done on the patch. Reviewed-by tags, when supplied by reviewers known to understand the subject area and to perform thorough reviews, will normally increase the likelihood of your patch getting into the kernel.