Building Debian packages with clang replacing gcc with Open Build Service

The production instance of our Open Build Service can be found here

This is the seventh post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:

• Post 1: My Google Summer of Code 2018 project
• Post 2: Setting up a local OBS development environment
• Post 3: Running OBS Workers and OBS staging instance
• Post 4: Notes on the OBS Documentation
• Post 5: Triggering Debian Builds on OBS
• Post 6: Towards Debian Unstable builds on Debian Stable OBS

My GSoC contributions can be seen at the following links

• https://irill8.siege.inria.fr/project/show/Debian:Unstable:Clang
• https://github.com/athos-ribeiro/salt-obs
• https://github.com/openSUSE/obs-docu/pulls?utf8=%E2%9C%93&q=author%3Aathos-ribeiro+
• https://github.com/openSUSE/obs-docu/commits?author=athos-ribeiro
• https://github.com/openSUSE/open-build-service/pull/5174
• https://salsa.debian.org/qa/distro-tracker/merge_requests/47

Triggering Debian Unstable builds on Open Build Service

As I have been reporting on previous posts, whenever one tries to build packages against Debian unstable with Debian Stretch OBS packages, the OBS scheduler would download the package dependencies and clean up the cacher before the downloads were completed. This would cause builds to never get triggered: OBS would keep downloading dependencies forever.

This would only happen for builds on debian sid (unstable) and buster (testing).

After some investigation (as already reported before), we realized that the OBS version packaged in Debian 9 (the one we are currently using) does not support debian source packages built with dpkg >= 1.19.

While the backports package included the patch needed to support source packages built with newer versions of dpkg, we would still get the same issue with unstable and testing builds.

We spent a lot of time stuck in this issue, and I ended up proposing to host the whole archive in our OBS instance so we could use it as dependencies for new packages. It ended up being a terrible idea, since it would be unfeasible to test it locally and we would have to update the packages on each accepted upload into debian unstable (note that several packages get updated every day). Hence, I kept studying OBS code to understand how builds get triggered and why it would not happen for unstable/testing packages.

After diving into OBS code, we realized that it uses libsolv to read repositories. In libsolv’s change history, we found that there was an arbitrary size limit to the debian control file, which was fixed in an newer version of the package.

After updating the libsolv0, libsolv-perl and libsolvext0 packages (using Debian Buster versions), we could finally trigger Debian unstable builds.

Substituting gcc for clang in build time

With OBS being able to trigger builds for Debian Unstable, we now needed to be able to substitute builds with gcc for builds with the clang compiler. As suggested by my mentor, Sylvestre, we wanted to substitute gcc binaries for clang ones. For that, we needed to find a way to run a script suggest by Sylvestre in our build environments prior triggering the builds.

Fortunately, Open Build Service has a Source Services feature that can do exactly that: change source code before builds or run specific tasks in the build environment during build time. For the latter (which is what we needed to substitute gcc binaries), OBS requires one to build an obs-service-SERVICE_NAME package, which will be injected in the build environment as a dependency and get whatever task specified by that package performed.

We built our obs-service-clang-build package with the needed scripts to perform the gcc/clang substitutions in build time.

To activate the substitution task for a package, it should contain a _service file with in its root path (along with the debian and pristine sources) the following content:

<services>
  <service name="clang_build" mode="buildtime" />
</services>

If everything is correct, we should see

[  TIME in seconds] Running build time source services...

in the log file, followed by the gcc substitution commands.

Automating clang builds for new Debian unstable uploads

Now, we must proceed to trigger Debian unstable builds for newly accepted uploads. To do so, we wrote a cron job to monitor the debian-devel-changes mailing list. We check for new source uploads to debian unstable and trigger a new build for those new packages (we added a 4 hour cool-down before triggering the build to allow the package to propagate through the mirrors).

The results can be seen at https://irill8.siege.inria.fr/project/show/Debian:Unstable:Clang

Updating links in Debian distro-tracker

Debian distro-tracker has links to clang builds, which would point to the former service at clang.debian.net. We now want to substitute those links to point to our new OBS instance. Thus, we opened a new pull request to perform such change.

Next steps

Now that we have an instance of our project up and running, there are a few tasks left for our final GSoC submission.

• Migrate salt scripts from my personal github namespace to https://github.com/opencollab/llvm-slave-salt. A few adjustments may be needed due to the environment differences.
• Write some documentation for the project and a guide on how to add new workers
• Create a separate github project for the cron which analyzes debian-devel-changes mailing list
• Create a separate github project for the obs-service-clang-build package