Google Summer of Code 2018 Final Report: Automatic Builds with Clang using Open Build Service

Project Overview

Debian package builds with Clang were performed from time to time through massive rebuilds of the Debian archive on AWS. The results of these builds are published on clang.debian.net. This summer project aimed to automate Debian archive clang rebuilds by substituting the current clang builds in clang.debian.net with Open Build System (OBS) builds.

Our final product consists of a repository with salt states to deploy an OBS instance which triggers Clang builds of Debian Unstable packages as soon as they get uploaded by their maintainers.

An instance of our clang builder is hosted at irill8.siege.inria.fr and the Clang builds triggered so far can be seen here.

My Google Summer of Code Project can bee seen at summerofcode.withgoogle.com/projects/#6144149196111872.

My contributions

The major contribution for the summer is our running OBS instance at irill8.siege.inria.fr.

Salt states to deploy our OBS intance

We created a series of Salt states to deploy and configure our OBS instance. The states for local deploy and development are available at github.com/athos-ribeiro/salt-obs.

Commits

• https://github.com/athos-ribeiro/salt-obs/commit/c6d5f4b7daf58d8e46c435f0d4b22e68e8629b13
• https://github.com/athos-ribeiro/salt-obs/commit/c5be18f8bf7fb483603388faa738fc8c771d7b97
• https://github.com/athos-ribeiro/salt-obs/commit/f38ac162509a7d8f2f532db1fb046194b963c639
• https://github.com/athos-ribeiro/salt-obs/commit/d194bf1b109d028934ab2a2cf144c171dee17fa9
• https://github.com/athos-ribeiro/salt-obs/commit/8da283492f315c7dc5303b7b82455ad9f6f4c534
• https://github.com/athos-ribeiro/salt-obs/commit/4bcbae274898f540a682882c8ccc42ece7af3e73
• https://github.com/athos-ribeiro/salt-obs/commit/690a6d8021ef77ab26c0afd162fd3e45f49abdf0
• https://github.com/athos-ribeiro/salt-obs/commit/3c78f1ab4a09210747185d593469c1c39064b136
• https://github.com/athos-ribeiro/salt-obs/commit/81ccc4171ee52aebe150943d2725a1dc9cc30d9a
• https://github.com/athos-ribeiro/salt-obs/commit/db97a2ca01760d4eb85c8e017da831752e942f7d
• https://github.com/athos-ribeiro/salt-obs/commit/37d539da7cd48c24f9f0ced43aba94004456a8a6
• https://github.com/athos-ribeiro/salt-obs/commit/04c7218b3b063d0b03cadb2944668db24e3618fc
• https://github.com/athos-ribeiro/salt-obs/commit/189083010d99fb7840122ad5e5a697ad6994fce8
• https://github.com/athos-ribeiro/salt-obs/commit/6caa9f809a0b4227e02f064ce0983e20920eff26
• https://github.com/athos-ribeiro/salt-obs/commit/53a66195a3f661c1096f582664a384f9509e40ec
• https://github.com/athos-ribeiro/salt-obs/commit/ddab227269864aeb3ad921f97bb4a3f07ae653c7
• https://github.com/athos-ribeiro/salt-obs/commit/6ac659eadcd9aaba5ecb8b72465944feb0e974de
• https://github.com/athos-ribeiro/salt-obs/commit/eb8905b10a7e630c6c14663da1e47dc2a34b1111
• https://github.com/athos-ribeiro/salt-obs/commit/809f511ffe78030aa96ac64fc7692c7771f1c1e7
• https://github.com/athos-ribeiro/salt-obs/commit/96be16540abce0da2da7dc211fb30b4cc2ba5ed2
• https://github.com/athos-ribeiro/salt-obs/commit/d3cec0bc02812d5dabc6217a663e86301b942c46
• https://github.com/athos-ribeiro/salt-obs/commit/640dc80909750516e06c6520baac75e3a49d72b9
• https://github.com/athos-ribeiro/salt-obs/commit/4a60e0974ee77de615a23a154c40fce4ce6c6c41
• https://github.com/athos-ribeiro/salt-obs/commit/1b549f704bef9a3a636eccee849c74f4a014bead
• https://github.com/athos-ribeiro/salt-obs/commit/c9d1af55fd5cacd60e265727e5d21437d6396974
• https://github.com/athos-ribeiro/salt-obs/commit/5afb38881d57b6d6bcbb2e041eeab44dbd7b2ed4
• https://github.com/athos-ribeiro/salt-obs/commit/b4db98a28a63abd442fb5137aa098d3238099d82
• https://github.com/athos-ribeiro/salt-obs/commit/dc49487843537eca9acdace821e1d477876b5352
• https://github.com/athos-ribeiro/salt-obs/commit/6bdb911487767acad777c689aee1670d2178116b
• https://github.com/athos-ribeiro/salt-obs/commit/ca5919b4dc624ccef986561340db084752706150
• https://github.com/athos-ribeiro/salt-obs/commit/ec95cb10af7548366479eb719fadb3523f600402
• https://github.com/athos-ribeiro/salt-obs/commit/a4a32d9176319b0469e9996c4cfd5c2717e56077
• https://github.com/athos-ribeiro/salt-obs/commit/88ea7f79022b6ba4dae5f923cb4fdc4a91221ae7
• https://github.com/athos-ribeiro/salt-obs/commit/27be63f03abf6bc4d4f77a9b05fc5bb1a341bb6d
• https://github.com/athos-ribeiro/salt-obs/commit/4382e1ebe80367dc099a8e3096c390820d247b99
• https://github.com/athos-ribeiro/salt-obs/commit/2a52be6ad72f198fee9c24d78d351282eb774183
• https://github.com/athos-ribeiro/salt-obs/commit/7f4a16aee01d38a25ebacac3da2e42d800047df0
• https://github.com/athos-ribeiro/salt-obs/commit/e5915647ac5454987c6c9718bd8aba2efb51769d
• https://github.com/athos-ribeiro/salt-obs/commit/a63462289b277fb01daacaefdb1db6c7c83f608d
• https://github.com/athos-ribeiro/salt-obs/commit/e6b0acead6daf120ecba57fc451649d2268743dd
• https://github.com/athos-ribeiro/salt-obs/commit/6248b856d83186ae8fb5261458b83c2674219278
• https://github.com/athos-ribeiro/salt-obs/commit/3eede2dd4652be0f40b8d87d5f54611a4c1efa76

The commits above were condensed and submitted as a Pull Request to the project’s mentor github account, with production deployment configurations.

• https://github.com/opencollab/llvm-slave-salt/pull/9

OBS Source Service to make gcc/clang binary substitutions

To perform deb packages Clang builds, we substitute GCC binaries with the Clang binaries in the builders chroot during build time. To do that, we use the OBS Source Services feature, which requires a package (which performs the desired task) to be available to the target OBS project.

Our obs-service-clang-build package is hosted at github.com/athos-ribeiro/obs-service-clang-build.

Commits

• https://github.com/athos-ribeiro/obs-service-clang-build/commit/7f4af746cf7f729ddabff38be02f72e86739d6f2
• https://github.com/athos-ribeiro/obs-service-clang-build/commit/29d112ee80f6233f59eec5d95d3ad1bcfdec5f49
• https://github.com/athos-ribeiro/obs-service-clang-build/commit/6f3dca53756523ce11fcf1b6c60bb09b7171cf60
• https://github.com/athos-ribeiro/obs-service-clang-build/commit/028ba5f589482db54e178954e3ac6c41b0dd6144

Monitor Debian Unstable archive and trigger clang builds for newly uploaded packages

We also use two scripts to monitor the debian-devel-changes mailing lists, watching for new package uploads in Debian Unstable, and trigger Clang builds in our OBS instance whenever a new upload is accepted.

Our scripts to monitor the debian-devel-changes mailing list and trigger Clang builds in our OBS instance are available at github.com/athos-ribeiro/obs-trigger-sid-builds.

Commits

• https://github.com/athos-ribeiro/obs-trigger-sid-builds/commit/fa2b467644f8285a3af5e7064b76e5debec9d9a0
• https://github.com/athos-ribeiro/obs-trigger-sid-builds/commit/ba0997d29eb9a84ffebd43366ae46e3442553ea5

OBS documentation contributions

During the summer, most of my work was to read OBS documentation and code to understand how to trigger Debian Unstable builds in OBS and how to perform customized Clang builds (replacing GCC).

My contributions

• https://github.com/openSUSE/obs-docu/pulls?utf8=%E2%9C%93&q=author%3Aathos-ribeiro+
• https://github.com/openSUSE/obs-docu/commits?author=athos-ribeiro

Pending PRs

We want to change the Clang build links at tracker.debian.org/pkg/firefox To do so, we must change Debian distro-tracker to point to our OBS instance. As of the time this post was written, we have an open PR in distro-tracker to change the URLs:

• https://salsa.debian.org/qa/distro-tracker/merge_requests/47

Reports written through the summer

• Post 1: My Google Summer of Code 2018 project
• Post 2: Setting up a local OBS development environment
• Post 3: Running OBS Workers and OBS staging instance
• Post 4: Notes on the OBS Documentation
• Post 5: Triggering Debian Builds on OBS
• Post 6: Towards Debian Unstable builds on Debian Stable OBS
• Post 7: Building Debian packages with clang replacing gcc with Open Build Service
• Post 8: Google Summer of Code 2018 Final Report: Automatic Builds with Clang using Open Build Service

Adding new workers to the OBS instance

To configure new workers to our current OBS instance, hosted at irill8.siege.inria.fr, just set new salt slaves and provision them with obs-common and obs-worker, from github.com/opencollab/llvm-slave-salt. This should be done in the top.sls file.

Future work

• We want to extend our OBS instance with more projects to provide Upstream LLVM packages to Debian and derived distributions.
• More automation is needed in our salt states. For instance, we may want to automate SSL certificates generation using Let’s encrypt.
• During the summer, several issues were detected in Debian Stable OBS packages. We want to work closer to OBS packages to help improving OBS packages and OBS itself.

Google Summer of Code experience

Working with Debian during the summer was an interesting experience. I did not expect to have so many problems as I did (see reports) with the OBS packages. This problems were turned into hours of debuging and reading Perl code in order to understand how OBS processes comunicate and trigger new builds. I also learned more about Debian packaging, salt and vagrant. I do expect to keep working with OBS and help maintaining the service we deployed during the summer. There’s still a lot of room for improvements and it is easy to see how the project benefits FLOSS communities.

Comments