Setting up a local OBS development environment

This is my second post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:

• Post 1: My Google Summer of Code 2018 project

Debian (testing) OBS API package

On the last post, I reported an issue with the OBS API package where the assets pipeline would trigger an error during compilation. Digging further, I realized that the problem lies within the fact that the assets group used in the package Gemfile was removed from Rails 4.

Removing the group from the Gemfile does solve the problem:

sed -i 's/^group :assets do$//' /usr/share/obs/api/Gemfile
sed -i '93s/^end$//' /usr/share/obs/api/Gemfile

But then, Debian testing has a different version of the Ruby sass gem, which leads to a different error when compiling the assets for the API. A dirty fix would be to update the version in the Gemfile.lock before compiling the assets:

sed -i 's/sass (3.4.22)/sass (3.4.23)/' /usr/share/obs/api/Gemfile.lock

After a few days working on this issue, I realized there is a script in the OBS API package, provided by the OBS Debian packagers, that:

• Sets the right permissions for the apache (www-data) user on the API files;
• re-generates the Gemfile.lock file, solving the sass issue;
• creates and sets the database;
• runs the assets precompile rake task.

The /usr/share/obs/api/script/rake-tasks.sh script does most of the work I have prepared on my local salt scripts to properly set the API, with one drawback: is it idempotent? i.e.: can we safely run the script every time we provision our nodes keeping the same state on every run? Running the create and setup rake tasks several times will trigger several warnings. I had solved the issue (before knowing about the rake-tasks.sh script) by making sure that the setup task would run only once by using salt grains. The problem of using the same approach for the rake-tasks.sh script is that if the obs-api package gets updated, the ownership of the package files will be set back to root and apache won’t be able to serve them, hence, we would need to re-run the related tasks available in the rake-tasks.sh script.

A possible solution for the issue would be to keep the rake-tasks.sh script running only once (on the first setup) and replicate the permissions setup present in the file for whenever the OBS package is upgraded.

My OBS salt configuration repository for local OBS deploy is available here.

Fetching new uploads information

Following up on the last post topic on how to fetch information about new uploads to the Debian archive, and if I understood the debian package workflow correctly, dak is the software the FTP masters use to accept packages in the archive. Checking dak’s code, I realized dak fires an email to recipients upon package acceptance. One of the recipients of these emails is the debian-devel-changes mailing list. Hence, it led me to believe that the shortest path to watching new uploads in the archive is to simply watch the mailing list. If we consider that email protocols do not guarantee a message was delivered, we would be accepting some margin of false negatives, where a package was accepted in the archive and an email was not fired (of course there would also be the cases where the mailing service fails).

The Debian IRC bot, which announces new archive uploads, just subscribes to the aforementioned mailing list and announces the upload whenever a new email is received.

When the time comes, we will also watch this mailing list to trigger new Clang builds in our OBS instance.

Setting up a default Debian packages builder

I could finally set up a local default Debian packages builder with a fully functional OBS instance (server, worker, api) by setting the configurations steps provided by Héctor Orón Martínez

When using OBS Download on Demand feature, my workers are still getting stuck at some point (no logs at all; to be debugged).

Next, I will proceed to aid Reshabh and Sylvestre on configuring our OBS instance at opencollab/llvm-slave-salt And start customizing a OBS project to build packages with Clang.

Next steps (A TODO list to keep on the radar)

• Verify the rake-tasks.sh script idempotency and propose patch to opencollab repository
• Solve remaining issue with local OBS worker
• Separate salt recipes for workers and server (locally)
• Properly set hostnames (locally)
• Configure Debian projects on OBS with salt, not manually
• Change the default builder to perform builds with clang
• Trigger new builds by using the dak/mailing lists messages