Towards Debian Unstable builds on Debian Stable OBS

This is the sixth post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:

• Post 1: My Google Summer of Code 2018 project
• Post 2: Setting up a local OBS development environment
• Post 3: Running OBS Workers and OBS staging instance
• Post 4: Notes on the OBS Documentation
• Post 5: Triggering Debian Builds on OBS

My GSoC contributions can be seen at the following links

• https://github.com/athos-ribeiro/salt-obs
• https://github.com/openSUSE/obs-docu/pulls?utf8=%E2%9C%93&q=author%3Aathos-ribeiro+
• https://github.com/openSUSE/obs-docu/commits?author=athos-ribeiro
• https://github.com/openSUSE/open-build-service/pull/5174

Debian Unstable builds on OBS

Lately, I have been working towards triggering Debian Unstable builds with Debian OBS packages. As reported before, We can already build packages for both Debian 8 and 9 based on the example project configurations shipped with the package in Debian Stable and with the project configuration files publicly available on OBS SUSE instance.

While trying to build packages agains Debian unstable I have been reaching the following issue:

OBS scheduler reads the project configuration and starts downloading dependencies. The deendencies get downloaded but the build is never dispatched (the package stays on a “blocked” state). The downloaded dependencies get cleaned up and the scheduler starts the downloads again. OBS enters in an infinite loop there.

This only happens for builds on sid (unstable) and buster (testing).

We realized that the OBS version packaged in Debian 9 (the one we are currently using) does not support debian source packages built with dpkg >= 1.19. At first I started applying this patch on the OBS Debian package, but after reporting the issue to Debian OBS maintainers, they pointed me to the obs-build package in Debian stable backports repositories, which included the mentioned patch.

While the backports package included the patch needed to support source packages built with newr versions of dpkg, we still get the same issue with unstable and testing builds: the scheduler downloads the dependencies, hangs for a while but the build is never dispatched (the package stays on a “blocked” state). After a while, the dependencies get cleaned up and the scheduler starts the downloads again.

The bug has been quite hard to debug since OBS logs do not provide feedback on the problem we have been facing. To debug the problem, We tried to trigger local builds with osc. First, I (successfuly) triggered a few local builds against Debian 8 and 9 to make sure the command would work. Then We proceeded to trigger builds against Debian Unstable.

The first issue we faced was that the osc package in Debian stable cannot handle builds against source packages built with new dpkg versions. We fixed that by patching osc/util/debquery.py (we just substituted the file with the latest file in osc development version). After applying the patch, we got the same results we’d get when trying to build the package remotelly, but with debug flags on, we could have a better understanding of the problem:

BuildService API error: CPIO archive is incomplete (see .errors file)

The .errors file would just contain a list of dependencies which were missing in the CPIO archive.

If we kept retrying, OBS would keep caching more and more dependencies, until the build succeeded at some point.

We now know that the issue lies with the Download on Demand feature.

We then tried a local build in a fresh OBS instance (no cached packages) using the --disable-cpio-bulk-download osc build option, which would make OBS download each dependency individually instead of doing so in bulks. For our surprise, the builds succeeded in our first attempt.

Finally, we traced the issue all the way down to the OBS API call which is triggered when OBS needs to download missing dependenies. For some reason, the number of parameters (number of dependencies to be downloaded) affects the final response of the API call. When trying to download too many packages, The CPIO archive is not built correctly and OBS builds fail.

At the moment, we are still investigating why such calls fail with too many params and why it only fails for Debian Testing and Unstable repositories.

Next steps (A TODO list to keep on the radar)

• Fix OBS builds on Debian Testing and Unstable
• Write patch for Debian osc’s debquery.py so it can build Debian packages with control.tar.xz
• Write patches for the OBS worker issue described in post 3
• Change the default builder to perform builds with clang
• Trigger new builds by using the dak/mailing lists messages
• Verify the rake-tasks.sh script idempotency and propose patch to opencollab repository
• Separate salt recipes for workers and server (locally)
• Properly set hostnames (locally)

Comments