This is the sixth post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:
My GSoC contributions can be seen at the following links
Lately, I have been working towards triggering Debian Unstable builds with Debian OBS packages. As reported before, We can already build packages for both Debian 8 and 9 based on the example project configurations shipped with the package in Debian Stable and with the project configuration files publicly available on OBS SUSE instance.
While trying to build packages agains Debian unstable I have been reaching the following issue:
OBS scheduler reads the project configuration and starts downloading dependencies. The deendencies get downloaded but the build is never dispatched (the package stays on a “blocked” state). The downloaded dependencies get cleaned up and the scheduler starts the downloads again. OBS enters in an infinite loop there.
This only happens for builds on sid (unstable) and buster (testing).
We realized that the OBS version packaged in Debian 9 (the one we are currently using) does not support debian source packages built with dpkg >= 1.19. At first I started applying this patch on the OBS Debian package, but after reporting the issue to Debian OBS maintainers, they pointed me to the obs-build package in Debian stable backports repositories, which included the mentioned patch.
While the backports package included the patch needed to support source packages built with newr versions of dpkg, we still get the same issue with unstable and testing builds: the scheduler downloads the dependencies, hangs for a while but the build is never dispatched (the package stays on a “blocked” state). After a while, the dependencies get cleaned up and the scheduler starts the downloads again.
The bug has been quite hard to debug since OBS logs do not provide feedback on
the problem we have been facing. To debug the problem, We tried to trigger local
builds with osc. First, I (successfuly) triggered a few local builds against
Debian 8 and 9 to make sure the command would work. Then We proceeded to trigger
builds against Debian Unstable.
The first issue we faced was that the osc package in Debian stable cannot
handle builds against source packages built with new dpkg versions. We fixed
that by patching osc/util/debquery.py (we just substituted the file with the
latest file in osc development version). After applying the patch, we got the same
results we’d get when trying to build the package remotelly, but with debug flags on,
we could have a better understanding of the problem:
BuildService API error: CPIO archive is incomplete (see .errors file)
The .errors file would just contain a list of dependencies which were
missing in the CPIO archive.
If we kept retrying, OBS would keep caching more and more dependencies, until the build succeeded at some point.
We now know that the issue lies with the Download on Demand feature.
We then tried a local build in a fresh OBS instance (no cached packages) using
the --disable-cpio-bulk-download osc build option, which would make OBS
download each dependency individually instead of doing so in bulks. For our
surprise, the builds succeeded in our first attempt.
Finally, we traced the issue all the way down to the OBS API call which is triggered when OBS needs to download missing dependenies. For some reason, the number of parameters (number of dependencies to be downloaded) affects the final response of the API call. When trying to download too many packages, The CPIO archive is not built correctly and OBS builds fail.
At the moment, we are still investigating why such calls fail with too many params and why it only fails for Debian Testing and Unstable repositories.
control.tar.xz