Feature #2385
closedvalidate debian rules/control as part of jenkins build testing
100%
Description
Right now one can make changes to the source code that will break the debian package builds.
It would be nice if the jenkins build testing for +Verified in gerrit would trigger a dpkg-buildpackage of the related code. This is probably a good case for a docker container that has all known/shared required packages installed?
Related issues
Updated by laforge over 5 years ago
- Assignee set to osmith
- Priority changed from Normal to Low
Updated by osmith over 4 years ago
- % Done changed from 0 to 10
I've done some testing and research to figure out the best way to implement this.
Caveats:- we must also build debian packages of all dependencies, and install them beforehand
- ideally we would build the debian packages in a fresh debian docker container, and use "apt-get build-dep ."
- building all dependencies and the project that was modified itself will take just as long as the current build job
So this is more than just a simple change in each project's jenkins.sh script.
Since this takes at least as long as running the current jenkins.sh, I think we should run it in parallel to jenkins.sh.
It would also be nice if we did not have to modify each repository to implement this, so I'd put a common script in osmo-ci.git instead.
Concept:- create a new script in osmo-ci.git, which:
- builds debian packages for all dependencies of a program, each in an empty docker container and by installing dependencies with "apt-get build-dep"
- already built dependencies get installed in that empty docker container first
- finally build the debian package for the project that we are currently verifying
- adjust gerrit-verifications.yml and master-builds.yml to run this script in parallel to the existing jenkins.sh for the projects where we want to test this (using build axes?)
This is not as trivial as I thought, so I will not continue with this issue right now.
Updated by laforge over 4 years ago
What about using the nightly debian package feed for all the dependencies? Yes, that
poses some kind of danger during a day, (i.e. libosmocore patch is merged, and one hour later
somebody wants to verify a patch using that feature in osmo-msc), but maybe that's acceptable?
In order to improve on that, we could also be something like a "master" package feed which rebuilds
packages as soon as a commit is made to master. This could probably be done using OBS, as rebuilding a
single package after a commit to our master (e.g. libosmocore.git) should be relatively quick compared to the
full-rebuild-everything on nightly? If OBS is not suitable, we could also have a different mechanism (like
pbuilder), but I would argue for trying OBS first. At least it should get the propagation delay from
"libosmocore patch was merged" from one day (nightly feed) to some (tens of) minutes.
This way we avoid rebuilding all dependencies all the time, which looks like a rather big waste
of time (and energy) to me.
Updated by osmith about 4 years ago
This way we avoid rebuilding all dependencies all the time, which looks like a rather big waste
of time (and energy) to me.
Indeed. I like the idea with the "master" package feed, using OBS. If a patch depends on a patch from another repo that was just merged, waiting ~10 minutes and manually re-triggering the jenkins verification seems like a very good trade-off to me.
(I would like to work on this, but since this has "Priority: Low" I won't start it now.)
Updated by osmith over 1 year ago
- Related to Bug #5654: OBS errors for libosmo-pfcp, osmo-upf, osmo-hnbgw added
Updated by osmith over 1 year ago
- Status changed from New to In Progress
- % Done changed from 10 to 40
Updated by msuraev over 1 year ago
- Related to Bug #5669: Test .deb packages built by our OBS added
Updated by osmith over 1 year ago
- % Done changed from 70 to 90
Updated by osmith over 1 year ago
- There's a new master repository, where packages get updated as soon as a patch is pushed to master: https://obs.osmocom.org/project/show/osmocom:master
- The new Osmocom_OBS_master job creates the source packages and uploads them to OBS. it runs often, but each run only takes 20-30 seconds as it skips all packages that were not modified.
- A docker container with build-essential installed and the master repository configured (and intentionally nothing else to catch missing deps) gets used to install deps and build packages
- As it takes forever to install osmo-gsm-manuals-dev + depends, as optimization there's another docker container that has it pre-installed; this gets used if the package has osmo-gsm-manuals-dev in the build depends
- I've rolled out the -rpm and -deb verification jobs for osmo-bsc-nat currently for testing (not yet for other repositories, will do after patches are merged): https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29492
Updated by osmith over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
As noted in code review, my first approach would have lead to lots of comments added to gerrit patches, and thus lots of mails. I've changed it so now a pipeline gets started, and only one comment is written at the end.
The patches are now merged, the jobs are rolled out, and although of course I tested carefully, I'm keeping an eye on the jobs in case there is any fallout.
I've written some documentation here:
https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_gerrit-verifications_Pipeline
And some minor follow-up patches that add back osmo-ci and add a few comments:
https://gerrit.osmocom.org/q/topic:ci-packaging+status:open
Updated by osmith over 1 year ago
- % Done changed from 100 to 90
Setting to 90% again since there are some more fixup patches.
https://gerrit.osmocom.org/q/topic:ci-packaging+status:open
There wasn't much fallout, the only problems I've seen so far and fixed:- gerrit-osmo-ttcn3-hacks-build wasn't matched by the regex, thus sending the result comment there didn't work (retriggered the ones manually that were missed)
- libosmocore deb/rpm packaging jobs failed on build2-deb11build-ansible in the socket tests, as docker wasn't configured for ipv6 on this node. I've used the ansible rules to deploy the ipv6 config.
- the notification for the comment was sent to everybody, not only the owner. turns out we can control that too, and now it only sends a mail to the owner on error, and no mail on success.
