Bug #5301
openRun TTCN3 docker tests with sanitizer enabled
70%
Description
After updating libosmocore I noticed that the TTCN3 GbProxy tests start to fail with an ASan issue when run locally.
I think at least for the TTCN3 tests on master we should enable *San to catch hidden bugs early. Unfortunately this has a large impact on how the osmo-*-master docker images are built if we want to enable it for the libraries as well - currently we install the nightly packages and build the target from master.
Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.
Not sure what the downsides are, any ideas?
Checklist
- initial build of osmocom:nightly:asan
- test running one testsuite with osmocom:nightly:asan packages locally
- submit all code changes to gerrit
- get patches merged
- roll it out (may need some fixups)
Related issues
Updated by daniel over 2 years ago
- Related to Bug #5302: ns2: ASan heap-use-after-free in ns2_nse_notify_unblocked() when running GBProxy_Tests.TC_bvc_reset_blocked_ptp_from_sgsn added
Updated by osmith over 2 years ago
Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.
If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.
How about we do what you described, but in addition to the tests that make use of distro packages?
We could have a new TTCN3-san view, next to:- https://jenkins.osmocom.org/jenkins/view/TTCN3/
- https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/
If we only test master and not latest too, then this new view has only half as much jobs.
laforge, what do you think?
Updated by laforge over 2 years ago
On Fri, Nov 12, 2021 at 03:36:30PM +0000, redmine@lists.osmocom.org wrote:
Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.
If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.
ACK.
How about we do what you described, but in addition to the tests that make use of distro packages?
We could have a new TTCN3-san view, next to:
- https://jenkins.osmocom.org/jenkins/view/TTCN3/
- https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/
If we only test master and not latest too, then this new view has only half as much jobs.
OK. I would argue to test master once per day only, and not every commit. I can add more
build slaves if we run into problems there.
Updated by daniel over 2 years ago
laforge wrote in #note-3:
On Fri, Nov 12, 2021 at 03:36:30PM +0000, redmine@lists.osmocom.org wrote:
How about we do what you described, but in addition to the tests that make use of distro packages?
We could have a new TTCN3-san view, next to:
- https://jenkins.osmocom.org/jenkins/view/TTCN3/
- https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/
If we only test master and not latest too, then this new view has only half as much jobs.
OK. I would argue to test master once per day only, and not every commit. I can add more
build slaves if we run into problems there.
Yeah, that would be great. I think TTCN3 tests every commit is a bit too much anyway (with or without sanitizer). Running it once per day is fine.
Updated by laforge almost 2 years ago
As this topic has come up again today: One alternative might also be to create a dpkg repo (for one distro/arch only to avoid tons of extra builds) with asan enabled. something like osmocom:nightly:asan or the like on https://obs.osmocom.org/
Updated by osmith over 1 year ago
- Checklist item initial build of osmocom:nightly:asan added
- Checklist item test running one testsuite with osmocom:nightly:asan packages locally added
- Checklist item submit all code changes to gerrit added
- Checklist item get patches merged added
- Checklist item roll it out (may need some fixups) added
- % Done changed from 0 to 70
I've created https://obs.osmocom.org/project/show/osmocom:nightly:asan and adjusted the osmo-ci scripts to generate debian source packages with --enable-sanitize.
When trying to build them on OBS, it failed to run the regression tests:
[ 237s] ==33619==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12) [ 237s] ==33619==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v
Using ulimit -v is hardcoded in OBS here: https://github.com/openSUSE/obs-build/blob/e82ad7023a03a81e0d48f44f0260ced7b6c0643e/build#L743-L744
(It runs through that code path, as it prints "Memory limit set to 66248548KB" at the beginning of the log.)
So as a workaround, I disabled the tests. I also had to disable building of documentation, as otherwise it would run into the same error while generating VTY docs. Now all Osmocom packages build successfully with sanitize flags in that new osmocom:nightly:asan repository.
I went on to adjust the scripts to actually run the testsuite with these binary packages, and in a quick local test it seems to be working fine.
Patches submitted:
https://gerrit.osmocom.org/q/topic:ttcn3-asan
Updated by pespin over 1 year ago
I'd probably change the osmo-*-master Dockerfiles to build all dependency libs from master with --enable-asan enabled, I have been doing that for ages in my local copy of docker-playground.git
This has the benefit that during development one can for instance, easily set LIBOSMO_NETIF_BRANCH=myownbranch when testing ie. osmo-stp-master.
Updated by pespin over 1 year ago
I think it makes more sense to have a -nightly using nightly repos with prebuilt binaries, and have a -master with libs+binary compiled with --enabe-asan.
Updated by osmith over 1 year ago
Discussions in code review:
- is the package repository the right approach? https://gerrit.osmocom.org/c/docker-playground/+/28902/1#message-30650d8450d2d129b6ef3f447f65e4b866d2f801
- when to run asan builds, really in addition to master builds? https://gerrit.osmocom.org/c/osmo-ci/+/28909/2#message-c3ce96b7775c1899a177b77437976d9a36bdd0ca
Updated by daniel over 1 year ago
What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.
The packages would only be rebuilt if something in git or a dependency changes, right?
Updated by osmith over 1 year ago
daniel wrote in #note-12:
What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.
The packages would only be rebuilt if something in git or a dependency changes, right?
They would be rebuilt every night, just like the other nightly packages.
We decided earlier to have all nightly Osmocom packages depend on a specific version (build date) of a meta package (e.g. osmocom-nightly). This ensures packages of different build days can't be mixed by accident, as we don't ensure ABI compatibility on master - only for latest. For that reason we need to also rebuild all nightly asan packages at once, even if they did not change.
FWIW the asan packages build pretty fast since they can't run tests or be built with documentation in OBS.
Updated by osmith over 1 year ago
- Status changed from In Progress to Stalled
Setting to stalled until discussed how to move forward. (Maybe in one of the next weekly meetings once everybody is back from holidays?)
Updated by laforge 10 months ago
I just updated the gerrit tickets linked above. I do think running daily/nightly asan builds is useful. Ideally we'd be able to figure out when to run them to avoid clashes with other busy times for our builders. Given that we now have grafana/prometheus, we should hopefully be able to figure out when would be a good time, or in general when to run which builds / tests (not just those expected new ones).
Updated by osmith 8 months ago
- Status changed from Stalled to In Progress
I forgot to update this issue.
Last time I looked into this, building all packages in OBS with asan did not work yet - specifically libosmo-pfcp fails because:- it runs a generated binary during the build (gen__pfcp_ies_auto)
- running asan binaries fails in our OBS (==16918==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)) (#6146)
Having another look at it.
Updated by osmith 30 days ago
- Status changed from In Progress to Stalled
To summarize the current status: I couldn't get the asan packages building on OBS last time I tried (asan binaries could not execute, and we need to run some binaries during build for some projects even if we disable tests). Meanwhile we have replaced the server that the OBS workers are running on (#6186) and I have a better understanding of how the OBS workers are set up. So it would be worth trying again when I'm done with issues with higher priorities.
