Feature #4212
closedTTCN3 docker containers: tell glibc to print segfault output to stderr
100%
Description
According to links below, glibc has a feature to log crash related output to stderr instead of /dev/tty (which in the case of running udner docker gets lost afaiu). The feature can be enabled by env var: LIBC_FATAL_STDERR_=1
The idea is to enable that var in docker containers we use to run osmo-* during TTCN3 tests (and unit tests) in order to be able to easily see that the program is crashing, and hopefully also get a backtrace.
We should also enable that in osmo-gsm-tester.
[1] https://stackoverflow.com/a/32057301
[2] https://stackoverflow.com/questions/32056387/catching-libc-error-messages-redirecting-from-dev-tty
[3] https://stackoverflow.com/questions/4290336/how-to-redirect-runtime-errors-to-stderr
Updated by osmith over 4 years ago
- Subject changed from TTCN3 docker containers: tell glibc to print segfault outputto stderr to TTCN3 docker containers: tell glibc to print segfault output to stderr
Note that right now, the osmo-* programs are configured to log to a file, e.g.
log file /data/osmo-msc.log
The osmo-* containers that run the programs are running in the background in the jenkins.sh scripts (docker run -d), which means printing the segfault output to stderr would get lost.
But it should work when we change the configs to log to stderr, and then start the osmo-* programs with redirected output:
export LIBC_FATAL_STDERR_=1 osmo-msc > /data/osmo-msc.log 2>&1
(A nice side-effect is, that we also get some output if the program does not start because the config can't be parsed etc.)
Updated by pespin over 4 years ago
I confirm we get nice output if the program fails to start (due to VTY CMD_WARNING for instance).
For instance, I run osmo-stp in a jenkins.sh like this:
[... here all the docker run cmd line ....] \ $REPO_USER/osmo-stp-$IMAGE_SUFFIX /bin/sh -c "/usr/local/bin/osmo-stp -c /data/osmo-stp.cfg >/data/osmo-stp.output 2>&1"
Updated by pespin over 4 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 80
- If process fails to start due to VTY misconfiguration, we get the failure issue
- If process crash, we get a backtrace. For instance see SGSN_Tests.TC_attach_req_id_req_ra_update which is expected to crash osmo-sgsn master (patch to fix it in gerrit):
... 20191113155240771 DMM <0002> gprs_gb.c:40 MM_STATE_Gb(0)[0x56519ab58850]{Idle}: Received Event E_MM_PDU_RECEPTION 20191113155240771 DMM <0002> gprs_gmm.c:1531 MM(262420000000026/f9ab395e) -> GMM RA UPDATE REQUEST type="RA updating" 20191113155240771 DMM <0002> gprs_gmm.c:1615 MM(262420000000026/f9ab395e) The MM context cannot be used, RA: 262-42-13135-0 Assert failed mmctx->gb.llme == NULL gprs_gmm.c:1620 backtrace() returned 15 addresses /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_panic+0xcb) [0x7f6973eefd2b] osmo-sgsn(+0xcf6f) [0x56519a014f6f] osmo-sgsn(+0xd66e) [0x56519a01566e] osmo-sgsn(+0x8672) [0x56519a010672] osmo-sgsn(+0x1f77d) [0x56519a02777d] /usr/lib/x86_64-linux-gnu/libosmogb.so.9(bssgp_rcvmsg+0x543) [0x7f6974337de3] /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0x8c0c) [0x7f6974331c0c] /usr/lib/x86_64-linux-gnu/libosmogb.so.9(gprs_ns_rcvmsg+0x4a) [0x7f697433369a] /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0xa807) [0x7f6974333807] /usr/lib/x86_64-linux-gnu/libosmocore.so.12(+0xba2f) [0x7f6973ee3a2f] /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_select_main+0x6) [0x7f6973ee40a6] osmo-sgsn(+0x813f) [0x56519a01013f] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f69721e72e1] osmo-sgsn(+0x842a) [0x56519a01042a] talloc report on 'vty' (total 159756 bytes in 8441 blocks) [... here the full talloc report ...] signal 6 received Aborted (core dumped)
The LIBC_FATAL_STDERR_ is actually not needed afaict, I get the backtraces anyway.
remote: https://gerrit.osmocom.org/c/docker-playground/+/16063 Update osmo-*/Dockerfile to log stdout+stderr of process to log file
remote: https://gerrit.osmocom.org/c/docker-playground/+/16064 Update ttcn3-*/jenkins.sh to log stdout+stderr of process to log file
Updated by pespin over 4 years ago
- Status changed from Feedback to Resolved
- % Done changed from 80 to 100
Merged, closing.