Miscellaneous Projects » Core testing infrastructure

Note that right now, the osmo-* programs are configured to log to a file, e.g.

log file /data/osmo-msc.log

The osmo-* containers that run the programs are running in the background in the jenkins.sh scripts (docker run -d), which means printing the segfault output to stderr would get lost.

But it should work when we change the configs to log to stderr, and then start the osmo-* programs with redirected output:

export LIBC_FATAL_STDERR_=1
osmo-msc > /data/osmo-msc.log 2>&1

(A nice side-effect is, that we also get some output if the program does not start because the config can't be parsed etc.)

Agree with osmith.

I confirm we get nice output if the program fails to start (due to VTY CMD_WARNING for instance).
For instance, I run osmo-stp in a jenkins.sh like this:

[... here all the docker run cmd line ....] \
$REPO_USER/osmo-stp-$IMAGE_SUFFIX /bin/sh -c "/usr/local/bin/osmo-stp -c /data/osmo-stp.cfg >/data/osmo-stp.output 2>&1" 

• If process fails to start due to VTY misconfiguration, we get the failure issue
• If process crash, we get a backtrace. For instance see SGSN_Tests.TC_attach_req_id_req_ra_update which is expected to crash osmo-sgsn master (patch to fix it in gerrit):
  

  ...
  20191113155240771 DMM <0002> gprs_gb.c:40 MM_STATE_Gb(0)[0x56519ab58850]{Idle}: Received Event E_MM_PDU_RECEPTION
  20191113155240771 DMM <0002> gprs_gmm.c:1531 MM(262420000000026/f9ab395e) -> GMM RA UPDATE REQUEST type="RA updating" 
  20191113155240771 DMM <0002> gprs_gmm.c:1615 MM(262420000000026/f9ab395e) The MM context cannot be used, RA: 262-42-13135-0
  Assert failed mmctx->gb.llme == NULL gprs_gmm.c:1620
  backtrace() returned 15 addresses
  /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_panic+0xcb) [0x7f6973eefd2b]
  osmo-sgsn(+0xcf6f) [0x56519a014f6f]
  osmo-sgsn(+0xd66e) [0x56519a01566e]
  osmo-sgsn(+0x8672) [0x56519a010672]
  osmo-sgsn(+0x1f77d) [0x56519a02777d]
  /usr/lib/x86_64-linux-gnu/libosmogb.so.9(bssgp_rcvmsg+0x543) [0x7f6974337de3]
  /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0x8c0c) [0x7f6974331c0c]
  /usr/lib/x86_64-linux-gnu/libosmogb.so.9(gprs_ns_rcvmsg+0x4a) [0x7f697433369a]
  /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0xa807) [0x7f6974333807]
  /usr/lib/x86_64-linux-gnu/libosmocore.so.12(+0xba2f) [0x7f6973ee3a2f]
  /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_select_main+0x6) [0x7f6973ee40a6]
  osmo-sgsn(+0x813f) [0x56519a01013f]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f69721e72e1]
  osmo-sgsn(+0x842a) [0x56519a01042a]
  talloc report on 'vty' (total 159756 bytes in 8441 blocks)
  [... here the full talloc report ...]
  signal 6 received
  Aborted (core dumped)
  

The LIBC_FATAL_STDERR_ is actually not needed afaict, I get the backtraces anyway.

remote: https://gerrit.osmocom.org/c/docker-playground/+/16063 Update osmo-*/Dockerfile to log stdout+stderr of process to log file
remote: https://gerrit.osmocom.org/c/docker-playground/+/16064 Update ttcn3-*/jenkins.sh to log stdout+stderr of process to log file

Merged, closing.