Project

General

Profile

Bug #4383

Jenkins build verification is non-deterministic

Added by fixeria 5 months ago. Updated 27 days ago.

Status:
Stalled
Priority:
High
Assignee:
Target version:
-
Start date:
01/29/2020
Due date:
% Done:

40%

Spec Reference:

Description

Recently I submitted a patch that is not supposed to change the unit test output:

https://gerrit.osmocom.org/c/pysim/+/16982/

and it actually does not, but Jenkins verification fails:

https://jenkins.osmocom.org/jenkins/job/gerrit-pysim/261/

Verifying card ...
Card contents do not match the test data:
Expected: sysmoUSIM-SJS1.ok
------------8<------------
Using PC/SC reader (dev=1) interface
Reading ...
ICCID: 1122334455667788990
IMSI: 001010000000102
SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
SPN: 
Display HPLMN: False
Display OPLMN: False
PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
PLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

OPLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

HPLMNAcT:
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused

ACC: 0008
MSISDN: Not available
AD: 00000002
Done !

------------8<------------
Got:
------------8<------------
Using PC/SC reader (dev=2) interface
Reading ...
ICCID: 1122334455667788990
IMSI: 001010000000102
SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
SPN: Magic  <-- The contents of this file do not match the expectations
Display HPLMN: True
Display OPLMN: True
PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
PLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

OPLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

HPLMNAcT:  <-- The contents of this file do not match the expectations
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

ACC: 0008
MSISDN: Not available
AD: 00000002
Done !

------------8<------------
Build step 'Execute shell' marked build as failure
[WARNINGS]Skipping publisher since build result is FAILURE
Finished: FAILURE

As I figured out, this is a side effect of the other changes previously submitted to Gerrit:

https://gerrit.osmocom.org/c/pysim/+/16941
https://gerrit.osmocom.org/c/pysim/+/16972

The problem is that a patch submitted to Gerrit may change the contents of SIM card's file system, so after that Jenkins fails to verify any other changes because the unit test output has changed. In this case both SPN and HPLMNAcT have been changed. Perhaps we need a 'smarter' way (than just comparing stdout and a file) to do build verification, e.g. we could match only those files which we're programming in the unit test. Python has a nice framework for unit tests - https://docs.python.org/3/library/unittest.html, so we could have one unit test per file.


Related issues

Related to pySim - Feature #4384: Jenkins program-read verification on daily (or weekly) basisFeedback01/29/2020

History

#1 Updated by laforge 3 months ago

dexter what about this, it has been 3 months? at least a one-line input/opinion from you would be much appreciated.

#2 Updated by dexter 3 months ago

I had a look at it, first of all the current master of pysim and sysmo-usim-tool test both fine.

From time to time we see problems when sysmo-usim-tool does changes to the file system which then cause the tests of pysim to fail and vice versa. At least in my opinion this wasn't to much of a problem yet, but I agree, the way we test is a bit clumsy.

When I get you right you want to use the APIs provided by pysim to check on the contents of the files and implement those checks as python unit-tests? Maybe we could also make pySim-read.py a bit more scripting friendly. Maybe have commandline arguments to specify which file/parameter to read out. Then we would also have some kind of separation and at the same time we woule improve pySim-read.py so that it can be used from scripts.

#3 Updated by fixeria about 2 months ago

One more change related to this ticket: https://gerrit.osmocom.org/c/pysim/+/18073.

#4 Updated by laforge about 2 months ago

  • Priority changed from Normal to High

#5 Updated by laforge about 2 months ago

I'm sorry, but this is starting to seriously impede the development of pySim.

From my point of view, what is needed is to properly reset the system to a sane state after executing tests. "system" here includes the contents of the SIM cards.

So if a patch makes some changes to the filesystem, fine - we will then have to re-write the entire filesystem with known data before starting another test. This should work
  • for user-writable files on all card types
  • for ADM1-writable files on all card types

If some files cannot be written even with ADM1, this shouldn't be a problem - it is safe to assume that any [future] patch we test will then also not be able to write those files.

#6 Updated by dexter about 2 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 40

The resetting of the file system contents should take place before the test executes. My idea is to use the erase() method in cards.py. I think at the moment this method is only programmed for one card. What I plan to do is to reset the contents of the files that pysim writes to 0xff, so that they are basically uninitalized. We can do this with a separate tool pySim-erase.py, this way we do not have to touch pySim-prog.py. The erase() method would use functions that do a read-after-write verification of the file contents, so that we can be sure that the card really has been erased. I already have a draft implementation of pySim-erase.py and some methods for the cards class that can do an erase of binary files and records in one line, so filling the erase() method for each card should be easy. For sysmo-isim-sjs2 I already filled the erase() method.

The following patches are in review now:
https://gerrit.osmocom.org/c/pysim/+/18205 cards: remove len calculation
https://gerrit.osmocom.org/c/pysim/+/18206 cards: reset uninitalized EF.AD
https://gerrit.osmocom.org/c/pysim/+/18207 pysim-prog: move ADM sanitation to utils.py
https://gerrit.osmocom.org/c/pysim/+/18208 utils: do not crash when all bytes of EF.IMSI are 0xFF
https://gerrit.osmocom.org/c/pysim/+/18209 commands: add method to determine size of a non record oriented file
https://gerrit.osmocom.org/c/pysim/+/18211 commands: add features to verify data written to files

#7 Updated by laforge about 2 months ago

dexter wrote:

The resetting of the file system contents should take place before the test executes. My idea is to use the erase() method in cards.py. I think at the moment this method is only programmed for one card. What I plan to do is to reset the contents of the files that pysim writes to 0xff, so that they are basically uninitalized.

The problem is that this will probably break once somebody submits code for a new file to gerrit, which is not yet understood by pysim master. This way the card contents changes juts because we tried to verify any random submitted but not yet approved patch.

Hence I think it would be better to really do a full dump / backup of all files on each of the cards once, and then restore that backup before each test run. The "Backup" part is basically what "osmo-sim-test -o" from libosmocore/utils is generating. Maybe all that's needed is the inverse part, i.e. restoring such a file/directory based backup?

We can do this with a separate tool pySim-erase.py, this way we do not have to touch pySim-prog.py.

I agree this test pre-setup should be kept out of pySim-prog. Whether it's C code or python, I don't really have a preference.

The erase() method would use functions that do a read-after-write verification of the file contents, so that we can be sure that the card really has been erased.

If you want to stick with erase rather than restore, you should make sure to generate a list of all files/paths on each of the cards once, and then erase all their contents [assuming one can do this with ADM1 pin on all relevant cards]

#8 Updated by fixeria about 2 months ago

  • Related to Feature #4384: Jenkins program-read verification on daily (or weekly) basis added

#9 Updated by dexter 27 days ago

  • Status changed from In Progress to Stalled

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)