Bug #5632
openbetter test result summary / notifications
10%
Description
10:55 < fixeria> regarding the testsuite monitoring in Jenkins... 11:19 < fixeria> not sure if you guys still use RSS/Atom, but this is a convinient way to monitor test results 11:20 < fixeria> https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/rssFailed 11:21 < fixeria> this one is specifically for ttcn3-bts-test, but you can also monitor all TTCN-3 stuff 11:21 < fixeria> https://jenkins.osmocom.org/jenkins/view/TTCN3/rssFailed 11:23 < LaF0rge> fixeria: i did a quick look, but this just gives you the high level overview of which job failed, right? 11:24 < LaF0rge> fixeria: what we'd want is down on the junit-xml level of granularity where we can track that sudenyl TC_foo_bar inside a jenkins job is failing 11:29 < LaF0rge> stuff like https://jenkins.osmocom.org/jenkins/job/ttcn3-bts-test/lastCompletedBuild/testReport/api/xml 11:29 < fixeria> LaF0rge: it simply says: N more tests failing, N less 11:29 < LaF0rge> fixeria: that cumulative count can be misleading if some old failures are passing but some new failures are added. 11:29 < LaF0rge> lynxis: ^^ 11:30 < LaF0rge> I think those testReport/api/xml URLs should cointain what we need 11:30 < LaF0rge> it even has the errorStackTrace with the verbose error message [if there is one] 11:31 < LaF0rge> and it has 'age' and 'failedSince' fields
Updated by lynxis over 1 year ago
- % Done changed from 0 to 10
I'll implement a irc bot and command line utility to check for ttcn3 failures.
use jenkins api to retrieve the junit xml fileparse junit xml files into junit object- convert junit object into data structure to get a sliding window over 10 days/jobs for
- "good" (first job succeed after fails, but job wasn't unstable)
- "fail" (first job fails after good, but job wasn't unstable)
- "unstable" (many fails & succeeds alternating (2x in a row is fine)
- generate a summary for every jobs
- generate a summary for all jobs
- connect a irc bot to the output
- check jenkins for updates every hour
Updated by laforge over 1 year ago
If possible, make the ir bot only one consumer of sll that information. As I mentioned, we also want some kind of overview web page at some point. You don't have to work on that if you dont want to, but it would be good to design the software in a way that multiplr consumers for the information exist so it can be extended later.
As a bonus, also considering obs.osmocom.org failures would be great
Updated by neels over 1 year ago
Maybe this bit of code can help:
https://gitea.osmocom.org/ttcn3/osmo-ttcn3-hacks/src/branch/master/compare-results.py
the proper way would be to do actual XML parsing though, this script cheats
around that with regexes, depending on the actual line feeds that jenkins
typically produces.