host2 disk space running low
/dev/md2 438G 377G 39G 91% /
- 199 GB are jenkins (build history / artefacts)
- 23GB is the deb8build jenkins slave lxc
- 79GB is the deb9build jenkins slave lxc
- 29GB are our docker image layers
We could add hard disks / SSDs to the server, but those are (rented) relatively expensive at ~ 8-10 EUR per month, which we'd have to double for RAID-1. It would be much more economic to upgrade from the AX60 to an AX51-NVMe (2x 1TB storage, faster CPU), which has only an EUR 5 per month price increase. However, that would mean migrating all data to a new machne and then finally switch over after everything is migrated.
#2 Updated by laforge about 2 months ago
I've reclaimed 5GB due to 'docker image prune'.In the debian9 build slave lxc,
- 33GB are docker images/layers
- 24GB are in /home/osmocom-build/jenkins/workspace
The workspace looks fine, but whether the 33GB docker layers are all needed remains to be investigated.
#3 Updated by laforge about 2 months ago
What's quite interesting is the difference in 'du' output within the debian9 lxc and outside of it:
root@deb9build-ansible:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/md2 438G 371G 45G 90% /
root@host2 /var/lib/lxc # du --max-depth=1 -h 78G ./deb9build-ansible
- Status changed from New to In Progress
- % Done changed from 0 to 10
the problem is becoming more critical. I've inquired wih Hetzner about an upgrade to AX51-NVMe.
The biggest consumer of space are the BTS tests at about 1.8 to 2 GB per build. We keep 35 at the moment, ant we test master+latest for debian+centos, adding to almost 100GB at the moment already only for the BTS test.
As an interim measure, I am bzip'ing all those pcap files of the BTS tests.
- Priority changed from Normal to Urgent
disk ran full tonight.
redmine for osmocom.org stalled out, so i tried to reload the docker compose unit which failed (looped)
i scraped off 2% reserved space of /dev/md2 via tune2fs to get it up again - this is urgent now.
/dev/md2 438G 415G 8.9G 98% /