NVMe and emerge compile

Black · Posted: Tue May 09, 2023 5:37 pm Post subject: NVMe and emerge compile

2 years ago, I got myself a new computer, this time with a NVMe drive mounted as root drive (/). I also have regular HDD set up in RAID1 for my /home. I did not create a separate partition for /var, so it's on the same NVMe drive as /.

That PC is on 24/7. At some point, I rebooted, and the BIOS complained about that drive failing the SMART test. I have now set up /var/tmp to be a tmpfs, taking 12GB out of the system's total 32GB of RAM. I also went looking on the net to see if leaving emerge to compile on a NVMe drive is bad. I came across a reddit page where people say it's not an issue, with one example saying he's got "38TB written in 7000 hours", with a warrantied TBW of 1200TB.

In my case, the drive is a Kingston SA2000M8250G - from what I find online, the TBW limit is 150TB. From what I can tell, I'm waaaaay past that, at 1.03PB (in almost 2 years - see below). Kingston's warranty is also apparently void since the "percentage used" is now 100%. So yeah, that drive is a failure waiting to happen (it still runs, despite the SMART test failure - I'm currently using this computer to post this message).

So my question is: is letting portage use a NVMe drive to compile killing such drives?

eccerr0r · Posted: Tue May 09, 2023 5:54 pm Post subject:

Today's TLC and QLC drives just don't have the endurance anymore but for most uses they are fine. 1PB written however is a LOT, what are you doing to the disk? Using it as a bittorrent dump?

I have machines on 24/7 but they're mostly idle. One of them is PVR and it's accumulated ~ 65TB written since last mkfs (it's a mechanical HDD however) and it's been over 10 years. I don't constantly do updates on it however, but it definitely gets emerge @world once in a while - but the vast majority of the writes are from downloading OTA TV programming.

Granted for me my Gentoo boxes typically use tmpfs when I can, but being RAM limited I cannot always use tmpfs. I do have a 180G SATA SSD that I've gone 23TB written according to SMART, but it has a minimum 540TBW estimate endurance limit
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Tue May 09, 2023 5:54 pm Post subject:

Black

eccerr0r · Posted: Tue May 09, 2023 6:00 pm Post subject:

Definitely not Gentoo doing that usage but I can't say it's unbelievable - but we don't know what else is using the disk. One thing that is suspicious is that the read/write ratio is oddly skewed to writes - meaning that it's written and never read back...

I found that I (accidentally) took a big chunk out of some of my SSDs by thrash swapping to them, and with an NVMe interface this can add up fast.

I do have to say that there are firmware bugs out there that lie about usage. One of my SSDs, according to its POH, says it was made when Edison made his first light bulb...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Black · Posted: Tue May 09, 2023 6:19 pm Post subject:

eccerr0r · Posted: Thu May 11, 2023 6:25 am Post subject:

Can these newer nvme SSDs sustain 1GB/sec written?
Writing through 2PB would take less than 1 month...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Black · Posted: Mon May 15, 2023 3:07 am Post subject:

Hu · Administrator Joined: 06 Mar 2007 Posts: 21726

Yes, logs are written and often not read, but typical logs should not be nearly large enough for that to be noticeable at this scale.

eccerr0r · Posted: Mon May 15, 2023 6:13 am Post subject:

only thing that could do this are:

- Backups (unless you verify)... I had one hard drive that I only wrote backups to (as well as it kept on getting dropped from the array for electrical problems, so it kept on getting resilvered and that's all writes)
- killer endurance testing
- sabotage...

mystery continues...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Black · Posted: Thu Jun 08, 2023 12:36 pm Post subject:

I have a new NVMe "drive" on my desk, and I will switch in the near-future, but in the meantime, here's some interesting data. I don't think it shows anything (other than it doesn't match). I have run smartctl twice, at 24-hour interval. iotop has been running (in accumulation mode) for that same period. md127 is a RAID1 array of spinning rust - so not the NVMe. Chrome, running under user "black" should be writing to the home folder, which is not on the NVMe (it's on md127 - the spinning rust). Syncthing's folders are also on md127.

/var/portable/tmp has been in tmpfs for a month now and doesn't appear to make a difference. I have included the relevant fstab line, in case I made a newbie mistake there.

eccerr0r · Posted: Thu Jun 08, 2023 2:02 pm Post subject:

You're still "writing" 100TB/month somehow!

is there anything funky show up in your dmesg?

What happens if you mount the disk from livecd (R/W) and wait out a similar period? Or probably at least an hour?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Black · Posted: Thu Jun 08, 2023 3:48 pm Post subject:

Filtering out the UFW (Uncomplicated FireWall) that I started using about 2 or 3 months ago (therefore, long after this excessive writing started), I get the output below.

I just found this page for ArchLinux stating there is an issue with that exact same drive with that exact same firmware revision. I don't have the exact same symptoms - the drive does not become unresponsive after a while, and I ran it for around 300 days without rebooting at some point. I'll have to try updating the firmware, or at least passing the kernel parameter to set a max latency to see if it changes anything.

Thanks for the livecd suggestion, I'll give that one a try as well.

Anon-E-moose · Posted: Thu Jun 08, 2023 7:35 pm Post subject:

That's an insane amount of disk writes, for that short a time.

I suppose it could happen if, you "emerge -e" several times a day or are using some part of the nvme for swap space and keep running out of memory or you have several things writing to /var/log/<something> constantly.

Edit to add: From kingston site about smart data

toralf · Posted: Thu Jun 08, 2023 8:38 pm Post subject:

At the tinderbox I made similar experiences.

From the smartctl values about 1.4 PB of data were written in the last 2 yrs to a BTRFS filesystem spawned at 2 partitions of 2 NVMe drives.
This are about 24 MiB/sec . The Grafana metrics node_disk_written_bytes_total (I do use it since 2 months) told me the same.
What is interesting, is that this value dropped down to 9-10 MiB/sec since kernel 6.3.x. And nothing else was changed at the server.

The emerge is made using a tmpfs for /var/tmp/portage.

FWIW, the nightly house keeping process here - which deletes about 10-100 GB of old data at that file system- is shown as node_disk_written_bytes_total too. So there's a big discrepancy between the house kept space and the reported written space, the factor is still 10x or 20x.

eccerr0r · Posted: Thu Jun 08, 2023 8:44 pm Post subject:

Based on the 24 hour sample, the iotop written bytes and the hard drive written bytes actually seem to correspond (assuming 512-byte logical blocks), but that 24 hour sample of 700MB/day, if sustained, would only 22GB/month, nowhere the 100TB that was measured.

Is your ext4 filesystem formatted for 512 or 4096 blocks? 512 byte blocks or perhaps partition alignment problems could cause some extraneous writes.

Did you even expect that 700MB writes that one day?

I never took a look at how much I write per day...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Black · Posted: Thu Jun 08, 2023 8:57 pm Post subject:

Anon-E-moose · Posted: Thu Jun 08, 2023 9:43 pm Post subject:

I suppose you could have buggy firmware.

If interested you could find the firmware rev and check google for that model and firmware version to see if there are reported problems.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

eccerr0r · Posted: Thu Jun 08, 2023 9:47 pm Post subject:

might be interesting to take a log snapshot every day and see if there's some anomalous behavior, but missing one day of 40MB/sec writes all day is kind of hard to make up - so it's probably not demand writes, more like firmware or consequential writes going on.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Anon-E-moose · Posted: Thu Jun 08, 2023 9:50 pm Post subject:

Do you have discard turned off and run fstrim periodically
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

Black · Posted: Fri Jun 09, 2023 12:57 am Post subject:

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2014

I've sometimes wondered if an overzealous combination of logging (or writing anything) and syncing can cause this sort of problem - the zeal being to sync after every line rather than let the kernel do its thing. Not syncing means the writes would be buffered, and there's a danger of losing the last buffer(s) if there's a power outage, but the cost of syncing on SSD or NVMe is a write (=new block allocated and written for some value of "block" for every single line...) I presume databases and other transactional mechanisms have some way round this for their journals; alternatively, just ensure the journal is on spinning rust.

FWIW I have a 5 disk RAID 10 array that I use for /home,and /var/tmp, and run emerges in a chroot in /home/packager/chroot to create binary packages, and then install the binpkgs into the root filesystem on NVMe, so all the compilation stuff happens on spinning disks.
_________________
Greybeard

Black · Posted: Fri Jun 09, 2023 2:51 pm Post subject:

Just another data point: I discovered the inotifywait command, so I'm running it on /tmp.

In the last 2.5 hours, Google Chrome is the only process that has written anything in /tmp - 740 times, either create, modify, or delete a file there. For most of the time, Chrome is actually just sitting there, as I'm working away on another PC. It doesn't mean it's all Chrome's fault, but it sure doesn't help. I guess I should run inotifywait on the entire partition.

I also ran inotifywait on /var/log, and only 75 writes were done in the same time period.

Anon-E-moose · Posted: Fri Jun 09, 2023 3:12 pm Post subject:

Do you have nvme-cli installed? It has lots of useful options for nvme investigation.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

Black · Posted: Fri Jun 09, 2023 3:41 pm Post subject:

eccerr0r · Posted: Fri Jun 09, 2023 3:50 pm Post subject:

Searching the web, I get a lot of hits on kingston ssds having this behavior...

Currently I only have Intel, Samsung(mPCIe) Patriot (mPCIe), HP, and Micron/Crucial SSDs ... they don't seem to exhibit this behavior though the Samsung I accidentally swap stormed on and ate a chunk of its life ...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?