BTRFS - Replacement of disks months ago - Problems on reboot

[matt2kjones]

This is more of a BTRFS rant than anything.

We have a large number of storage servers, some are fast NVMe arrays which are used for live access to files, while others, like this one, are hard drives made arrays made up of 26 spindles used as backup storage.

We use BTRFS on these systems and they form a large btrfs RAID10 array used as bacula storage for our backups.

Over time, we've replaced failing disks in our BTRFS pool using the replace command. The enclosure holds 28 discs with 26 active at any one time so that we can have two spare disks in the enclosure ready to go.

The last couple of times we replaced disks, we didn't take the old ones out. After a reboot, we've realised that btrfs has put disks that were replaced back into the pool, rather than the active disks causing a huge amount of errors:

[Hu]

When you declare a disk decommissioned, but do not physically remove it, what commands do you use to instruct btrfs not to use that disk anymore? Do I understand correctly that after issuing such an instruction to btrfs, it proceeded to put the disk back into service on next reboot, in direct contradiction of what you told it to do?

[matt2kjones]

Yes this seems to be the case... So we had a drive fail (/dev/sdm) a few months back. I used the btrfs command to replace that disk with /dev/sds (which was a new empty drive). On reboot today, after a kernel change, /dev/sdm went back into the pool and an error came up in dmesg from btrfs saying it was unable to activate /dev/sds: file exists. Luckily, in this case I realised what was happening, so I shut it back down, pulled out /dev/sdm and on reboot, /dev/sds went back in.

Unfortunately, this has also happened with another disk as well.... From the logging, I think /dev/sdr has gone in to the pool instead of another disk.

I'm guessing that whatever information btrfs uses to identify disks as part of the pool and individual mirrors within the raid10 are identical on the removed disk and the disk that was set as it's replacement.

All these disk replacements were done when an older kernel (a few years old), so maybe it's a bug that has been addressed now, I'm not sure.

Partially my fault I suppose. I should have removed the disks from the enclosure once the drive replacement processes completed and replaced them with blank drives ready for the next drive failure, but I never expected it to be an issue.

None of this data is critical, it's backups of our lives systems and we have other backups as well, so I can destroy this partition and recreate it, but I think I will attempt to fix it before wiping it just to try and understand repairing btrfs more. Or maybe switch to ZFS as it seems there is a lot more information out there because of it's wide adoption.

[matt2kjones]

Also, to answer your question. I did nothing other than use the command to replace the disk. I assumed once the disk was replaced, btrfs wouldn't put that disk back in the pool on reboot. In hindsight, maybe I should have, at the very least, used DD to zero out the start of the drive or simply removed it.

[Hu]

I was hoping to see the exact command, well-formed and with all parameters, so that someone else could run it or research it. I could guess from context that you ran btrfs replace /dev/sdm, but this is only a guess, so any attempt to read up on it might go down the wrong path. Knowing the versions of the btrfs command line tool and of the kernel on replace day might be helpful too, for looking up whether there are now known issues with those versions.

[matt2kjones]

BTRFS before replaced disk:

[matt2kjones]

I'm going to be binning this partition and recovering from a backup...

But interestingly I run a scrub on the array...