How to: convert your VPS root filesystem to btrfs (using rescue boot)

I’m moving from (a kind of…) a dedicated server to a VPS, to decrease my frightful anticipation of hardware failures.
Honestly though, that server had been freezing up and restarting spontaneously for several months now, causing sometimes really long down-times…
That server is now about 6-7 years old, built with off-the-shelf components, some of which (the HDD Image may be NSFW.
Clik here to view. ) had weird noises from the very start.
Definitely time to move!

I’ve purchased a fairly cheap VPS with an easy, one-click upgrade option for after I’m done configuring it.
It comes with a wide selection of OSes to pre-install; I’ve chosen Debian Jessie, version 8.3 as of this writing.

I wanted to use btrfs from the beginning, so could have installed Debian myself, but… VPS provider does some initial configuration (like their Debian mirror and some other things), so I’ve felt that converting to btrfs after the fact would be easier. Now that I’ve done this – I guess it was fairly easy, although preparation did take some time.

Below, I’m providing step-by-step instructions on how to convert your root filesystem from (most likely) ext4 to btrfs.

I’ll be using my provider’s rescue boot mode: this is a live Debian system which is network-booted on my own VPS, and thus has access to the SSD/HDD of my VPS. Hopefully, your provider has a similar feature.

Preparing for this conversion, I had mostly used two sources:

an older howtoforge conversion tutorial, and
Sysadmin Guide

Before we begin with actual conversion, here is a

List of hints and useful practices/commands for a future btrfs user

Changing the default subvolume with btrfs subvolume set-default will make the top level of the filesystem inaccessible, except by use of the subvol=/ or subvolid=5 mount options.
If top level is no longer your default subvolume, it is useful to have an fstab entry for the top level (note the noauto option!):
LABEL=toplevel /root/btrfs-top-lvl btrfs subvol=/,defaults,noauto,noatime 0 0
scurbbing and repairing btrfs; most of the hints below originate from Marc’s 2014 btrfs presentation
scrub script: http://marc.merlins.org/linux/scripts/btrfs-scrub
mount -o compress=lzo is fast and best for SSDs; mount -o compress=zlib compresses better but is slower
you can turn off COW (copy-on-write) for specific files and directories with chattr -C /path (new files will inherit this)
for a highly fragmented VM, btrfs filesystem defragment vbox.vdi could take hours; cp -reflink=never vbox.vdi vbox.vdi.new; rm vbox.vdi is much faster
cp -reflink=always copies within and between subvolumes without duplicating data
run scrub nightly or weekly on all btrfs filesystems
noatime is the best fstab mount option for btrfs IF USING SNAPSHOTS, because relatime still causes at least 1 write every 24h
if you have errors=remount-ro in your fstab for extX-filesystem, you should remove it after converting to btrfs as it does not understand the option.

We are almost ready to start! Just one last thing: partitioning!

On my previous server, I did have a certain partition scheme, with separate /boot, /home, /var, /usr, and everything else under / partition.
It worked very well for me, never ran into problems (although this is more like a Debian’s feature, not that of my partition scheme Image may be NSFW.
Clik here to view. ).
However… With the current VPS, it started initially with just everything under /, in a single partition.
I could have re-partitioned everything from the rescue system while converting to btrfs, but what for? What would be the benefits?

Let’s have a look and evaluate

Advantages/disadvantages of keeping specific partitions separate.

/boot comes the first. Normally I format it with ext2, and set aside just 100-200 megabytes for this small partition.
This improves boot-reliability of the system. With btrfs, it was also the case that grub could not use /boot on a btrfs volume.
However,
Newer GRUBs can handle a /boot partition which is btrfs, so you need not have a separate /boot partition formatted as ext3/4.
So this time I’m going without a separate /boot.
I will not even create a separate subvolume for it: for snapshotting purposes, /boot fits perfectly under the / snapshots.
Next comes /home.
The reason to separate it is to safeguard user data when upgrading/reinstalling the OS.
This, however, can be achieved with btrfs subvolumes – no need for partitions!
I’m going with a “subvolume for home” here, then.
Next comes /var, or more precisely /var/log.
One of the reasons to keep /var/log separate is to prevent filesystem overfill if logs suddenly start growing too fast.
This, however, can be achieved with subvolume quotas. No need for a partition!
There is also /var/lib/ (and also /var/lib/lxc/…), which hold fairly big data files, often with random write access patterns.
There are two reasons to make them subvolumes:
1. for data-only snapshots
2. to enable nocow/nodatacow at mount time, to avoid high fragmentation with random writes
However, this can be done later, when these databases/LXC get installed – in just a few commands, without rebooting.
Finally, / itself should be a separate subvolume – mostly for snapshot and reinstall/upgrade reasons.

Ok, now we are really going to

Convert the filesystem

Enable rescue system, write down access credentials. You may need to restart your VPS to boot into rescue system, but check with your VPS provider first.
Connect to and log in to the rescue system.
Install btrfs-tools in the rescue system, if not there yet. We’ll need btrfs-convert, which should be included in btrfs-tools.
I’m assuming that your SSD/HDD is device /dev/sda, and that the only (root) partition there is /dev/sda1; adjust the next steps if your setup is different. Only proceed with commands if you understand what they are doing!
It never hurts to fsck /dev/sda1 before doing filesystem conversion Image may be NSFW.
Clik here to view. You should see something like /dev/sda1: clean, 27282/1564672 files, 351853/6249472 blocks.
btrfs-convert -l toplevel /dev/sda1; note that I’m adding a “toplevel” LABEL to the top level of btrfs, for easier mounting it later. The output should look like this:
creating btrfs metadata.
creating ext2fs image file.
cleaning up system chunk.
conversion complete.
At this point the system is already btrfs! Image may be NSFW.
Clik here to view.
If you forgot to set btrfs label, or decide to do it later: list the filesystems with btrfs filesystem show, or just look for the label with btrfs filesystem label /dev/sda1 (which will produce an empty line if there is no label), then simply btrfs filesystem label /dev/sda1 toplevel; it will produce no output on success.
Let’s change into the new system, and start configuring it:
mount /dev/sda1 /mnt for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done chroot /mnt
ls now should show your root filesystem files and directories, plus the ext2_saved “directory” (which is a subvolume, actually):
bin dev ext2_saved initrd.img installimage.debug lib64 media opt root sbin sys usr vmlinuz
boot etc home installimage.conf lib lost+found mnt proc run srv tmp var
Run blkid /dev/sda1 to find out the UUID of the new btrfs top-level volume. You can also see it with btrfs filesystem show:
/dev/sda1: LABEL=”toplevel” UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ UUID_SUB=”ffad5668-2ac7-4ea4-83ad-e6ccba7ccf96″ TYPE=”btrfs” PARTUUID=”750b31a7-01″
Edit /etc/fstab (with e.g. nano /etc/fstab):
- change UUID of / to the new one
- change the filesystem type from ext3/ext4 to btrfs
- change the options to subvol=root,defaults,noatime,compress=lzo,ssd,discard (yes, subvol=root does not exist yet… also, ssd,discard are for SSDs only, and instead of adding discard here it might be better to setup an fstrim cronjob. Finally, noatime can be left out if you are not going to make snapshots.)
- change the last number in the line from 1 to 0 (the fsck passes – btrfs does not want boot-time filesystem checking)
Some guides mention editing /etc/grub.d/00_header in a specific way, running grub-mkconfig | grep ” ro “ to see if it adds the proper rootflags=subvol=root, then even run update-grub and grub-install /dev/sda here… But all of this seems redundant or plain unnecessary. First of all, the root subvolume simply does not exist yet, so grub-mkconfig will not produce correct rootflags. Secondly, we are not yet done with setting up subvolumes, and we are not going to reboot right now, so running update-grub/grub-install is a bit premature at this stage Image may be NSFW.
Clik here to view.
Now we are going to create some subvolumes. As I want to name the root subvolume simply root, I should first move the root user’s home files somewhere else (otherwise btrfs snapshot complains “ERROR: incorrect snapshot name ‘/’”):
mv /root /root_orig btrfs subvolume snapshot / /root
The same problem with /home, need to move it as well: mv /home /home_orig
I can NOT snapshot /home like I did with /, because it is just a directory, not a volume like /, so:
btrfs subvolume create /home
Note: my /home (/home_orig) is still empty, as this is a brand-new installation with only the root user, so double-check steps below, as I haven’t really run them.
rsync --progress -aHAX /home_orig/* /home ls -la /home/ ls -la /home_orig/ rm -fr /home_orig/* ls -la /root/home/ rm -fr /root/home/* # be sure to leave the empty /root/home/ in place: it will be the mount point.
nano /etc/fstab to add the /home mount line, using the same UUID as earlier for the / partition, but a different subvol option:
UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /home btrfs subvol=home,defaults 0 0
Feel free to add more mount options, similar to those for root subvolume.
Follow the same procedure to create the /var/log subvolume. I’m using flat subvolumes layout here:
btrfs subvolume create /log rsync --progress -aHAX /var/log/* /log ls -la /log/ ls -la /var/log/ rm -fr /var/log/* ls -la /root/var/log/ rm -fr /root/var/log/*
Add /var/log entry to fstab:
UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ /var/log btrfs subvol=log,defaults,ssd,discard,compress=lzo,noatime 0 0
/var/lib/ – databases, LXC, nodatacow/nocow – can be created later, not doing it here and now.
cp /etc/fstab /root/etc/fstab
exit # chroot
for fs in proc sys dev/pts dev; do umount /mnt/$fs; done umount /mnt
Let’s enter the new root subvolume!
mount -o subvol=root /dev/sda1 /mnt for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done chroot /mnt
Oh, we forgot to move back root_orig into /root! That’s easy: mv root_orig root. If you want your root’s aliases/profile to load, then just exit and chroot /mnt again (I had to do that for the nice pre-configured prompt colours Image may be NSFW.
Clik here to view. )
btrfs subvolume list / to see our new and shiny subvolumes:
ID 256 gen 8 top level 5 path ext2_saved
ID 260 gen 49 top level 5 path root
ID 261 gen 35 top level 5 path home
ID 262 gen 42 top level 5 path log
Let’s make root the default subvolume! If suddenly rootflags get lost during grub configuration, then the system should still be able to boot. Subvol ID comes from the output of the command above (260 in my case):
btrfs subvolume set-default 260 / # no output
btrfs subvolume get-default / # to verify
ID 260 gen 49 top level 5 path root
Now let’s check grub-mkconfig output: grub-mkconfig | grep " ro ". Now that the subvolume exists, it does show all the desired rootflags=subvol=root.
Let’s make sure our system will be bootable:
update-grub grub-install /dev/sda
Leave the chroot:
exit for fs in proc sys dev/pts dev; do umount /mnt/$fs; done umount /mnt
Now let’s mount top-level again, for cleanup. If you remember, we had all the / files there – which are now in the root subvolume:
mount -o subvol=/ /dev/sda1 /mnt
Navigate to /mnt (cd /mnt, yes Image may be NSFW.
Clik here to view. ) and delete from here everything that is NOT a subvolume.
The command which supposedly does this is find /mnt -xdev -delete (-xdev is supposed to not descend into mounts of other filesystems), but I haven’t used it – was a bit tired and did not want to screw up the so-far-successful conversion, so I just did rm -rf bin boot dev etc initrd.img lib lib64 lost+found media mnt opt proc run sbin srv sys tmp usr vmlinuz var root_orig – basically, deleted everything except for ext2_saved, home, root, and log – which are all subvolumes.
Confirm that only subvolumes are left now:
ls -la:
drwxr-xr-x 1 root root 10 Feb 13 21:30 ext2_saved
drwxr-xr-x 1 root root 0 Feb 13 22:04 home
drwxr-xr-x 1 root root 510 Feb 13 22:13 log
drwxr-xr-x 1 root root 318 Feb 13 22:18 root
btrfs subvolume list /mnt:
ID 256 gen 8 top level 5 path ext2_saved
ID 260 gen 52 top level 5 path root
ID 261 gen 35 top level 5 path home
ID 262 gen 42 top level 5 path log
btrfs subvolume get-default /mnt:
ID 260 gen 52 top level 5 path root
We can leave now: cd / && umount /mnt
Reboot into the new system. If this fails: can boot rescue again, chroot and check what went wrong…
In my case, conversion seemed to have succeeded, BUT: a) for some reason /toplevel was mounted at boot and full of root-like files which should not have been there anymore; b) I could not delete ext2_saved, at all, neither from / nor from /toplevel. The problem was with the toplevel mount line in /etc/fstab (initially, I forgot to specify the filesystem type). After fixing this and rebooting once more, everything was perfect: /toplevel not mounted, and when mounted it only had subvolumes in it. And of course the system booted without any problems. Now I could delete the previous filesystem snapshot: mount /toplevel; btrfs subvolume delete /toplevel/ext2_saved; umount /toplevel. Note that there was also another, different ext2_saved in the root subvolume: it is NOT a subvolume, which was not carried over when we created the / snapshot – only the empty directory was left in place. (Yes, snapshots do not descend into subvolumes.)

The last 2 commands to issue (after mount /toplevel) are btrfs filesystem defragment -r /toplevel and btrfs balance start /toplevel.
I’m not including them in the instructions above: they are not strictly required, and if your filesystem had some heavy use before conversion, you may want to run these 2 commands more intelligently, in phases (metadata first, then big files, then smaller, etc). Another reason is that it may produce significant system load. On my fresh system both finished very quickly.

…And we are done!

Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.