The Linux Philosophy for SysAdmins, Tenet 12 — Use separate filesystems for data

Author’s note: This article is excerpted in part from chapter 14 of my book, The Linux Philosophy for SysAdmins, with changes.

There is a lot to this particular tenet and it requires understanding the nature of Linux filesystems and mount points. If you missed Tenet 4, “Use the Linux FHS,” you should go back and read it now.

Has your computer hard drive ever crashed leaving your Linux computer unable to boot; with all of your data on the on the crashed hard drive; with no recent backups? Most of us have. And our friends, co-workers, or customers have all likely experienced this as well.

Keeping data of all types safe is part of the SysAdmin’s job. Using separate filesystems for storing that data can help us accomplish that.

Note: The primary meaning for the term “filesystem” in this article is a segment of the directory tree that is located on a separate partition or logical volume that must be mounted on a specified mount point of the root filesystem to enable access to it. We also use the term to describe the structure of the metadata on the partition or volume such as, EXT4, XFS, or other structure. These different usages should be clear from their context.

Laptop lament

This happened to me back on December 26, 2017 – a friend of mine sent me a text to let me know that the laptop computer I have been supporting for her would not boot.

A few years prior to that, my friend, Cyndi, who is also my Yoga instructor, was becoming unhappy because her computer was constantly slowing down due to malware infestations. She asked me to help and I agreed. I told her about Linux, and she decided to try it.

Over the years I have helped Cyndi keep her computer running with hardware repairs, software updates, and upgrades to newer releases of Fedora, which is what I started her out with. She always calls me first when she has a computer problem that she cannot figure out on her own. Today was no exception.

The first text said that her computer would not boot and it was printing a repeating series of messages on the screen. After a couple text exchanges I determined that I should look at the laptop and she brought it to my home office. Based on the messages that showed a long series of I/O errors on /dev/sda, I determined that the hard drive was failing. I also told her that I was not sure I could salvage the data from the hard drive but would try.

At this point I checked her external USB backup drive and determined that the latest backups were several months old. That part is my fault and I did fix it. We discussed the situation and she told me to do what I could. After she left, I removed the 320GB hard drive from the laptop and inserted it in one of the slots in my hard drive docking station for exploration.

Here is what I already knew. When I installed Linux on the laptop, I used separate filesystems for /usr, /var, /tmp, /swap, and /home, like I always do when I install Linux. That meant that the /home filesystem was on a different logical volume than the root (/) filesystem. Because the problem was during the startup sequence – technically after boot and during startup while systemd was starting various services – there was a good possibility that the other logical volumes were unaffected, including the volume on which I had installed her home directory, /home.

After the hard drive spun up, I used the lvscan tool to locate all of the logical volumes on my main workstation. The results included the logical volumes on the defective hard drive as shown in Figure 1.

[root@david ~]# lvscan
  ACTIVE            '/dev/fedora_mobilemantra/home' [<48.83 GiB] inherit
  ACTIVE            '/dev/fedora_mobilemantra/root' [<29.30 GiB] inherit
  ACTIVE            '/dev/fedora_mobilemantra/tmp' [19.53 GiB] inherit
  ACTIVE            '/dev/fedora_mobilemantra/var' [19.53 GiB] inherit
  ACTIVE            '/dev/fedora_mobilemantra/usr' [<34.18 GiB] inherit
  ACTIVE            '/dev/fedora_mobilemantra/swap' [7.81 GiB] inherit
  ACTIVE            '/dev/vg_david2/home' [50.00 GiB] inherit
  ACTIVE            '/dev/vg_david2/stuff' [130.00 GiB] inherit
  ACTIVE            '/dev/vg_david2/Virtual' [590.00 GiB] inherit
  ACTIVE            '/dev/vg_david2/Pictures' [75.00 GiB] inherit
  ACTIVE            '/dev/david1/root' [9.31 GiB] inherit
  ACTIVE            '/dev/david1/tmp' [<28.63 GiB] inherit
  ACTIVE            '/dev/david1/var' [<18.63 GiB] inherit
  ACTIVE            '/dev/david1/usr' [<46.57 GiB] inherit
  ACTIVE            '/dev/david1/swap' [14.90 GiB] inherit
  ACTIVE            '/dev/vg_Backups/Backups' [3.63 TiB] inherit

Figure 1: lvscan displays the logical volumes including those on the defective hard drive.

The results from the lvscan command also lists the device files assigned to the logical volumes on the laptop hard drive. These device files were created by udev¹ as soon as the hard drive was up to speed and could be read. As Seth mentions in his article, “How to use udev,” udev is responsible for detecting when new devices are plugged into the system, and creating device files for them in /dev.

Among the listed logical volumes is /dev/fedora_mobilemantra/home which is the home directory for my friend’s hard drive. At this point I knew all I needed in order to mount the home directory on /mnt on my workstation. I did that and started exploring the home directory. Everything looked good and I was able to read several of the files without a problem.

I was fortunate that the hard drive had not failed catastrophically. In this case the errors were apparently bad sectors on the / (root) partition which left the other partitions intact so they could be recovered.

The simplest way to both create a backup of the home directory and discover if any of the many files located there have errors is to use the tar command. No errors occurred during the creation of the tarball so I was able to retrieve all of the data from the home directory.

Cyndi was very happy when I called her with this news of the continued digital existence of her data.

I used the dd command, as shown in Figure 2, to perform a quick test of the entire hard drive. An I/O error occurred after reading only 115GB out of a total of 320GB. I could have continued to focus in on the location of the error but it is enough to know at this time that the errors caused problems with startup which indicates they are located in the / (root) filesystem. Preceding the dd command with the time command enables me to see how much time was used by the dd command.

[root@david ~]# time dd if=/dev/sdi of=/dev/null
dd: error reading '/dev/sdi': Input/output error
224078480+0 records in
224078480+0 records out
114728181760 bytes (115 GB, 107 GiB) copied, 1551.07 s, 74.0 MB/s

real    25m51.071s
user    0m23.646s
sys     2m36.991s

Figure 2: Using the dd command to test the hard drive. This also shows the I/O error that occurred.

Sending the “outfile” data to /dev/null prevents it from being displayed in the terminal session as STDOUT. Displaying STDOUT to the terminal also significantly slows down the entire process. However any STDERR messages indicating I/O errors will still display on the terminal. Neither the hard drive nor any of its partitions were mounted while running this command.

I ordered a new hard drive for her laptop and installed that. I then installed Fedora 27, the most current release at that time, and restored the saved data to her home directory. Everything worked fine and all her data proved to be intact.

This story is a perfect illustration of one reason to use separate filesystems for data. It also shows quite nicely that understanding the Linux Filesystem Hierarchical Structure can be important to us as SysAdmins. It shows an appropriate use for the /mnt mount point and the /dev/null device. It is also a nice example of the fact that everything is a file and the device special files in /dev can be used with simple tools.

Data security

This tenet is about data security as much as anything else. In this context I mean security in the sense of maintaining the continued existence and integrity of the data. Today’s hard drives are huge, with some running into multiple terabytes. And hard drives are one of the most common devices to fail in a computer, along with others that have mechanical moving parts such as fans. So the larger the hard drive is, the more data that could be lost when it fails.

Of course one part of securing your data is to make backups. Another very important part is ensuring that the data – the actual data like documents, project files, financial files, graphics, video, audio, user configuration files, and more, is as safe from the results of corruption as possible. Because backup systems fail, too.

According to the Linux Filesystem Hierarchical Standard, “Disk errors that corrupt data on the root filesystem are a greater problem than errors on any other partition. A small root filesystem is less prone to corruption as the result of a system crash.²” The reasoning behind this is that in most systems the greatest number of disk writes takes place in the root partition so it is most likely to be corrupted by a problem. This seems to be the case in the previous example.

The corollary to that is that directories that are part of the root filesystem are more likely to suffer from the side effects of those system crashes than directories that are not. This is demonstrably the case because I was unable to mount the root filesystem so could not recover any of the files on it. I was able to mount the /home filesystem.

And that leads us to the conclusion that maintaining separate filesystems for directory trees that contain user data is a good idea. It also reinforces the statement in the above quote that the root filesystem should be as small as reasonably possible. The amount of used space in the root filesystem of my primary workstation is only 1.5GB, which is not a lot these days. Nevertheless, I do recommend, considering the huge size of current hard drives, that the root filesystem be allocated at least 5GB of disk space to allow for unexpected occurrences.

Recommendations

There are some specific parts of the Linux directory tree that I recommend being placed on separate filesystems. Sometimes I even recommend placing them on a separate hard drive to further ensure their safety and to facilitate recovery because the data on the filesystems maintained on a separate drive would not need to be restored from backup if the drive containing the root filesystem needs to be replaced. This can also speed access to the data on the storage drives.

All that is required to mount the preexisting filesystems as part of the re-installed directory tree is to add appropriate entries in the /etc/fstab file. This enables them to be mounted when the system is rebooted after the installation of the operating system in a new drive. Configuring this can be accomplished during the Linux installation using a “custom” disk configuration. The details of that procedure are outside the scope of this book.

Current Red Hat based installations like RHEL and Alma Linux use a default disk configuration that can be far from ideal. However, Fedora uses BtrFS³ as the default filesystem and it has a unique method for allocating storage space. All of the data space on the storage device is used as a pool of available file extents and is not allocated to any filesystem mount point. This strategy lumps all historical filesystems into a single data space on the storage device — totally ignoring the advice proffered by the Linux Filesystem Hierarchical Standard.

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  120G  0 disk 
├─sda1   8:1    0    1M  0 part 
├─sda2   8:2    0    1G  0 part /boot
└─sda3   8:3    0  119G  0 part /home
                                /
sr0     11:0    1 1024M  0 rom  
zram0  252:0    0  7.8G  0 disk [SWAP]

Figure 3: The /dev/sda3 device special file contains two filesystems. / (root) and /home. They share the pool of available file extents.

The actual numbers vary depending upon the size of the hard drive. But the result is still the same. I dislike BtrFS as it’s very unstable and I won’t use it.

Those defaults can result in sub-optimal disk usage and lead to problems later in the life of the host. Even though /home is a separate filesystem on its own logical volume in these default installations, its size is far too large for many environments. For example, the home directory on my main workstation has only about 30GB of data including this book and lots of photos, all going back over 25 years. There are others filesystems that need to be considered as well.

Three of the main branches of the Linux filesystem hierarchy are specifically designed to be located on separate partitions or volumes as a separate filesystem⁴. This is possible because conforming to the Linux FHS makes it so. These three branches are /usr, /opt, and /var. I have also found other branches that work well as separate filesystems, /home, /tmp, and /opt.

Let’s look at the branches of the directory tree that I recommend as being separate filesystems. All of these filesystems can be placed on one or more hard drives separate from the drive that contains the root filesystem. This helps to ensure the overall survivability of those directory branches that are not located on the failing drive.

You can refer to the Filesystem Hierarchical Standard V3.0 for more details.

/boot

The /boot directory tree is an interesting one because it cannot be part of a logical volume configuration. It must be a separate disk partition with a Linux EXT3, EXT4, or XFS filesystem. Other filesystems may be supported, but Reiser and HFS are explicitly stated as not supported, and BtrFS is not supported by my empirical observations.

Because most modern distributions use logical volume management, this directory must be a separate filesystem. If this is the case, you won’t have a choice. However if you do not use logical volume management and use filesystems such as EXT4 that are directly created on the raw partitions, you could make /boot part of the root (/) filesystem.

I recommend that /boot always be a separate filesystem even if logical volumes are are not used for other filesystems on the host.

/home

Clearly /home should be a separate filesystem. The experience I discussed above, as well as others like it over the years have made it abundantly clear to me that /home should always be a separate filesystem from the rest of the directory tree. The default filesystem configurations of the distributions with which I am familiar make it clear that this is a best practice aside from any size issues.

When upgrading from one version of Fedora to the next, for example, having my data, especially /home on a separate filesystem has made release level upgrading easy even when I choose not to use the provided upgrade tools and simply install a new release of Fedora over an old one. This requires the same procedure as recovery from root filesystem corruption; it would be necessary to perform a custom configuration of the filesystem and choose to keep the existing home directory without formatting it.

/home is also one of the filesystems I recommend be placed on a hard drive separate from the one on which the operating system is located. This helps to ensure that the data on /home is safe if the primary drive fails. It also improved performance by spreading disk access over more than one hard drive so that the operating system does not have to wait for user data access and vice-versa.

Although the FHS specifies /home as the “home” directory, it also recognizes that the location of the home directories in the directory tree has typically been subject to the standards of the organizations and the discretion of the SysAdmins managing the systems. I have encountered /var/home, /opt/home, /homedirs, and others. I like to follow the standard in this.

Having the home directory as a separate filesystem enables it to be moved to a different mount point if the need arises. Other things may need to be changes as well in order to make this work, such as default paths.

The root user’s home directory is /root and this directory should remain part of the root filesystem. The reason for this is to ensure that the tools and files that we store in the root home directory for convenience will still be available in recovery modes and runlevels that don’t mount the other filesystems.

Many non-login accounts related to system services also have home directories in other locations. The specific location varies by service but is generally a directory that is part of the service itself.

/usr

The /usr branch contains non-essential commands, i.e., user commands and not system administration commands that would be required during boot and startup or when running in recovery mode.

I typically make /usr a separate filesystem as much to prevent problems with it being part of the root filesystem as to ensure the security of the data here. Most user-level commands and libraries are located here. This is considered to be a directory tree with static files – ones that typically do not change in the course of the host’s operation.

One of the reasons this tree is called the “secondary hierarchy” is that in many ways it has a structure similar to the main tree starting with the root directory. There is one subdirectory tree that is for local files, /usr/local.

The /usr/local directory tree contains subdirectories etc, bin, sbin, include, lib, lib64, share, and more. The purpose of the /usr/local tree is to be a place where local programs and configuration files can be stored.

This is where I place all of my locally written scripts and any configuration files they require. The scripts themselves are located in /usr/local/bin and the configuration files are placed in /usr/local/etc. Documentation such as man pages written for these scripts would be located in /usr/local/share.

Another option would be to make /usr part of the root filesystem but to make /usr/local a separate filesystem. I only back up /usr/local anyway and since I never do a bare metal recovery of the entire system this makes some sense. If I have an issue with the operating system or the hard drive on which it is installed that would require a complete re-installation, everything except for the /usr/local tree is recreated during the installation.

It is also possible for both /usr and /usr/local to be separate filesystems. The /usr filesystem would be mounted on the /usr mount point and then the local branch would be mounted on the /usr/local mount point.

The /user tree is explicitly not for large programs such as commercial software or large open source applications. It is for small to moderately sized locally coded programs to meet the needs of SysAdmins or local regular users.

Large software applications should be installed in another location. The /opt directory tree is recommended.

/opt

Large commercial programs should be installed in the /opt directory tree. This directory should be created as a separate filesystem so that it can be easily expanded in size if that should become necessary.

The /opt branch supports a complete hierarchy of subdirectories for multiple vendors to install their software as well as a complete set of directories reserved for local SysAdmin use, /opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man⁵.

/var

The /var branch of the Linux filesystem hierarchy is an interesting combination of – well, stuff. It is intended to contain “variable” data – data that can change, but that is not configuration data. Because the data in /var is not needed by the operating system in any type of recovery or maintenance mode, or during the initial boot up sequence, it can be safely created as a separate filesystem.

The data located in /var is user data and databases. We find many different types of data in /var. For example /www would contain the files required for a web site if the host were a web server. In fact, I have multiple sites running on my web server and they each have a directory of their own such as, /var/wwwboth, /var/wwwlinuxdatabook, and so on. This makes it easy to determine which directory branch contains the data for each web site. The only configuration needed to accomplish this is in the web site configuration files. I use the Apache web server so that would be /etc/httpd/conf/httpd.conf.

MariaDB – a fork of MySQL – maintains its databases in /var/lib/mysql. SendMail stores user inboxes in /var/spool/mail. BIND provides name services with the databases located in /var/named. And there is much more data stored there as well.

/tmp

The /tmp directory is a place where users and system services can store files temporarily. Think of /tmp a a place where data of any type can be stored by any user or process — temporarily. There is a strong likelihood that files stored in /tmp will be deleted, usually at the next boot or after a predetermined time on the order of a day or two.

I use /tmp for downloading large files such as ISO images like the installation images for various distributions. These can be quite large and, along with files created by various system processes can fill a small /tmp filesystem. Because of this I like to make my /tmp filesystem very large, usually 10GB or more. For my main workstation I currently have 15GB allocated to /tmp. In these days of multi-terabyte hard drives 15GB is a very reasonable size.

If the /tmp filesystem does fill up, strange things can happen. I once managed to fill up the /tmp filesystem. GUI desktop logins failed because the desktop could not create new files in /tmp, but the console and remote SSH logins continued to work because they don’t store files in /tmp.

However, the problem would be worse if /tmp were part of the root filesystem. In that case many other symptoms would be likely as various additional services were unable to find sufficient disk space in which to work.

The other branches

All of the other branches of the Linux filesystem hierarchy must be an atomic part of the root filesystem. They cannot be created as separate filesystems and mounted to the directory tree at the appropriate mount point. The programs and data stored in these other branches of the directory tree are required to be available during early stages of boot and when running in low-level recovery or maintenance modes.

Starting with separate filesystems

The best time to set up separate filesystems for one or more components of the Linux filesystem hierarchy is when the operating system is first installed. Most Linux installers, such as Fedora’s Anaconda, provide the ability to do a custom disk configuration during the installation. At that time you can specify separate logical volumes to contain one or more of the filesystems we have discussed in this article that are capable of being mounted separately.

The installers — at least the ones with which I am familiar – also have the ability to recognize existing partitions, volumes, and filesystems and display identifying information about them. This makes it easy to re-install Linux after the hard drive containing the OS crashes and needs to be replaced. It also enables easy release upgrades — from Fedora 39 to Fedora 40, for example – when a complete re-installation is desirable. The OS can be installed and the root filesystem can be reformatted without touching the /home and /var filesystems. That would keep intact all of my personal data, email inbox, and my web site data, for example.

Adding separate filesystems later

It is not particularly difficult to convert one or more of the directories discussed here to a separate filesystem to a system well after the original installation. It just requires some foresight and planning.

The basic process is simple. In fact there are multiple ways to do this. Here is what one process would look like for the /home directory. This procedure assumes that /home is not currently on a separate filesystem from the root filesystem.

1. Install a new hard drive if necessary.
2. Create a partition or logical volume on the drive.
3. Add a filesystem label to the new partition or volume. This makes the new filesystem easily identifiable when it is not mounted and allows mounting by label.
4. Back up the data in the current home directory. If there is room, store the backup in /tmp. This is one good reason to make /tmp large.
5. Delete the data from the current /home directory. This step frees up that space which would not be accessible after the new filesystem is mounted at /home on the main trunk of the filesystem hierarchy.
6. Add an entry in /etc/fstab that specifies the mount for the new filesystem on /home.
7. Mount the new volume on /home.
8. Restore the data to /home.
9. Test and verify that all data was correctly restored.

Final thoughts

Separate filesystems make our jobs as SysAdmins easier. Maintaining parts of the directory tree in separate filesystems we provide more flexibility in the case of drive crashes, enable flexibility in moving filesystems to different mount points, and make it easier to perform complete operating system re-installations when they are needed. It also improves the survivability of other parts of the directory structure in case one hard drive crashes.

It did take me a a while to figure out what a good idea this is, but since that time I have always maintained separate filesystems for the directories discussed here. That has saved my data more than once.

---

1. Unnikrishnan A, Linux.com, Udev: Introduction to Device Management In Modern Linux System, https://www.linux.com/news/udev-introduction-device-management-modern-linux-system ↩︎
2. LSB Work group – The Linux Foundation, Filesystem Hierarchical Standard V3.0, 3, https://refspecs.linuxfoundation.org/FHS_3.0 ↩︎
3. In my experience, the BTRFS filesystem has multiple functional problems which have caused issues every time I’ve tried it. I consider it unsafe and unusable. ↩︎
4. LSB Work group – The Linux Foundation, Filesystem Hierarchical Standard V3.0, 3, https://refspecs.linuxfoundation.org/FHS_3.0, 3. ↩︎
5. ibid., 13. ↩︎