The Linux Philosophy for SysAdmins, Tenet 04 — Use the Linux FHS

Author’s note: This article is excerpted in part from chapter 6 of my book, The Linux Philosophy for SysAdmins, with some changes.

The Linux Filesystem Hierarchical Standard (FHS) defines the structure of the Linux directory tree. It names a set of standard directories and designates their purposes.

This standard has been put in place to ensure that all distributions of Linux are consistent in their directory usage. Such consistency makes writing and maintaining shell and compiled programs easier for SysAdmins because the programs, their configuration files, and their data, if any, should be located in the standard directories. This chapter is about storing programs and data in the standard and recommended locations in the directory tree and the advantages of doing so. You will learn how to refer to the Linux FHS documentation and use that knowledge in problem solving.

Definitions

Before we get too deep into this subject, let’s put some definitions of the word “filesystem” in place to try and sort some of the confusion you are likely to find about terminology. You may hear people talk about filesystems in a number of different and confusing ways. The word itself can have multiple meanings and you may have to discern the correct meaning from the context of a discussion or document.

I will attempt to define the various meanings of the word, “filesystem” based on how I have observed it being used in different circumstances. Note that, while attempting to conform to standard “official” meanings, my intent is to define the term based on its various usages.

1. The entire Linux directory structure starting at the top (/) root directory.
2. A specific type of data storage format such as EXT3, EXT4, BTRFS, XFS, and so on. Linux supports almost 100 types of filesystems including some very old ones, as well as some of the newest. Each of these filesystem types uses its own metadata structures to define how the data is stored and accessed.
3. A partition or logical volume formatted with a specific type of filesystem that can be mounted on a specified mount point – a directory – on a Linux filesystem.

I will be using the term “filesystem” in the context of all of these definitions at some point in this chapter.

The standard

As SysAdmins our tasks include everything from fixing problems to writing CLI programs to perform many of our tasks for us and for others. Knowing where data of various types are intended to be stored on a Linux system can be very helpful in resolving problems as well as preventing them.

The latest Filesystem Hierarchical Standard (3.0) is defined in a document maintained by the Linux Foundation¹. The document is available in multiple formats from their web site as are historical versions of the FHS.

Figure 1 provides a brief list of the standard, well known, and defined top-level Linux directories and their purposes. These directories are listed in alphabetical order. I suggest that you read the entire document in order to understand the roles played by the many subdirectories of these top-level ones.

Note column 2, the middle column, in Figure 1. All of the directories with a “Yes” in this column must be an integral part of the root (/) filesystem. None of these directories can be located on separate partitions or logical volumes; they must all be located in the same partition or logical volume as the root filesystem because they are an integral part of it. These directories must all be mounted at the beginning of the boot process as a single unit with the root filesystem.

The directories that have a “No” in column 2 can be created on separate partitions or logical volumes – although they do not have to be separate but they can be. These filesystems, when they are separate from the root filesystem, are mounted later in the startup sequence based on the information contained in the /etc/fstab file. There are some very good reasons for mounting these directories as separate filesystem and I will discuss those later in this chapter.

The /media and /mnt directories are mount points for temporary filesystem maintenance or for external devices such as USB thumb drives that contain filesystems.

There is actually one exception to the “top level” statement about Table 1. That is the /usr/local directory. I will discuss that directory in more detail a bit later in this chapter.

Using a well-defined filesystem structure

There are some excellent reasons for following the Linux Filesystem Hierarchical Standard. All of them make my life as a SysAdmins easier. Don’t worry, I am not going to discuss the functions of each of the directories defined in the standard – after all, you can read what I have written and the more detailed on-line version just as easily as I can. What I am going to do is discuss how a couple specific features of this FHS affect how I do my work.

The purpose of the Linux FHS is to provide a well-defined structure in which to store files, whether executables, data, or configuration files. The structure defined in the document, Filesystem Hierarchical Standard (3.0), and previously referenced, sets forth guidelines for file locations in Linux which are based in historical context dating back to the early days of Unix, as well as new, updated, and changing standards and conventions.

The fact is that usages do change. It is also true that the Filesystem Hierarchical Standard has changed with the times. Even further, not all distributions and software vendors interpret the FHS in the same way and some software vendors may just be ignorant of the standard.

Regardless of these facts, it is incumbent upon us as SysAdmins to adhere to the current standard in all the ways that are under our control. We cannot always control the usage by vendors, but we can certainly have our say. Don’t misunderstand me — I see no widespread problems here but if there is a problem I believe that as responsible SysAdmins we should report those issues to the proper vendor.

We should also adhere to these standards ourselves when we write code even when it seems to be just a small, insignificant bit of CLI programming.

Linux unified directory structure

The Linux filesystem directory structure consists of a hierarchy of mountable filesystems. This results in easier and more consistent access to all of the directories in the hierarchy. It also provides some very useful side effects.

The Linux filesystem unifies all physical hard drives, partitions, and logical volumes into a single directory structure. It all starts at the top – the root (/) directory. All other directories and their subdirectories are located under the single Linux root directory. This means that there is only one single directory tree in which to search for files and programs.

This can work only because a filesystem, such as /home, /tmp, /var, /opt, or /usr can be created on separate physical hard drives, a different partition, or a different logical volume from the / (root) filesystem and then be mounted on a mountpoint as part of the root filesystem directory tree. Mountpoints are just empty directories with nothing special about them. Even removable drives such as a USB thumb drive, an external USB or an ESATA hard drive will be mounted onto the root filesystem and become an integral part of that directory tree.

One good reason to do this is apparent during an upgrade from one version of a Linux distribution to another, or changing from one distribution to another. In general, and aside from any upgrade utilities like dnf-upgrade in Fedora, it is wise to occasionally reformat the root and other partitions containing the operating system during an upgrade to positively remove any cruft that has accumulated over time. If /home is part of the root filesystem it will be reformatted as well and would then have to be restored from a backup. By having /home as a separate filesystem, it will be known to the installation program as a separate filesystem and formatting of that filesystem can be skipped. This can also apply to /var where database, email inboxes, web site, and other variable user and system data are stored, and the /opt filesystem where commercial applications are intended to be stored. Thus none of that data is lost and the applications should not require reinstallation unless the vendor is incredibly stupid.

I once had a situation where a Linux host continued to run, but prevented the user from logging in using the GUI desktop. I was able to login using the command line interface (CLI) locally using one of the virtual consoles, and remotely using SSH. The problem was that the /tmp filesystem had filled up and some temporary files required by the GUI desktop could not be created at login time. Because the CLI login did not require files to be created in /tmp, the lack of space there did not prevent me from logging in using the CLI. In this case the /tmp directory was a separate filesystem and there was plenty of space available in the volume group of which the /tmp logical volume was a part. I simply expanded the /tmp logical volume to a size that accommodated my fresh understanding of the amount of temporary file space needed on that host and the problem was solved. Note that this solution did not require a reboot and as soon as the /tmp filesystem was enlarged the user was able to login to the desktop.

Special filesystems

Linux has some special filesystems when running, two of which are particularly interesting to SysAdmins, /proc and /sys. These are virtual filesystems that exist only in RAM while the Linux host is running; they do not exist on any physical disk. Because they exist only in RAM these filesystems are not persistent like filesystems that are stored on the hard drive. They disappear when the computer is turned off and are recreated anew each time Linux starts up.

Each of the special filesystems has a unique role to play in a Linux host. The /proc filesystem is most likely the one with which you will become well acquainted as a SysAdmin so we are going to explore it just a bit.

The /proc filesystem

The /proc filesystem is defined by the FHS as the location for Linux to store information about the system, the kernel, and all processes running on the host. It is intended to be a place for the kernel to expose information about itself in order to facilitate access to data about the system. It is also intended to be provide access to view kernel configuration parameters and to modify many of them when necessary.

When used as a window into the state of the operating system and its view of the system and hardware, it provides easy access to virtually every bit of information you might want as a SysAdmin. For best results with this experiment it must be performed as root.

Let’s look at the top-level contents of the /proc filesystem of a running Linux host. On your host you may see color coding to differentiate files from directories. First, look at the numeric entries. The names of these directories is a PID, or process ID number. Each of those PID directories contains information about the running process that it represents.

[root@testvm1 ~]# cd /proc ; ls

The data from the cpuinfo file includes the processor ID and model, its current speed in MHz, and the flags that can be used to determine the CPU features.

Now lets look at memory. First cat the meminfo file and then use the free command to do a comparison.

[root@testvm1 proc]# cat meminfo ; free

There is a lot of information in the /proc/meminfo file. A few bits of that data are used by programs like the free command. If you want the complete picture of memory usage, look in /proc/meminfo. The free command, like many other core utilities, gets its data from the /proc filesystem.

Because the data in /proc is a nearly instantaneous picture of the state of the Linux kernel and the computer hardware, the data may change rapidly. Look at the interrupts file several times in a row.

The /sys filesystem

The /sys directory is another virtual filesystem that is used by Linux to maintain specific data for use by the kernel and SysAdmins. The /sys directory maintains the list of hardware hierarchically for each bus type in the computer hardware. A quick look at the /sys filesystem shows us its basic structure.

[root@testvm1 ~]# cd /sys ; ls

There are different types of disk (block) devices in /sys/block and the sda device is one of them. This is usually the first, and in this case the only, hard drive in this VM. Explore the contents of the sda directory.

[root@testvm1 sys]# ls block/sda

Here’s a command that displays the hard drive device model information.

[root@david ~]# cat /sys/block/sda/device/model
ST320DM000-1BD14

If you have SSD storage, that information would be in a slightly different but logical location.

root@david:~# cat /sys/block/nvme0n1/device/model
INTEL SSDPEKNW512G8

I sometimes find it helpful to find specific hardware devices, especially newly added ones. As with the /proc directory there are some core utilities like lsusb and lspci that make it easy for us to view information about the devices connected to the host.

Problem solving

One of the best reasons I can think of for adhering to the Linux FHS is that of making the task of problem solving as easy as possible. Using the Linux Filesystem Hierarchical Standard promotes consistency and simplicity which makes problem solving easier. Knowing where to find things in the Linux filesystem directory structure has saved me from endless flailing about on more than just a few occasions.

I find that most of the Core Utilities, Linux services and servers provided with the distributions I use are consistent in their usage of the /etc directory and its subdirectories for configuration files. This means that finding a configuration file for a misbehaving program or service supplied by the distribution should be easy to find.

I typically use a number of the ASCII text files in /etc to configure SendMail, Apache, DHCP, NFS, NTP, DNS, and more. I always know where to find the files I need to modify for those services, and they are all open and accessible because they are in ASCII text which makes them readable to both computers and humans.

There would appear to be an inconsistency with BIND DNS because its zone, reverse, and the root hints file, named.ca, are located in /var/named. This is not inconsistent because those are not configuration files, they are database files, which, as you can see in Figure 1, is one of the functions of /var. Also, those “variable” files may be modified by external servers such as when a primary name server updates the database of a secondary name server. Keeping those external servers out of the main configuration directory, /etc, on our computer is a really good idea.

The location of the BIND database files is consistent with the FHS. But it did take me a while to figure that out and why it is so, not to mention extensive research into the FHS. Sometimes my curiosity can take me on long detours but I have always learned a great deal from those journeys that has been useful later on.

Using the filesystem incorrectly

One situation involving the incorrect usage of the filesystem occurred while I was working as a lab administrator at a large technology company in Research Triangle Park. One of our developers had installed an application in the wrong location, /var. The application was crashing because the /var filesystem was full, and the log files, which are stored in /var/log on that filesystem, could not be appended with new messages that would indicate that the /var filesystem was full because due to the lack of space in /var. However the system remained up and running because the critical / (root) and /tmp filesystems did not fill up. Removing the offending application and re-installing it in the /opt filesystem, where it was supposed to be, resolved that problem. I also had a little discussion with the developer who did the original installation.

Adhering to the standard

So how do we as SysAdmins adhere to the Linux FHS? It is actually pretty easy and there is a hint way back in Table 1. The /usr/local directory is where locally created executables and their configuration files should be stored. By local programs, the FHS means those that we create ourselves as SysAdmins to make our work or the work of other users easier. This includes all of those powerful and versatile shell programs we write.

The programs should be located in /usr/local/bin, and the configuration files, if any, in /usr/local/etc. There is also a /var/local directory in which local programs can store their own database files.

I have written a fair number of shell programs over the years and it took me at least five years before I understood the appropriate places to install my own software on host computers. In some cases I had even forgotten where I installed them. In other cases, I installed the configuration files in /etc instead of /usr/local/etc, and my file was overwritten during an upgrade. It took a few hours to track that down the first time it happened.

By adhering to these standards when writing shell programs, it is easier for me to remember where I have installed them. It is also easier for other SysAdmins to find things by searching only the directories that we as SysAdmins would have installed those programs and their files.

Summary

This article has explored the Linux filesystems. The hierarchical directory structure has standards that apply to the usage of the directories in that structure. Adhering to the standard usage conventions as outlined in the Linux Filesystem Hierarchical Standard as maintained by the Linux Foundation provides some significant benefits to SysAdmins. This can be especially true when portions of the directory tree containing data are created as independent filesystems and mounted separately.

The Linux filesystem is not simply a place to store programs and data. It is a place where data and statistics about the operating system, running programs, and even the hardware can be found and put to good use. The Linux FHS defines the directories where this information can be found so we know it will always be there for us when we need it.

Knowledge of what is contained in the Linux filesystem and where it is located can be an indispensable tool in performing problem determination.

1. The Linux Foundation maintains documents defining many Linux standards. It also sponsors the work of Linus Torvalds. ↩︎