Author’s note: This article is excerpted in part from chapter 17 of my book, The Linux Philosophy for SysAdmins, with some changes to update the information in it and to better fit this format.

https://www.cnn.com/travel/millau-viaduct-tallest-bridge-construction/index.html

Strive for elegance

Elegance is one of those things that can be difficult to define. I know it when I see it but putting what I see into a terse definition is a challenge. Using the Linux dict command, Wordnet provides one definition of elegance as, “a quality of neatness and ingenious simplicity in the solution of a problem (especially in science or mathematics); ‘the simplicity and elegance of his invention’”

I think that elegance is a state of beauty and simplicity in the design and working of both hardware and software. When a design is elegant, software and hardware work better and are more efficient. The user is aided by simple, efficient, and understandable tools.

Creating elegance in a technological environment is hard. It is also necessary. Elegant solutions produce elegant results and are easy to maintain and fix. Elegance does not happen by accident; you must work for it.

The quality of simplicity is a large part of technical elegance. So large, in fact that it deserves an article of its own, Chapter 18 in my book, “Find the Simplicity,” but we don’t ignore it here. This article discusses what it means for hardware and software to be elegant.

Hardware elegance

Yes, hardware can be elegant — even beautiful — pleasing to the eye. Hardware that is well designed is more reliable as well. Elegant hardware solutions improve reliability.

Many of us SysAdmins have responsibility for hardware as well as software. This is especially true of those of us who work in smaller organizations, but it can also be true in larger environments as well. In my 5-year stint at Cisco, the SysAdmin part of my job required me to rack and cable new servers, identify and fix hardware problems, help design rack layouts and power requirements, and more hardware related tasks.

Understanding hardware elegance is just as important as understanding software and operating system elegance.

Motherboards

Hardware elegance can mean the simple and well laid out design of a motherboard, which is a fairly large PCB. As previously discussed good motherboard design can improve reliability.

In my opinion a well laid out motherboard, one that looks good with sleek lines of the conductors and land patterns (mounting pads to which components are soldered) on the surface is elegant. A motherboard with well placed components that do not interfere with each other or with additional components that might be added later such as powerful but long video adapters, a CPU socket that is placed on the motherboard so that RAM memory and other motherboard components do not interfere with adding high-capacity air cooling fans or liquid cooling equipment – that is elegant design. Frankly, I have always appreciated the look of a well designed motherboard. These motherboards are truly works of art.

Computers

A well-designed computer is elegant. This includes a case that is designed with easy access to the internal components and that provides for plenty of unrestricted airflow, lots of locations to mount fans and liquid cooling radiators, and plenty of options for easy cable routing.

You can see one of my computers in Figure 1, with an ASUS PRIME H370M-PLUS motherboard in a CoolerMaster case with a large CPU air-cooling tower and multiple network interface cards (NICs). The cables are all routed to the rear of the main deck where they can be easily managed and out of the way. The motherboard is well designed and laid out so that the CPU cooler with it’s fan don’t impede access to the memory slots.

A computer is elegant when the external power and storage device activity LEDs are easily visible and a decent brightness so they can be seen in all lighting conditions. The power and reset switches are also easily accessible but not obtrusive so that they might be accidentally bumped and cause the computer to reset or power off.

I could go on but you get the idea.

Data centers

Hardware elegance also means a well planned and constructed computer room or data center. Rack enclosures are laid out in such a way that multiple power sources can reach them with ease and access to both front and rear are unimpeded. The cabling is neat and orderly; it is cut to length and flows without kinks or tangles through the cable channels and trays throughout the computer room. Figure 2 is an example of horrible cabling.

Uninterruptible power supplies should be used to maintain power to all devices until a generator can be brought on line and power switched over to that for long-term stability in the event of power failures.

Power and grounding

It is hard to imagine that power and grounding feeds to a computer should have a term like “elegance” applied to them but here is something to consider about that.

In about 1976 I was working for IBM as a Customer Engineer in Lima, Ohio. In addition to fixing broken computers and installing new ones, one of my duties was to assist customers in planning for the installation of new computers. This included planning for appropriate power and grounding. In this particular instance I had a long discussion with my customer about the requirements for power and a good ground. Good grounding is essential for the proper electronic operation and stability of computers.

So what constitutes a good ground? It is a large gauge wire with green insulation (green wire ground) that runs from the device being protected, the computer, to a copper stake that is embedded at least 10 feet deep into moist earth. The green wire ground must not have any other grounding wires connected to it and it must not be connected to any grounding or neutral buses in any of the power distribution boxes through which it passes. Note that this is IBM’s definition for the integrity of the logical operation of the computer as well as for human safety.

After our discussion, the customer said that they had an old well on the grounds that was lined with copper and that was at least 80 feet deep with over 60 feet of water in it. This was a great ground and I agreed that bonding the green wire ground from the computer to the well casing was appropriate grounding. Using this preexisting and amazingly good ground point was an elegant solution.

After the installation we had nothing but problems. These were seemingly random but frequent problems that one day would be indicative of memory failures, the next day a disk problem, the next day a processor problem, and so on. We replaced memory, CPU boards and pretty much everything we could think of over a period of about two weeks and the customer was understandably getting upset.

I was by now fairly certain that this was a grounding problem. I discussed the ground with the plant electrician and he said that he had run the ground wire just as we had discussed. But I had to see for myself to be sure. I got out my oscilloscope and used an induction clamp on the ground wire so that I could see any electrical noise on the ground line. As I got this all hooked up, the vice-president of IT came by and I told him what I was doing. He was a bit skeptical, to say the least.

Just as we were finished with our discussion, we both heard an electrical motor start up and a large burst of electrical noise showed up on the scope. A moment later someone ran out of the computer room and yelled that the computer was down. I could not have asked for a better audience to this than the disgruntled vice-president. Especially since the motor we had heard starting up was the compressor motor on the large soft drink machine that we were standing next to.

We had the electrician begin removing the front panels to all of the electrical distribution boxes in order to see and verify the integrity of the green wire ground. In the first box we looked at, I immediately saw the problem. There was a big, ugly, nasty looking, very old wire that had been grafted onto our once pristine green wire ground.

About this time someone came into the little room in which we had been standing during all of this and ran a few letters through the postage machine. The resultant noise on the ground wire said it all because both of these devices were plugged into the same power outlet. We had the electrician clip that ugly, old ground wire off of my nice new, clean green wire ground and the customer never had another problem caused by grounding issues.

Sometimes elegance is a pristine green wire ground.

Software elegance

Here we’re talking specifically about shell scripts which is the type of coding that SysAdmins typically do. It would be very unusual for a SysAdmin to write code in a language such as C which requires much more development effort and needs to be compiled. This is a poor use of time for a SysAdmin.

There are many opinions on software elegance. What makes software elegant, and what does “elegant” even mean in the software world? Here are some of my opinions with a bit of explanation for each.

In general, elegance is code that looks good, even beautiful, and follows the tenets outlined in this series of articles. In my opinion, software is elegant when you use these guidelines. This is my list of characteristics and I am willing to bet that other SysAdmins have their own ideas about what constitutes software elegance. In any event these are not hard and fast rules – just guidelines. The most important aspect of any software is that it should perform the task you wrote it to do. Using these guidelines makes it easier for others – and you – to understand what you did and maintain the code you wrote.

1. Use consistent indenting – Code should be consistent in the indenting of procedures and flow control structures. This helps to make it easier to visualize the structure of the program and the flow of execution under various circumstances. I know that some developers use tabs for indentation and others spaces. The number of tabs or spaces people use vary as well. That is mostly irrelevant so long as the code can be easily read by anyone who did not write it – and by those who did write it as well.
2. Design with a clear layout – Code should be well laid out and sequenced so that it is easy to see the flow of execution under various conditions. The most efficient code is that which executes in a straight-through fashion and which does not jump around or have unnecessary flow control structure which slow it down. There are good reasons for using procedures, such as to prevent replication of the same code in multiple places. However the main body of the program should flow in a straightforward fashion where possible.
3. Use STDIO – We have already seen that STDIO is a powerful enabler; it allows us to chain many small programs together in order to perform complex tasks that no single program can do. A program with a captive user interface (CUI) such as a menu, does not provide for STDIO. Such a program is limited to a stand-alone existence and cannot work as a within a data stream. Captive user interfaces should be avoided because they are so limiting and do not play well with the command line pipes and redirection. The fdisk program is one example of a useful and powerful utility that uses a menu interface. The problem with this is that fdisk cannot be used in scripts. Someone wrote a separate program for performing fdisk functions from within scripts. The current tool for this is sfdisk.
4. Add meaningful comments – The program is well commented with meaningful information. This helps to make the purpose of the code clear for maintainers and to ensure that problems can be located and fixed quickly.
5. Each program should do one thing well – This guideline has long been a tenet of the Unix and the Linux Philosophies and has resulted in the Core Utilities, and other core utilities that are small, targeted to a single task and that perform that task well. This results in powerful and flexible command line programs that can be combined into pipelines to perform complex tasks that a single program cannot. One side effect of this is that programs that do one thing tend to be small. This makes them easy to understand and modify when necessary. The corollary to this tenet is that adding more features to these small programs is not usually a good idea. The need for a so-called “new feature” should really be seen as the need for a new program which should also follow these guidelines. Most new features, when patched onto existing programs, simply create code bloat and make the programs harder to use and maintain.
6. Silence is golden – Linux command line tools usually do not display a message to the SysAdmin that all has gone well. This keeps undesired messages from entering the STDOUT data stream feeding a pipeline and causing confusion for later programs.
7. Always use the least amount of code necessary – The minimum amount of code necessary to perform the desired task is used. Everything else is cruft and should be eliminated. This is the crux of simplicity and the opposite of complexity. Some programmers like to show off with a twisty maze of complex code that is impossible to determine the entrances and exits. This type of code is poor practice and is subject to bugs.
8. The output is easy to read and understand – When any output is necessary at all, it should be easily interpreted by the user. For many programs the output is their reason for being. Output that is cluttered with messages and other information that have little or nothing to do with the purpose of the program obfuscates the important data. The actual structure of the output is irrelevant so long as it serves the intended purpose with clarity.
9. Use meaningful variable names – I like to use meaningful variable names in my command line and shell programming. Random variable names or names like $X have little meaning to whoever needs to debug code a couple years down the road; that includes the person who originally wrote the code. Names like $AccountTotal and $NumberOfUsers are far more meaningful than $A1, $B3, for example. They make it much easier to read the code. They also serve as a good starting place for the tenet “Document everything,” in Chapter 20. Well named variables tell program maintainers what to expect in terms of how the variable fits into the logic of the program as well as the kinds of values to expect when debugging the program. Going back to the Perl programs that I was tasked with cleaning up, the variable names were so random that it turned out several of those variable names pointed to the same thing. I renamed all of the variables in the program and then was able to substitute a single variable name for those other different names for the same variable. Just that little step made a big leap forward in cleaning up that particular program.
10. Follow Eric S. Raymond’s 17 Unix Rules¹ – These are 17 rules that should be read and understood by all developers including SysAdmins. Raymond expounded at length on these rules in his book, The Art of Unix Programming². Wikipedia has a nice summary of these rules (see footnote 1). If you think an important guideline is absent from my list, it is probably in Raymond’s list of rules. Be sure to read these rules because they apply to SysAdmins as well as developers.
11. Test everything – Is this not blindingly obvious?! Apparently not, because I have encountered plenty of software that has clearly not been well tested. My job at Cisco was twofold. Part of the time I was assistant to the lab manager where the testing department’s tests were run. The rest of the time I was one of the testers, assigned to test Linux powered appliances. Testing is not just running a series of test programs to verify that the software under test could perform its design tasks – it is also ensuring that the software does not fail when it encounters unexpected input. One of the most common vulnerabilities that hackers use to obtain unauthorized access to computers and other devices that are run by software, is the inability of the software to handle unexpected input. Other testing that I did was to simply peruse the documentation and the code to determine whether the code met the specifications outlined in the documentation. If it did not I had to fail it or the development team would have to obtain an exception, which was rare. Part of what I did while reviewing both the code and the documentation was to ensure that the design of the code supported the Linux Philosophy and well-documented standards such as the filesystem Hierarchical Standard, and standards created to ensure consistency of usage among all Linux distributions.
12. Clean out the cruft – Cruft is all of the old code in a program that is never used. Many programs evolve over time and sometimes code that was once useful is no longer needed. As I fix my own scripts or add new features or options I sometimes find myself with code and variables that are no longer used and which need to be cleaned out.

Following these guidelines will help ensure that the code you write remains easy to read and modify. It will look good and it will run well. It will be elegant.

LaTex

Elegance is very subjective and it may not be the same for everyone. Ann Barcomb sent an email to the writers’ list on Opensource.com to which we both subscribe. She discussed about LaTeX and its elegance. I asked her to elaborate and share her thoughts about what constitutes elegance in software.

LaTeX³ is a powerful document preparation system that is used for high quality typesetting. LaTeX has been around for over 30 years and is used in many different fields of academia, science, and technology.

Barcomb wrote:

“Speaking of LaTeX, when I speak of elegance I think of:

• Parsimony: The possibility of doing most of the things a beginner could want, with a subset of the capabilities
• Expressiveness: Being able to do just about everything you do want, once you go beyond the basics
• Simplicity: The user does not need to know how typesetting works to produce a beautiful document, which uses the precise characters required. Presentation logic is largely separated from context logic – Intuitive: the interface/logic is fairly internally consistent, and makes sense to me
• Clean: I can write programs to generate LaTeX without learning its storage format. I can see what I last changed with diff or looking back in my repo. I can logically divide content into multiple files and I can repeatedly include sections. I consider it in contrast with HTML and Libreoffice, both of which I endure but in no way enjoy using. LaTeX is a pleasure to use.”

Removing cruft

Creating elegance is hard work. Maintaining it can be even more difficult. Cruft is superfluous programs and code within programs, old data files, and directories with files left over from programs that have been removed.

Cleaning out cruft is an important part of our job as SysAdmins. Among the things we can search for cruft are old or unused software, old code in our own scripts, and old configuration and data files. Fortunately we have some tools that can assist us in this task.

Old or unused programs

I just removed some programs I don’t use. I was using the KDE Application Launcher and noticed that several of the Calligra office suite programs were in the list. I never use Calligra, preferring LibreOffice. It had been installed by default with Fedora and I did us it several months ago for a test. Because I will never use it for my productive work I decided to delete it.

I used the rpmorphan utility to list RPM packages that are not dependencies for any other packages installed on the host. I found a couple and deleted them.

Caution! Don’t remove orphan packages indiscriminately. This may result in removing needed packages. Just because they are orphans does not mean they are not needed. Be judicious when removing orphan packages. The rpmorphan tool simply allows us to identify packages that should be investigated further so that we may make a determination of whether they are truly safe to remove.

The deborphan tool can be used for Debian distributions. In fact, rpmorphan is based on denorphan. The rpmorphan tool only locates the orphans it does not remove them. If you do decide to remove any orphan packages you would use a package manager such as yum or dnf.

You can use the Application Launcher for your desktop to locate user level packages that you never use. This can also help you find some GUI administration tools you never use.

If you do decide to remove the packages you find, be careful that you will not remove other software that is needed. I once tried to remove a single package I thought I did not need. The list of packages that would have also been removed as dependencies ran into the hundreds and would have completely removed the KDE desktop from my system. That was definitely not what I wanted to do.

Old code in scripts

Finding cruft code in scripts is also a task that should be undertaken at least occasionally by SysAdmins. Getting rid of unused code and locating syntax errors can be challenging but there are some tools we have to help us.

The shellcheck utility is like lint⁴ for C and other languages. It scans bash and bash-like shells, sh, bash, dash, and ksh, for cruft and syntactical improvements that can be made. It is, as always, your choice as to whether you make the suggested changes or not.

Beyond anything that shellcheck can tell you about syntax, orphan variables, and other things, sometimes you just need to look through the code. For example, one type of cruft that shellcheck does not find is superfluous procedures that are not called from anywhere in the script. These can also be removed if they are not required.

Do you see an unused procedure in the script template? There is one, SelectPkgMgr(), that is not used in the template and shellcheck did not find that it was superfluous.

Old files

Sometimes old software is removed when it is no longer needed. In many cases the package removal procedures leave behind their user level configuration files. These are usually the hidden “dot” files that we find in our home directories.

The good thing about these left-over configuration files is that if that software package is ever reinstalled we will not have lost our personal configuration. The bad thing is that, over a long period of time, a large number of these files can accumulate. For example the personal configuration files for Calligra, which I had removed were still located in my home directory.

Old data files also are left on our hard drives long past any usefulness they might have had. This is usually because we seldom take the time to assess all of the files we have in order to determine whether they can be deleted, archived, or retained. One easy way to find old files is to use the find command to determine the last time files were accessed.

Summary

It is not always possible to do everything discussed in this article and the references to which I have referred. It would be great if we could but in real life we cannot always do so. Our scripts will never be completely free of cruft and they will never reach the highest levels of elegance. The title of this article should hint at that. Elegance is something to strive for but we will probably never achieve the pinnacle in which all cruft has been remove, all code made as efficient as possible, added the exact number of perfect comments that are clear and concise to our code, and all of the programming rules and suggestions followed.

This is not possible for a number of reasons. The two I run into most frequently are that the PHBs don’t care and won’t allow us the time, and that some of these guidelines are – at least to some extent – in conflict.

We do have some tools available to assist with locating cruft in scripts and as files on hard drives. Although those tools can be helpful they are imperfect and can only do so much. It is really up to us as SysAdmins to search out cruft in our code and directories; sometimes this means manually looking through things to see what is there that can be eliminated. This is time-consuming and I dislike doing it but it does need to be done.

Using these tools to locate the largest and oldest files in our home directories – or those of the other non-root users – on our systems can be the first step in cleaning out the cruft. It gives us a starting point where we can get the best results for the least amount of effort. After deleting the largest and oldest files, it becomes less effective to continue to look for smaller and newer files to delete or move to archival storage.

1. Wikipedia, The Unix Philosophy, Section: Eric Raymond’s 17 Unix Rules, https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond%E2%80%99s_17_Unix_Rules ↩︎
2. Raymond, Eric S., The Art of Unix Programming, http://www.catb.org/~esr/writings/taoup/html/ ↩︎
3. The LaTeX Project, Home page, https://www.latex-project.org/ ↩︎
4. Wikipedia, Lint, https://en.wikipedia.org/wiki/Lint_(software) ↩︎