Author’s note: This article is excerpted in part from chapter 20 of my book, The Linux Philosophy for SysAdmins, with some changes to update the information in it and to better fit this format.

“Real programmers don’t comment their code, if it was hard to write, it should be hard to understand and harder to modify.”

— unknown

I, too, would want to remain anonymous if I had written that. It might even have been meant to be sarcasm or irony. Regardless, this does seem to be the attitude of many developers and SysAdmins. There is a poorly disguised ethos amongst some developers and SysAdmins that one must figure everything out for themselves in order to join the club — whatever club that might be. If you cannot figure it out, they imply, you should go do something else because you don’t belong.

First, that is not true. Second, most developers, programmers, and SysAdmins that I know definitely do not subscribe to this view. In fact, the best ones, some of whom have been my mentors over the years, exemplify the exact opposite. The best of the best make documentation – good documentation – a high priority in everything they do.

I have used a lot of software whose creators subscribed to the philosophy that all code is self-explanatory. I have also been required to fix code that was completely uncommented and otherwise undocumented as well. It seems that many developers and SysAdmins figure if the program works for them it does not need to be documented.They all tend to espouse the idea that documentation is neither needed nor should it be. Yet throughout my career I have seen the disastrous results of this lack of documentation. I have been the SysAdmin assigned to fix uncommented code on more than one occasion. That is one of the least enjoyable tasks I have ever had to do.

Part of the problem is that many PHBs do not see documentation as a high priority. I have been involved in many aspects of the IT industry and fortunately most of the companies I worked for believed that documentation was not only important, but that it was crucial to the task at hand, regardless of what that task was.

I don’t think I have ever heard anyone say, “This documentation is great.” Mostly I hear how badly some specific documentation sucks. And I have repeated that refrain myself many times.

And yet there is a lot of really good documentation out there. For example the documentation for LibreOffice is excellent. It includes several documents in multiple formats including HTML and PDF that range from “Getting Started” to a very complete user’s guide for each of the LibreOffice applications.

The documentation for RHEL and Fedora – which are all very closely related distributions – are also among the best I have seen in my more than 40 years of working in the IT industry.

Good documentation is not easy and takes time. It also requires an understanding of the audience – not only in relation to the purpose of the documentation, but also the technical expertise of the intended readers as well as the languages and cultures of the readers. Rich Bowen covered that quite nicely in his fine article at Opensource.com, “RTFM? How to write a manual worth reading¹”.

There is also the question of what constitutes good documentation for a SysAdmin. We explore these things in this article which is mostly about documenting the scripts we write.

The Red Baron

One of the more frustrating incidents in my IBM career as a Customer Engineer was to assist with resolving some problems on an IBM 1800² process control computer in an oil refinery.

This particular computer was connected to many sensors out in the refinery and it was used to make adjustments to various components of the processes taking place. Based on the sensor readings, this computer would adjust things such as temperatures and flow rates to ensure that the products of the processes were correct and of high quality. But when things went wrong, it could have been disastrous. I mean, come on — This was process control for a freakin’ oil refinery!

It seems that the programmer who wrote the code did not comment his code very well – or not at all as far as I could tell – not that I had direct access to his proprietary source code. That developer was apparently not a fan of informative error messages, either.

I do have to say that the code was good at detecting errors. It also seems to have been good at shutting off the affected processes out on the ground in the refinery. Nothing blew up, after all.

My favorite Line in the movie, Cool Hand Luke is, “What we have here is a failure to communicate” and that applied here. No matter what the error, no matter what was wrong, the only message printed on the console was, “Curse you, Red Baron,” along with a numeric error message that we had to look up in a very long list of error codes. The resulting message from the list was never much help, either.

My documentation philosophy

My philosophy is one that has been drilled into me by my best mentors over the years, “The job is not done until the documentation is complete.” This means that everything must be documented. And documentation is definitely not the place to skimp on typing. Still — good documentation means something different to a SysAdmin than it does to an end user.

In the context of the Linux Philosophy for System Administrators, we shall consider documentation for the intended audience of our code – ourselves and other SysAdmins. There are two primary types of documentation we SysAdmins need. Some form of decent command line help option and well commented shell code.

The help option

The first place I go when looking for documentation to help me understand a shell scripts is the help facility because my most common need is to understand the syntax of the command that launches the program and available options and required or optional arguments to the command. This type of information is usually available by using the -h option for the desired command.

Good help is the primary line of documentation for the scripts we write as SysAdmins. All of the operational documentation must be included in the help procedure. It also means that the user interface of the script should be blindingly obvious and extremely simple so that the need to refer to any form of help is minimized.

Comment code liberally

Comments within code are a form of documentation. In fact, they should be the primary form of documentation for SysAdmins.

As part of my own need to document everything, I add many comments to my scripts. When tempted to cut back on comments, I think back to what it was like when I had to interpret and fix uncommented and undocumented code that was written by someone else.

I know many SysAdmins and other developers who think that their code is self-explanatory, even without comments. No matter how good our code is, and even with profuse and well-written comments, code will never be self-explanatory. We all think about problems differently, we write our code differently, and we solve problems differently. Because of the different ways in which we perceive things like code and its structure, the purpose of code that might be perfectly obvious to you even without comments, may be impenetrable to me.

I start with a program description that defines the purpose of the program and a little about its primary functions and options. Then I use a change history that tells future SysAdmins that may need to perform maintenance on the script, what features were added or removed, bugs that were fixed, who did the work, and when these things took place.

I also include a license statement to document the license under which the script is distributed and made available for other users. This is important so that there can be no question about the conditions under which the script can be used, modified, and distributed.

Comments embedded in the code should describe the functions of the code segments to which they refer. They should also contain information about why things are done in a certain way, and explanations of logic where that might not be obvious.

My code documentation process

Which came first? The program or the documentation. Ideally the documentation should be first. Then the code can be developed to meet the specifications outlined in the documentation. You do create specifications before you write code, don’t you? That is another common problem I encounter; the lack of clear specifications for scripts.

I like to start coding by creating an outline of my proposed code using comments. This lets me see the structure of the program and determine whether it is clean and elegant, allowing me to change the structure if necessary before I have written any code. Whether I am writing new code or maintaining existing code, comments are the first thing I add. These comments become the specifications for the script I am writing or maintaining. Then I can write the code that enables the actions described in the comments.

However, I don’t always do all of the comments to begin with. I first create a basic outline containing a bare framework of comments describing the logic of the program. I create an outline of the main body of the program, as much as I can. If I envision using additional procedures, I create and name the empty procedure, then add comments to describe its internal functions.

I then create the code to implement that basic framework. I usually start with the main body of the program, adding new comments as it becomes necessary and then filling in the code to implement the comments. When I arrive at a call that branches to an incomplete function, I write that function and add any comments that might still be required, then write the code to implement the procedure.

This is the answer to the question I posed at the beginning of this section. For me, at least, the documentation comes first. I can hear the Agile proponents’ keyboards typing their contrary opinions already. But in a very real sense, what I am doing is Agile, because I write just the documentation I need, just in time to write the code. And then the comments become the documentation as well.

Not everyone will want to work this way or will find it as well suited to their modus operandi as I do. There are as many ways of creating code and documenting it as there are people doing it. Do what works best for you but do it!

Man pages

Where do man pages fit into this philosophy of documenting everything? Frankly, not very well for scripts written by SysAdmins.

Early on we discussed the time constraints under which we SysAdmins work, and the fact that most of the scripts we write tend to start out as minimalistic solutions to operational problems. In this type of environment we have little or no time to spend on creating man pages. The bottom line is I do not spend the time to create man pages.

Systems documentation

This type of documentation is not about documenting scripts or programs. It is about documenting the state of the network, the connected hosts, and any work I perform on them. This documentation is critical and important to the customers of my former consulting business, and to any employer where I have worked as a full-time employee or contractor.

At one time I owned a small LLC through which I used to do a bit of consulting on Linux and open source. When working with customers I always documented my interactions with them and the work I performed. Documentation like this serves me the same way that a doctor’s notes of my visits serve her. It is a permanent record of the customer’s environment that I can refer to when talking to them on the phone or engaging in email conversations. It provides me with a running commentary of the problems I find and what actions I take to resolve them.

In some cases I have years of documentation that covers everything from my first contact with them to the information I discover about their network while I am working on projects for them, the details of hardware I install for them, details of my work on projects, and a record of each time I install updates. I include data in these records such as network diagrams, tables of network IP and MAC addressed with notes about the function of each node. I also keep the output from a script I have written that lists the hardware and some of the configuration details of each Linux host that I work on.

This information has multiple uses. It gives me a record so that I can go back and recall what I have done and the structure of my customer’s environment — it is a memory aid for me. I can use it to support my recommendations for additional work when needed. Keeping detailed records also can be useful in the event of a dispute with a customer.

I always create a task list before performing work for a customer so that I do not forget anything that needs to be done. I take notes on that list and then, at the end of the work, the task list becomes part of the documentation of the work I have performed and is supplemented by the notes I took during my performance of the work. For some of my customers I ended up with over 40 pages of this type of documentation.

Document existing code

Creating the documentation for existing code requires different approach than any other type.

The first thing I do is read the source code, which is almost always Perl or bash scripts for me. Then I can use the comments as a starting point to create external documentation — if there are any comments and if the comments have any meaning at all.

One job I took a goodly number of years ago, I was to take over maintenance and fix a large number of preexisting bash scripts. Those scripts were part of a series of complex internal applications used by the company. The code worked — mostly — but was excessively convoluted spaghetti code and it lacked usable comments and documentation of any kind.

My first task was to fix a few bugs in several of the scripts. I started reading these scripts to determine what they actually were supposed to do. As I determined what each section of code did, I added comments that would describe the code I had just read and interpreted. And just in the process of doing that I was able to determine the cause of some the bugs and correct them.

During this initial stage I determined from reading the bash scripts and questioning the IT folks that the scripts were originally written by several different contractors and had been maintained over a period of years by a series of other contractors. Each of the contractors added on little sections of code that were obviously designed to circumvent the problems that they encountered. None of these add-on bits of code made any attempt to fix the root causes. Each contractor had their own way of doing things such as naming schemes for variables, indents, coding style, and comments. Those scripts were a complete disaster.

That project was a nightmare. It took me weeks to analyze the code and add appropriate, understandable, and useful comments to the code. That task was tedious and made more difficult by apparently random naming of the variables. That was one of the inevitable results of having so many different people working on the scripts without any type of guidance or oversight in terms of either the project objectives or programming style.

After I completed the task of commenting each of these scripts, renaming as many variables as I could, it became much easier to resolve the remaining problems.

Of course code written by others is not the only code with these problems. My own code, especially much of my older code, is subject to these same problems. This occurred because I had not yet learned about the Unix or Linux philosophy. My code did improve over time and when it becomes useful to revisit some of my old code to revise it of fix a problem, I revise it to follow the better programming practices I have learned since I was a baby SysAdmin.

Keep docs updated

I have had a few issues with my own documentation. First among them is neglecting to update documentation in a timely or complete manner. This has caused problems when information I needed had not been properly recorded.

When I discover I have been lax in my documentation, I try to go back and correct it as soon as possible. This usually means correcting and updating the comments embedded in my scripts. It also means fixing the help procedures to be consistent with the changes made to the code.

Updating my customer documentation is also a task I need to keep up with. I sometimes forget to do this as I always seem to be rushing to my next task.

It takes discipline to keep my documentation current. Without constant upkeep, documentation can become hopelessly out of date.

A few thoughts

Documentation is very important to System Administrators. While executing our daily duties we depend on the documentation that others have left for us. The quality and speed with which we can do our work is directly affected by the quality of that documentation. Here are some guidelines for documenting our scripts.

1. Scripts should be documented with lucid and meaningful comments.
2. Scripts should be easy to read. This is a form of self-documentation.
3. Scripts should have a useful and concise help feature.
4. Following these guidelines results in elegant scripts.

Whatever you do and however you choose to work, just remember that the job is not done until the documentation is complete.

1. Bowen, Rich, Opensource.com, RTFM? How to write a manual worth reading, https://opensource.com/business/15/5/write-better-docs ↩︎
2. Engineering and Technology Wiki, IBM 1800, http://ethw.org/IBM_1800 ↩︎