Programming Bash #4: Using Loops

Last Updated on December 29, 2023 by David Both

The ability to iterate over a section of code using various types of loops is one of the most important tools we have for performing repetitive tasks. We’ve already seen the for loop but there are others available to us in Bash, the while and until structures also provide iterative looping.

Using for Loops

Every programming language I have ever used has some version of the for command. The Bash implementation of this structure is, in my opinion, a bit more flexible than
most because it can handle non-numeric values while the standard C language for loop, for example, can only deal with numeric values.

The basic structure of the Bash version of the for command is simple – for Var in list1 ; do list2 ; done. This translates to: for each value in list1, set the variable $Var to that value and then perform the program statements in list2 using that value; when all of the values in list1 have been used, we are done so exit the loop. The values in list1 can be a simple explicit string of values, or it can be the result of a command substitution as we have seen in the previous article of this series.

I use this construct frequently so we’ll explore it in some detail. First, make ~/testdir6 the PWD. Then try this simple example of a for loop.

[dboth@testvm1 testdir6]$ for I in a b c d 1 2 3 4 ; do echo $I ; done
a
b
c
d
1
2
3
4
[dboth@testvm1 testdir6]$

Here’s a bit more useful version along with a more meaningful variable name for the task of creating directories for multiple departments. We’ll just start by ensuring that the basic loop works as expected but won’t create the directories.

Note that it is necessary to enclose the $Dept variable in quotes in the echo and mkdir statements or the two part department names such as “Information Technology” will be treated as two separate departments. That highlights a best practice that I like to follow and that is that all file and directory names should be a single unbroken string with no spaces or other whitespace in them. Although most modern operating systems can deal with spaces in those names, it takes extra work for SysAdmins to ensure that those special cases are considered in scripts and CLI programs. But they almost certainly should be considered, even if they’re annoying, because you never know what files you’re actually going to have.

[dboth@testvm1 testdir6]$ for Dept in "Human Resources" Sales Finance "Information Technology" Engineering Administration Research ; do echo "Department $Dept" ; done
Department Human Resources
Department Sales
Department Finance
Department Information Technology
Department Engineering
Department Administration
Department Research
[dboth@testvm1 testdir6]$

Now let’s actually make some directories and show some progress information while doing so. Then list the contents of the testdir6 directory.

[dboth@testvm1 testdir6]$ for Dept in "Human Resources" Sales Finance "Information Technology" Engineering Administration Research ; do echo "Working on Department $Dept" ; mkdir "$Dept" ; done ; ll
Working on Department Human Resources
Working on Department Sales
Working on Department Finance
Working on Department Information Technology
Working on Department Engineering
Working on Department Administration
Working on Department Research
total 28
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Administration
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Engineering
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Finance
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08 'Human Resources'
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08 'Information Technology'
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Research
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Sales
[dboth@testvm1 testdir6]$ 

Delete everything in testdir6 and recreate the directories using names with no spaces. We’ll use dashes instead.

[dboth@testvm1 testdir6]$ for Dept in Human-Resources Sales Finance Information-Technology Engineering Administration Research ; do echo "Working on Department $Dept" ; mkdir "$Dept" ; done ; ll
Working on Department Human-Resources
Working on Department Sales
Working on Department Finance
Working on Department Information-Technology
Working on Department Engineering
Working on Department Administration
Working on Department Research
total 28
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Administration
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Engineering
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Finance
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Human-Resources
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Information-Technology
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Research
drwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Sales
[dboth@testvm1 testdir6]$

Suppose someone asks for a list of all RPMs on a particular Linux computer and a short description of each. This happened to me while I worked at the State of North Carolina. Since Open Source was not “approved” for use by state agencies at that time and I only used Linux on my desktop computer, the pointy haired bosses (PHBs) needed a list of each piece of software that was installed on my computer so that they could “approve” an exception. How would you approach that?

Here is one way, starting with the knowledge that the rpm –qi command provides a complete description of an RPM including the two items we want, the name and a brief summary. We will build up to the final result one step at a time. Note that this can be done as a non-privileged user since we are not going to add or remove any packages. First we list all RPMs.

[dboth@testvm1 testdir6]$ rpm -qa
fonts-filesystem-2.0.5-11.fc38.noarch
google-noto-fonts-common-20230201-1.fc38.noarch
xkeyboard-config-2.38-1.fc38.noarch
tzdata-2023c-1.fc38.noarch
google-noto-sans-vf-fonts-20230201-1.fc38.noarch
liberation-fonts-common-2.1.5-4.fc38.noarch
hyperv-daemons-license-0-0.40.20220731git.fc38.noarch
hunspell-filesystem-1.7.2-3.fc38.x86_64
abattis-cantarell-fonts-0.301-9.fc38.noarch
web-assets-filesystem-5-19.fc38.noarch
mobile-broadband-provider-info-20221107-2.fc38.noarch
fedora-logos-38.1.0-1.fc38.noarch
adwaita-cursor-theme-44.0-1.fc38.noarch
js-jquery-3.6.3-2.fc38.noarch
python-systemd-doc-235-2.fc38.x86_64
liberation-mono-fonts-2.1.5-4.fc38.noarch
google-noto-sans-mono-vf-fonts-20230201-1.fc38.noarch
google-noto-serif-vf-fonts-20230201-1.fc38.noarch
adobe-source-code-pro-fonts-2.030.1.050-14.fc38.noarch
dejavu-sans-mono-fonts-2.37-20.fc38.noarch
<SNIP>

This gives the list of RPMs installed on the host, so we can use this as the input list to a loop that will print all of the details of each RPM.

[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done

This code produces way more data than was desired because it displays all of the data about each RPM. We only need the name and summary lines.

Note that our loop is complete. The next step is to extract only the information that was requested. To do that we add the grep -Ei command.¹ The -E option is used to specify extended regular expressions will be used on the pattern. In this case it is used to select either ^Name or ^Summary. Thus any line with Name or Summary at the beginning of the line (the carat ^ specifies the beginning of the line) is displayed. The -i option tells grep to ignore case in the strings being checked. The vertical bar ( | ) is a logical or.

[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done | grep -Ei "^Name|^Summary"

Our final command sequence looks like this. The only change here is to add redirection so the data stream is stored in the RPM-summary.txt file.

[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done | grep -Ei "^Name|^Summary" > RPM-summary.txt

This command line program uses pipelines, redirection, and a for loop – all on a single line. It redirects the output of our little CLI program to a file that can be used in an email or as input for other purposes. This process of building up the program one step at a time allows you to see the results of each step and to ensure that it is working as you expect and provides the desired results.

Note that the PHBs received a list of over 1,900 separate RPM packages. I seriously doubt that anyone actually read that list. But I gave them exactly what they asked for and I never heard another word from them about this.

Other loops

There are two more types of loop structures available in Bash. The while and until structures, which are very similar to each other in both syntax and function. The basic syntax of these loop structures is simple.

while [ expression ] ; do list ; done

and

until [ expression ] ; do list ; done

The logic of these reads as follows. “While the expression evaluates as true, execute the list of program statements. When the expression evaluates as false, exit from the loop.” And “Until the expression evaluates as true, execute the list of program statements. When the expression evaluates as true, exit from the loop.”

while

The while loop is used to execute a series of program statements while (so long as) the logical expression evaluates to true. Your PWD should still be ~/~.

The simplest form of the while loop is one that runs forever. In the following form we use the true statement to always generate a “true” return code. We could use a simple “1” and that would work just the same but this illustrates use of the true statement. Using the head statement displays only the first 10 results. If we don’t use the head statement the count will continue until it reaches the maximum integer size for the Bash implementation.²

[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $X ; X=$((X+1)) ; done | head
0
1
2
3
4
5
6
7
8
9
[dboth@testvm1 testdir6]$

This CLI program should make more sense now that we have studied its parts. First we set $X to zero just in case it had some leftover value from a previous program or CLI command. Then, since the logical expression [ true ] always evaluates to 1, which is true, the list of program instructions between do and done is executed forever – or until we press Ctrl-C or otherwise send a signal 2 to the program. Those instructions are an arithmetic expansion that prints the current value of $X and then increments it by one.

One of the tenets of “The Linux Philosophy for SysAdmins” is to strive for elegance and that one way to achieve elegance is simplicity. We can simplify this program by using the variable increment operator, ++. In this first instance the current value of the variable is printed and then the variable is incremented. This is indicated by placing the ++ operator after the variable.

[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $((X++)) ; done | head

Now delete “| head” from the and of the program and run it again. Press Ctrl-C when you tire of watching the count.

In this next version, the variable is incremented before its value is printed. This is specified by placing the ++ operator before the variable. Can you see the difference?

[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $((++X)) ; done | head

We have reduced two statements into a single one that both prints the value of the variable and increments that value. There is also a decrement operator, –.

We need a method for stopping the loop at a specific number. To accomplish that we can change the true expression to an actual numeric evaluation expression. So let’s have our program loop to 5 and stop. In the code below you can see that -le is the logical numeric operator for “less than or equal to.” This means that so long as $X is less than or equal to 5, the loop will continue. When $X increments to 6 the loop terminates.

[david@testvm1 testdir6]$ X=0 ; while [ $X -le 5 ] ; do echo $((X++)) ; done
0
1
2
3
4
5

until

The until command is very much like the while command. The difference is that it will continue to loop until the logical expression evaluates to true. Let’s look at the simplest form of this construct.

[david@testvm1 testdir6]$ X=0 ; until false ; do echo $((X++)) ; done | head

We can also use a logical comparison to count to a specific value.

[dboth@testvm1 testdir6]$ X=0 ; until [ $X -eq 5 ] ; do echo $((X++)) ; done
0
1
2
3
4
[dboth@testvm1 testdir6]$ X=0 ; until [ $X -eq 5 ] ; do echo $((++X)) ; done
1
2
3
4
5
[dboth@testvm1 testdir6]$

Be sure to note the difference between these two forms of the until statement.

Summary

We have explored the use of many powerful tools that we can use to build command line programs and Bash shell scripts. Despite the interesting things we have done in this article, Bash command line programs and shell scripts can do so much more. We have barely scratched the surface. The rest is up to you.

I have discovered over time that the best way to learn Bash programming is to actually do it. Find a simple project that requires multiple Bash commands and make a CLI program out of them. SysAdmins do many tasks that lend themselves to CLI programming this way so I am sure that you will easily find tasks to automate.

Many years ago, despite being familiar with other shell languages and Perl, I made the decision to use Bash for all of my SysAdmin automation tasks. I have discovered that – perhaps with a bit of searching – I have been able to accomplish everything I need.

Series Articles

This list contains links to all eight articles in this series about Bash.

1. Programming Bash #1 – Introducing a New Series
2. Programming Bash #2: Getting Started
3. Programming Bash #3: Logical Operators
4. Programming Bash #4: Using Loops
5. Programming Bash #5: Automation with Scripts
6. Programming Bash #6: Creating a template
7. Programming Bash #7: Bash Program Needs Help
8. Programming Bash #8: Initialization and sanity testing