My first Linux in the enterprise

I’ve used Linux at home since 1993. Even in those pre-1.0 days, Linux was a very powerful operating system, loaded with tools that let me work the way I wanted to. I also recognized Linux as a feature-complete replacement for Big Unix systems. Running Linux at home meant I could run the same Unix tools as our “Big Iron” Unix systems on campus, but for free and with all the source code included.

When I graduated in late 1994, my first job was managing Big Unix systems for a local company. These systems were already pretty well managed, but I added a few Linux servers at the “edge” to support things like NIS and NFS, which made it easier to support a Unix network. At my next job, I implemented a few other Linux servers here and there to support other “backoffice” jobs like printing.

But my first experience of deploying Linux in the enterprise was a few years later, in 1998.

Web registration challenges

In 1998, I managed the server administration group for a new “Web Team” at the University of Minnesota. The U of M was and is a very large enterprise; at the time, we supported about 60,000 students across four campuses. Until then, the university managed all student records on an aging mainframe system. But that was about to change.

The mainframe wasn’t Y2K compliant, so we were working to migrate to a new records system. The new system was “new” in many ways; it was so new, it lacked one key feature: students couldn’t register for classes on a web browser. In 1998, the “Web” was still too new.

Our job on the Web Team was to add that missing web registration component to the new system.

I’ll spare you the technical details which are now ancient history, but we partnered with a vendor to deploy the new web registration system on a Big Unix system. But we quickly learned one big box couldn’t support the rush of students who were all trying to register for classes at the same time. The system came crashing down several times.

To help with the load, our partner donated a total of three Big Unix systems, plus a network load balancer that divided the web traffic among them. Each web server connected to several “application servers” to run the back-end business logic, and queried data directly from the student database.

Even these three massive computers couldn’t accommodate the student traffic. On day one of the next registration period, over 400 students signed into the system within a few minutes. Overwhelmed by the unexpected load, the three web servers crashed. We found ourselves constantly restarting the web servers as the high web traffic continued to crush them. As soon as we restarted one web server, the next would go down. And so on, for the entire month-long registration period.

Without a reliable way to register for classes on the web, students had to sign up for classes the old-fashioned way: by going to the registrar’s office. Lines to register went down the hallway and out the door. It wasn’t long before the bad news hit the local news, with headlines such as “Computer failure forces students to register in person.”

Finding a new solution

Without a reliable way to register for classes on the web, students had to sign up for classes the old-fashioned way: by going to the registrar’s office. Lines for students to register in person went out the door of the registrar’s office and down the hallway. It wasn’t long before the bad news hit the local news, with headlines such as “Computer failure forces students to register in person.”

With another six months before our next go-live, we felt trapped. No one could figure out why the system was constantly crashing under load. We knew it would fail again at the next registration period. We had to do something, anything, to improve the system. But what to do? Every option was on the table.

I asked, what if we tried it on Linux?

My Unix team was already familiar with Linux, and they were all fans. We were most familiar with Red Hat Enterprise Linux, which several of us were already running on our desktop systems.

It was also convenient timing. IBM had recently embraced Linux, releasing Linux versions of its Java and WebSphere products. All products were certified for Red Hat Enterprise Linux. We realized we now had the ecosystem to run the web registration system on Linux, as a supported platform. But would it perform any better on Linux than the Big Unix systems?

After setting up a test Linux server and running a series of tests, we were stunned to see one Linux server could easily support loads that could only be managed by dividing traffic across the three Big Unix systems.

We shared our findings with the registrar and the CIO, who approved our plan to migrate the web registration system to Linux. It was our first time running Linux in the University of Minnesota enterprise, but we had nothing to lose. If we changed nothing, we knew the BIg Unix systems would fail on the next registration cycle. At worst, Linux would fare just as poorly, but on less expensive hardware. At best, Linux could run it better.

We immediately ramped up new Linux servers for production. After a restless two months, we finally went live. And it was a resounding success! The web registration system performed flawlessly on Linux, despite heavy usage. At our peak during that registration period, the Linux servers managed over 600 concurrent users, with barely a blip. Linux had rescued web registration at the University of Minnesota.

Lessons learned

As I look back on that massive rescue operation, I find several themes you can use to introduce Linux in your own organization:

Solve a problem, don’t stroke an ego: When we proposed running Linux in the enterprise, we weren’t doing it because we thought Linux was cool. Sure, we were Linux fans and we already ran Linux on the desktop and at home, but we were there to solve a problem. Our registrar and other stakeholders appreciated that Linux was a solution to a problem, not just something we wanted to do because Linux was cool.

Change as little as possible: Our success hinged on the fact that IBM had finally released versions of its Java and WebSphere products for Linux. This allowed us to minimize changes to the system as we migrated from Big Unix to Linux. Comparing the Big Unix configuration to the Linux configuration, only the hardware and operating system changed. Every other component on the system remained the same. It was this “known” quantity that instilled confidence in making the change.

Be honest about risks and benefits: Our problem was obvious: Web registration had failed in our previous two registration cycles and would likely fail again. When we presented our idea to our stakeholders, proposing that we replace the Big Unix web servers with Linux, we were open about the expected risks and benefits. The bottom line was if we changed nothing, we would fail. If we tried Linux, we might fail or we might not. We shared our findings from our initial load tests, which demonstrated that Linux was more likely to succeed than fail.

But even if Linux failed, we could easily put the old Big Unix servers back into production. That “fallback” preparation reassured the registrar that we had appropriately measured the benefits and the risks and were prepared in case things went wrong.

Communicate broadly: In making our pitch to migrate to Linux, we cast a wide net. We wrote an executive white paper that clearly communicated what we planned to do and why we thought it would work. The key to this white paper’s success was its brevity. Executives do not want to read a “novel” about a technical idea, nor do they want to get mired in the technical details. We intentionally wrote the white paper for the executive level, describing our proposal in broad strokes.

As we replaced the system with Linux, we provided regular updates to inform our stakeholders about our progress to build the new Linux system. After we finally went live on the Linux web registration system, we posted daily updates, reporting how many students had registered for classes on the new system, and if we saw any problems.

This article is based on Linux to the rescue: How I introduced my organization to Linux by Jim Hall, and is republished with the author’s permission.