Fear and Loathing on the UNIX Trail

Confessions of a Berkeley system mole.

by Doug Merritt with Ken Arnold and Bob Toxen

We were enhancing the system in the middle of the night because we had no official sanction to do the work. That didn't stop us, though, since UNIX had just freshly arrived from Bell Labs, where computer security had never been an issue. The system was now facing its first acid test -- exposure to a group of intelligent, determined students -- and its security provisions were failing with regularity.

I was lying face down because I'd gone without sleep for over two days, and the prone position somehow seemed the most logical under the circumstances. Bob was still working because he'd napped not 30 hours before, giving him seniority under the "Hacker-best-able-to-perform" rule of our informal order. We might have called our group "Berkeley Undergraduate Programmers for a Better UNIX", or, less euphemistically, "Frustrated Hackers for Our Own Ideas". But, in truth, our group was never named. It was simply a matter of Us versus Them.

"Them" was the bureaucracy -- the school administrators, most professors, some grad students, and even the legendary Implementors themselves at Bell Labs.

"Us" was a small, self-selected group of undergraduates with a passion for UNIX. We were interested in computers and in programming because it fascinated us; we lived for the high level of intellectual stimulation only hacking could provide. Although some in our group never expressed an interest in breaking computer security, others invested thousands of fruitful hours in stealing accounts and gaining superuser access to various UNIX systems. Our object? To read system source code.

For the most part we stayed out of trouble, although one of our rank once had his phone records subpoenaed by the FBI -- after a minor incident with a Lawrence Livermore National Laboratory computer. The Feds seemed to think our comrade had been diddling with top secret weapons research, but he actually hadn't.

Our group could probably best be characterized by its interest in creating and using powerful software, regardless of the source of the idea. Our battle cry, thanks to Ross Harvey, was "FEATURES!!!", and we took it seriously. Well, Ross may have been a little sarcastic about it, since he was referring to superfluous bells and whistles. But I used the expression as a shorthand for "elegant, powerful, and flexible". We were always bugging Them to add "just one more feature" to some utility like the shell or kernel. Although They accepted some suggestions, They didn't think twice about most.

One example stands out. In early 1977, Ross, Bob, and I spent months collaborating on a new and improved shell, just before Bill Joy had started on what is now known as the C shell. The most historically significant features we designed were Ross's command to change the shell's prompt, Bob's command to print or chdir to the user's home directory, and my own edit feature, which allowed screen editing and re-execution of previous commands. What we did was smaller in scope than what Bill later included in the C shell, but to Us it was unarguably better what was then available. We ceased work on our projects only when it became clear that Bill was developing what would obviously become a new standard shell. Our energies then were re-focused on persuading him to include our ideas. Some of our features ultimately were incorporated, some weren't.

We modified the kernel to support asynchronous I/O, distributed files, security traces, "real-time" interrupts for subprocess multitasking, limited screen editing, and various new system calls. We wrote compilers, assemblers, linkers, disassemblers, database utilities, cryptographic utilities, tutorial help systems, games, and screen-oriented versions of standard utilities. User friendly utilities for new users that avoided accidental file deletion, libraries to support common operations on data structure such as lists, strings, trees, symbol tables, and libraries to perform arbitrary precision arithmetic and symbolic mathematics were other contributions. We suggested improvements to many system calls and to most utilities. We offered to fix the option flags so that the different utilities were consistent with one another.

To Us, nothing was sacred, and We saw a great deal in UNIX that could stand improvement. Much of what We implemented, or asked to be allowed to implement, is now a part of System V and 4.2 BSD; others of our innovations are still missing from all versions of UNIX. Despite these accomplishments, it seemed that whenever We asked The Powers That Be to install Our software and make it available to the rest of the system's users, We were greeted with stony silence.

Fred Brooks, in The Mythical Man-Month, describes the NIH (Not Invented Here) Syndrome, wherein a group of people will tend to ignore ideas originated outside their own social group. However, there was a stronger force at work at Berkeley, where a certain social stratification prevails that finds Nobel Laureates and department chairs ranking as demigods, professors functioning as high priests, graduate students considered as lower class citizens, and undergraduates existing only on sufferance from the higher orders -- and suffered very little at that. Now, the individuals cannot be blamed for what is, in essence, an entire social order. But this is not to say that we did not hold it against them -- for we most assuredly did. Unfortunately, it took time for us to appreciate the difficulties of Fighting City Hall.

This is why We were frustrated. This is why We felt We HAD to break security. Once We did, We simply added Our features to the system, whether The Powers That Be liked it or not. Needless to say, They didn't. This is why We felt like freedom fighters, noble figures even when found in the ignoble position of lying face down on the floor of Cory Hall at two in the morning.

We were on a mission that morning to install our new terminal driver. With the old, standard terminal driver, the screen gave you no indication that the previous character had been deleted when you pressed the erase character. You had to accept it on faith. This remains true on many UNIX systems today. Most people on Cory Hall UNIX changed their erase character to backspace so that later characters would overwrite the erased ones, but even that was not sufficient. This was especially true when erasing a backslash, which counter-intuitively required two erase characters. We wanted the system to show that the character was gone by blanking it out. We also wanted the line-erase character to display a blanked-out line. Some UNIX systems such as 4.2 BSD and System V now support this, but it was not then available anywhere under UNIX version 6.

Bob and I had argued, somewhat sleepily, for hours as to the correct method of erasing characters, and Bob had started putting our joint design into effect just as I collapsed on the floor for "a short nap". I awoke around dawn to find Bob asleep over the terminal. When he woke up, he said he was pretty sure he'd finished the job before falling asleep, but neither of us had enough energy to check. It was time for food and 14 hours of sleep.

When we finally checked our handiwork the next day, we found some serious flaws in the implementation -- not an uncommon situation with work performed under extreme conditions. But the system was up and running, and although the new features were flawed, they didn't seem to cause any problems, so we forgot about it for the time being. A week later, I was consulting in Cory -- we all offered free programming help to other students in the time-honored tradition of hackers everywhere -- when Kurt Schoens called me over to the other side of the room.

"Hey Doug," he said. "Look at this. It looks like someone tried to put character deletion into the terminal drivers, but only half finished."

My heart raced. Did he suspect me? Or was he just chatting? I could never tell whether Kurt was kidding; he had the most perfect poker face I had ever seen. But he quickly made the question academic, and proved again that he was one of Them.

"I showed this to Bill, and he wanted to fix it", Kurt said. "Oh, really?" I stammered. "Sounds good to me," thinking that it was a real stroke of luck that Bill Joy would be interested in the half-completed project. If Bill finished it, then it would be in the system on legitimate grounds, and would stay for good.

Kurt paused for effect. "Yeah, he was all fired up about it, but I talked him out of it, and I just deleted it from the system instead."

Oh, cruel fate! Kurt must know that I was involved; he just wanted to see me jump when he said "boo!"

Although I'm sure Kurt thought the whole incident very funny, all I could think of was that yet another of my features had gone down the drain. I discussed this latest setback with others in the group, and we shared a sense of frustration. More than ever before, we were determined to get our contributions accepted somehow.

Kurt was both a graduate student and a system administrator, but I liked him all the same -- chiefly because of his practical jokes. We had recently cooperated in a spontaneous demonstration of Artificial Intelligence at the expense of an undergraduate named Dave who had joined Them as a system administrator. Dave had watched Kurt as he typed pwd to his shell prompt and received /usr/kurt/mind as the response. His next command had been mind -i -1 english. During all this time, Kurt was double-talking about psychology and natural language processing and some new approach to simulating the human mind that he'd thought of. Dave looked dubious, but was willing to see how well Kurt's program worked.

What Dave didn't realize was that Kurt had not been typing commands to the system at all; although we were sitting not 10 feet apart, Kurt and I had been writing to each other and chatting for half an hour, and as a joke I had been pretending I was Kurt's shell, sending him prompts and faking responses to commands. Dave had walked in at just the right time. So when Kurt typed mind -i -1 english, I had naturally responded with:

"Synthetic Cognition System, version 17.8"
"Interactive mode on, Language=english"
"Please enter desired conversational topic: (default:philosophy)"

Dave couldn't help looking a little impressed; Kurt's "artificial intelligence" system was off to a great start. Kurt had talked to his budding mind for several minutes, and Dave of course had grown more and more impressed. Kurt and I faced the greatest challenge of our lives in keeping a straight face during the demonstration, but we eventually made the mistake of making the mind altogether TOO smart to be believable, in effect sending Dave off to tackle more serious work.

There was one practical joke that was notable for the length of time that it was supported by the entire group. The target was system administrator Dave Mosher. Dave had been suspicious of bugs in our system's homebrewed terminal multiplexer for some time. Ross decided to persecute Dave by having random characters appear on his screen from time to time, which of course convinced Dave that the terminal multiplexer did indeed have problems. To help Ross with the prank, each of us sent Dave some garbage characters at random intervals whenever any one of us was on the system. We had settled on the letter "Q" so that Dave would be sure it was always the same bug showing the same symptoms. Since Dave had these problems no matter which terminal he was on, day or night, no matter who else was logged onto the system, he was positive there was a problem, and he spent much time and effort trying to get someone to fix it.

Unfortunately for Dave he was the only one who ever saw these symptoms, so everyone thought he was a little paranoid. We thought it was pretty funny at first, but after a few months of this, it seemed that Dave was really getting rattled, so one day Ross generated a capital "Q" as big as the entire screen and sent it to Dave's screen. This made it pretty obvious to poor Dave that someone, somehow, really had been persecuting him, and that he wasn't paranoid after all. He had an understandably low tolerance for practical jokes after that.

The numerous practical jokes we played were probably a reaction to the high level of stress we felt from our ongoing illicit operations; it provided some moments of delightful release from what was, at times, a grim battle. There were many secret battles in the war; if Our motto was "Features!", Theirs was "Security for Security's Sake" and the more the better. We were never sure how long our victories would last; on the other hand, They were never sure whether They had won. The war lasted almost three years.

We were primarily interested in the EECS department's PDP 11/70 in Cory Hall, since that was the original UNIX site and continued to be the hotbed of UNIX development, but We "collected" all the other UNIX systems on campus, too. One peculiar aspect of the way the Underground had to operate was that we rarely knew the root password on systems to which we had gained superuser access. This is because there were easier ways to get into, and stay into, a system than guessing the root password. We tampered, for instance, with the su program so that it would make someone superuser when given our own secret password as well as when given the usual root password, which remained unknown to us. In the early days, one system administrator would mail a new root password to all the other system administrators on the system, apparently not realizing that we were monitoring their mail for exactly this kind of security slip. Sadly, they soon guessed that this was not a good procedure, and we had to return to functioning as "password-less superusers", which at times could be a bit inconvenient.

Late one night on Cory Hall UNIX, as I was using my illegitimate superuser powers to browse through protected but interesting portions of the system, I happened to notice a suspicious-looking file called /usr/adm/su. This was suspicious because there were almost never new files in the administrative /usr/adm directory. If I was suspicious when I saw the filename, I was half paralyzed when I saw it contained a full record of every command executed by anyone who had worked as superuser since the previous day, and I was in a full state of shock when I found, at the end of the file, a record of all the commands that I'd executed during my current surreptitious session, up to and including reading the damning file.

It took me perhaps 10 minutes of panic-stricken worry before I realized that I could edit the record and delete all references to my illicit commands. I then immediately logged out and warned all other members of the group. Since nothing illicit ever appeared, the system administrators were lulled into a sense of false security. Their strategy worked brilliantly for us, allowing us to work in peace for quite a while before the next set of traps were laid.

The next potential trap I found was another new file in /usr/adm called password, that kept track of all unsuccessful attempts to login as root or to su to root, and what password was used in the attempt. Since none of us had known the root password for months and therefore weren't going to become superuser by anything as obvious as logging in as root, this wasn't particularly threatening to us, but it was very interesting. The first few days that we watched the file it showed attempts by legitimate system administrators who had made mistakes of various sorts. One of Them once gave a password that We discovered, through trial and error, to be the root password on a different system. Several of Them gave passwords that seemed to be the previous root password. Most of them were misspellings of the correct root password. Needless to say, this was a rather broad hint, and it took Us less than five minutes to ascertain what the correct spelling was.

One might think that, since we had several ways to become superuser anyway, it wouldn't make any real difference whether or not we knew the actual root password as well. The problem was that our methods worked only so long as nothing drastically changed in the system; the usual way that They managed to win a battle was to backup the entire system from tape and recompile all utilities. That sometimes set Us back weeks, since it undid all of our "backdoors" into superuserdom, forcing us to start from ground zero on breaking into the system again. But once we knew the root password, we could always use that as a starting place.

We worked very hard to stay one step ahead of Them, and we spent most of our free time reading source code, in search of either pure knowledge or another weapon for the battle. At one time, I had modified every single utility that ran as superuser with some kind of hidden feature that could be triggered to give us superuser powers. Chuck Haley once sent a letter to Jeff Schriebman commenting that he "had even found the card reader program" to show signs of tampering. I thought that I had disguised it well, but it was extremely difficult to keep things hidden from a group of system administrators who were not only very intelligent, but also highly knowledgeable about the inner workings of UNIX. As an indication of the caliber of the people we were working against, I should note that Chuck Haley is now a researcher at Bell Labs; Bill Joy is VP of Engineering at Sun Microsystems; Kurt Schoens is a researcher at IBM; Jeff Schriebman is founder and President of UniSoft; and Bob Kridle, Vance Vaughn, and Ed Gould are founders of Mt. Xinu.

This was an unusual situation; system administrators are not usually this talented. Otherwise, they'd be doing software development rather than administration. But at the time, there was no one else capable of doing UNIX system administration.

As a result, we had to move quickly, quietly, and cleverly to stay ahead, and planting devious devices in the midst of standard software was our primary technique. Normally trusted programs which have been corrupted in this way are called "Trojan Horses", after the legend of the Greeks who were taken in by a bit of misplaced trust. One of our favorite tricks for hiding our tracks when we modified standard utilities was the diddlei program, which allowed us to reset the last change time on a modified file so that it appeared to have been unchanged since the previous year. Bob modified the setuid system call in the UNIX kernel so that, under certain circumstances, it would give the program that used it root privileges. The "certain circumstances" consisted simply of leaving a capital "S" (for Superuser) in one of the machine's registers. Bob was bold enough to leave this little feature in the system's source code. We usually put our Trojan Horses in the system executables only -- to decrease the chance of it being noticed. But Bob took the chance so that the feature would persist even if the system were recompiled. Sure enough, it lasted several months and through more than one system compilation before Dave Mosher noticed it (undoubtedly with a sense of shock) as he was patiently adding comments to the previously undocumented kernel.

This sort of battling continued for several years, and although They were suspicious of most of Us at one time or another, none of Us was ever caught red-handed. It undoubtedly helped that we never performed any malicious acts. We perhaps flouted authority, but we always enhanced the system's features. We never interfered with the system's normal operation, nor damaged any user's files. We learned that absolute power need not corrupt absolutely; instead it taught us restraint.

This is probably why we were eventually accepted as members of the system staff, even though by then several of Us had confessed to our nefarious deeds. Once we were given license to modify and improve UNIX, we lost all motivation to crack system security. We didn't know it at the time, but this has long been known to be one of the most effective ways of dealing with security problems; hire the offenders, so that there is no more Us verses Them, but simple Us.

It worked well in our case; under the auspices of the System Development and Research Group, created by the ever-industrious Dave Mosher, we went happily to work on UNIX development. The development of UNIX at Berkeley, always fast-paced, exploded once everyone -- including undergraduates -- were participating.

The only fly in the ointment was the introduction a short while later of UNIX Version 7. While it was a vast improvement in many ways over the Version 6 that we had been working with, most of the enhancements we had developed were lost in the changeover. Some were reimplemented under Version 7 by those of the group who remained at Berkeley, but by then many of us were leaving school, and the impetus behind our ideas left with us.

Ken Arnold is, perhaps, the most famous of our original group. He stayed at Berkeley longer than any of the rest of us, and became well known for such contributions as Termlib, curses, fortune, Mille Bourne, and of course his co-authorship of Rogue. But somehow it seemed a Pyrrhic victory even for Ken; much of his best work in the early years never saw the light of day.

We could not help but feel that we had passed through a sort of Dark Age for UNIX development, and even with the Renaissance in full bloom, We ponder what might have been, and bewail the features that UNIX will now never have.

Best known as the author of curses and co-author of Rogue, Ken Arnold was also President of the Berkeley Computer Club and the Computer Science Undergraduates Association during his years at UC Berkeley. He currently works as a programmer in the Computer Graphics Lab at UC San Francisco and serves as a member of the UNIX Review Software Review Board.

Copyright © 1984 by Doug Merritt, Ken Arnold, and Bob Toxen. All rights reserved.