Lurking in Code
People who develop or maintain software know how easy it is to let a system become such a complex patchwork that nobody completely knows what it does. I can count the number of software groups I’ve worked in that did it right on the fingers of one hand and not use all those fingers. Keeping a growing software system orderly, well controlled, fully tested and focused on what it needs to do requires effort, patience, good tools and self-discipline. As you can imagine, a lot of organizations aren’t willing to spend the money or take the time to do it right.
Even when software is dealt with rigorously, it’s created by people.
People make mistakes or overlook things, no matter how hard we try not to. The best software groups put enormous effort into keeping that as small as possible, but even then, problems can sneak through… or more nefarious deliberately concealed features, which can be worse. Toward the end of this post I’ll tell you about one that affects many of us, directly or indirectly.
Small Bugs Can Be Mighty
Once upon a time, I was responsible for testing the fix for a particular bug in 368,000 lines of executable FORTRAN. Just one character was wrong. I forget whether it was > when it should have been < or the other way around. About 60 factories around the world had been using that software for about 20 years before one of them hit the line that contained the mistake. When that line executed, it blew a hole in a high temperature chlorine reactor and evacuated a town.
The person who had made the mistake stayed up all night finding what had gone wrong. He wasn’t the only one who was aghast. After letting the software run amok, the group responsible for it had spent years reining it in, putting in strong software management and extensive testing. Somehow this bug had squeaked through it all.
Although it was a dangerous accident, no one got hurt. Sometimes the consequences are worse.
Bugs Can Be Deadly
A number of people died in crashes of Toyota cars that suddenly and radically accelerated. The company persistently blamed driver error, saying drivers must have pressed the accelerator rather than the brake pedal or must have improperly installed floor mats that got stuck under a pedal or a pedal must have been sticky. They drastically underreported incidents to regulators, winnowing 60,000 reports to well under 200. They tried to blame a supplier who came into their supply chain three years after the incidents began. When review of their software was done by NASA, they withheld crucial portions of the source code.
They got away with it until Jean Bookout, a woman who was severely injured but survived, took them to court. (Friend Barbara Schwarz riding with her died in the crash.) Bookout insisted the car accelerated suddenly and uncontrollably despite her efforts to slow it down. She left a 150 foot skid mark trying to slow it. Unlike others who had taken Toyota to court over such crashes, she succeeded in getting the court to order the company to submit its electronic throttle code for inspection by software experts. All of the code.
What experts found was damning. Click here if you understand software and want more details about what was wrong with the software.
The case was settled for an undisclosed amount. It is the reason I would not consider a Toyota when I was looking for a used car recently. They focused resources on covering up the problem instead of straightening out their software. They make great cars, but I can’t bring myself to trust their cars when I know how they dealt with the software.
But… I did buy a used car of another make. It probably has more software in it than an Apollo mission to the moon. I’m not comfortable with it yet. This is the first time I’ve had a car for which it is not possible for me to do most of the maintenance myself, regardless of whether I want to do it. I am uneasy about placing my faith in software I don’t personally know.
Sneaky Stuff Happens Too
When I was in transition from communications test engineering to computer programming, my first truly challenging assignment came because a programmer was sneaky. He thought setting up his code so that no one else could touch some parts of it would give him job security. Instead, it caused management to decide they couldn’t rely upon any of his work. They fired him, threw away his code and assigned me to rewrite it.
One of their worries was that he might have hidden a backdoor in his programs. A backdoor is a secret access path that bypasses the usual security.
It would have been of very limited use or interest. That particular program only put together the data stream sent down to Earth from certain types of experiments on certain types of space flights. At worst it could ruin the data from those experiments, wasting time and money. It didn’t command anything. It couldn’t endanger the crew.
What’s an example of a more serious place to put a backdoor?
Guess what? There is a backdoor in TETRA (Terrestrial Trunked Radio), a European radio standard used since the 1990s for encrypted communications, primarily in Europe but also to a lesser extent in the USA.
TETRA is in radios commonly used for critical infrastructure such as pipelines, power grids, railways, transit systems, chemical factories and oil refineries. It is also in specialized radios for police, prison, emergency services, intelligence agencies and military purposes. Something nefarious there can do a great deal of harm.
TETRA’s encryption algorithms used to be secret until Dutch researchers got into them. The researchers found severe flaws plus a deliberate backdoor. These allowed snooping on communications, decryption of encrypted communications, and transmission of fraudulent commands or messages.
What does this mean in practical terms? It means that in the midst of a crisis, a person or agency that knew about these flaws could use them to redirect emergency responders or military personnel, or spread misinformation, or spy on crisis response movements to interfere with them.
Manufacturers of affected radio units have known about this for a while. Patches for at least some of the problems have been issued. This was on the agenda for discussion at Black Hat last week (a regular conference about cybersecurity), so I haven’t spilled beans that aren’t already spilled.
Software doesn’t have to be artificial intelligence to do marvelous things, but the devil is in its details. To be good, the effort that goes into creating and testing it has to be very good. And even with the best of efforts, we’re only human. Sometimes, even with our best efforts, a few gremlins lurk in the code.
Am I sure I want to sell my old non-computerized car?
It's all very scary. This is why it's a good idea to have some cash on hand. We are overly reliant on computer systems with potential glitches that can be exploited, as you describe!
Ancient adage: show me some software with no bugs and I'll show you some software that doesn't do anything. Sadly in the world of gigabit/sec downloads, the end user is now responsible for the beta testing. Rolling out a fix is too easy. When I wrote software in the early 80s, rolling out a fix meant getting on a plane with a 256MB mass storage module and delivering it by hand. I think it's only recently that the same approach for car firmware has become feasible with the infamous "over the air upgrade" so beloved of Mr Musk. Fine for sat nav data, less great for fly-by-wire systems. The other problem is that what most people call "code" these days actually isn't - it's a sequence of calls to pre-written routines that do the actual work and which are themselves full of bugs. Which are on the whole invisible because the software is proprietary.
The first "test" of anything my team produced in the 80s was a code inspection. I'm also happy to say that the application would have worked through y2k (even if the underlying operating system wouldn't have). Now we have governments that pass laws demanding back doors into encrypted comms and we blindly accept it.