Your Software is Never Perfect - Comments

[D H]

d-h submitted a new PF Insights post

Your Software is Never Perfect

Continue reading the Original PF Insights Post.

 

[jedishrfu]

Wonderful article! It brings back memories of working with GE and crashing the system while running in master mode with some software that was never designed to run in that environment.

Guess who got the credit/blame for the crash. Even after it was explained to the technical staff, one customer service rep pipes up at the end of the meeting: "So it was xxxx who crashed the system right?".

The crash error actually had a bright side as it illustrated how another service was crashing things and we found the bug there too. But still the one who crashed it lives on...

 

[Borg]

Nice article DH. One of the biggest lessons that I had to learn as a junior developer was that complexity did not mean that previous developers knew what they were doing. My first reaction would be that they must be really good programmers if they could write code that was difficult to follow. I would be afraid to make changes for fear of breaking something in all that complexity. These days, simplicity is my goal and I have no problem taking a chain saw to overly complex code as I'm doing on my current project. The link in my signature is my mantra for what not to do when writing software.

 

[Pepper Mint]

I share the same thoughts with Borg. I realize the best part in coding or software development in general is how to simplify things as much as possible but they must be guaranteed to have their basic provided functions preserved at the same time. This also helps to reduce costs for business development and resource management.

 

[QuantumQuest]

Really nice article. I share same thoughts with Borg too. What I tried to do for myself to improve my skills from the outset, was to be pedantic enough on testing and especially proper commenting and documentation in the web world, where I began my professional coding endeavor. Back in that era, there was a lot of improvisation, catchy things to include and proprietary things to follow, in order to survive. Fortunately this "web 1" era has long gone. After a specific point , I jumped onto the bandwagon of enterprise software - mainly Java, but I kept on taking some medium web projects, sometimes just for the fun of it. I found many times poorly documented code and ad-hoc solutions, courtesy of top developers. This does not help anyone. When such developers leave a company, their leftovers is just mess. Surviving these things, although part of the process, is something I personally try to avoid recreating. There is always room to write good code, that can be followed by others. I also vote for simplicity, provided that fits sufficiently to the project at hand. I definitely agree to the three final points that are made in this insight. Especially to the fact that there hasn't been and can never exist perfect code. There are bugs that will survive almost every testing process and manifest themselves at the wrong time. If we talk about real time software this can be fatal. Continuous testing and adaptations - including modifications, must me done in a regular basis.

 

[eltodesukane]

"It’s your name that will be on the code, so take full responsibility for it."
Good advice, but usually this can not be done.
How many times does a programmer says the code is not ready, but the employer says we release it now anyway?
Same problem in design, architecture or else..
If an architect is hired to replace a wonderful bay windows with a concrete wall, then the architect will do that.
If this concrete wall is viewed as an ugly abomination, the decision maker has to be blamed, not the architect or designer hired to do it.

 

[Hornbein]

I've been a pro programmer but my main interest is music. The standards are completely different. In music your stuff has to be pretty close to perfect if you want to make a living. In software, a total incompetent with credentials can make a living. The demand for programmers is so high that you can get away with anything. In music, demand is so low that you can be very talented and lose money.

If the situation were reversed, software WOULD be perfect. You'd starve if it weren't.

Steve Morse won a ton of Best Guitarist polls. Keyboardist T Lavitz said of guitarist Steve Morse that in five years of rehearsing and performing very difficult music he never heard Steve make a mistake. Not once. Nevertheless Steve couldn't make it as a solo act. Classical music is even more demanding, to a degree that's almost inconceivable. If programming were like that, you'd have to start at age three then do it five hours a day for the rest of your life in order to have a chance to make it. And not a very good chance at that.

Guitarist Mick Goodrick advised people to stay out of music if they could do anything else. Indeed, those who make it a career generally can't do anything else. I think there is no space left over in the brain for anything else. It's too demanding.

When I grew up I found that many of those famous jazz musicians like Barney Kessell really made their money playing for TV commercials and stuff like that. It's kept secret because it's depressing. Entertainers can't be depressing.

Yes, I know, lots of pop stars have little musical ability. That's different. Pop stardom has almost nothing to do with music.

 

[Borg]

"It’s your name that will be on the code, so take full responsibility for it."
Good advice, but usually this can not be done.
How many times does a programmer says the code is not ready, but the employer says we release it now anyway?

In my experience, there is rarely a case of not being able to do the job within the alloted time. If a developer gets sidetracked by other priorities, then the deadline is extended or someone else picks up the slack. Quite often, when I hear this excuse, it's from someone who isn't doing their job correctly - either they're goofing off and not doing their job or they're stuck and are too afraid (or too proud) to ask for help. I have no pity for the goof-offs and the prideful can be their own worst enemy. For everyone else, a simple five minute discussion of how to tackle a problem can make all the difference. I have no problem asking a junior developer how something works if I think that he has a better insight into it.

 

[anorlunda]

Wow. that was a fun read.@D-H. That's one Insights article that is really insightful.

Life isn't fair. Most developers are forced to follow orders and meet the requirements handed down, as you were. More fortunate developers, are ahead of the curve. They create the future, then show users what they really wanted, contrary to what they asked for. Perhaps the most famous example of that was Steve Jobs and his team with the iPhone. But an even better example was Dan Bricklin and Bob Frankston with Visicalc.

 

[rcgldr]

In my experience, there is rarely a case of not being able to do the job within the allotted time.

Who's setting this allotted time? The two main issues I've seen are overly optimistic schedules set by management, or unexpected technology issues, usually related to hardware.

In DH's example, the issue wasn't software bugs, as the software was doing what it was asked to do, which was to mask certain types of anomalies.

I was spoiled by my first job, back in 1973. It was a multi-processor / multi-tasking online database system for pharmacies (mostly prescriptions in the database, with support for insurance billing features). The system never experienced data corruption. There were instances of hardware failures that resulted in temporary down time, but the backup / recovery procedures developed using a test system, worked the first time they were used in an actual failure occurrence. This was a near mission critical environment. At the other extreme, I've worked in environments where a project was thrown together just to release something, and most of the software effort was spent tracking down and fixing bugs. In a few rare cases, there was a dual effort, the quick and dirty approach just to release something (like a beta release), in parallel with a proper development cycle to produce code that would replace the quick and dirty release.

 

[rootone]

There was a time when I made most of my living from freelance programming.
On one occasion I was tasked with putting right a buggy application when the original coder had departed and was seemly uncontactable,
furthermore there was little in the way of any documentation other than a description of what the system was supposed to achieve.
I told the client that it looked like there would need to be at minimum a couple weeks just going through the code, testing things and making notes.
Client was not happy to be told that and said they would get somebody else to do the job.
In the end I think what happened is the whole thing got rewritten from scratch by somebody.

 

[Borg]

Who's setting this allotted time?

I've been on projects where management set the time and where the developers were asked to provide an estimate. I've never felt that I couldn't question an estimate even if I was the provided it in the first place. I haven't had to deal with technology issues with respect to my estimates.

 

[D H]

Wow. that was a fun read.@D-H. That's one Insights article that is really insightful.

Thanks, and thanks to everyone else who liked my article.

Life isn't fair. Most developers are forced to follow orders and meet the requirements handed down, as you were.

That was my first job out of college. That's to be expected for a freshout. My career has evolved since then. I've learned that "debugging the blank sheet of paper" is my personal briar patch. (Debugging the blank sheet of paper: "There's supposed to be a design or an architecture for X on this blank sheet of paper. Fix it!")


A couple of random thoughts I did not put in my Insights article:

The article I sited in my Insights article missed a key point. The end of that article suggested all of ten of those "historical software bugs with extreme consequences" would not have occurred with improved testing. That was not that case in the case of my "historical bug." We had tests, and the software passed those tests. I thoroughly understand the concept of filtering out bad data from remote sensors. While filtering is essential, its usually reserved for egregiously bad values such as what appears to be a 6+ sigma outlier. Digital noise is not Gaussian, thank you cosmic rays.

Even the lowest person on the totem pole working on software intended to kill, but not everyone if written correctly (counterexample: software that erroneously suggests starting WWIII; see the sited article), software that most not kill if written correctly (counterexample: The Therac-25 software), or software that must not lost 400 million dollars if written correctly (counterexample: The Knight Capital fiasco) bears the burden of taking ownership of ones code. These are fields where you do not release just because the boss says "ship it, now!"

 

[.Scott]

I also like your article. Although I would quibble about the difficulty in getting code down to the 1 bug per 10,000 lines.
With thorough code reviews, thorough black-box modular testing, thorough review of code-coverage results, and good programming talent, you can get to that level and still have good productivity. Also, having been involved in several such efforts, those quality procedures themselves are not the bear. Making all of that auditable is the bear. After all, someone is paying for that quality and they want evidence that they're getting it.

 

[newjerseyrunner]

I think code rot is the biggest producer of bugs and complicated code. Lots of times developers will come up with beautifully simple designs for complicated problems, but then requests start trickling for changes that were never expected to be made. This causes refactoring of small parts of the code, which ends up making it more rigid after a while.

 

[anorlunda]

I was once involved with a service organization that helped companies deal with the Y2K bug. When it came to the conclusion, the IT workers were fired without even a thank you handshake, the business managers who created the service company went for a week long celebration in Bermuda, and the public and the media said, "See nothing happened. The Y2K bug was a myth in the first place."

I also think of the initial launch of Lotus 123 in 1983.. The 1.0 release was limited and buggy. The news reported that the Lotus startup spent $5 million on advertising and only $1 million on the software itself. Their retort was simple. They said "If this product is successful, we will have truckloads of money to abandon 1.0 and write a proper 2.0. If not successful, we will quickly abandon it anyhow So every penny spent on debugging and quality is wasted." As it turned out, customers like the buggy initial release enough, that the 2.0 version was indeed financed.

My point is that society is very hypocritical about bugs and flaws. We get so easily indignant when hearing of bugs in so-called "mission critical" places. But the reality is that those programmers who slave to check for bugs (or to not create bugs in the first place) are among the least valued members of the profession. We will never have a Turing award for one of those people.

Debugging is a very thankless task.

 

[elusiveshame]

Nice article. It's interesting to see how major firms design and code their software. You're right, though, in that there is not perfect software.

 

[Sherwood Botsford]

"It’s your name that will be on the code, so take full responsibility for it."
Good advice, but usually this can not be done.
How many times does a programmer says the code is not ready, but the employer says we release it now anyway?
Same problem in design, architecture or else..
If an architect is hired to replace a wonderful bay windows with a concrete wall, then the architect will do that.
If this concrete wall is viewed as an ugly abomination, the decision maker has to be blamed, not the architect or designer hired to do it.

So you litter the code with comments like,
/* Somewhere in this block is a bug that bites when processing partial data segments */

/* Untested code: proceed at own risk */

 

[elusiveshame]

So you litter the code with comments like,
/* Somewhere in this block is a bug that bites when processing partial data segments */

/* Untested code: proceed at own risk */

How is that littering? It's providing insight to the next programmer who either has to troubleshoot or fix code that was untested. Would you rather look for a needle in a haystack, or have some guidance to help you fix troubled code?

 

[Sherwood Botsford]

The most arcane bug I've tracked was on an MS-DOS system. At that point drives larger than 33 MB were just coming into play. You had to split them into two drives because 32 and change MB was all that Dos could address.

Twice a day a BAT file would run backing up the department accounting data to an external tape drive.

I was called in because the backup would crash every time the BAT file ran. Olliveti, the machine's maker, had been around and swapped mother boards, and this and that.

It took me a day to reproduce the problem. I basically made a clone of the orgininal machine including tape drive and controller.

The problem didn't manifest itself if the machine had a single virtual drive. So the apparent data fault was due to the logical partition. I kept cutting the data on that drive in half.

Two days later: If the first file on the second logical drive was under 509 bytes in length the driver for the tape drive would crash.

***

Some languages are more error prone than others. Something to be said for "Discipline and Bondage" languages like Pascal with strong typing and range checking

I knew a programmer who kept his tabs set at 8 characters, but used a 132 character window. If he found that his code was wandering off the right edge, he'd abstract a chunk of it out. His claim was that you could only understand so many levels deep.

Another time a grad student came to me. "Can you help me get this code to run?" I looked at it. Fortran. Written with identifiers like R6 Q27. "Look: First thing, give each variable a name that is at least 5 characters long, has a vowel, and is meaningful to you. Exception: You can use single characters for loop counters if the ohter end of the loop is in the same page. Second, for each subroutine write a 10 line description of what it is supposed to do. He grumbled and went away.

Several days later he came back, and I gave him another lesson from my first programming course. "No goto's. No abnormal loop exits." That took him longer.

While writing good code is hard, there are lots of ways to write bad code. He did eventually get his code to run, and his thesis involving black hole physics got him his masters.

 

[Sherwood Botsford]

How is that littering? It's providing insight to the next programmer who either has to troubleshoot or fix code that was untested. Would you rather look for a needle in a haystack, or have some guidance to help you fix troubled code?

Littering in a somewhat different sense is to scatter randomly. Trees litter the ground with leaves. Of late it has come to mean trash.

It is likely to get you in trouble with your boss. If you leave comments that are perjoritive about the code, and it results in an accident or financial loss, then the company as a group knew of the bug, and therefore is more culpable.

On a complex project change submissions are reviewed by someone, so it often takes a conspiracy or very enlightened policy to get this sort of comment embedded.

 

[Borg]

I knew a programmer who kept his tabs set at 8 characters, but used a 132 character window. If he found that his code was wandering off the right edge, he'd abstract a chunk of it out. His claim was that you could only understand so many levels deep.

Putting a chunk of code in a separate method to keep the code clean isn't the worst thing in the world. 8 space tabs is pretty excessive though.

You probably don't mean abstraction the way that I have to deal with it. In my current project, I mostly have to deal with code that is accessed through abstract interfaces even though there may only be a single implementation of the class. The interfaces are then created through multiple configuration files that are themselves abstracted. There is no organization as to where various code is located in the source tree such that html-based code can often be found at the database layers making it extremely difficult to modify and test without running the server. Then, toss in a few undocumented expectations of OS environment variables for extra fun. Figuring out the code when someone does this can be a nightmare.

 

[mfb]

I knew a programmer who kept his tabs set at 8 characters, but used a 132 character window. If he found that his code was wandering off the right edge, he'd abstract a chunk of it out. His claim was that you could only understand so many levels deep.

Putting a chunk of code in a separate method to keep the code clean isn't the worst thing in the world. 8 space tabs is pretty excessive though.

Doing it only after ~8-15 indentation steps (not sure how to interpret the statement) is way too late, however.


You can certainly overdo it with making new classes and functions for everything, of course - a function that does nothing but calling a different function can be useful, but if you do that 5 layers deep something could be wrong.

 

[Borg]

You can certainly overdo it with making new classes and functions for everything, of course - a function that does nothing but calling a different function can be useful, but if you do that 5 layers deep something could be wrong.

Definitely agree. There is seldom a reason for code to be nested so deep. Usually when I see something like that, the person should have written a recursive function.

 

[anorlunda]

My personal favorites for obscure bugs came on two occasions when (the same) someone asked me for help.

1. He couldn't find the problem despite numerous re-compiliations adding PRINT statements for diagnostic purposes. He tried to re-compile once again, and out of the corner of my eye I caught the card reader shuffling the order of his cards.
   
   
2. He couldn't find another bug. I helped him to narrow it down to a single statement, then a particular expression, then to floating multiply of two specific values. I wrote a new program to multiply those two values and compare the result with the known answer. It failed 0.003% of the time. The floating point unit was replaced and the bug vanished. Even in today's world, I suspect that programmers expect the CPU to faithfully execute the code with zero error rate.

 

[StatGuy2000]

DH, a well written article - thank you for sharing.

I quote the following from your post above: "Well-crafted, well-inspected, and well-tested code typically has one bug per one thousand lines, or perhaps one bug per every ten thousand lines if done very carefully.

Reference https://www.physicsforums.com/insights/software-never-perfect/
"

This raises a question for me. Software plays a critical importance in systems where safety and/or reliability is of critical importance (e.g. nuclear power plants, electric grids, air-traffic control, medical equipment, etc.) Under those circumstances, the threshold for tolerance of bugs in software would be extremely low (if not close to non-existent). Under these circumstances, how could engineers or software developers ensure that the system would be as close to "perfect" as possible?

 

[mfb]

There are a lot of guidelines that help making bugs less frequent and easier to find. As an example, some commands or concepts are known to be bug-prone, and you can avoid them. Apart from that: code reviews, extensive test cases (typically written earlier than the code itself) and so on.
A common way to avoid hardware issues (e. g. from radiation damage) is the majority vote system: have three identical systems, do what at least two of the systems suggest to do. If one of them fails, its output is overridden by the other two systems.

 

[Sherwood Botsford]

@StatGuy20

If the figures are correct, then one way to do this is to minimize code size. Fewer lines = fewer bugs.

For really critical code, do the code in different languages by different teams. Do they get the same results? Code in 3 languages, and run all at the same time against real time data. two out of three separate systems have to agree to take action.

Use discipline & bondage languages that enforce strong typing, bounds checking, strict calling conventions. Sure, it slows down the code. You want it in a microsecond, or you want it right in 3 microseconds?

Use standard libraries whenever possible. Some one else has found the bugs. Or most of them.


A large number of problems aren't with the code, but with the operating system libraries. The obvious response to that, is use a well audited operating system. OpenBSD last I looked, had an enviable reputation for exploitable bugs.

 

[Svein]

If you are producing drivers for some high-speed peripheral (say Gigabit Ethernet hardware), your software is going to be run thousands of times every day. And - if the product is successful - in thousands of locations every day. The chance of even a small bug going unnoticed are close to zero.

A hardware-related anecdote: I once wrote an Ethernet driver for a system-on-a-chip based on one of the ARM CPUs. The Ethernet driver worked perfectly, but somehow no IP packet made it to the IP handler. The cause turned out to be a combination of the Ethernet packet format and a peculiarity of the ARM CPU:

• The 32 bit ARM CPU could only access words on a 32 bit boundary. If you violated this rule, the CPU did not protest, but read the closest 32 bit word, scrambling the contents along the way.
• The Ethernet packet format starts with two 6-byte addresses plus a 2-byte packet type. Add this together, and you get a 14-byte offset from the start of the Ethernet packet to the payload.
• The IP software assumes that the IP packet is aligned on a 32 bit boundary (or that the CPU can access the contents as if it were on a 32 bit boundary).

Those three facts together resulted in the ARM CPU trying to make sense of a scrambled IP address. The solution: Copy payload to a buffer aligned on a 32 bit boundary (a bad solution, since copying a packet from one place in memory to another is slower than transmitting the packet across the Ethernet).

 

[D H]

This raises a question for me. Software plays a critical importance in systems where safety and/or reliability is of critical importance (e.g. nuclear power plants, electric grids, air-traffic control, medical equipment, etc.) Under those circumstances, the threshold for tolerance of bugs in software would be extremely low (if not close to non-existent). Under these circumstances, how could engineers or software developers ensure that the system would be as close to "perfect" as possible?

By working very carefully, by using thorough testing, by using processes that reduce the number of bugs that arise, by having a top-to-bottom attitude toward quickly stomping those few bugs that do arise, and by having others check everything.

 

[Svein]

Software plays a critical importance in systems where safety and/or reliability is of critical importance

Yes, and that is a specialist area all by itself. I am not going to list all the standards that must be met, but you can get an overview here: https://en.wikipedia.org/wiki/Functional_safety .

 

[anorlunda]

Often neglected is the fact that many (perhaps most?) bugs are benign. They may never get triggered, or their negative effect not noticeable.

An interesting example came up during the Y2K remediation. Some software in nuclear power plants had operated successfully for 35 years or more. Presumably, software standards were much better in 1999 than in 1965, so new software is expected to have many fewer bugs. On the other hand, the old software had amply demonstrated that any bugs remaining must be benign. New software may have few bugs (never say zero), but they may not be benign.

So which is safer, the old or new software? That question should not be answered flippantly.

That same question arose in many critical applications in many industries during the Y2K years. It is the age old debate between new and better versus proven.

 

[QuantumQuest]

Often neglected is the fact that many (perhaps most?) bugs are benign. They may never get triggered, or their negative effect not noticeable.

An interesting example came up during the Y2K remediation. Some software in nuclear power plants had operated successfully for 35 years or more. Presumably, software standards were much better in 1999 than in 1965, so new software is expected to have many fewer bugs. On the other hand, the old software had amply demonstrated that any bugs remaining must be benign. New software may have few bugs (never say zero), but they may not be benign.

So which is safer, the old or new software? That question should not be answered flippantly.

That same question arose in many critical applications in many industries during the Y2K years. It is the age old debate between new and better versus proven.

In my opinion, each software - old vs. new, must be judged according to its domain and problem/s it solved/solves, at its time.

In the old days, there were not so many programming languages and dialects of them, a few programming paradigms, testing was all but trivial and maintenance was difficult at best - talking about big software. Positive thing is that developers, had the time and opportunity to focus better on what they were creating, so for that instant there were mostly benign bugs.

In the present days on the other hand, there are huge demands for software to solve way more difficult/demanding problems, operate on a lot of newer domains, operate in an inter-domain fashion, and although there is a multitude of programming languages - this holds for multi-branch descendants too, a lot of programming and testing tools, libraries and frameworks, its complexity increases at a very steep fashion and even with the best designed tool chains for full cycle development, bugs are inevitable and this I think also justifies that are not so benign, as in the old days.

I don't think that a direct comparison can be done between old and new software. There is a whole lot of independent and interdependent factors between old and new, as well as in each of them, that makes a direct comparison very difficult.

 

[anorlunda]

In the present days on the other hand, there are huge demands for software to solve way more difficult/demanding problems, operate on a lot of newer domains, ...a

Where it comes to embedded real time control software, I dispute that so much has really changed. In a few cases, modern software may have difficulty doing as well as some of the vintage stuff.

 

[QuantumQuest]

Where it comes to embedded real time control software, I dispute that so much has really changed. In a few cases, modern software may have difficulty doing as well as some of the vintage stuff.

I agree. I was talking about software at large and not particularly for embedded software.