Tags:
1. May 29, 2016

D H

Staff Emeritus
2. May 29, 2016

Staff: Mentor

Wonderful article! It brings back memories of working with GE and crashing the system while running in master mode with some software that was never designed to run in that environment.

Guess who got the credit/blame for the crash. Even after it was explained to the technical staff, one customer service rep pipes up at the end of the meeting: "So it was xxxx who crashed the system right?".

The crash error actually had a bright side as it illustrated how another service was crashing things and we found the bug there too. But still the one who crashed it lives on...

Last edited: May 29, 2016
3. May 30, 2016

Borg

Nice article DH. One of the biggest lessons that I had to learn as a junior developer was that complexity did not mean that previous developers knew what they were doing. My first reaction would be that they must be really good programmers if they could write code that was difficult to follow. I would be afraid to make changes for fear of breaking something in all that complexity. These days, simplicity is my goal and I have no problem taking a chain saw to overly complex code as I'm doing on my current project. The link in my signature is my mantra for what not to do when writing software.

4. May 30, 2016

Pepper Mint

I share the same thoughts with Borg. I realize the best part in coding or software development in general is how to simplify things as much as possible but they must be guaranteed to have their basic provided functions preserved at the same time. This also helps to reduce costs for business development and resource management.

5. May 30, 2016

QuantumQuest

Really nice article. I share same thoughts with Borg too. What I tried to do for myself to improve my skills from the outset, was to be pedantic enough on testing and especially proper commenting and documentation in the web world, where I began my professional coding endeavor. Back in that era, there was a lot of improvisation, catchy things to include and proprietary things to follow, in order to survive. Fortunately this "web 1" era has long gone. After a specific point , I jumped onto the bandwagon of enterprise software - mainly Java, but I kept on taking some medium web projects, sometimes just for the fun of it. I found many times poorly documented code and ad-hoc solutions, courtesy of top developers. This does not help anyone. When such developers leave a company, their leftovers is just mess. Surviving these things, although part of the process, is something I personally try to avoid recreating. There is always room to write good code, that can be followed by others. I also vote for simplicity, provided that fits sufficiently to the project at hand. I definitely agree to the three final points that are made in this insight. Especially to the fact that there hasn't been and can never exist perfect code. There are bugs that will survive almost every testing process and manifest themselves at the wrong time. If we talk about real time software this can be fatal. Continuous testing and adaptations - including modifications, must me done in a regular basis.

Last edited: Jul 12, 2018 at 6:31 PM
6. May 30, 2016

eltodesukane

"It’s your name that will be on the code, so take full responsibility for it."
Good advice, but usually this can not be done.
How many times does a programmer says the code is not ready, but the employer says we release it now anyway?
Same problem in design, architecture or else..
If an architect is hired to replace a wonderful bay windows with a concrete wall, then the architect will do that.
If this concrete wall is viewed as an ugly abomination, the decision maker has to be blamed, not the architect or designer hired to do it.

7. May 30, 2016

Hornbein

I've been a pro programmer but my main interest is music. The standards are completely different. In music your stuff has to be pretty close to perfect if you want to make a living. In software, a total incompetent with credentials can make a living. The demand for programmers is so high that you can get away with anything. In music, demand is so low that you can be very talented and lose money.

If the situation were reversed, software WOULD be perfect. You'd starve if it weren't.

Steve Morse won a ton of Best Guitarist polls. Keyboardist T Lavitz said of guitarist Steve Morse that in five years of rehearsing and performing very difficult music he never heard Steve make a mistake. Not once. Nevertheless Steve couldn't make it as a solo act. Classical music is even more demanding, to a degree that's almost inconceivable. If programming were like that, you'd have to start at age three then do it five hours a day for the rest of your life in order to have a chance to make it. And not a very good chance at that.

Guitarist Mick Goodrick advised people to stay out of music if they could do anything else. Indeed, those who make it a career generally can't do anything else. I think there is no space left over in the brain for anything else. It's too demanding.

When I grew up I found that many of those famous jazz musicians like Barney Kessell really made their money playing for TV commercials and stuff like that. It's kept secret because it's depressing. Entertainers can't be depressing.

Yes, I know, lots of pop stars have little musical ability. That's different. Pop stardom has almost nothing to do with music.

8. May 30, 2016

Borg

In my experience, there is rarely a case of not being able to do the job within the alloted time. If a developer gets sidetracked by other priorities, then the deadline is extended or someone else picks up the slack. Quite often, when I hear this excuse, it's from someone who isn't doing their job correctly - either they're goofing off and not doing their job or they're stuck and are too afraid (or too proud) to ask for help. I have no pity for the goof-offs and the prideful can be their own worst enemy. For everyone else, a simple five minute discussion of how to tackle a problem can make all the difference. I have no problem asking a junior developer how something works if I think that he has a better insight into it.

9. May 30, 2016

Staff: Mentor

Wow. that was a fun read.@D-H. That's one Insights article that is really insightful.

Life isn't fair. Most developers are forced to follow orders and meet the requirements handed down, as you were. More fortunate developers, are ahead of the curve. They create the future, then show users what they really wanted, contrary to what they asked for. Perhaps the most famous example of that was Steve Jobs and his team with the iPhone. But an even better example was Dan Bricklin and Bob Frankston with Visicalc.

10. May 30, 2016

rcgldr

Who's setting this allotted time? The two main issues I've seen are overly optimistic schedules set by management, or unexpected technology issues, usually related to hardware.

In DH's example, the issue wasn't software bugs, as the software was doing what it was asked to do, which was to mask certain types of anomalies.

I was spoiled by my first job, back in 1973. It was a multi-processor / multi-tasking online database system for pharmacies (mostly prescriptions in the database, with support for insurance billing features). The system never experienced data corruption. There were instances of hardware failures that resulted in temporary down time, but the backup / recovery procedures developed using a test system, worked the first time they were used in an actual failure occurrence. This was a near mission critical environment. At the other extreme, I've worked in environments where a project was thrown together just to release something, and most of the software effort was spent tracking down and fixing bugs. In a few rare cases, there was a dual effort, the quick and dirty approach just to release something (like a beta release), in parallel with a proper development cycle to produce code that would replace the quick and dirty release.

11. May 30, 2016

rootone

There was a time when I made most of my living from freelance programming.
On one occasion I was tasked with putting right a buggy application when the original coder had departed and was seemly uncontactable,
furthermore there was little in the way of any documentation other than a description of what the system was supposed to achieve.
I told the client that it looked like there would need to be at minimum a couple weeks just going through the code, testing things and making notes.
Client was not happy to be told that and said they would get somebody else to do the job.
In the end I think what happened is the whole thing got rewritten from scratch by somebody.

12. May 30, 2016

Borg

I've been on projects where management set the time and where the developers were asked to provide an estimate. I've never felt that I couldn't question an estimate even if I was the provided it in the first place. I haven't had to deal with technology issues with respect to my estimates.

13. May 30, 2016

D H

Staff Emeritus
Thanks, and thanks to everyone else who liked my article.

That was my first job out of college. That's to be expected for a freshout. My career has evolved since then. I've learned that "debugging the blank sheet of paper" is my personal briar patch. (Debugging the blank sheet of paper: "There's supposed to be a design or an architecture for X on this blank sheet of paper. Fix it!")

A couple of random thoughts I did not put in my Insights article:

The article I sited in my Insights article missed a key point. The end of that article suggested all of ten of those "historical software bugs with extreme consequences" would not have occurred with improved testing. That was not that case in the case of my "historical bug." We had tests, and the software passed those tests. I thoroughly understand the concept of filtering out bad data from remote sensors. While filtering is essential, its usually reserved for egregiously bad values such as what appears to be a 6+ sigma outlier. Digital noise is not Gaussian, thank you cosmic rays.

Even the lowest person on the totem pole working on software intended to kill, but not everyone if written correctly (counterexample: software that erroneously suggests starting WWIII; see the sited article), software that most not kill if written correctly (counterexample: The Therac-25 software), or software that must not lost 400 million dollars if written correctly (counterexample: The Knight Capital fiasco) bears the burden of taking ownership of ones code. These are fields where you do not release just because the boss says "ship it, now!"

14. May 31, 2016

.Scott

I also like your article. Although I would quibble about the difficulty in getting code down to the 1 bug per 10,000 lines.
With thorough code reviews, thorough black-box modular testing, thorough review of code-coverage results, and good programming talent, you can get to that level and still have good productivity. Also, having been involved in several such efforts, those quality procedures themselves are not the bear. Making all of that auditable is the bear. After all, someone is paying for that quality and they want evidence that they're getting it.

15. May 31, 2016

newjerseyrunner

I think code rot is the biggest producer of bugs and complicated code. Lots of times developers will come up with beautifully simple designs for complicated problems, but then requests start trickling for changes that were never expected to be made. This causes refactoring of small parts of the code, which ends up making it more rigid after a while.

16. May 31, 2016

Staff: Mentor

I was once involved with a service organization that helped companies deal with the Y2K bug. When it came to the conclusion, the IT workers were fired without even a thank you handshake, the business managers who created the service company went for a week long celebration in Bermuda, and the public and the media said, "See nothing happened. The Y2K bug was a myth in the first place."

I also think of the initial launch of Lotus 123 in 1983.. The 1.0 release was limited and buggy. The news reported that the Lotus startup spent $5 million on advertising and only$1 million on the software itself. Their retort was simple. They said "If this product is successful, we will have truckloads of money to abandon 1.0 and write a proper 2.0. If not successful, we will quickly abandon it anyhow So every penny spent on debugging and quality is wasted." As it turned out, customers like the buggy initial release enough, that the 2.0 version was indeed financed.

My point is that society is very hypocritical about bugs and flaws. We get so easily indignant when hearing of bugs in so-called "mission critical" places. But the reality is that those programmers who slave to check for bugs (or to not create bugs in the first place) are among the least valued members of the profession. We will never have a Turing award for one of those people.

Debugging is a very thankless task.

17. Jun 8, 2016

elusiveshame

Nice article. It's interesting to see how major firms design and code their software. You're right, though, in that there is not perfect software.

18. Jun 8, 2016

Sherwood Botsford

So you litter the code with comments like,
/* Somewhere in this block is a bug that bites when processing partial data segments */

/* Untested code: proceed at own risk */

19. Jun 8, 2016

elusiveshame

How is that littering? It's providing insight to the next programmer who either has to troubleshoot or fix code that was untested. Would you rather look for a needle in a haystack, or have some guidance to help you fix troubled code?

20. Jun 8, 2016

Sherwood Botsford

The most arcane bug I've tracked was on an MS-DOS system. At that point drives larger than 33 MB were just coming into play. You had to split them into two drives because 32 and change MB was all that Dos could address.

Twice a day a BAT file would run backing up the department accounting data to an external tape drive.

I was called in because the backup would crash every time the BAT file ran. Olliveti, the machine's maker, had been around and swapped mother boards, and this and that.

It took me a day to reproduce the problem. I basically made a clone of the orgininal machine including tape drive and controller.

The problem didn't manifest itself if the machine had a single virtual drive. So the apparent data fault was due to the logical partition. I kept cutting the data on that drive in half.

Two days later: If the first file on the second logical drive was under 509 bytes in length the driver for the tape drive would crash.

***

Some languages are more error prone than others. Something to be said for "Discipline and Bondage" languages like Pascal with strong typing and range checking

I knew a programmer who kept his tabs set at 8 characters, but used a 132 character window. If he found that his code was wandering off the right edge, he'd abstract a chunk of it out. His claim was that you could only understand so many levels deep.

Another time a grad student came to me. "Can you help me get this code to run?" I looked at it. Fortran. Written with identifiers like R6 Q27. "Look: First thing, give each variable a name that is at least 5 characters long, has a vowel, and is meaningful to you. Exception: You can use single characters for loop counters if the ohter end of the loop is in the same page. Second, for each subroutine write a 10 line description of what it is supposed to do. He grumbled and went away.

Several days later he came back, and I gave him another lesson from my first programming course. "No goto's. No abnormal loop exits." That took him longer.

While writing good code is hard, there are lots of ways to write bad code. He did eventually get his code to run, and his thesis involving black hole physics got him his masters.

21. Jun 8, 2016

Sherwood Botsford

Littering in a somewhat different sense is to scatter randomly. Trees litter the ground with leaves. Of late it has come to mean trash.

It is likely to get you in trouble with your boss. If you leave comments that are perjoritive about the code, and it results in an accident or financial loss, then the company as a group knew of the bug, and therefore is more culpable.

On a complex project change submissions are reviewed by someone, so it often takes a conspiracy or very enlightened policy to get this sort of comment embedded.

22. Jun 8, 2016

Borg

Putting a chunk of code in a separate method to keep the code clean isn't the worst thing in the world. 8 space tabs is pretty excessive though.

You probably don't mean abstraction the way that I have to deal with it. In my current project, I mostly have to deal with code that is accessed through abstract interfaces even though there may only be a single implementation of the class. The interfaces are then created through multiple configuration files that are themselves abstracted. There is no organization as to where various code is located in the source tree such that html-based code can often be found at the database layers making it extremely difficult to modify and test without running the server. Then, toss in a few undocumented expectations of OS environment variables for extra fun. Figuring out the code when someone does this can be a nightmare.

23. Jun 8, 2016

Staff: Mentor

Doing it only after ~8-15 indentation steps (not sure how to interpret the statement) is way too late, however.

You can certainly overdo it with making new classes and functions for everything, of course - a function that does nothing but calling a different function can be useful, but if you do that 5 layers deep something could be wrong.

24. Jun 8, 2016

Borg

Definitely agree. There is seldom a reason for code to be nested so deep. Usually when I see something like that, the person should have written a recursive function.

25. Jun 8, 2016

Staff: Mentor

My personal favorites for obscure bugs came on two occasions when (the same) someone asked me for help.
1. He couldn't find the problem despite numerous re-compiliations adding PRINT statements for diagnostic purposes. He tried to re-compile once again, and out of the corner of my eye I caught the card reader shuffling the order of his cards.

2. He couldn't find another bug. I helped him to narrow it down to a single statement, then a particular expression, then to floating multiply of two specific values. I wrote a new program to multiply those two values and compare the result with the known answer. It failed 0.003% of the time. The floating point unit was replaced and the bug vanished. Even in today's world, I suspect that programmers expect the CPU to faithfully execute the code with zero error rate.