Breathalizer Source Code to Get a Day in Court

Here's an interesting news story at the intersection of embedded systems and due process:

http://www.news.com/Police-Blotter-Intoxilyzer-code-must-be-disclosed/2100-1030_3-6227951.html

How many potential bugs might a knowledgable expert witness spot in your code? Are the comments in your source accurate and clean enough for a judge or jury to read?

Building Reliable Systems

I'm writing this live from the Embedded Systems Conference in Boston, while participating in a birds of a feather discussion moderated by Jack Ganssle. The subject of the session is Building Reliable Systems.

The discussion here amongst perhaps 80 engineers (about 75% electrical engineers by education) initially focused on resources and schedules and the inevitability of bugs, but has now turned to what seems to a more productive thread: specific processes and tools that produce higher reliability.

One gentleman, had a great way of summarizing what needs to be done at a high level:

* Prioritization
* Process
* Metrics

In this view, Prioritization is an input to the Process. That is, prioritization of features relative to one another--as well as the accurate definition of properties such as quality and reliability. These are to be provided by the customer or from engineering management. The Process used for design and development and testing are then guided by these input parameters. Metrics are an output of such a Process, if you strive to generate them. Metrics include the schedule time it took to complete specific feature implementations, as well as bug rates.

On the specific point of process and tools, we've discovered that just a few companies represented in the room are using code coverage tools and test-driven development, though most appear to agree these might be helpful in raising reliability.

My bottom line fear: One of the products designed by someone in the room with me right now will kill or maim eventually!

Educating Engineers

Engineering is a fast-moving field. Even embedded systems design, which is known to use decades old tools, sees new processors and new peripherals and higher speeds and new languages and new techniques arrive continuously.

Unlike the medical profession, which requires continuing education to keep pace with changes in the knowledge base, engineers generally stop learning formally once they obtain a University degree. There is no such thing as Continuing Engineering Education credits.

Symptoms of the resulting intellectual stasis include not only a failure to keep up with evolving best practices, but also the perpetual desire of businesses to hire younger, freshly-educated engineers whether here or abroad.

If we as engineers are to continue to deliver value commensurate with our growing salaries, we must dedicate ourselves to the kind of perpetual learning required in other professions. Only then will the trend toward the use of more fresh-out and offshore engineers be slowed.

What do you think?

Firmware Ethics

We the members of the embedded software development community, in order to protect our customers (and their customers) from the risks inherent in our products, extend product lifetimes, lower long-term maintenance costs, promote the general welfare of society, and advance the integrity and reputation of our profession, do ordain and establish the following code of ethical conduct.

Check return codes. If a function can return an error code, it will almost certainly return one eventually. How will our software recover if it doesn’t even know an error has occurred? We will, therefore, always check all return codes. When an error is detected, we will take whatever action is required to recover safely.

Document everything. We make dozens of design and implementation decisions each week; we forget the reasoning behind others at a similar pace. We will, therefore, document all of our reasoning in the code itself. Furthermore, we will update these comments if our reasoning changes.

Use version control. Even when done carefully and with discipline, making backup copies of source directories, commenting out code that might be needed again, and tracking changes in comments is insufficient. The use of a good configuration management tool is a must. Furthermore, we agree to always perform a diff and provide a good description of the explanation for all changes each time a file is checked in.

Enforce coding standards. No matter how proud we may be of our abilities, we do not “own” the code we produce. Our employers and clients own it; we and others are hired to develop and maintain it. The true owners have the right to enforce a coding standard upon our work. If they do not choose to exercise this right, we will self-enforce a standard in line with generally accepted programming practices.

Run lint regularly. Compilers are more concerned with generating code than the correctness of your syntax. Only a tool like lint can watch over our shoulders to ensure we’re not straying into dangerous or non-portable programming constructs. We will, therefore, include lint in our development process. If we don’t remove all code with warnings, we will at least understand why each and every warning remains.

Perform code inspections. Automated tools are helpful, but nothing is better than a peer review for finding logic errors. Code inspections have been shown to be many times more efficient than any other single debugging tool. We will always review our code as a group at least once before each release.

Assign one tester to every developer. Programmers make lousy testers, especially when it’s their own code they’re testing. For best results, one independent tester should be hired for every developer. And the testing team must maintain and rerun regression tests regularly, to certify that a new release is at least as good as the previous one.

Measure performance. Of course, there are some kinds of tests that are best performed by the engineers themselves. In real-time systems, we shall directly measure interrupt latency and the time it takes each of our critical subroutines to complete. An oscilloscope and a spare I/O pin are quite handy in this regard.

These best practices are easy things to do, and all are inexpensive to implement. A 1 or 2 person design team can follow these rules as easily as a 50 person engineering department. If you aren’t doing these things already, there are no excuses.

Going Pro

The recent introduction of the .PRO top-level Internet domain, short for “professional,” and especially the narrow restrictions placed upon its use have got me thinking: Why don’t engineers qualify as professionals? According to the charter of .PRO, only “doctors, lawyers, and accountants” qualify to register such domains. I could not, for example, register the domain MICHAELBARR.PRO and thus market my services as an engineer on the Web.

Why are such folk as doctors, lawyers, and accountants often considered professionals by distinction from other workers? Why aren’t engineers members of that elite group? Like doctors, engineers attempt to debug complex systems and prescribe solutions and workarounds that may or may not work. Like lawyers, we are masters of arcane languages and skilled in making stuff work even in the face of seemingly bad precedents. And like accountants, we sit in our cubicles and crunch numbers—and thus make someone else’s life easier. All four professions are known to pay well and consist predominantly of white collar work.

It cannot simply be that only doctors, lawyers, and accountants are certified to practice in a particular state, as there are professional licensing examinations and boards for engineers in all states too. Engineers who’ve been licensed in this way are generally termed “P.E.’s”, an abbreviation that is short for Professional Engineer. In some states and fields of engineering P.E. licenses are, in fact, requirements for the performance of related engineering work.

I wonder if the crux of the matter isn’t simply that engineering, as a profession, hasn’t been around as long as those other fields. Or perhaps it’s that engineers and the role they play in our society hasn’t, until fairly recently, been as large or important as it is today. In a sense, perhaps the empowering of doctors, lawyers, and accountants with the imprimatur of the state—which developed as the importance of these professions to large numbers of people increased over a long history—is what really made them into elite professionals.

Based on the demonstrated long-term success of the three professional fields, it seems like anything we engineers, as a group, could do to make ourselves more like professionals would be good for individual engineers, the wider group, and society as a whole. For example, during the current downturn there has been much worry that engineering jobs are disappearing overseas. If, however, it was a requirement that any engineer involved in the design of a safety-critical product (or one simply to be approved by the FDA or FAA or another government agency) be certified by the state, such jobs would be tied to this country.

Another upside of the professionalization of engineering might be the wrenching of the power over engineering out of the hands of corporations and management and into the hands of independent engineering firms. Like law firms, such engineering companies would be skilled in specific areas of engineering and licensed to practice in one or more states. They might work on all sorts of different products for all sorts of different companies—essentially based on an hourly billing system that would properly emphasize the quality of the work over meeting a specific deadline.

To accomplish this, of course, more and greater emphasis would need to be placed on continuing education, engineering ethics, and state licensing. Most engineers today get basically no continuing education, which might have something to do with why older engineers as a group are often sidelined or pushed into management. Most engineers today are likewise also not schooled in ethics or licensed in any way; but maybe we should be.

What do you think? Do individual engineers, such as embedded systems designers, have more to gain from state licensing and continuing education than they have to lose? What about society or the greater profession?

Get Rich Slow

By a show of hands, how many of you jumped ship from a stable engineering company to a startup in the late 1990s? I bet if you didn’t, you at least thought about it or had a few offers. I never jumped ship myself, but was straddling two boats at once trying to make a quick extra million on a tight budget of night and weekend hours.

I guess I always knew the air would come out of the bubble at some point. And I sure as heck knew it wouldn’t be good to be on top if that happened. So my partners and I kept our day jobs and focused on long term issues in our business planning. How would we profit from our ideas, in ways other than a quick sale of the company or an IPO? We figured we’d identified a product and a market for it, so needed only develop the code and keep expenses lower than revenues while we tried to increase sales.

Still, though, we crossed our fingers and hoped as much as the next guy that we would time our business just right and make a bundle somehow. We certainly weren’t going to turn down a multi-million dollar purchase offer—and even felt confident enough it was worth that to turn down a bona fide small private funding source and free help from an experienced CEO that would’ve valued the company far less initially.

In the end, unfortunately, the bursting of the dot com bubble took not only the really bad ideas but also many good ones (including ours?) down with it. By the time we had filed our patent application, developed our prototype, and written our business plan, all of the funding sources for search engine enhancements had dried up. I suspect several of the venture capital firms and search engine companies we talked to would’ve jumped at the chance to be involved with our idea just a few months earlier. But the game was up. And two years later—long after we wrote off our personal investments in the company—the venture capital we needed to quit our jobs and work toward profitability still isn’t available. So I have a new plan.

My new plan is to get rich slow. To play the part of the tortoise rather than the hare. Engineering is a good stable profession, and one that generally pays well—especially if you have a specialty as in demand as real-time embedded systems design. It’s really not a bad life, if you can get it.

So rather than try to outwit or outplay, I’ll just try to outlast—and I’ll save every penny I can along the way. Besides, it’s quite a lot easier to stick to your core values and make the world a better place little by little when you’re not busy making an end run. To see how extremely disjoint the two paths can become, witness any of the recent corporate scandals.

Honesty, integrity, and responsibility should be the core values of all practicing engineers. And we should practice them outside of work as well. As fun as both are, there’s more to life than engineering and money. So I’m stopping to smell the roses now more too. It sure is nice to have my nights and weekends back! And that’s worth a lot more than a million dollars to me!

Bad Code

Enough with the bad code already! While I’ve been discussing the subject of which language to use for embedded programming and how best to ensure a quality result the past few months, millions of lines of “bad code” have been newly written.

You’ve seen the kind of code I mean: modules and procedures carelessly divided (broken up as if to meet some arbitrary length limit, for example, rather than by purpose); variables randomly named, mostly global, and with a large percentage no longer in use; compiler warnings flagging a myriad of suspicious pointer and type conversions unheeded; comments—what few there are—mostly outdated and in conflict with the nearby code; other comments full of code that once did or meant something to somebody, but now doesn’t (or does it?).

Bad programmers can write bad code in any language. It’s time they and their code were dragged into the light. I’ve encountered bad assembly code, bad C code, and bad C++ code. I’m sure those who program regularly in Ada, Java, and every other language have uncovered bad code in those languages as well.

To achieve the best long-term results, it is often necessary to have the courage to discard such code and rewrite it. If an organization can accept that the existing code was never worth the money spent to develop it in the first place, they can move on and look forward to a brighter future. Ultimately, the costs (including the rewrite) will probably be much lower.

I’ve replaced bad assembly code with new C code that was smaller, more efficient, and easier to maintain. It was also developed more quickly and cheaply than the bad code and had far fewer bugs at integration. I’ve similarly replaced bug-ridden C code with new C++ code that required half the code and data memory—and was just as efficient.

I’m not trying to suggest that C is better than assembly, that C++ is better than C, or even that the original authors in these examples chose the wrong languages to begin with. (I’ve also rewritten bad code in the same language as the original.) I’m just trying to make the point that assembly doesn’t always result in the most compact code; there’s skill involved in achieving that result. And C++ code can be just as compact and efficient as C code—if you know what you’re doing.

That, of course, is the important part: The programmer must know what he is doing. Too often that isn’t the case. However they manage to get themselves hired, bad programmers seem to exist in every organization. The decisions they make and the code they write create more problems, hassles, and bugs than any interviewer can imagine. The costs are unbearable, particularly in real-time/embedded devices.

Well? Don’t just stand there. Do something. If it’s your own code that needs the fixing: read a book like Code Complete and start learning how to write well-structured easy-to-read code, obey the Ten Commandments for C Programmers, and get a copy of lint. The version control system you aren’t using should help you feel comfortable deleting no-longer needed code rather than commenting it out. If the fault lies elsewhere: tell someone who can do something about it before any more serious damage is done.

Is C Passing?

One of the very first comments I received in response to my editorial for the May 2002 edition of Embedded Systems Programming magazine (subsequently revised and posted to this blog as “Firmware Ethics”) was the following:

You're obviously not a very good programmer and are using an archaic language. Nearly everything you said was biased toward mediocrity. Those of us Ada professionals wish that you would speak for yourself.

Though quite rudely put, the author does suggest an interesting possibility. Could the choice of programming language alone significantly improve the quality and safety of our finished products?

Of the eight suggested professional “ethics,” three (check return codes, enforce coding standards, and run lint regularly) might possibly be removed from the list if we all used a more strongly enforced language. Such a language would need to at least support exceptions, have strict syntax rules, and be strongly typed.

The C programming language fails on the first two counts. C++ adds support for exceptions but does not require programmers to use them. Among the “well-known” languages only Ada and Java meet all three requirements. Both also offer language-level support for multithreading, which enhances program portability. Yet few embedded programmers use either language.

A recent Embedded Systems Programming study suggests that almost three quarters of subscribers use C regularly. C++ is used by about half, with a slight erosion of C’s relative numbers in recent years. Assembly language remains almost as popular as C++, though the trend with that language is clearly toward decreasing use. Despite its strengths, the use of Ada—currently below 5%—is also on the decline. And, though its use is higher and on the increase, Java has a long way to go before it achieves acceptance within the embedded community.

Why not consider a switch? Not all of us develop systems with safety aspects; but perhaps those who do ought to take the issue of language selection seriously at the outset of new projects. How many bugs need to be preventable to make such a transition cost effective?

Many recognize C’s weaknesses and some, like the U.K.’s Motor Industry Software Reliability Association, have even laid down complex ground rules for its use in safety-critical systems. So why stick with C at all? Why should we allow past practice to dictate future language choices? Will it take a future catastrophe to get us to make the change we should today?

Don’t get me wrong. I love C. It was my first programming language and the one I use most competently. In an ideal world, though, the language decision should not be made based on our personal biases and experience. This is a decision that should be based solely on professional standards. But how can we compare languages analytically and measure the results of a transition from one language to another? It would be nice if there were easy answers.

Unfortunately, even if every one of us did switch to some “safer” language, miscommunication and logical errors would continue to be part and parcel of our discipline. To produce quality maintainable code, it would still be necessary to comment our work well, use version control, perform code inspections and regular testing, and measure real-time performance. Though compilers might be able to protect us from shooting ourselves in the foot, they’ll never stop us from being entirely too human.