RTOS Myth #4: The RTOS is in Charge

The Myth: The operating system is in charge and it decides when to switch from one application task to another.

The Truth: A real-time operating system (RTOS) is a very different beast than a multi-user desktop operating system, such as Linux. In fact, an RTOS is simply a library of functions plus a timer tick interrupt handler.

The only opportunities for an RTOS to effect a context switch from one task to another are:

1. If a running task deletes itself (or exits, if your OS allows that). In this case, a function in the RTOS library detects the lack of a running task and can directly invoke the scheduler function to select the next task to run.

2. When a running task blocks, which can only happen by making a function call into the RTOS library.

3. If a running task creates a new task with a priority higher than it's own.

4. When a previously blocked task of higher priority is unblocked, which could happen as a result of:

a. The running task made a function call into the RTOS library (e.g., semaphore post).

b. An interrupt service routine executed with that side effect.

These four (or, five, if you prefer) points of entry into the RTOS are the only mechanisms by which control of the CPU can be transferred from one task to another. They are called "scheduling points".

The implication here is that your application code is actually in charge. If it were to avoid calling into the RTOS library while simultaneously disabling interrupts, the running task could steal control of the CPU for any length of time.

Real-Time Java is Dead

A few months less than ten years ago, I presented a paper at the Embedded Systems Conference (ESC) for the first time. My 1.5 hour course was entitled "Embedded Java" or something similar. This was in Silicon Valley, and the audience was standing room only--despite a rather large room to start with.

I've tracked technical and business developments in the world of embedded and real-time Java for even longer--and written a number of articles on the subject. And so I didn't miss that over the years the audiences for courses on either variant of Java dwindled. After a hopeful year or two too long, I gave up on Java in our space and stopped proposing the topic at ESC.

Then, last Spring, I had the refreshing experience of teaching a two-day hands-on real-time Java programming class in the Netherlands. The room was packed. There was enthusiasm. And these guys were really using Java to develop software (though it wasn't truly embedded code). So I thought I'd try again at ESC and reproposed the topic.

Last week I taught a course called "Real-Time Java Programming" at the ESC Silicon Valley venue. A lot had changed about the audience size. This time, in a room of a similar size, there were many empty chairs and tables. I think there were perhaps 15 people attending this time.

By my reckoning, Java is officially dead in the embedded systems community--especially in the U.S.

Educating Embedded Engineers

One of the biggest problems affecting the embedded systems industry is a shortage of properly skilled firmware developers. I spoke of this in a recent video interview.

Embedded software developers need some skills taught to Electrical Engineers and others taught to Computer Scientists. They also need practical hands-on experience. Unfortunately, even the best of the Computer Engineering programs at Universities fail to deliver the right mix for embedded work.

The demand for such skilled help is growing rapidly--with the latest estimates put at over 4 billion embedded products manufactured per year.

Fellow Embedded Guru Mike Anderson wrote a nice piece about all this for an IEEE-USA publication called "Today's Engineer." I happened across Mike's article in print, but you can find it online by the title "Help Wanted: Embedded Engineers."

Toward a Better Mutex API

A few months ago I blogged that mutexes and semaphores are distinct RTOS primitives. Unfortunately, the APIs of today's most popular commercial RTOSes only add to the confusion for application programmers.

For example, consider the VxWorks API, which not only forces mutexes and semaphores into an inappropriately common-looking API (semMXxx vs. semCXxx) but also adds a third "binary semaphore" type (semBxxx). Micrium's popular uC/OS-II API is preferable to this in that it at least has just two primitive types (OSMutexXxx and OSSemXxx). But the uC/OS-II API also forces programmers to use similar function name suffices--Post() and Pend()--with each.

I propose a new and clearer API such as the following, which is based loosely on uC/OS-II's current API:

int OSSemCreate(SEM * phSem, int cnt)
int OSSemPost(SEM hSem)
int OSSemPend(SEM hSem, int timeout)

int OSMutexCreate(MUTEX * phMutex)
int OSMutexGet(MUTEX hMutex)
int OSMutexPut(MUTEX hMutex)

Wind River would do well to eliminate semBxxx functions and rename the semCXxx functions as semXxx. In addition, the VxWorks API for mutexes should be changed to something like mutexXxx.

Note, too, that this new API is intended to force each mutex object to be created in the 'available' state (i.e., value = 1), as it already is in uC/OS-II. An additional feature of the mutex API should be that any OSMutexPut() call by a task that does not currently own that mutex should fail with an appropriate error code. Together these easily used mutex functions ensure correct usage of mutexes by application developers.

RTOS Myth #3: Mutexes are Needed at the Task Level

The Myth: Mutexes are a useful intertask synchronization primitive, which you should expect to use frequently.

The Truth: Mutexes are a necessary feature of all real-time operating systems. However, best practice is to use them only inside functions that must be reentrant. That is, you should use mutexes only inside functions that are or could be called by two or more tasks.

Mutexes, as their name suggests, enforce mutual exclusion and thus eliminate race conditions between tasks. A mutex can be used to protect a global data structure accessed by two or more tasks. However, doing this properly requires that each user task have the actual address of the global data and allows for bugs in the failure of one task to acquire and release the associated mutex properly.

It is preferable always to abstract or encapsulate said global data structure as an object, which can only be read or written through a set of reentrant function calls. That way, both the address (and possibly internal format) of the data structure can be hidden and the mutex calls can be ensured to be coded correctly.

Note that mutexes are also necessary inside functions that control hardware through I/O registers, which are effectively global data structures.

Of course, a priority inheritance capability should be present within the mutex API (put there by the RTOS vendor) to ensure that priority inversions cannot occur.

RTOS Myth #2: RMA is for Academics

The Myth: The Rate Monotonic Algorithm (RMA) is an interesting theory but it has no practical meaning for users of real-time operating systems.

The Truth: For starters,

• All of the popular real-time operating systems (e.g., VxWorks, ThreadX, and uC/OS-II) feature fixed-priority preemptive schedulers
  
• RMA is the optimal fixed-priority scheduling algorithm (and note that dynamic-priority algorithms do not degrade gracefully)
  
• Unless you use RMA to assign priorities to RTOS tasks, there are no task-specific performance guarantees; if the processor becomes overly busy in a brief period of time, a critical task may miss its deadline
  

In a nutshell, RMA is the one and only proper way to assign relative priorities to RTOS tasks with deadlines. (Shock of shocks: Deferring to your boisterous colleague Bill's insistence that his task is the most important isn't guaranteed to work!) There's a nice introduction to the RMA technique at http://www.netrino.com/Embedded-Systems/How-To/RMA-Rate-Monotonic-Algorithm/.

The principal benefit of RMA is that the performance of a set of tasks thus prioritized degrades gracefully. Your key "critical set" of tasks can be guaranteed (even proven a priori) to always meet its deadlines--even during periods of transient overload. Dynamic-priority operating systems cannot make this guarantee. Nor can static-priority RTOSes running tasks prioritized in other ways.

Too many of today's real-time systems built with an RTOS are working by accident. Excess processing power can mask a lot of design sins. But if you haven't used RMA to assign priorities, it could just be a matter of time before you get burned.

VDC Counts 4 Billion Embedded Products Shipped in 2006

Read Venture Development Corporations OnTarget Blog for more details. But in a nutshell, in the last year for which data are available, VDC estimates 4 billion total units of embedded products, most without a commercial operating system inside. By other estimates, total CPU shipments in 2006 were around 9 billion, but some of those are PCs/Macs and there are also numerous systems containing two or more CPUs.

More Bad RTOS Information

The Internet (and magazines and conferences) are filled with bad information about when to choose an RTOS. In short, the world wants to sell you an RTOS, even when you don't need one or the use of one would overly complicate your software design.

Here are two generalizations from a recent whitepaper:

Operating systems make programming more efficient and better structured, and their use is now frequently justified even in embedded solutions that are relatively small.

and

A clear benefit of using an RTOS is that it reduces time to market, because it simplifies development.

At best, this is misguided advice.

Here's the straight scoop. An RTOS may either "make programming more efficient and better structured" or less efficient and poorly structured; it depends on the nature of the requirements. In many cases, a design composed entirely of state machines is easier to code and works more reliably than one using an RTOS. In other cases, particularly closed-loop control systems, a simple main+ISR approach will work even better.

Embedded Systems History

The editors of Embedded Systems Design recently put together an interesting historical timeline of the embedded systems industry. To learn some history, take a stroll over to http://www.embedded.com/timeline/.

Embedded C Quiz Results

When we redesigned the Netrino.com website late last year, we thought it'd be fun to challenge our more than 20,000 monthly visitors (mostly embedded software engineers) to a skills test. So we developed a ten question multiple-choice quiz (http://www.netrino.com/Embedded-Systems/Embedded-C-Quiz). And it has been a popular feature of the new site, with a couple hundred participants just in the first two months.

And now the results are starting to come in. We analyzed the early results a couple of ways and discovered something worth talking about: Quiz takers from India did about the same as quiz takers in the U.S. But the rest of the world lagged behind these two groups quite a bit.

There are ten questions in our quiz, and we consider a passing score to be 8 out of 10. A handful of quiz takers have scored 100%, but most score in the 30-90% range, with an overall average at 60.4%. (A little scary, huh?)

Statisically speaking, there were three significant groups of quiz takers by geography. The average score of those taking the quiz from the United States was just shy of 64%. The average for India was not far behind at about 61.2%. However, the rest of the world scored an average of just 55.9%.

What does this say about the state of the profession of embedded software development? Offshoring? The quiz itself?

Breathalizer Source Code to Get a Day in Court

Here's an interesting news story at the intersection of embedded systems and due process:

http://www.news.com/Police-Blotter-Intoxilyzer-code-must-be-disclosed/2100-1030_3-6227951.html

How many potential bugs might a knowledgable expert witness spot in your code? Are the comments in your source accurate and clean enough for a judge or jury to read?

Twenty Years of Embedded Systems Design

It's hard to believe, but industry magazine Embedded Systems Design, which I edited for five years from 1999 to 2004, is this year celebrating its twentieth continuous year in print.

The popular magazine, which was titled Embedded Systems Programming for about the first 16 years, started its run in 1988, at a time when the widespread use of assembly language was challenged by some engineers as inefficent compared with coding in raw hex. There were then dozens of dialects of the C language, which few embedded developers used, but a committee at ANSI was actively working on the standard that would ultimately win.

The passing of this event brings two interesting thoughts to my mind. One about the past, the other about the future.

Looking backward, I wonder just how much the embedded developer community owes to Ted Bahr and Tyler Sperry (the founding publisher and editor-in-chief, respectively) for giving us the name "Embedded Systems". Although the concept of embedding microcontrollers into products dates back 17 years earlier (to the introduction of the 4004 single-chip micro by Intel in 1971), the name embedded systems was made popular by the success of the magazine and associated Embedded Systems Conference.

Looking forward, though, I wonder just how much longer the print magazine is for the world. Tremendous changes have rocked the universe of print media--even just in the four years since I left the editor-in-chief role at Embedded Systems Programming. Both the number of subscribers and the number of pages in the print magazine peaked during my tenure, at about 70,000 and over 120, respectively. Today, the circulation is down considerably and the typical number of pages is around a third of that peak. Online advertising has changed the game for most technology companies, who no longer find the unmeasurable and expensive investment in print ads their only option.

What do you think?

RTOS Myth #1: Mutexes and Semaphores are Interchangeable

The Myth: Mutexes and semaphores are similar--even interchangeable--operating system primitives.

The Truth: Mutexes and semaphores should always be used for distinct purposes, and should thus feature distinct APIs. (My recommendations to RTOS vendors are at the end.)

The cause of the confusion between mutexes and semaphores is historical, dating all the way back to the 1974 invention of the semaphore by Djikstra. Prior to that date, the interrupt-safe task synchronization and signaling mechanisms known to computer scientists were not efficiently scalable for use by more than two tasks. Dijkstra's scalable semaphore could be used for task synchronization (including mutual exclusion) as well as signaling.

After the introduction of commercial real-time operating systems (beginning with VRTX, ca. 1980) and the publication of a 1990 paper on priority inheritance protocols it became apparent that mutexes needed to be more than just semaphores with a binary value. Because of the possibility of unbounded priority inversion, which would break RMA assumptions, ordinary semaphores cannot be used for mutual exclusion.

Many bad sources of information add to the general confusion by introducing the alternate names binary semaphore for mutex and counting semaphore. The current wikipedia entry for semaphore is a prime example.

The correct and appropriate solution is a distinct set of RTOS primitives: one for semaphores and another for mutexes. Mutexes must prevent unbounded priority inversion. The APIs for semaphores and mutexes should be as distinct as possible, as their use is quite different.

Are all RTOSes the Same?

Having just, coincidentally, returned from teaching a two-day hands-on RTOS course in Florida, I was greeted this morning by the following message from an RTOS company president in my inbox:

Recently, I have had a statement by you thrown at me. The statement essentially said that all RTOSes are the same, or something to that effect. Obviously, you and I both know that there are differences, some large and some small. The problem is that people listen to what you say and I think they may have misunderstood you. So what were you trying to say? I'd like to know so I can rebut them with your own words when they quote you.

This reply is probably best handled publicly. Having used several commercially successful RTOSes (including both of the current top two according to Embedded Systems Design); written one of my own (ADEOS) for a book; taught OS theory as adjunct faculty at the University of Maryland; and also written and spoken of non-preemptive RTOS alternatives in several venues, I am quite opinionated on the subject. A few years back I was even interviewed about RTOSes on a PBS television show called American Business Review.

Here are a few of my past RTOS articles, which may provide additional background for this post:

• Introduction to Preemptive Multitasking (2003)
  
• How to Choose a Real-Time Operating System (2003)
  
• The Perils of Preemption (2006)
  

I believe the opening paragraph of that last article concisely sums up an opinion I've often expressed--and which may have been the basis of the remark aimed at the RTOS vendor who e-mailed.

Every commercial RTOS employs a priority-based preemptive scheduler. This despite the fact that real-time systems vary in their requirements and real-time scheduling doesn't have to be so uniform. Multitasking and meeting deadlines is certainly not a one-size-fits-all problem.

But my e-mail correspondent is correct that there are differences large and small. Here are some of the most obvious differences between the various commercial RTOSes:

1. At the API level, each RTOS is unique. Though every RTOS has functions for creating a new task, acquiring a mutex, and posting to a message queue, the specific function names and parameter lists differ "by brand". Individual programmers may find one RTOS' API more comfortable or logical than another.
   
2. To support true real-time scheduling via RMA, each RTOS must provide the following:
   
   
   
   1. A guarantee that the highest-priority task ready to use the CPU is the one actually running at all times
      
   2. A bounded worst-case context switch time
      
   3. A bounded worst-case interrupt latency
      
   4. A mechanism to automatically prevent unbounded priority inversion during mutex contention
      
   
   An RTOS that doesn't do one of these things is, obviously, different from the others--but most do. But the specifics may vary. In particular, the precise timing of those worst-case times may differ from one RTOS to the next on one processor to the next. In addition, the details of the chosen priority inversion workaround will make a difference in the RMA calculation mathematics.
   
3. Some RTOSes can use the MMU and others can't
   

Hopefully, this clarifies both that I think commercial RTOSes are somewhat commodity products and that there are, nonetheless, obvious differences.

Embedded Software and Product Recalls

Here's an example of a bug in an embedded system (a dive computer for use in Scuba) that might have killed http://www.cpsc.gov/cpscpub/prerel/prhtml06/06194.html



The design of bridges and roads and the engineers who work on them are regulated by states and the federal government. The modern equivalent of bridges and roads are often embedded computers, but there is little regulation outside of medical devices (FDA) and avionics (FAA). Sooner or later, this unregulated posture may get us into big trouble as a society.

Compiler Quality and C's volatile Keyword

At at a meeting with a client yesterday, I was reminded of a conversation we'd had about eighteen months ago at an Embedded Systems Conference. At that time the client, I'll call him John, was having a problem with C's volatile keyword on a PIC microcontroller.

John had written a few lines of C code to swap the contents of two peripheral registers, perhaps something along these lines:

uint8_t  temp;

temp = *pRegisterA;
*pRegisterA = *pRegisterB;
*pRegisterB = temp;

where pRegisterA and pRegisterB were pointers to memory-mapped I/O locations.

Since the data pointed to by pRegisterA and pRegisterB wasn't being used anywhere else in the code, the compiler's optimizer was completely ignoring the above code; outputting zero opcodes of machine code for that sequence.

As John and I discussed then, redeclaring the pointers as pointer to volatile integers (e.g., uint8_t volatile * pRegisterA = /* address */;), should have instructed the compiler to treat those lines of code as sacred and output corresponding machine instructions. Having tried this already, though, his compiler continued to output nothing.

Wow! John didn't realize it at the time but he had just found a bug in his compiler.

I advised John to try another compiler, rather than the free compiler he was using. You generally do get what you pay for with C cross compilers. And, indeed, his problem went away with the new compiler.

Although it is common for embedded programmers to believe their compiler's optimizer is broken when the flaw is really their failure to declare certain variables volatile, this was a genuine case of an internal compiler bug.

Public Course on Multithreaded Programming

This coming January, I'll travel from chilly Baltimore to sunny Miami to teach an in-depth training course about the proper use of real-time operating systems to design multithreaded firmware. The aim of the class is to clarify the safe and correct use of RTOS primitives, such as mutexes, semaphores, and mailboxes.

The two-day course, called Multithreaded Programming with uC/OS-II, will be held January 22-23, 2008 at the Weston, Florida headquarters of RTOS vendor Micrium, just east of the Everglades. Registration is open to the public, but the total number of seats is limited.

The hands-on course involves a mix of lectures and a coordinated series of programming exercises. The target hardware is an ARM9 development board from STMicro. The increasingly popular uC/OS-II real-time operating system will serve as the reference API with compiler and debug tools from IAR Systems.

Full details, including registration instructions, are available at the Micrium website: http://www.micrium.com/support/training.html

Building Reliable Systems

I'm writing this live from the Embedded Systems Conference in Boston, while participating in a birds of a feather discussion moderated by Jack Ganssle. The subject of the session is Building Reliable Systems.

The discussion here amongst perhaps 80 engineers (about 75% electrical engineers by education) initially focused on resources and schedules and the inevitability of bugs, but has now turned to what seems to a more productive thread: specific processes and tools that produce higher reliability.

One gentleman, had a great way of summarizing what needs to be done at a high level:

* Prioritization
* Process
* Metrics

In this view, Prioritization is an input to the Process. That is, prioritization of features relative to one another--as well as the accurate definition of properties such as quality and reliability. These are to be provided by the customer or from engineering management. The Process used for design and development and testing are then guided by these input parameters. Metrics are an output of such a Process, if you strive to generate them. Metrics include the schedule time it took to complete specific feature implementations, as well as bug rates.

On the specific point of process and tools, we've discovered that just a few companies represented in the room are using code coverage tools and test-driven development, though most appear to agree these might be helpful in raising reliability.

My bottom line fear: One of the products designed by someone in the room with me right now will kill or maim eventually!

Embedded Industry Survey Results

I was quoted (mostly accurately :-) over at Embedded.com, in an analysis of the results of the 2007 survey of embedded system designers. There are some interesting year-on-year trends, including an increase in the use of C vs. C++ and a decrease in the use of both commercial and open-source operating systems such as VxWorks and Linux.

Embedded.com Gets a Facelift

The popular website Embedded.com just got a long-awaited makeover. Check it out!

Medical Product Opportunity

My wife and I recently had a new baby boy (our second; thanks! =). At the hospital, I couldn't help noticing just how primitive much of the equipment in use is by modern standards. Of course, from the point of view of the hospital staff this stuff is positively state-of-the-art! But as an electronics designer, I know better.

Take for example the fetal heart monitor. The state-of-the-art features include that the doctors and nurses can simultaneously monitor up to nine fetal heartbeats in laboring women from one PC screen--from wherever they are on the floor. The hopelessly outdated reality is that the sensor itself is constantly losing the heartbeat; one nurse must actively attend each laboring woman simply to move the sensor around.


What a pain for a mom-to-be! Here she is in some of the worst discomfort/pain of her life. She's thankful there is such a system as it's reassuring that a professional is keeping tabs on the stress level of the baby insider her during all this. But then every 5-10 minutes the nurse is climbing all over her to move the sensor around until it can "lock onto" the baby's heartbeat again.

Better idea (free to a good home): how about a sensor "belt" with an array of heartbeat pickups (simple mics?) and an attached embedded computer. The computer could simply monitor all of the mics and route the strongest signal up to the printer and network.

Even better idea: make this belt contain the wireless broadcast feature--instead of attaching it to a five pound box on the IV cart.

What is your great product idea?

Embedded Systems Rule the Earth

It is estimated (for example, by WSTS) that 9 billion processors were manufactured last year. That's a 50% jump from the pre-2001 peak (i.e., prior to the Dot Com bubble crash). This is quite an impressive growth rate when considered in light of the financial market indices; DJIA and S&P500 have only recently come back to their pre-2001 peaks while Nasdaq remains solidly underwater.

Part of the reason for this is that there is growth in sales of the most visible computers. PCs, Macs, and other general-purpose computers sell in the 100-200 million unit per year range. And these generally each contain several DSPs and microcontrollers in the peripherals attached to the main microprocessor. Being generous then, perhaps 10% of the world's annual processor production can be explained by general-purpose computer sales.

However, the bulk of the growth is happening in the field of invisible computers; that is, processors embedded within other kinds of products--from cell phones to appliances to medical devices. Every year, as the price of basic computing power goes down, new applications emerge at the low end--in products that didn't historically have processors inside them. For example, Freescale has noticed a recent trend like this in mechanical systems and launched several marketing efforts (see, Mechatropolis.com and their own microsite) to reach mechanical engineers needing processors for the first time. Texas Instruments is focusing its microcontroller marketing efforts on a similar trend in medical devices.



Although an article on the "invisble processors" phenomenon at ExtremeTech is several years out of date, I still value its analogy of embedded systems as insects. In biology, it is well known that insects far outnumber birds, fish, mammals, and all the other "visible" life on Earth. It seems that in the world of computers, embedded systems are the invisible class that outnumbers all others. And like insects, embedded systems come in millions of shapes and sizes.

Coming Software Crisis?

Next Tuesday (6/19) I'll be participating in an online panel discussion entitled: "Solving the Embedded Code Crisis" with Jack Ganssle, Dan Saks, and National Instruments' Joel Sumner.

Here's the blurb advertising this:

There's an ever-increasing need for more code, which is greatly hindered by the fact that the numbers of developers remains relatively flat. This phenomenon is nearing crisis level. What do we do about it? Potential solutions include everything from off-shoring to [productivity-enhancing] tools to developer discipline. This live Webinar will feature renowned experts in the embedded systems industry, as well as real-world case studies and solutions from leading organizations.

After the panel discussion--which is expected to last about 45 minutes--there will be a live question-and-answer period.

All of this will be hosted at TechOnline.com. If you might be able to attend online, I recommend that you register in advance.

MicroTCA Summit

During a brief visit earlier today, I was quite impressed by the size of the MicroTCA Summit here in Baltimore. I had expected a much smaller gathering. It seems there is quite a lot of interest in this new backplane technology, which radically reduces the space required to connect boards together. It appears MicroTCA is really set to "take off" in the mil/aero community, in particular.

What are you doing with MicroTCA?

To India and Beyond

Much has been said and written in recent years about the offshorting trend for embedded software development. One of the destinations for some work has apparently been India, which has the triple advantage of world class educational institutions (e.g., the IIT system), an ultra-low cost of living, and English-speaking engineers.

Later this month I'm headed to Bangalore and Mysore in Southern India. I may or may not blog from there, but I will certainly be taking note of the embedded software companies and opportunities there. So please know that though my blog is silent I am indeed still active.

SecureRF == SnakeOil?

I'm pleased to report that after years of reading Bruce Schneier, I am finally catching on. A few weeks back at the Embedded Systems Conference I attended a lunch for startup companies and venture capitalists. One of the presenting companies, SecureRF, claimed to have invented a security protocol for resource-constrained systems (such as RFID devices) that was (a) cheap mathematically and (b) even more secure than existing security techniques.

I was skeptical. I even thought of blogging here or e-mailing Bruce, but just never found the time. Encryption is mathematically hard for reasons I had learned in grad school. The company's hand-waving about a new faster way of doing it was either bullsh*t or a major cryptology breakthrough we should all be reading about elsewhere.

It seems that Schneier has now caught wind of SecureRF. Here's his blog post about the company, which labels the "breakthrough" nothing more than snake oil.

The End for Embedded Linux?

Last week at the Embedded Systems Conference in Boston, I moderated a panel discussion premised on the recent downward trending slope of Linux use in such systems. The panelists were Dr. Inder Singh (CEO, LynuxWorks), consultant Bill Gatliff, and John Carbone (VP of Marketing, Express Logic).

The graph to the left shows the operating systems use data. The source of this data is an annual (except 2003) subscriber survey by Embedded Systems Design (nee Embedded Systems Programming) magazine. To create this graph, I aggregated individual Linux distribution numbers, as well as combining data for pSOS and VxWorks under ISI acquirer Wind River Systems and Nucleus and VRTX under Accelerated acquirer Mentor. Similarly, all variants of DOS and Windows are lumped into Microsoft.

The question for the panel discussion revolved around the future trend: Will Linux's share growth return or has it peaked? Whatever the answer, Linux is clearly very popular with embedded software developers. And other surveys support this finding.

An interesting subplot concerns Wind River Systems (Nasdaq:WIND). When Wind acquired competitor Integrated Systems (ISI), the combined market share of ISI's pSOS and Wind's VxWorks products (according to the data cited above) was more than 30%. Today the combined share for the same two products has fallen to about 10%. Over the same era the company's stock price has fallen from a high of $60 to about $10. I see little reason to be optimistic about the company's future and noted that they were not even present at the aforementioned industry gathering.

Is VxWorks dead? Is the company's recurring market share around 10% simply due to past users at large companies continuing to use the product? How much has Linux contributed to the early demise of a previous market share leader? What do you think about the future of either operating system?

Embedded Systems Bloggers

There are currently just a handful of bloggers writing about embedded systems development. So I thought I'd take a minute to introduce the others I've run across.

There are, of course, three of us blogging at EmbeddedGurus.net. The others are: Nigel Jones who writes Stack Overflow and Dr. Miro Samek who authors State Space. Each of these guys has unique expertise that is very different from my own.

Another embedded developer who blogs is Harvey Sugar; he calls his blog Nerd1951. I'm hoping to run into Harvey at the Embedded Systems Conference next week in Boston--an event from which I'll be blogging quite a bit.

On a related but different note, industry analyst Christopher Lanfear calls his blog about the tool vendors and their funding On Target.

Please let me know if you've found other related blogs (or you write one).

Perils of Preemption

Embedded.com just picked up a paper I wrote for the upcoming Embedded Systems Conference in Boston. The paper is about the downsides of the dominant RTOS (real-time operating system) scheduling algorithm. It turns out that priority-based preemptive scheduling has one key benefit but more than ten important caveats.

Unfortunately, the formatting and editing was screwed up in several ways in Embedded.com's publication of this paper. But I have republished it at http://www.netrino.com/Embedded-Systems/How-To/Preemption-Perils.

I'll be speaking about alternatives to priority-based preemptive RTOSes in Boston on Tuesday, September 26.

Firmware Ethics

We the members of the embedded software development community, in order to protect our customers (and their customers) from the risks inherent in our products, extend product lifetimes, lower long-term maintenance costs, promote the general welfare of society, and advance the integrity and reputation of our profession, do ordain and establish the following code of ethical conduct.

Check return codes. If a function can return an error code, it will almost certainly return one eventually. How will our software recover if it doesn’t even know an error has occurred? We will, therefore, always check all return codes. When an error is detected, we will take whatever action is required to recover safely.

Document everything. We make dozens of design and implementation decisions each week; we forget the reasoning behind others at a similar pace. We will, therefore, document all of our reasoning in the code itself. Furthermore, we will update these comments if our reasoning changes.

Use version control. Even when done carefully and with discipline, making backup copies of source directories, commenting out code that might be needed again, and tracking changes in comments is insufficient. The use of a good configuration management tool is a must. Furthermore, we agree to always perform a diff and provide a good description of the explanation for all changes each time a file is checked in.

Enforce coding standards. No matter how proud we may be of our abilities, we do not “own” the code we produce. Our employers and clients own it; we and others are hired to develop and maintain it. The true owners have the right to enforce a coding standard upon our work. If they do not choose to exercise this right, we will self-enforce a standard in line with generally accepted programming practices.

Run lint regularly. Compilers are more concerned with generating code than the correctness of your syntax. Only a tool like lint can watch over our shoulders to ensure we’re not straying into dangerous or non-portable programming constructs. We will, therefore, include lint in our development process. If we don’t remove all code with warnings, we will at least understand why each and every warning remains.

Perform code inspections. Automated tools are helpful, but nothing is better than a peer review for finding logic errors. Code inspections have been shown to be many times more efficient than any other single debugging tool. We will always review our code as a group at least once before each release.

Assign one tester to every developer. Programmers make lousy testers, especially when it’s their own code they’re testing. For best results, one independent tester should be hired for every developer. And the testing team must maintain and rerun regression tests regularly, to certify that a new release is at least as good as the previous one.

Measure performance. Of course, there are some kinds of tests that are best performed by the engineers themselves. In real-time systems, we shall directly measure interrupt latency and the time it takes each of our critical subroutines to complete. An oscilloscope and a spare I/O pin are quite handy in this regard.

These best practices are easy things to do, and all are inexpensive to implement. A 1 or 2 person design team can follow these rules as easily as a 50 person engineering department. If you aren’t doing these things already, there are no excuses.

Verticality

More than 6 billion processors of all types (4-bit to 64-bit plus DSPs) were sold in 2002. That astonishing number was actually off about 25% from the record volume of over 8 billion recorded two years earlier. Only about 100 million of those chips (just 1.5%) became the brains of PCs, Macs, and UNIX workstations born each year; embedded systems designers are the cause of the rest.

With about one new processor taking hold per person per year, it would be fair to say embedded systems are everywhere—or soon will be. The technology is only three decades old at this point—imagine the ultimate potential! Applications span the realm of imagination. Think of the number and diversity of insects and you’re on the right track.

There are so many different applications, and thus so many different types of embedded processors, in fact, that it’s becoming increasingly valuable in some circles to break that huge market up into more easily digested chunks. Applications can be grouped roughly into about seven key categories: communications, computer peripherals, industrial controls, military/aerospace, consumer, medical, and automotive, plus the obligatory catchall other/miscellaneous. These are termed vertical markets.

We should give some thought to what it is that ties all of the many vertical markets together. There is significant overlap between the work of designers of embedded systems for applications as seemingly diverse as consumer gadgets and military/aerospace systems. I suspect it is in this overlap that you find your identity as an embedded system designer.