Real-Time Java is Dead

A few months less than ten years ago, I presented a paper at the Embedded Systems Conference (ESC) for the first time. My 1.5 hour course was entitled "Embedded Java" or something similar. This was in Silicon Valley, and the audience was standing room only--despite a rather large room to start with.

I've tracked technical and business developments in the world of embedded and real-time Java for even longer--and written a number of articles on the subject. And so I didn't miss that over the years the audiences for courses on either variant of Java dwindled. After a hopeful year or two too long, I gave up on Java in our space and stopped proposing the topic at ESC.

Then, last Spring, I had the refreshing experience of teaching a two-day hands-on real-time Java programming class in the Netherlands. The room was packed. There was enthusiasm. And these guys were really using Java to develop software (though it wasn't truly embedded code). So I thought I'd try again at ESC and reproposed the topic.

Last week I taught a course called "Real-Time Java Programming" at the ESC Silicon Valley venue. A lot had changed about the audience size. This time, in a room of a similar size, there were many empty chairs and tables. I think there were perhaps 15 people attending this time.

By my reckoning, Java is officially dead in the embedded systems community--especially in the U.S.

Educating Embedded Engineers

One of the biggest problems affecting the embedded systems industry is a shortage of properly skilled firmware developers. I spoke of this in a recent video interview.

Embedded software developers need some skills taught to Electrical Engineers and others taught to Computer Scientists. They also need practical hands-on experience. Unfortunately, even the best of the Computer Engineering programs at Universities fail to deliver the right mix for embedded work.

The demand for such skilled help is growing rapidly--with the latest estimates put at over 4 billion embedded products manufactured per year.

Fellow Embedded Guru Mike Anderson wrote a nice piece about all this for an IEEE-USA publication called "Today's Engineer." I happened across Mike's article in print, but you can find it online by the title "Help Wanted: Embedded Engineers."

VDC Counts 4 Billion Embedded Products Shipped in 2006

Read Venture Development Corporations OnTarget Blog for more details. But in a nutshell, in the last year for which data are available, VDC estimates 4 billion total units of embedded products, most without a commercial operating system inside. By other estimates, total CPU shipments in 2006 were around 9 billion, but some of those are PCs/Macs and there are also numerous systems containing two or more CPUs.

Embedded Systems History

The editors of Embedded Systems Design recently put together an interesting historical timeline of the embedded systems industry. To learn some history, take a stroll over to http://www.embedded.com/timeline/.

Kill the Patent Office

I'm not sure what to make of all the criticism of and suggested improvement to the U.S. patent system. However, I found this article thought-provoking:

http://www.hosteny.com/archive/Hosteny%2005-05.pdf

Embedded C Quiz Results

When we redesigned the Netrino.com website late last year, we thought it'd be fun to challenge our more than 20,000 monthly visitors (mostly embedded software engineers) to a skills test. So we developed a ten question multiple-choice quiz (http://www.netrino.com/Embedded-Systems/Embedded-C-Quiz). And it has been a popular feature of the new site, with a couple hundred participants just in the first two months.

And now the results are starting to come in. We analyzed the early results a couple of ways and discovered something worth talking about: Quiz takers from India did about the same as quiz takers in the U.S. But the rest of the world lagged behind these two groups quite a bit.

There are ten questions in our quiz, and we consider a passing score to be 8 out of 10. A handful of quiz takers have scored 100%, but most score in the 30-90% range, with an overall average at 60.4%. (A little scary, huh?)

Statisically speaking, there were three significant groups of quiz takers by geography. The average score of those taking the quiz from the United States was just shy of 64%. The average for India was not far behind at about 61.2%. However, the rest of the world scored an average of just 55.9%.

What does this say about the state of the profession of embedded software development? Offshoring? The quiz itself?

Breathalizer Source Code to Get a Day in Court

Here's an interesting news story at the intersection of embedded systems and due process:

http://www.news.com/Police-Blotter-Intoxilyzer-code-must-be-disclosed/2100-1030_3-6227951.html

How many potential bugs might a knowledgable expert witness spot in your code? Are the comments in your source accurate and clean enough for a judge or jury to read?

Twenty Years of Embedded Systems Design

It's hard to believe, but industry magazine Embedded Systems Design, which I edited for five years from 1999 to 2004, is this year celebrating its twentieth continuous year in print.

The popular magazine, which was titled Embedded Systems Programming for about the first 16 years, started its run in 1988, at a time when the widespread use of assembly language was challenged by some engineers as inefficent compared with coding in raw hex. There were then dozens of dialects of the C language, which few embedded developers used, but a committee at ANSI was actively working on the standard that would ultimately win.

The passing of this event brings two interesting thoughts to my mind. One about the past, the other about the future.

Looking backward, I wonder just how much the embedded developer community owes to Ted Bahr and Tyler Sperry (the founding publisher and editor-in-chief, respectively) for giving us the name "Embedded Systems". Although the concept of embedding microcontrollers into products dates back 17 years earlier (to the introduction of the 4004 single-chip micro by Intel in 1971), the name embedded systems was made popular by the success of the magazine and associated Embedded Systems Conference.

Looking forward, though, I wonder just how much longer the print magazine is for the world. Tremendous changes have rocked the universe of print media--even just in the four years since I left the editor-in-chief role at Embedded Systems Programming. Both the number of subscribers and the number of pages in the print magazine peaked during my tenure, at about 70,000 and over 120, respectively. Today, the circulation is down considerably and the typical number of pages is around a third of that peak. Online advertising has changed the game for most technology companies, who no longer find the unmeasurable and expensive investment in print ads their only option.

What do you think?

Verticality

More than 6 billion processors of all types (4-bit to 64-bit plus DSPs) were sold in 2002. That astonishing number was actually off about 25% from the record volume of over 8 billion recorded two years earlier. Only about 100 million of those chips (just 1.5%) became the brains of PCs, Macs, and UNIX workstations born each year; embedded systems designers are the cause of the rest.

With about one new processor taking hold per person per year, it would be fair to say embedded systems are everywhere—or soon will be. The technology is only three decades old at this point—imagine the ultimate potential! Applications span the realm of imagination. Think of the number and diversity of insects and you’re on the right track.

There are so many different applications, and thus so many different types of embedded processors, in fact, that it’s becoming increasingly valuable in some circles to break that huge market up into more easily digested chunks. Applications can be grouped roughly into about seven key categories: communications, computer peripherals, industrial controls, military/aerospace, consumer, medical, and automotive, plus the obligatory catchall other/miscellaneous. These are termed vertical markets.

We should give some thought to what it is that ties all of the many vertical markets together. There is significant overlap between the work of designers of embedded systems for applications as seemingly diverse as consumer gadgets and military/aerospace systems. I suspect it is in this overlap that you find your identity as an embedded system designer.

Firmer-ware?

Too much of the terminology embedded systems engineers use in their everyday oral communications and written documentation is only vaguely defined—at best. For example, terms like mutex, binary semaphore, and semaphore are often interchanged by software developers. In a related context, task, thread, and process are also tossed around as if they all represented the very same construct.

Hardware designers too are terminologically challenged. For example, does the new board need a 1 kb, 1 kB, or 1 KB FIFO? Will it have flash or Flash, or is that flash really an EEPROM? There are also terms like emulator, which are overloaded with multiple meanings and must be expanded to be fully understood.

About a year ago, Jack Ganssle and I teamed up to finally do something about all this linguistic nonsense. We have since combined our years of experience as embedded hardware and software developers and precisely defined more than 2,800 terms used in the field of embedded systems design, development, and testing. The results, called the Embedded Systems Dictionary, should be available from CMP Books about the time you read this.

But this is not meant to be a sales pitch. I bring all this up only as background. You see, at a joint “book launch” attended by Jack and me at the CMP Books booth at the Embedded Systems Conference San Francisco, we had a very interesting conversation with a hardware designer who considered the results of his Verilog coding to be firmware. Neither Jack nor I had encountered such a usage of the term before.

In fact, Jack and I have both often seen the use of the term firmware somewhat restricted to either specifically DSP code or embedded software written entirely in assembly. When compiling the dictionary we agreed, however, that the definition ought properly to include code written in any programming language:

firmware - n. Executable software that is stored within ROM. (Usage Note: This term is interchangeable with “embedded software” and sometimes used even when the executable is not stored in ROM.)

But this fellow brought up a very interesting point. At what point does hardware written in Verilog (or VHDL) and compiled into an executable binary format become indistinguishable from software written in C (or any other language) and compiled into an executable binary format. Is the hardware executable “firmer”-ware than the software executable? Or are they both just different flavors of firmware?

What will happen when, ultimately, a special-purpose C compiler can generate hardware? Or when a UML tool can automatically (and optimally) partition the description of a system’s requirements into hardware and software and output a pair of binaries for the processor and the programmable logic in a single-chip platform FPGA?

At that point, will the hardware designers and software developers even be able to distinguish themselves from each other? As the line between hardware and software continues to blur, perhaps it is only the hours we keep and the forms of caffeine we favor that will belie our EE or CS backgrounds. That’s when things will get really interesting.

Distributed Development

Though the trend toward overseas development has been brewing for more than a decade, I've just lately been noticing a number of IT-sector layoff announcements in the U.S. featuring near-simultaneous announcements of increases in overseas outsourcing by the same companies. It’s not entirely clear if there's an active migration of engineering jobs from the U.S. to overseas, but there’s certainly a decent case to be made that something like that is happening.

According to the Bureau of Labor Statistics, over 120,000 electrical engineers and computer scientists were unemployed at the end of 2002. That represents almost a three-fold increase in just the past two years, and a near record unemployment level. Yet even as skilled engineers remained in good supply, companies such as Microsoft, Sun, and HP recently announced major expansions of their overseas development operations.

To be honest, I am not sure what to make of this. I favor free markets and believe in the equality of all people in all nations. I traveled to India in 2001 and was impressed by the entrepreneurial spirit the new engineering jobs have generated there. I'm also pleased that engineers there and in many other parts of the world have increasing job prospects and standards of living.

You may be thinking that outsourcing is obviously a negative trend and that “the American engineer” will suffer. If you’re unemployed right now and are personally affected, hang in there. You’ll almost certainly disagree with what I have to say next, but I’ll say it anyway.

The very technologies we’ve been developing and improving for the past few decades are key enablers of distributed development. As the world becomes more interconnected, it becomes increasingly reasonable to bring together a group of geographically-diverse individuals with the collective skill set needed to get the job done. If some of these minds are on the other side of the world, so be it. If they’ve got the same skills as someone here but will work for a lot less, we’ll lose that job.

But in the long run we’ll win too. Increasing standards of living for workers in other parts of the world do more than just take jobs from better developed countries. Those workers spend the money they make in a variety of ways and that expands markets. Things also get cheaper here as a result of their labors. The ensuing economic growth creates more opportunities and jobs here too. Unfortunately, the process doesn’t happen as quickly or seamlessly as anyone likes—and some individuals do get caught in the crossfire.

Fortunately, U.S. engineers continue to be among the best in the world. Those who continue to improve their skills will always be in high demand. They’ll also be well poised when the global economy eventually does turn up again, which I’m confident it will.

Moving Targets

There are currently so many interesting operating systems and alternatives that it’s hard to choose—as we must for each project—just one. Within the priority-based preemptive category, you can choose based on worst-case latency, source code availability, upfront and/or recurring cost, memory usage, API/features, and numerous other criteria. Indeed, the realm of possible price/feature combinations has fragmented the market into many tiny niches.

Though there are a handful of well-known names that have the bulk of the market tied up, a sizeable number of smaller RTOS providers do quite a nice business on just a tiny fraction of total market share. And as the demand for embedded operating systems continues to accelerate, these smaller vendors need not even hold their market share numbers to continue to increase profits. That’s good—because they will continue to lose market share.

There are a lot of forces that will shape the RTOS marketplace going forward—as it goes even more toward the big guys. Not the least of these factors is that more of us will go off-the-shelf. Among subscribers to Embedded Systems Programming magazine, for example, the percentage using no OS or a proprietary alternative has fallen from 38% to about 18% in just the past five years. Extrapolating, perhaps we’ll all be using off-the-shelf OS code by 2007.

Competition from “desktop-lite” operating systems has also picked up. There are a large number of embedded designs that look (or can look) an awful lot like a PC inside, benefit from the low component costs in that market, and no more than dabble in the realm of real-time. What used to be a small ROM-DOS market has morphed into today’s WinCE/XP and Linux market—almost entirely in the last three years. In 2002, some 17% of you fell into that category; I suspect it’s not a coincidence that x86 architectures continued to dominate the list of 32-bit processor choices.

And then there’s consolidation. Though the pace of consolidation has slowed with the business cycle, the effects continue to be felt. Mostly it was the vendors of 32-bit solutions that picked up 8- and 16-bit competitors and debugging tools when times were good—so they could offer one-stop shopping. An up-and-coming Linux player even spent some of its paper wealth to acquire a commercial RTOS vendor for that same purpose. To compete, a large commercial vendor picked up an open source, though non-Linux, OS. When the buying resumes, as it surely will, where will it ultimately end?

Several of the technologies positioned to profit from these trends are not what we traditionally think of as “embedded.” Microsoft, which—love ‘em or hate ‘em—correctly understands they must make it in the embedded space to stay relevant in the coming decade, is trying hard to find the right combination of OS features and vertical markets. In many of these markets, they’re competing directly against the open source alternatives—and apparently losing in some. According to a recent article in EE Times, Linux is also beating out traditional RTOSes in key markets like consumer electronics.

Of all the traditional vendors, Wind River is probably in the best position to compete with these market forces and survive. We are fortunate in the embedded space to have lots of choice when it comes to operating systems. But the future may hold far less technological alternatives. It’s not clear to me that QNX, VxWorks, Nucleus, or any other RTOS is really distinguishable from another in the boardroom—or that the smaller players have enough to gain by staying in the business longer term. What do you think?

Let’s Go Wireless

Being an electrical engineer with a home office who’s moved five times in a decade, I’ve gotten quite adept at setting up and maintaining a small network of computers, telephones, printers, and other office equipment on my own. In fact, I’d say I rather enjoy doing these things. I typically set aside one day at the beginning of each move—before bringing in any boxes or furniture—to configure the phone jacks for separate home and work lines, fax, and Internet access, and to run CAT5 cables and install jacks in each room for Ethernet.

I used to have to buy a few new cables, connectors, or tools each time I moved, but in recent years I’ve generally found that everything I need I already own. And as my resources and experience in this area have grown, I’ve even developed a filing system of sorts for the necessary equipment and cables. In my system there are precisely four categories of cables: power, audio/video, telecom/datacom, and computer. Related equipment such as power strips, antennae, network hubs, and spare hard drives are kept with the cables of their type.

I realized only during my most recent move that a significant change is in the air—literally. It turns out that the time required to connect all this stuff together is going down not up. And that’s not simply because of my increasing experience. The fact is that wireless connections are becoming a mainstream reality—particularly in the communications area. And this is making my pile of telecom/datacom cables and cabling equipment increasingly irrelevant.

In our new home we have 900 MHz and 2.4 GHz wireless telephones as well as a wireless 802.11 LAN. Our early adoption of DSL a few years back eliminated a dedicated incoming phone line, which used to be dedicated to low-speed dial-up (at about the same monthly cost overall). The wireless phones and 802.11 LAN make the physical location of the remaining two incoming phone lines irrelevant. So long as I go wireless, the DSL modem, router/firewall, and hub need not even be located near the computers in the office.

In fact, the whole concept of a home office is changing. That term used to mean that I was a slave to the phone and computer on my desk—just like most office workers. As I write this, however, I’m out on our new patio seated comfortably with my laptop, cordless phone by my side. I can access files on the local server, the Internet, and even over on the publisher’s side of a VPN. All of this took just minutes to set up, instead of the usual hours.

But now that I’ve seen the future, I’m disappointed that there are still so many wires left in other areas of my home and office. Ultimately, I’d like to see my collection of coaxial and RCA cables made obsolete along with those for serial, parallel, and USB devices. I won’t be happy until my Palm and server can sync simply by being in the same room, I can debug embedded software remotely from the patio as well, and my TV can show movies from a DVD player (or hard drive) anywhere.

As a consumer, I don’t care how any of this is accomplished. And though I want it to be secure, I don’t care how that is accomplished either. Whether it’s ultimately 802.11, Bluetooth, the proposed 802.15 hybrid of the two, or some other technology doesn’t much matter from the consumer perspective. As Nike says, we need to just do it.

Get Rich Slow

By a show of hands, how many of you jumped ship from a stable engineering company to a startup in the late 1990s? I bet if you didn’t, you at least thought about it or had a few offers. I never jumped ship myself, but was straddling two boats at once trying to make a quick extra million on a tight budget of night and weekend hours.

I guess I always knew the air would come out of the bubble at some point. And I sure as heck knew it wouldn’t be good to be on top if that happened. So my partners and I kept our day jobs and focused on long term issues in our business planning. How would we profit from our ideas, in ways other than a quick sale of the company or an IPO? We figured we’d identified a product and a market for it, so needed only develop the code and keep expenses lower than revenues while we tried to increase sales.

Still, though, we crossed our fingers and hoped as much as the next guy that we would time our business just right and make a bundle somehow. We certainly weren’t going to turn down a multi-million dollar purchase offer—and even felt confident enough it was worth that to turn down a bona fide small private funding source and free help from an experienced CEO that would’ve valued the company far less initially.

In the end, unfortunately, the bursting of the dot com bubble took not only the really bad ideas but also many good ones (including ours?) down with it. By the time we had filed our patent application, developed our prototype, and written our business plan, all of the funding sources for search engine enhancements had dried up. I suspect several of the venture capital firms and search engine companies we talked to would’ve jumped at the chance to be involved with our idea just a few months earlier. But the game was up. And two years later—long after we wrote off our personal investments in the company—the venture capital we needed to quit our jobs and work toward profitability still isn’t available. So I have a new plan.

My new plan is to get rich slow. To play the part of the tortoise rather than the hare. Engineering is a good stable profession, and one that generally pays well—especially if you have a specialty as in demand as real-time embedded systems design. It’s really not a bad life, if you can get it.

So rather than try to outwit or outplay, I’ll just try to outlast—and I’ll save every penny I can along the way. Besides, it’s quite a lot easier to stick to your core values and make the world a better place little by little when you’re not busy making an end run. To see how extremely disjoint the two paths can become, witness any of the recent corporate scandals.

Honesty, integrity, and responsibility should be the core values of all practicing engineers. And we should practice them outside of work as well. As fun as both are, there’s more to life than engineering and money. So I’m stopping to smell the roses now more too. It sure is nice to have my nights and weekends back! And that’s worth a lot more than a million dollars to me!

Clash of Titans

In the March 2002 edition of Embedded Systems Programming magazine, Jim Turley predicted “The Death of Hardware Engineering.” For evidence, he pointed to the fact that hardware design, especially chip design, has become almost exclusively the domain of programmers using Verilog and VHDL. An emerging trend toward the use of more popular software development languages—such as C, C++, or Java—to perform “system design” (with the compiler automatically deciding what is best done in hardware and software) could indeed put an end to much of hardware design as we know it.

However, I’m not convinced the trend is as one-sided as Jim says. At the same time that hardware designers are moving toward writing more code, an increasing amount of software designers are moving toward more graphical forms of design. Automatic state machine generators and similar tools have started us down this path. A technology called Executable and Translatable UML offers capabilities that could ultimately make this a broader industry trend.

Of course, the increasing overlap between hardware and software does lead to some present confusion, and much uncertainty about the future. Hardware itself is becoming increasingly “soft.” Custom hardware on a board has already given way to custom hardware on a chip. And we’re now seeing a direct change toward integrated chips with a fixed processor surrounded by a flexible array of programmable logic. That’s the perfect target platform for a “system design language” to dominate.

Traditional hardware and software tool vendors are eyeing each other nervously across this narrowing digital divide. If you only design a system, instead of a system consisting of separate hardware and software designs, where will you go for your tools? To Wind River or Mentor Graphics? Though not traditionally direct competitors, companies like these are concerned they may increasingly vie with one another in the near future.

The named companies, both leaders in their respective domains, are beginning to position themselves accordingly. Wind River has been working with Xilinx, through a partnership. Apparently, their goal is to create a version of the Tornado development tool suite that includes hardware design and synthesis capabilities for Xilinx's programmable logic surrounding an embedded processor-plus-VxWorks software environment. They’ve already delivered a set of tools for working with the current generation of Xilinx FPGAs.

Meanwhile, in another market, Mentor Graphics is laying a path for its current codesign/coverification customers to follow toward more involvement in software development. Hence, in my opinion, their recent acquisition of Accelerated Technology. The threat to Wind River is implicit in that acquisition. Though Mentor already had an RTOS product of its own (VRTX), they no longer had the software development perspective or inhouse experience they will need to compete in the not-so-distant future. The former Accelerated, led by its former president Neil Henderson, is now Mentor’s Embedded Systems Division.

Working separately, from their own markets, but both following this convergence trend, these companies (and others) will inevitably collide as the hardware and software do. That’s when things will get really interesting in the tools market. But I doubt it will truly be the end of hardware engineering.