RoweBits

Real-time Embedded Linux and POSIX RTOSs For Microcontrollers (MCUs)

Sunday, March 24, 2013

Japan Is Not China

When I was small I got a plastic toy car in my serial. It was the first thing that I ever saw that was marked "made in Japan". I asked my aunt about this and she said "Japan makes cheap junk" in typical 50's style. I played with that free car and noticed it was not so badly made. It was certainly the best one that I'd ever gotten in a box of cereal. A few weeks later someone stepped on it and broke it and someone said "see - cheap junk." A decade later a strange car appeared at the local garage. It had gold trim not chrome and very distinct styling. I'd never heard of the manufacturer - Toyota. The mechanic laughed at us we we suggested it was not such a great car. It had 200K miles on it and needed a minor repair. This was in the days when the "big three" cars were scrapped at 100K miles or less. A few days ago a family member wanted a very loud non electronic bike bell. I did find one but not in my city. The best one I could find that everyone raved about was from - you guessed it - Japan. As far as my experience goes, Japan should be known world wide for exceptional quality. My experience with anything from China is far from the same. As a young engineer, I looked at a piece of electronics stamped made in China and half the components were missing. They simply left off everything that they could to create a minimally functional device - quality was not an issue. As I have been deluged with products over the past fifteen years marked "made in China" my experience has been similar to my first experience. Anything that can make the device cheaper is a "good idea". LED lights that are unserviceable. Faucets that need replacement before installation, electronics that are unusable with a manual that is unreadable. This is "made in China". I ask you - why do we buy it? Made in Japan, made in Germany, made in Sweden, and many other countries provide great quality and a good customer experience. When will local manufacturers (the few that are left) realize that there is a great local market for their products as long as they deliver quality. As for China, to protect consumers and to protect the environment, minimum quality standards should be mandatory for all imported durable goods.

Wednesday, September 26, 2012

Free - Software Based Trojan Horse


It is always interesting to me how large companies offer "free" products to lock in naive customers. Somehow they are oblivious to one of the most famous tricks of all time.

In the past couple of years both Freescale and now TI have launched "free" operating system support in the MCU space. The Freescale offering called MQX provides a range of features that is quite broad and solves many problems. It is only available on Freescale MCU products. It has been used on many projects.

Now, TI has just announced that its proprietary OS which is years old is available for its MCU products. It offers similar features to that of MQX. Over the years, many have used their OS and there is clearly demand for this. Again it is a vendor specific product.

Both offerings are completely proprietary, locking the customer into the vendor's MCU offerings. As a customer you should ask yourself "Why is this semi-conductor company investing millions of dollars in proprietary OS development and giving this to my company?" The reason should be obvious - they are making money doing it. It gives them account control if nothing else.

When an OEM builds a product, they will stick with the same basic hardware for many years. If the OEM product purchase for their MCU based devices is 100K units per year, then over the life of the product line, there will be many hundreds of thousands of units purchased. When the OEM is in a single supplier position, they have no negotiating position and the margins are much better. Also, if development is started with the vendor OS, and the OS is proprietary, the switching costs are high, and other options are eliminated.

How can OEMs avoid this obvious cash grab? Well, the first thing is to educate their engineering managers on the basics of software engineering. Open system solutions for operating systems have been the norm in all but very deeply embedded systems for at least 20 years. By choosing an open system operating system, any lock in to a specific set of hardware and software is eliminated and the purchasing department has maximum leverage in all negotiations. This can easily mean hundreds of thousands of dollars in cost reduction over the life of a product line.

Does this really save money? Yes, it does if the operating system product pricing reflects the needs of the OEM on a single product. If it costs $8000-$10000 to purchase software for an MCU product development which will utilize three programmers and take four months to develop, then this cost must be considered as part of the overall product development budget and amortized over the total number of units sold. Note that total development costs should be used in this calculation which means using fully loaded labor rates for development cost calculations.

Generally, the software costs will outstrip the electronics hardware development costs in an 80/20 or 90/10 ratio. If the operating system software is correctly priced, on a single product development, the operating system cost will be more in the range of 5%-10% of the overall software development cost and less than half of the core electronics development cost. The operating system software purchase price will be easily recouped in decreased hardware costs, elimination of training, enhanced software reuse, reduced time to market and increased flexibility independent of unit sales volume.

The operating system component cost will be recouped just in reduced hardware costs if product volume exceeds a few thousand units. This is the real measure of whether the Trojan horse is a trick or not. Intangible benefits are not measured here so the total savings of an open operating system are much greater but you should be able to see benefits in the range of $1-$2 today in terms of pricing if you can go to any vendor to purchase your MCU if you have a low volume solution. The cost savings are much more obvious with volume MCU purchases although the absolute savings per MCU is much smaller.

The market share gained through more rapid market access totally dwarfs the hardware savings costs. What this means is use an off the shelf, completely integrated operating system as the overall savings will be huge. Market share is correlated with time to market and the savings that this brings diminishes the cost of the operating system to zero in the scope of things.

Other factors are important too. By using open standards like POSIX, knowledge and software reuse is maximized. This approach leads to reusable applications, elimination of training, faster development, lower cost development, lower time to market, larger market share and total cost of ownership minimization.

With open system solutions, platform based approaches and lean product development is possible. Now the software platform is POSIX based which supports the use of any of the various embedded operating system platforms for larger systems (embedded Linux, QNX, VxWorks, ...) and also supports POSIX based MCU solutions including DSPnano and Unison from RoweBots. Most larger and sophisticated companies understand this and use this approach today.

The real message here is "Don't get a free Trojan horse." The overall costs of locking in to a closed solution stopped making sense over twenty years ago and certainly make no sense today. By keeping your long term interests in mind, you can maximize your company's flexibility and profits.

Friday, August 12, 2011

MCU and MPU Survey Results - Few Surprises

I recently reviewed the 2011 survey on embedded systems sponsored by EETimes, EmbeddedSystems and others and was surprised that there were no real surprises. Well, not no surprises, just very few.

Of course, I am very interested in MCUs and how they are changing in the market. It seemed to indicate that the processors were getting the peripherals right or better anyway and hardware designers liked this. It is no surprised that ARM is doing better and better; however the weak showing by Renesas, particularly after the NEC merger was a bit of a surprise. It is clearly a North American dominated survey.

There was a couple of other surprises. The rise of embedded Linux is no surprise - I expect this trend to continue and I also think that this has directly lead to the decline of other larger embedded operating systems like VxWorks, QNX Nucleus and Integrity.

The continued popularity of TI's OMAP is significant. It is clear that they have the correct mix of ARM and DSP functionality for a broad set of applications combined with the right tools and pricing. I expect that the Sitara and Integra lines will do well too - levering off this success with more specialized versions.

The apparent growth of freeRTOS - which isn't an RTOS but is just a kernel, combined with the growth of roll your own RTOS was to be expected. The reason for this is the very high pricing of the two best known MCU RTOS brands coupled with the lack of understanding of the true costs of integration, debug and test. The survey put test and debug numbers at 25% of the overall effort. It is likely closer to 50% and if managers computed the real lost market share for this period of delay, it is very likely that this would change.

Another related set of comments were made that the RTOS market was moving towards domination by the semiconductor vendors providing their own RTOSs for MCUs. Some even thing that they will move into hardware. It seems that the volume for most applications does not justify the loss of flexibility so hardware implementation will be limited to specialized applications.

Freescale has include the MQX RTOS free with its new MCUs. It is provided as a software component and it is likely that others may follow this trend. The real story here though is that many offerings of "free" this or that part of the total solution have been made: a free OS (and I mean a real OS) with an IDE sold by seat is one example. In any case, the offerings are always tied to something: a silicon vendor's hardware, an IDE lock-in and cost per developer, or extensive user development which masks the real cost. There is no free lunch!

Along with all of this, we see that users often don't pay attention to the real total cost of ownership, the largest component of which is time to market delay. Lean product development and platform based designs focused on open standards will still win the day for small, medium and large company development.

Thursday, August 4, 2011

Myopic MCU Software Approaches - Part 1

I talk to people doing MCU software everyday and the variance in the skill set and understanding of the key issues is large. I spend my days educating others on what is really important for their product development and companies in terms of MCU software.

With an MBA and an MEng along with 25 years of experience applying these skills in embedded software, I have an advantage over most others. I've been there and done that 100 times and seen all the mistakes several times. For the next few weeks, I thought that I would highlight the repeated mistakes I see others making in the marketplace.

The biggest mistake that I see is that technical people often don't understand the effects of time to market. Even very experienced and accomplished managers who apparently understand time to market often miss apply this knowledge because the decision criteria are complex.

Basic ideas on Time to Market costs are found here.

On a company basis, managers have to plan for Total Cost of Ownership minimization for lines of products. This involves lean product development or a platform based approach. This approach should not lock them into a single vendor if done correctly, or may do that but with unlimited rights, it is not expected to be an issue for many years. Most understand and strive towards this but fail to make sure that the decision minimizes time to market for upcoming projects.

Often, managers don't understand software engineering. Their experience is in hardware and are less familiar with software due to the technology shift in the market. Some managers know enough to listen and others do not. By using open standards in software, we have learned for the past 30 years that costs are minimized because we can quickly adapt the software to fit new customer requirements. This leads directly to minimal time to market for development. It is an often missed issue.

The reasons people use for locking their company into a single vendor, proprietary solution are varied. Generally they ignore Total Cost of Ownership and Time to Market. Here is a sample:

- The operating system is free with MCU-X and it is so expensive from other vendors we should just do this and the overall project cost will be minimal. This ignores the cost of training and development for a proprietary system as well as the main benefits of open systems - software reuse.

- We don't need an operating system - the application is simple. Here, the manager is thinking about past implementations on MCUs where the software was minimal. This is not the case today. As users demand new features, the lack of operating system will push out time to market and hamper the introduction of competitive features.

- A kernel is an operating system. Actually, most don't even know the difference between a kernel and an operating system. They start with a kernel, which the vendor may call an operating system even when its not, and then find out later that they are wasting many person years of time because they did not get the correct platform to develop their products. An operating system includes a complete set of standardized I/O modules and drivers along with other libraries to provide high level abstractions which eliminate many person months of effort.

- Performance of the kernel/operating system doesn't matter, they are all the same. Driven by hardware oriented thinking, software performance is often ignored. Managers are unaware that the cost of optimization for a specific function is often 10X the cost of a non optimized solution. Out of the box modules optimized for size and performance can save huge amounts of time, minimizing time to market. If the kernel is inefficient, this inefficiency can drive performance optimization everywhere with big delays and lost market share.

- We don't need feature X today; a minimal solution is enough, we will add it later. Although this is apparently a good choice today, the long term cost of adding feature X may be significant. By using an operating system which has provisions for adding a broad set of features, time to market is minimized as customers demand new and unexpected features.

By simply planning ahead and using the same ideas that have minimized time to market and total cost of ownership for larger systems, optimal results can be achieved for MCU based products as well. Here is a short list of things to do in order to minimize time to market and total cost of ownership for your product line.

- Use mainstream languages and tools.
- Use open standards where practical.
- Pay attention to off the shelf solutions which minimize time. The cost of purchase is far less than the cost of development in 99% of cases if time to market is considered.
- Fight not invented here - software reuse is your friend while in house development is extremely expensive.
- You can use an OS provided by a vendor, just make sure you have hardware options and other sources for the OS to avoid lock in.
- Get off the shelf features if possible.
- Planning is required even with a few developers. Make a good project plan.


-

Wednesday, April 14, 2010

A POSIX RTOS vs Lemming and Ostrich Behaviors

I've been thinking about why engineers that develop MCU based systems don't do their homework on software alternatives while analyzing hardware decisions to death. I think in part that they are just happy to find a solution that works because traditionally, only the hardware was a problem. Today, MCU based systems are 80% software. Engineers must recognize and use new approaches to maximize company benefits.

Software engineering economics is an important area when building MCU based systems. Often technical people ignore the business side because they are more comfortable with the technical side. Sometimes they ignore the technical side. When they ignore both the business and the technical side, the company pays big time.

One key area that engineers have completely ignored is total cost of ownership or TCO. This concept has been well known and used in IT systems and now is entering the realm of embedded systems. The concept of TCO is simple: when making software decisions consider the total life cycle costs for your OEM development and minimize costs thereby maximizing profit.

TCO should be like apple pie and homemade bread - so universally appealing that everyone loves it; however this is seldom done today. Why is this the case?

Lemming behavior is most common. If a solution becomes fashionable, often others pick it up and use it without analysis of their requirements. The current case of Lemming behavior in the MCU world is underway with a well known proprietary kernel. Its consistently abysmal and disappointing performance and proprietary API should preclude its use on any system except a hobby system where development time, longevity, engineering costs and TCO are all subservient to immediate out of pocket costs.

When are these engineers going to realize free is not necessarily free? Why didn't they calculate TCO before the commitment rather than after the fact?

Ostrich behavior is common too. In this case, engineers use obsolete and proprietary technologies when they should know better. The world is built on open systems and compatibility. For the MCU world and the RTOS world, two key factors which should influence all software development are discussed here.

The first fact is that the world has moved to POSIX for every major operating system and RTOS excluding those RTOS solutions in the MCU world. This should not be surprising - we have known for 20+ years that software reuse and portability of software are critical and POSIX delivers on this. In addition, it minimizes training, eases access to trained people, ensures robust and reliable systems and much more.

The MCU world is now adopting POSIX solutions quickly for all the same reasons that we use them in larger systems. Up until the announcement of Unison and DSPnano, a native POSIX solution was not available for MCUs, but today a broad set of MCUs is supported.

The second issue is the use of a single loop of control or a "scheduler", use of a kernel (not to be confused with an RTOS), use of a proprietary rtos and use of a POSIX based RTOS. For MCU developers this tradeoff has been complex because many developers failed to understand system complexity. For trivial systems, an RTOS is overkill but for anything but a trivial system, a POSIX micro-kernel architecture is the best you can get. You can use just the kernel if that is all you need, or you can add a complete I/O system - all absolutely free.

The third issue which is often ignored is time to market. Being late or being locked into a proprietary product which limits your ability to change with the market can cost significant market share. The entire profitability of the product line is often at stake. POSIX micro-kernels and commercially supported products minimize time to market and maximize your profits.

Just because it will work eventually and the download is free does not mean that the TCO will be optimal and time to market will be minimized. By thinking ahead and doing your own analysis you can save your company substantial time, money and effort.

Tuesday, April 13, 2010

Avoid Ostrich Behavior, Understand RTOS POSIX Standards

Recently I saw that many naive users are using a proprietary real-time kernel. It makes no sense to me and I really wondered why. I've been trying to get more data on this and it seems that it based on ostrich theory.

http://www.embedded.com/224201284?cid=NL_embedded


MCU developers need a real-time kernel and simply follow others into a poor solution because this solution is free to download. Users completely fail to understand the real costs associated with a choice like this, largely because they don't understand the economics and don't think about better alternatives. (Learn more)



First, standards like POSIX mean that you can hire people that know how to use it right away; actually, most well educated engineers on you staff will already know this and have used it. In terms of both cost and risk, POSIX offers a significant reduction. Every major OS now uses POSIX as an API. All the RTOS solutions that run with an MMU also use this as an API simply because it reduces cost and risk. POSIX is the mainstream of APIs for RTOS.

For RTOS solutions without an MMU, there is a variety of choices for POSIX based solutions. Generally the RTOS vendors push their proprietary solutions to lock people in; however, they are available in POSIX flavors, albeit with a performance hit.

There is no reason you can't use POSIX.

Second, if you move to a proprietary solution you loose the ability to reuse software easily. With a non POSIX RTOS you must do extensive work to make it operational on your system. Why would you intensionally create extra cost and risk for current and future development on your project? It makes no sense.

Third, all MCUs now come with source code to support peripherals. In the Unison environment, it is fast and easy to port any driver source code that is well done to Unison or DSPnano. It is so inexpensive that we offer this service at the cost of a single license; provided the vendor has quality code to port. If you don't want us to do it, you can do it yourself. This means drivers are not a risk item as they have been in the past.

Forth, ease of use is very important to get up and going quickly. In the Unision and DSPnano world, the Quick Start Guide gets you up and going with eight to twelve demos (FREE version) in less than 10 minutes each. Many drivers are out of the box and no development is required. Typically Unison and DSPnano come with complete API documentation and eight to twelve demo programs (commercial version has 33+ demos) with a detailed Quick Start Guide which covers all demos and options.

Fifth, optimization to meet real-time needs is one of the most expensive things that you can do as a developer. For this reason, you want a very fast RTOS (with POSIX standards). Unison and DSPnano are the equal of the fastest MCU RTOS on the market and substantially better than most. Simply by choosing this solution, you have reduced your risk and increased your chances for success substantially.

http://www.embeddedstar.com/weblog/2008/01/31/express-logic-threadx-mcu/


Sixth, DSPnano and Unison are FREE for version 4. All of these benefits are completely FREE without cost or risk. Do some research and get the very best free solution that you can. An ostrich avoiding threats in the environment is not a pretty site.

Try here for DSPnano and Unison.

Sunday, February 7, 2010

Proprietary Lock in With Software

The other day I became aware that a vendor with a proprietary RTOS for MCUs is busy locking in clients with full featured but proprietary offerings. They are doing so with the help of a major semiconductor vendor. The typical asking price is $120 000 US for the license. There is another vendor that does the same thing with a $65 000 license but without the semiconductor vendor.

Can you imagine finding out two years later that you were completely locked in to a set of APIs that everyone else in the industry is abandoning? When they discover that they can't hire trained people and can't easily reuse software that their competitors can, how will they feel. For good reason, the customers will be pissed.

I don't know what the semiconductor vendor is thinking. They will surely loose customers as they shoulder the blame months later.

The customers of the 65K product can only blame themselves.