What does it cost to test?

Posted by 
in Blogs
11 October 2017

Developing HW is a long and costly process, designing the board, layout, or the creation of a new ASIC. All these require months, or even years, of engineering and production setup. Mistakes are of course costly, often requiring a do-over of the whole process and causing significant delays in projects. So, what is the right solution to this expensive problem?
Everyone knows that testing is needed and that coverage is of the utmost importance. Most projects in the market will have a dedicated budget for testing, and a sizeable one. For those developing hardware devices, there are several options when getting ready to test a new product.

1. Build a full testing environment.
2. Use a scripting language to directly access the hardware (a bit different than #1, and simpler).
3. Use of off the shelf tools supplied by test equipment manufacturers (i.e., LabVIEW)
4. Use 3rd party tools for test management and execution.

Let’s examine each of these options, their benefits, and pitfalls.

Building your own full test environment

Building a test environment often seems like the best solution, available only to the rich and prosperous, although sometimes a method used by the poor and hasty. The advantages are clear:
1. Your testing environment will be designed to fit your product like a glove.
2. The control is yours, tools and technologies that are favoured by the team can be used.
3. Design for testability is at its maximum, the development teams and the test developers can easily coordinate their mutual needs.

However, many times the result is a tad different.
1. The testing environment is suitable for only your current variety of product, whereas, for the next version, a new development will be needed for the test environment.
2. Engineers will often select technologies according to what they know best or especially favour, instead of industry standards, proving to be hard to maintain in the long run.
3. Developers and test developers working closely together may copy bugs from the design directly onto the test environment, resulting in bugs going through the process.

But the real problem here, what does this all cost? Assuming a total of only 4 man years of engineering, this effort could easily go between $500K to even $1M. And that is just to get to a first working environment. There are further maintenance and development awaiting down the road. Not to mention the costs of de-focusing a group of engineers within your organization that is now working on something that is not the core business.

Use standard scripting and basic tools

Many projects do not require elaborate testing environments, and it’s a simple project, the requirements are only 1 page long; everything is clear and simple. If the product is simple, why go into lengthy and costly processes for testing it? Sometimes, that may be the case. Scripting tools such as visual basic and python, offer direct access to peripherals, thus allowing access from a standard PC to your DUT. Additional solutions, such as using excel to manage your test lists allow for simple test management shared amongst team members. Let’s take a look at how this usually works.
One (usually junior) engineer is assigned to the project. They create the infrastructure needed, such as low level calls to the device, some basic testing scenarios, such as read/verify, write/verify from/to the device. Once that is done, a few basic tests are created and voila! You have your testing environment. Indeed, all goes well, but then, one day the junior grows up and gets ready to move up, or even worse, move out. You are left with an orphan testing environment and the need to maintain it. True, you can bring in another junior, get them up to speed and continue from the same spot. But let’s look at the cost of that over 5 years.
1. Junior engineer $50K annually
2. Average turn-around of a junior engineer is 1.5 years
3. Total cost over 5 years: ~$160K

And that is assuming that everything goes perfectly smooth! Have you ever seen an engineering project go perfectly smooth?

Use off the shelf tools supplied by test equipment manufacturers

Test and measurement equipment are an important part of testing, of almost any hardware device. On simple devices you may need to measure frequencies and temperature, sometimes, you need an injection of data or network streams. There is no doubt that testing equipment makes this all possible. In recent years, the equipment has developed to become truly remote controlled and easily accessible. The software purchased alongside the equipment usually provides very good building blocks for an environment centred on the equipment itself, it is often difficult to connect custom hardware to the mix and requires further software development.
Here is a direct quote from one manager at a leading hardware maker: “Getting them to customize what we needed was both complicated and expensive, running at $50K per project and taking several months to get what we wanted, the process is just too difficult”.
When choosing this option check the following:
1. Does the software provided connect to everything you need in your test scenario?
2. What is the contract for service offered under your license?
3. Can you do custom work on your own? What training does that course of action require, and is that readily available?
4. Can you meet all your testing requirements, including real-time, data collection, feedback loops, etc.?

Be sure you have tied all loose ends and get ready to shell out at least $100K on the software itself and engineering work around it.

Use 3rd party tools for test management and execution

I know what you’re thinking: ”Of course this is the best option, they are a software testing tool company, surely they will recommend their own stuff.” Well, yes, but we are doing this based on years of experience. Over the past 7 years, we have been meeting with companies that have chosen one of the 3 solutions depicted above. We are almost never the first at the scene. Our customers have usually spent a great deal of time and money on creating a solution that is just “right for them”, only to discover that they have their hands tied and cannot move forward past a certain point. Have they shipped products before? Yes. Did they have problems? Sometimes. So, why do they turn to 3rd party tools as a solution?
Most often, it is because they have reached a dead end. A need to evolve the testing environment to a new version, new test equipment, a new scenario discovered in the field, that they simply cannot do without significant further development and time delays. Sometimes it’s the understanding that
they are wasting enormous resources on something that is not their core business. You are developing a product, not a test execution and management tool.
What about cost? Here’s the trick, this does mean you need to pay a sizeable amount at the start and spend the effort of converting legacy tests and environments to a new setup. But in the long run, this is much cheaper, sometimes 10% of the cost, than creating a custom testing system using your own precious resources. You know this is true when fixing your car, doing electrical work at home and many other places where you would rather hire a professional than do it yourself. So why the different approach for a test and validation environment?
A dedicated testing environment (like SimVisible) was created with the full consideration of HW test flow.
SimVisible is built to support multiple projects and scenarios, its flexibility is key to be able to get your project up and running quickly, while allowing for complex test scripts, connecting to multiple test equipment and running several DUT from one station.
Test environment makers will be open to adding custom features that you ask for, since such features enrich their product. Enjoy years of experience derived from dozens of projects: Instead of learning from your own mistakes, learn from others’. It’s cheaper and faster, and less painful!
Don’t believe me? (Why should you?) Drop us a note, and we will do the math specifically for your unique, completely different, totally special case. Put your time and energy on the tasks that are important to you, streamline your development and testing process.

Yonatan (Yoni) Passwell

VP Sales & Marketing, SimVisible