Whistling Past the Boneyard
With work, and some financial nudging, failures can be resurrected.
Remember that moment in high school when every student in the class was terrified at the prospect of being selected by the teacher? Nobody was confident of the correct answer; therefore, the only acceptable survival strategy was to not raise one’s hand for fear of immediate embarrassment and subsequent withering ridicule by one’s peers.
Bonepiles are like that. At least the acknowledgment of them is.
Behold the contemporary spectacle of six adults, many years removed from high school, arranged around a conference table. One of the six is The Boss. The rest are underlings. All are worldly wise.
In this scenario they might as well be back in high school. Ask them to admit they have a bonepile, and let the entertainment begin.
No one raises their hand. Including The Boss. Lots of furtive glances. Nobody dares reveal weakness.
Problem is somebody has to raise their hand and admit they have a problem. It’s not unlike a 12-step program. You first have to own up to the reality of the situation. Only then can the therapy ensue.
So begins another bonepile rehabilitation project.
Bonepiles in engineering parlance are those boards that have failed somewhere along the way, usually in functional or system test following manufacturing, or occasionally in the field while under warranty. In either scenario (production or field return) some effort was exerted by the OEM to troubleshoot the boards, once returned, but that effort came up empty. Perhaps key personnel were needed to address higher priorities; or technicians lacked essential troubleshooting skills; or the functional test apparatus simply stated the board failed, with no diagnostic feedback (red light/green light). Whatever the case, interest rapidly shifted elsewhere, thus leaving a bonepile of neglected failed boards.
So there they sit in a warehouse or MRB cage. One failed board doesn’t merit much attention. One hundred failed boards, especially if they are expensive, get a lot of attention, painfully so when the time comes to write them off the books. Tough to hide them too, although the desire for concealment often brings out peoples’ creative side (see not raising one’s hand in front of The Boss, above). Eventually the bonepile grows too big to ignore, and this usually draws the CFO’s attention. At which point alarms sound.
Something must be done!
We get the call. The pitch to the CFO is easy: She buys into the concept in mid-first sentence. There is money waiting to be found – in certain cases hundreds of thousands of dollars of it – if only an external resource existed to address it. That would be us. Restore the boards to life, subtract the cost of that resource, and the rest is pure profit. The CFO gets it. Already visions of bonuses dance in her head.
We are commissioned to undertake the project. The CFO is excited; the money’s already spent, figuratively. It’s Christmas in July, so results are eagerly awaited. Carry on, technical people, and let me know when the check hits the bank (cue sound of door closing as CFO exits the meeting, not wanting to be bothered with the details). On to more important work.
Thus exhorted, we carry on. We tell the technical and operations people we require several things to make this project successful. They are:
- All the usual documentation and data files necessary to make test programs (CAD, schematics, bills of materials, etc.).
- Failure data. As much as possible. And for goodness’ sake no printouts that simply state “No boot,” or “Com Failure,” or worse, “Doesn’t work.” The latter are worse than worthless.
- A functional test apparatus, to be loaned to us for the duration of the project, preferably onsite at our facility. It makes validation of troubleshooting and proving of hypotheses much easier and quicker.
- A Champion within the customer’s organization who can be our primary liaison to all others in that organization, and who has the clout to make things happen. This is important because bonepiles overwhelmingly consist of older generation boards and, therefore, do not always command high-priority attention from management on down. It often falls to an overworked underling to act as the go-between – not a recipe for success. This task gets added to the underling’s ever-expanding list of tasks. In this case it is No. 322. Successful completion of this task has no bearing on the underling’s chances for advancement or of opportunities for increased remuneration. In the mind of the underling, it will be addressed by the time Chelsea Clinton runs for president.
It is with such high hopes and disparate expectations (depending on your position as either underling or management) that we usually start our bonepile projects. They typically go nowhere in the first month or two because our liaison is, predictably, nonresponsive. Cage-rattling time frequently occurs at about the 60-day mark, when we go directly to Management and warn them the project is going nowhere. Invoices also concentrate the customer’s mind. Billing them for test programs suddenly makes them keen to anticipate results.
So, on or about Day 60, we really get going. Techs get assigned to our project. Prior test results and project history are shared. A functional test apparatus changes hands and lands at our facility. We get trained how to use it or, more often, figure out how to use it ourselves because the person who developed the apparatus is retired. Programs are completed. (It is not uncommon to wait months to create programs because the customer lacks CAD data, does not know what CAD data are, or fears we will use CAD data to somehow reverse-engineer their product. Suspicions must be overcome.) We actually start testing boards.
Results are grim. Missing components. Wrong values. Backwards capacitors and resistors. Solder splashes. Voids in QFNs and BGAs in excess of the IPC-A-610 30% threshold.
Now we know why nobody on the customer’s team raised their hands at our initial meeting. How did these boards pass their initial production tests? How were they ever shipped?
In the course of unearthing this mess, the customer reveals that their previous contract manufacturer, the same one who built the boards presently at hand, was disqualified during the prior fiscal year. It seems there were problems with quality and delivery. New operations regime came in and cleaned house.
No kidding.
Oh, but the price was undoubtedly right at the time!
Chances are the buyer who selected this unfortunate supplier no longer works at the company either.
It seems the rays of sunlight, metaphorically speaking, expose all sorts of management deficiencies.
We do our thing and diligently repair as many boards as we can. Of 100 units submitted to us, 60 to 70 have promise. The rest are basket cases, beyond hope of resurrection, and used, with customer permission, for part cannibalization. This is also important because with older boards many components are obsolete and no longer available on the open market.
Troubleshooting is the art of detecting patterns of failure, and categorizing those patterns into probabilities, which become more focused with time and experience. At the beginning we know nothing about the board. After two to four months of concentrated work, we generally know as much as the customer’s bench technicians.
In this specific case, we are able to return 65 good boards to the customer that had previously been written off. The CFO is happy.
So are we.