Why Prototypes can work on one board but fail on the next
You built one and it worked. You built another and is doesn’t…
Nothing changed, the design is the same, and the deadline hasn’t moved.
This is one of the most common and frustrating positions in electronics development, and it is almost always the same handful of causes. The bad news is that the most likely explanation is the one nobody wants to hear. The good news it’s mostly avoidable.
Start with the uncomfortable possibility
The most common answer is that the design was built on optimistic assumptions and datasheet headlines, and was never guaranteed to work across the full range of parameter variation. If your design has enough margin, board to board variation doesn’t matter. Every unit passes because every unit is comfortably inside the limits. If your design has almost no margin, then whether a board passes is decided by where its particular collection of components happens to land in their tolerance distributions.
This is why the problem gets worse with volume. The spread of the parts does not widen as you build more, but you sample more of it, so the tails you never saw in a batch of one start turning up in a batch of a hundred. It’s a margin problem showing you its distribution.
So, the first question isn’t “what’s different about the failing boards”. It’s “how much margin did the passing board actually have”. If the answer is “we don’t know, it passed”, that’s the finding.
Why “it worked on the first one” is weaker evidence than it feels
Board one is almost never representative, for reasons that have nothing to do with luck.
It’s usually the board an engineer hand assembled or hand reworked. It was probably tuned, possibly with a component swapped during bring up, tested on a particular bench, with a particular probe, by the person who designed it and knows where not to poke.
There’s also the plain statistics. One sample tells you the design can work, not that it will when you manufacture hundreds. Taking margin as two sided, a design with three sigma of it fails roughly once in every four hundred units. A design with one sigma fails about a third of the time. Both designs pass board one. You cannot tell them apart from a single success, and the temptation after board one passes is to treat the design as proven and move on. That’s where most of these programmes go wrong, months before the failures appear.
The seven sources of that variation, roughly in order of likelihood…
1. Assembly and process variation.
Solder paste volume, reflow profile, placement offset, voiding under a thermal pad, insufficient wetting, a moisture sensitive part that was not baked to J-STD-033 before reflow, or ESD damage that left a part working but degraded. On instrumentation and any high impedance design, add flux residue, which combined with humidity can put a leakage path across a node where you have carefully designed for picoamps. If board one was hand built and boards two to five went through a line, you have changed process, not just quantity, and that is the single most common reason a first article and a first batch behave differently.
2. A different part than you think you have.
Much more common since the shortage years. A different date code, a different die revision, a change that was notified formally but stopped at the distributor rather than reaching you, or an “equivalent” substituted by the assembly house without notice. If any part came through a broker rather than a franchised distributor, authenticity is also on the table. Check the markings and lot codes physically on the failing boards against the BOM, and against what was on board one. This costs an hour and occasionally ends the investigation.
3. A different build than you think you have.
The same question applied to firmware, and it is asked far less often than it should be. Board one is frequently running a bring up image with different clock configuration, different peripheral timing, debug code left enabled, a watchdog disabled, or per unit calibration and trim values that later boards were never given. Read the image back off the failing boards and diff it against the one that worked, and check what is in non-volatile storage as well as what is in flash. “The design is the same” is a claim about the schematic, not about the product.
4. The measurement is varying, not the board.
Not a design fault at all, but it accounts for a surprising share of these investigations. A different probe, a longer ground lead, different scope bandwidth or probe loading, a different bench power supply and current limit, a different operator, a different warm up time. It sits here on likelihood but it should be first on order of work, because it is the cheapest of the seven to rule out and the only one that invalidates every other line of enquiry if you skip it. Reproduce the failure on your own bench, with your own kit, before you trust any of the data.
5. Component tolerance stack up.
Designs done to typical datasheet values fail at the tails. The important distinction is between the deterministic and the random. Ceramic capacitor DC bias loss is deterministic: a small case X5R or X7R can lose well over half its nominal value under bias and more again over temperature, so a nominal 10 µF decoupling capacitor may be delivering under 3 µF once initial tolerance, bias, temperature, AC level and ageing are multiplied together. That happens identically on every board, so it does not explain divergence by itself. What it does is remove the margin that the genuinely varying terms then push you through. Those are the ±20% initial tolerance, lot to lot dielectric variation, and ageing, which runs at roughly 2.5% per decade hour for X5R and X7R and restarts at reflow, so a board built in March is not electrically the same as one built last week. Add oscillator initial tolerance and drift, load capacitance error, MOSFET threshold voltage spread, op amp offset and drift, and regulator reference accuracy stacked against the feedback divider tolerance, and the worst case corner can sit a long way from where you designed.
6. PCB fabrication variation.
Controlled impedance tolerance is commonly ±10%, with ±5% available only at extra cost and with verified material. Standard FR4 dielectric constant carries a tolerance of around ±0.2 to ±0.3, dielectric thickness and copper weight vary within spec, plated through hole copper only has to average 20 µm for IPC 6012 Class 2 and 25 µm for Class 3, and at high speed the glass weave itself introduces variation depending on where your trace happens to sit. If the second build went to a different fabricator, or came back on a different stackup or a substituted laminate because someone accepted an equivalent, that is a genuine design change and should be treated as one.
7. Marginal timing, sequencing or settling.
Power sequencing races, reset and brown out thresholds, crystal startup where the oscillator negative resistance margin against the crystal’s ESR varies by crystal lot, I²C bus capacitance against the 400 pF limit for Standard and Fast mode, and settling time after a multiplexer switch. These look unit dependent but are usually condition dependent and often temperature dependent, and they are the reason a board that failed on Monday works on Tuesday.
How to characterise it, in this order
The instinct is to put a scope on the suspect node, that belongs at step five, where it is worth far more than it is at step one.
1. Stop changing things – Freeze the boards, label them individually, and stop reworking. Every undocumented change destroys evidence you’ll want later.
2. Define a pass or fail test that produces a number – Not “it works” or “it’s noisy”. A number, measured the same way every time, with a written procedure. If two people can’t get the same reading from the same board, nothing downstream is trustworthy.
3. Check the measurement, not the board – Measure one board twice, with a break in between. Measure a known good board, then a known bad board, then the good one again. If the good board’s number moved, your bench is the variable.
4. Build a distribution rather than a verdict – You have five numbers, not one pass and four fails. Where does each board sit relative to the limit? If the failures are marginally out and the pass is only just in, you have a margin problem and you can stop looking for a defect. If one board is wildly out and the rest are clustered, you’re looking for a defect on that board. These two findings lead to completely different work, and a pass or fail column can’t tell them apart.
5. Vary one environmental axis at a time – Temperature, supply voltage, load, with the scope on the node you suspect. This is the single highest value test on the list. If the failure moves predictably with temperature or supply, it’s margin, and you now know which direction to look in. If a passing board fails when you warm it by fifteen degrees, it was never a passing board.
6. Swap parts between boards, don’t just replace them – Move the suspect component from a failing board onto a working one. If the fault follows the part, it’s the part. If it stays with the board, it’s the board or the layout. Replacing a part with a fresh one tells you much less, because it changes two things at once.
7. Compare the builds, not just the boards – Different fabricator, different stackup, different reel, different date code, different assembly house, different reflow profile, different firmware image. Get the AOI and X-ray records if they exist. Programmes routinely spend weeks on the bench for something that was visible in a build record on day one.
When to stop investigating and re-spin
Investigation is open ended and consumes your most expensive engineers. A re-spin is a known cost with a known lead time. That asymmetry means the decision is usually made too late, and it also means it is often made for the wrong reason, because the team is out of ideas rather than because it has a finding.
Test as much as is physically possible before you spin. Don’t spin a board out of hope, spin it on the back of test results and facts, and be able to write down what you are changing and why. A re-spin you can specify is a bounded cost buying a defined improvement. A re-spin you can’t specify costs you the board, the lead time and the fortnight, and leaves you exactly where you started but later.
Most of what people think requires a new board can be proven on the one in front of you, and the bench techniques are more invasive than teams usually allow themselves.
When to bring someone in
Three signals, and any one of them is enough, you’ve been round the diagnostic loop twice without narrowing it. The characterisation plan you’d need is longer than the time you have left. Or the current best theory in the room is “we think it’s noise”, which is generally what a team says when the measurement discipline hasn’t been established yet.
This is a large part of what we do.
A meaningful proportion of our work arrives as somebody else’s stalled board, and we start with characterisation rather than a proposal, because the answer is sometimes that a re-spin is cheaper than another month of investigation and sometimes that it’s a two-week fix.
You’re better off knowing which before you commit the budget.
Sound like you? Contact us today for a quick no obligation chat about how we can support your project and help you deliver with confidence.