Useful
Hints for System Designers
by Joerg Schulze-Clewing
System designers and management have to decide early on how to lay out a system, how design the enclosure and so on. Usually Sales and Marketing will have a say in this as well, and they should. While it would be best to have a consultant in on that phase we realize that this typically does not happen. That is one reason why I have compiled these hints. Check back at times as this section will grow.
The hints below represent my personal opinion and are based on experience. However, they are not intended as advice because every situation is unique and may require a different approach. Some of the techniques outlined below require a thorough understanding of the theory behind them and cannot simply be implemented as a slam dunk.
Ideally documentation should be performed in parallel to any design. This should include not only the module specification but also the hazard analysis, service instructions and other documents that will be needed for a engineering change order release (ECO or ECR). Doing these in hindsight and under increasing schedule pressure carries the risk of mistakes.
In a consulting relationship and also within a company it is important to frequently share these documents while they are being created. One reason is that trending into a dead end is caught earlier on. The other reason is that something unforeseen can happen. A case in point was when the frame of a client engineer's motor cycle disintegrated at full speed. Thank God he survived even though it was a close call. But since we both document early on I could jump in easily and then hand back the baton once he was released from the hospital, avoiding a major schedule setback.
Another important factor is that the documentation be written in a way that any of the engineers can understand. We can try to avoid expressions such as "LO phase shifter for IQ mixers". Sure, an RF engineer knows that this is the phase shifter for the local oscillator of a Doppler receiver. But does the software engineer understand? Service manuals and production procedures have to be crafted even more carefully, always anticipating that the reader may not have an engineer's level of education and possibly grew up speaking a foreign language.
Engineers and Sales & Marketing are often at odds over what to include in a design. S&M appears to want every bell and whistle, and soon. Engineers groan under the load of a seemingly impossible timeline. We have got to reach a consensus here. After all, sales to paying and happy customers is what brings in the money in the end.
As engineers we can try to think as if we were the user of our product. Did you ever wonder why nobody thought about providing a reset function for that Laser printer so you don't have to reach in back, pull and reinsert the power cord? That is clearly a design process failure and it could have been avoided at no additional production cost. Many times it is beneficial for the design engineer to tag along a sales team and talk directly with end customers. It can be pretty eye opening when, for example, an engineer realizes that medical customers simply cannot reach certain locations on a back panel because they wear a lead vest all day long. Only after wearing one ourselves for 10 or 12 hours do we really understand.
Products for export need an even higher scrutiny. An example that comes to mind is a fairly large system that needs to be powered down like our "modern" computers before shutting it of. This turned out to be very unfortunate as such a scenario is impossible to follow when the system is shipped to a country with a less than optimal electricity grid. One location had power for only about four to eight hours a day and the users would never know exactly when the power would be going out.
Is it easy to test the system and its components? Can someone with limited language skills do it? Can it be designed so that no alignment is needed? Can it be tested without expensive lab gear?
As engineers we should strive to keep things as simple as possible. If we can make it happen with 20 parts less then these 20 parts won't have a chance to die, or even be bad upon arrival. The same goes for alignment. We have redesigned boards where two dozen potmeters and trim caps had to be tweaked and only the chief tech was able to do it. After the redesign the boards calibrated themselves while the system was being used, without anyone noticing. Production just had to test the boards and then install them.
It can't hurt to discuss a design with people from Manufacturing. In terms of mass production they know best what works and what doesn't.
Which parts of the design should be serviceable, which ones would better be tossed and replaced? Is it easy to get to things? Can a person with limited skills safely perform some tasks? Without costly test gear? What about foreign countries?
Every system that isn't a disposable should be easy to service. So as engineers we have to think about what it takes to disassemble things. This doesn't always have to mean costly hardware such as screws and washers, there are other options like snap flanges or pins. For larger systems the diagnosis is the most important part of service since the technician has to find out within a reasonable time frame what is damaged.
Another often overlooked need is to provide access points that are short circuit proof. A simple resistor can prevent a falling scope ground clip from frying a circuit. Often engineers use expensive gold plated coax connectors such as SMB for test points. That is nice but they can cost more than $5 a piece. Why not a plain RCA jack, just like those in the back of a stereo? It doesn't even need to be the gold plated luxury edition. The impedance mismatch rarely ever matters but now the service tech can buy a replacement test cable at any radio store. Then, use LEDs wherever that makes sense. Buzzers are also helpful and very cheap. All they require is a free port pin on a controller and some firmware. Sure, many folks won't understand morse code anymore but if the processor "talks" slowly it'll sink in quickly.
In the old days we sometimes designed a telephone jack onto a board. You could hook up a touch tone phone and "communicate" with the circuits. Nowadays the more appropriate method might be a USB port or something that can be hooked up to a laptop or PDA. Still, many complicated system don't have any of these. Be creative, but most of all talk with the service folks before designing much. Chances are they'll be your friends forever if you do.
Often it is tempting to use that wonderful "panacea chip" that does it all. However, this kind of chip is usually expensive and single sourced. Besides high cost this can lead to a real nightmare: At times Purchasing might be unable to find any of these wonder chips on the market. Production comes to a screeching halt and the manager scrambles for that pack of heart burn pills.
Therefore, we try to stay away from specialized devices or tightly toleranced components whenever possible. Take a low offset op amp as an example. If the circuit is used in intermittent applications such as a video path or a hard drive head, why not incorporate the good old clamp method to get rid of the offset error? Suddenly any old "jelly bean" amplifier will do and it can be purchased for pennies from half a dozen manufacturers. Same for situations where we need an adjustable signal delay. There are integrated devices that will do that nicely for $30 or more. The good old discrete design with capacitors, inductors and maybe PIN diodes can serve us for under a Dollar here, including that 8 bit register chip.
Passive components need to be considered, too. We have seen designs where circuit boards were literally peppered with tantalum capacitors. That is not only expensive but also prone to stuffing errors. One board sounded like a popcorn machine when installed in the test set and some caps even cinged the technicians shirt. Should have worn a cotton shirt, he grumbled. Thank God he didn't hurt his eyes. This board needed so many caps because a couple of large RAM banks were ping-pong loaded. All we did was change the timing so that the RAM bank loading cycles were just short of overlapping. That reduced the needed energy storage on the 5V rails by a few orders of magnitude and almost all cap locations went unoccupied from then on. Along the way this saved almost $10 per board and reduced the electromagnetic interference substantially.
"Jelly bean" parts: It is amazing what can be done with plain old transistors, CD4000 series logic or things like motor driver chips that are cheap and were intended for a totally non-related market. Take CD4000 for example. It was already there when most of us started to seriously tinker with electronics. To this day almost nothing beats their extremely low power consumption. These magic chips let us design circuits without a power switch. Many of the chips are in production since decades, cost only a few nickels and there is no end in sight because of consistently heavy sales. How heavy? That's something you absolutely have to find out before a design-in because nobody wants the purchasers breathing down their neck when a part goes obsolete too soon. Distributors can be a good source for an honest answer here and so can application engineers.
The first design may be mostly discrete and logic chips. But what if success sets in and production reaches thousands per month? Can the idea be implemented using a micro controller?
A frequently overlooked technology are thick film hybrids. Defense electronics engineers are very familiar with those ceramic substrate carriers but the civilian world often misses out. Resistors are screen printed, no need for parts. They can be laser trimmed to very high precision at little cost. Printed layer capacitors of a few pF are also feasible. The best came out when we had to design a gain controlled amplifier with a dynamic range that just could not be realized on a chip. The amplifiers were actively laser trimmed while running at frequency in a test rig so their gain curves tracked to within fractions of a dB. Therefore, the 10,000th amp was literally the same as the first one.
What if sales really take off and tens of thousands roll off the line per month? Can a design be condensed into an integrated circuit? This may sound easier than it is. ICs do not allow certain techniques, for example it is merely impossible to provide absolute values for resistance on a chip. The largest practical on-chip capacitors won't be more than a few hundred femto-Farad, mainly because they require a lot of real estate and chips become expensive with larger die size. However, other things can be achieved much easier than with discrete circuitry. Ratios between resistors or active devices on the same chip are very accurate. Also, chips from the same wafer track each other pretty well. Therefore, it would be nice if the architecture of a new design can be laid down with a future migration to an IC in mind.
Systems and modules used in aircraft, safety critical applications or defense have to undergo rigid endurance tests before a release. Thump, thump, up a slide rail and back down smack onto a concrete pad. Experience in that field of electronics is an eye opener. Unfortunately that stiff test regime is not common with other electronics. The result is often that many systems do not arrive at the customers in working order. I have seen large electrolytic caps that broke off, pieces of fan blades sloshing around and programmable chips that worked themselves out of their sockets. Tiny shards of glass and some sand at the bottom of the chassis indicated that this must have been a fuse before it left the loading dock.
When designing a system or circuit board imagine what happens when this product is exposed to shock. When the cargo pilot has to "nail it to the runway" in bad weather that shock for a brief moment makes a component many times heavier than it actually is. A rough truck ride for hours along a worn out "wash board" desert road wreaks even more havoc. What can we do? First of all we should avoid anything that transfers stress to weak spots. Take those tall heat sinks. They are fastened pretty safely but there are always semiconductors mounted to them. When these heat sinks start to vibrate that stress transfers right into the pins of those semiconductors and can transform the solder in their vias into pudding. I never use this type of heat sink but if you must, why not provide relief bends on the transistor pins? Surprisingly I almost never see that being done in actual designs. The same goes for large electrolytics. All that holds these poor beasts are usually two tiny wires into the circuit board. Can the cap be split up into a few smaller ones? Is there anything else in the design that could vibrate or work itself loose? Those are the questions we should always ask ourselves.
Large ballgrid arrays should be positioned where the circuit board will not flex much. The same goes for any large parts that are surface mounted. Pot cores are another concern. On some of them the top half is merely secured by two small clips that grip into notches in the rather brittle ferrite material. The better choice is the type with a spring clamp all across the top half. Way too often I see circuit boards the size of a large pizza being held in place by two tiny 3mm screws up front. That is calling for trouble. Why not at least provide a bracket across the whole card cage front, similar to those that hold a car battery in place?
Back in college most of us learned the single point grounding scheme, the separation of analog and digital grounds, isolation. This works nicely in theory and may be a good strategy when dealing with frequencies under 100 KHz. However, above several MHz all this blurs. It becomes impossible to predict whether spikes and other unwanted signals follow the path they were supposed to take. Pretty soon noise problems take over and an EMC certification seems a nightmare. That is when our phone usually rings.
Just imagine a ground separation between analog and digital on a large AD converter board. The stray capacitance between these separated layers might be as low as 50 pF. But to an unwanted 10 MHz signal that looks like a low impedance path of barely more than 300 ohms. At 100 MHz, well, you get the picture. Suddenly a 50 pF barrier appears like what a 2ft fence looks to a large dog.
This is an example why one common ground is often best, provided that it is a massive ground plane and the plane is tied into chassis at multiple points. No pigtail wires but with bolts and hardware. Just look at an old military radio from the bone yard. Typically that is the way it is built.
Common ground includes the chassis. It is important that there is enough conductivity. Sounds easy but when Marketing wants a nice molded plastic enclosure things become challenging. Not only do we have to deal with the intricacies of metallization but also with environmental regulations concerning the application of such coatings.
There are situations where all the above just won't cut it. One case was a small aircraft that had the engine up front but the propeller in back, with a metal shaft connecting them. The shaft had joints where the electrical path had not been considered. Electronics were in both areas and cables abound. However, this aircraft was made from composite materials and not the usual aircraft aluminum. Oh boy. Noise everywhere. Here the options are few. Much of it boils down to module level work and breaking down the "efficiency" of unwanted antennas created by the cabling.
Often it is preached that good EMI performance requires a #6 screw every inch or so. Not only does that look ugly, it is often not necessary at all. The problems are usually more in a lack of panel overlap and the wrong coating. There are even systems where panels are bent inwards for stability and instead of overlap there is a gap. That can lead to electromagnetic pandemonium.
Paint on the overlap contact surfaces is a no-no and can easily be avoided by masking. Not enough overlap is more serious. On most systems half an inch will do, others better have one inch. The goal is to provide conductive contact where mounting screws are and enough capacitive contact everywhere else. Ideally it should be a circumferential conductive contact but that might not always be possible.
Coating is frequently ignored. That nice anodized finish looks great and prevents fingerprints and other messes. But it also prevents conductivity. Just lay ohm meter tips gently on the surface and watch. Unless piercing the surfaces, chances are that it won't show any conductivity. There are alternatives such as alodine. Best is to check what the aircraft industry does while making sure that there are no safety or regulatory hurdles.
Designers often default to aluminum. However, during EMI assignments we have often redesigned for galvanized steel. It was certainly not easy to convince clients here. As with aluminum coatings there can be little issues when environmental regulations prevent a certain galvanizing process but so far it had not been a real problem. The upsides afterwards were stunning: No more chassis induced EMI problems, quite a cost savings over aluminum and the whole system became almost as strong as a tank.
We need to keep ourselves educated in regards to materials, chemistry, corrosion and other topics that typically are not part of an EE's education. The same goes for mechanical stress analysis. One of the most frequent field issues is stuff coming off or connections working themselves loose. A very typical example is a large heat sink with a few transistors. Sure, it is usually secured with screws but any ever so slight vibration will work on the solder joints and turn them into pudding.
Most systems require cables to enter the enclosure. This could be the mains power connection, a cable going to a peripheral or to a terminal, or a patient cable. We have often found cables to enter with or without connector and then continue to run inside for several inches until reaching the destination point on a circuit board. That is another EMC concern because these few inches inside the enclosure can pick up enough clock harmonics and other noise that the EMC test will be failed by tens of decibels.
Ideally a cable should terminate right behind the point of entry. The circuit board could be laid out in a way that it stretches to that point of entry and provides a nice grounded mounting hole right there. On some connectors it is possible to install a number of capacitors with short paths to ground. But that is not only ugly and expensive, it can also compromise signal integrity on that cable.
Toroids are another option but that should be the last resort. There are cases such as defibrillator proof patient interfaces where toroids and pot cores are unavoidable but this topic would go beyond scope here.
Many back planes and circuit boards look like an aerial photo of a busy truck stop. Dozens upon dozens of 220/330 ohm mass terminator strips. Every conceivable bus or signal line is connected to one. Just consider what that does to the power budget: Every 16 bit bus consumes 1.5 W of power when run on 5 V and terminated at both ends. Without a single bit changing. When it is driven this consumption rises dramatically and we have a gigantic space heater.
What do we need to terminate here? Only that part of the spectrum where transients can occur. A regular back plane bus trace which stretches 15 inches will not generate any transients in the low MHz range, so why terminate all the way down to DC? The answer to many situations like these can be an AC termination. A resistor matching the characteristic trace impedance in series with a small capacitor of a few ten pF to ground. For best results the cap should be towards ground. If the line can go tri-state there might be a need to add a 10 K to ground to prevent it from floating. Of course, as with the usual termination this would be needed at both ends of the line, except for the 10 K which is only needed once. Just make sure that the RC constant of the new AC termination is several times higher than needed for the length of line but not as high as to cause data content related floating. If in doubt, deliberately mismatch a little and hook up a fast scope.
Usually a great deal of skepticism comes up whenever I suggest AC termination, almost as if it was some kind of snake oil. After clients agree to give it a shot it instantly solves a ton of electromagnetic problems and the systems run much cooler. Once we had a weird outcome: AC termination gave the same great results but the power supply started making noises like a cat in distress. Then "Phhhht...poof", followed by silence. This was in the 90's when many switchers required a minimum load and we had just managed to drop below. It wasn't supposed to die though and the vendor replaced it for free. We spec'd in a smaller model, saving over $100 per system. The only remaining problem was what to do with all that room.
Whatever you do in regards to termination always make sure you thoroughly understand what is going on. Proper AC termination can save lot of power and component wear but if done wrong it can also lead into serious trouble. This might be a situation where a good analog engineer can help.
This occurs a lot more than we may think and it often goes unnoticed. Here is a story that illustrates it: A client called because new regulations came out and their products now had to pass EMC class B. One of their systems just wouldn't make it and since it was in production for many years a major enclosure change was not in the cards. This meant we had to attempt a module level approach, something that is usually not desirable. One of the peaks was extremely stubborn. So we used a magnetic probe kit which showed it to originate from a digital processor board. But where? The probes weren't small enough for this high density board so I took a small BNC cable where one end had been cut off, taped that end so accidental touches wouldn't fry anything and sniffed around. Here they were again, the snake oil looks.
Pretty soon we found that the emission came off a single long trace going from a programmable logic device (PLD) to a micro controller. Probing with a fast scope revealed something ugly. At very brief but regular intervals the digital signal would go into an intermediate state although this was not a tri-state line. Aha. The situation had gone unnoticed for years. Turned out this must have been the reason why this controller died more often than all the others. The PLD had to be reprogrammed, everything was fine and the system passed class B with flying colors. As a bonus the controllers lived happily ever after.
These often meander across a whole board. Ideally nicely buried and of controlled characteristic impedance. Maxwell's equations can be used here to figure out trace width and other parameters but an older Motorola ECL data book has nice "cheat sheets" in there. Don't throw away all your data books in this Internet age. Ever done a design during a power outage?
Unfortunately a controlled trace geometry is where clock line etiquette often ends. All kinds of FPGAs, registers and latches are connected just like street lanterns. What happens if we do this to a TV cable is ghosting caused by mounting reflections. Same here. On a clock line the edges become skewed and often it reaches a point where the actual clock signal becomes hard to discern on a scope. One chip sees the edge a few hundred picoseconds later than the simulator says it does and the whole timing scheme is off kilter. This is one of the most prevalent of the problems found when called out to a client whose fast digital designs are not working reliably. Hold a hand near one FPGA and it works, move a scope probe to another and everything freezes up.
This is one clear cut case where analog meets digital and also a reason why it is sometimes good to ask for analog help on a seemingly digital problem. One very nice remedy is to make sure the clock line sees only a miniscule amount of capacitance everywhere it is tapped into. A regular chip can bog it down with 5 pF or more. High input impedance doesn't mean much, it is the capacitance that hurts. A transistor follower between clock line and chip can reduce that capacitance dramatically. I prefer UHF receiver transistors of the bipolar kind here because they are cheap and plentiful, and very low in capacitance.
Another approach is to split up the clock line into several. However, that requires beefy driving and costs real estate.
Most digital designers resort to clock oscillators "in
a can". These are nice, except that they cost a lot and can be out of
stock for weeks at a time. It would be nice if we all kept alive the art of
designing our own oscillators, using a crystal and some parts. Pretty much like
it is good to be able to whip up a delicious meal instead of a take-out. More
healthy at lower cost.
When multiple oscillators play into the game things can become dicey. Would it be possible to use one central oscillator with very high phase stability and then derive other needed clocks from that using PLLs? A PLL isn't that hard to design, there are chips galore thanks to the wireless craze. Once you have done a few it becomes easy.
If you are critically close to EMC margins you might want to look into dithering. That is hard to do with an oscillator in a can although there are chips that do that as a hindsight task. It is easier when you design one from scratch. Here the clock is moved around the center frequency. Not as far out as to disrupt operation yet far enough to "spread" the electromagnetic noise so that the quasi peak detector registers less for each "frequency bin". Just make sure there aren't any IP or patent infringement issues with your particular application. You could compare this strategy to some of the engine hush kits where the noise is spread out into spectral areas that don't bother people as much. Listen to a large waterfall and it's a great and soothing sound. Concentrate all that sound energy into a band of a few hundred Hertz and you'd run away screaming.