It Works! In Theory…

For most of my IT career I’ve been known as The Troubleshooter. Have a problem that’s stumping everyone? Give it to Steve. Been getting the run-around with tech support? Hang up and call Steve! I may have developed a reputation.Wondering what magic I use to do it? Want to learn how to troubleshoot? Read on.

There is a trick to troubleshooting and problem resolution and it’s something I learned from my dad. Now, my dad was an amazingly good automotive electrician – best in the city. In the old days, cars didn’t have as much electronic circuitry as they do now, but there was still plenty to do: alternators, starters, ignition systems, air conditioning, all the accessories – and once customers had exhausted the knowledge of “the guy down the street”, they brought their cars to my dad. That included the guys at the local dealerships. At some point they couldn’t just keep replacing parts hoping to get it right, they had to know what was going on. Enter “The Troubleshooter”.

I started working for my dad a few years after high school; he had his own business – Cardinal’s Auto Electric. First I started working a few hours each afternoon once I got out of my other job. I’d rebuild alternators and starters, mostly. Yank ’em out, break ’em apart on the bench, test ’em, and put ’em back in.

Eventually I joined him full time and started learning the big stuff: fuel injection, computer-controlled emissions systems, etc. Sure, we did the occasional maintenance task, but we were troubleshooters – we took the tough problems and figured them out. We didn’t just replace parts hoping to get it right. We didn’t flail about wondering why we got into the business in the first place. We got down to brass tacks and knocked it out of the park. How did we do this?

Theory.

Right. My dad emphasized learning theory: how is it supposed to work. If you don’t know how it works – and I mean really knows how it works – how can you figure out why it broke? And we’re talking system level here. Sure, I learned what every part inside of an alternator did, but I also learned how the entire charging system worked and what the alternator did and what it didn’t do. I wasn’t born knowing it, I learned it.

But computer systems are so much more complex, you say! (Have you looked under the hood of a modern car recently?) How can I learn the theory?

Back in the day when we were learning automotive systems there was no Google, no Bing, no websites of any kind. We paid top dollar to get Mitchell manuals and they were occasionally flawed. What frustration! But it was important as you worked on the car to recognize that there was an engineered system in front of you and not a random bunch of parts thrown together. Computer systems are the same. There is a rhyme and a reason for a system. And that’s the key!

When you’re working on an intractable problem it’s easy to get frustrated and start flailing about. Every error message looks like it must be the cause. You can head down a rabbit hole real quick if you don’t stay on track and it’s much easier to stay on a track when you realize you’re working on a structured, engineered system.

Sure, you can joke that Windows (or Mac or Linux) just runs on hope and prayers but, even if you disagree with some of their choices, you can be sure that they were talked about ad infinitum and that they were developed with a plan. Don’t lose sight of this, because it’s your saving grace. There is underlying logic to how it works. All you have to do is figure out what that logic is and, given the maturity of software development, they’re all pretty darn similar.

My first job when I left the automotive field and started doing computers was working in a call center for Microsoft. I went through 2 weeks of in-depth training on Microsoft Word. I learned everything about it – every feature, every function. Not just how to use it, but how and why it worked that way. I eventually got promoted to the Windows NT team when they were getting ready to release 3.5. That was when I learned how their (soon to be) flagship operating system worked. You know what? It followed the exact same kind of logic as their word processor did!

It’s been around 20 years since those call center days and I’ve worked on numerous operating systems, network devices, protocol stacks, and other technological marvels and they all work in logical and predictable ways. When they work! Access to modern search engines can bring the knowledge of theory to my desktop in seconds. All I have to do is figure out what to search for and how to cull out the chaff (bad advice by people who don’t understand theory) from the grain.

If you’re having a bad time with a troublesome error, write down (or talk it out with a coworker) what you know about how the system should work. You won’t always know! Most of the problems I’ve been assigned recently are things I’ve never worked with before. Single Sign-On using SAML? WTH! But I poured over Google search results for the various errors I saw to build a picture of how the system should work. Working with custom code as I do means I also get custom error messages. Often badly written. So I can’t rely on finding a solution in a single search. I also comb through error logs – not just looking for errors but for what is working, because that’s information necessary for the big picture. Remember I said systems are logical? Logs help you uncover that logic.

There may be a certain amount of art to troubleshooting, but it’s far more a science and therefore something you can learn. If you aren’t putting your efforts into learning the theory of the systems you support you’re asking for a frustrating and limiting career. Don’t stop at how to configure a network card, learn how the network stack works. You’ll be more valuable and get a greater sense of accomplishment when you start getting called “The Troubleshooter”.

At least that’s my theory.

Steven Cardinal

Steve is the founder of Straight Talk Entertainment and currently produces and writes for the audio drama Aural Traditions, recently voted Charleston City Paper’s Best Local Podcast. He’s also an Information Security professional and avid shark tooth hunter.