Month: September 2021

Posted on

Gullible?

We are in the midst of a series of critical repairs at the Cox home. The pandemic forced us to postpone many of them, but slowly they are starting back up.  This last week involved the removal and upgrade of our failing main electrical breaker box panel.  Of course, that meant an extended power outage and the hourly “When will power be back on?” questions from my girls.  I must say, we all gained a greater appreciation for our ancestors who navigated the 1800s with gas lights, candles and no air conditioning.

Speaking of appreciating our 21st century lifestyle, I love using Apple Pay!  With the team of contractors working hard in the heat to get our power back on, I decided to make a run to our local grocery store to pick up some bottled water, Gatorade and snacks for them.  As is my custom, I paid with my Apple watch.  The customer behind me was shocked and struck up a conversation.  

I grew up in the Midwest where you expected friendly conversations with random fellow human travelers all the time.  However, that’s typically not how we do things in California.  Here, everyone tends to be more focused on their own business, mostly without even making eye contact.  But I find I still revert to my Midwest roots on occasion, much to the embarrassment of my kids, especially when there happens to be a cute baby in line with us.  I just can’t help myself.  Babies are irresistible.  In any case, I happened to run into this concerned citizen in line with me at the grocery store who was seriously worried about my Apple watch.   The conversation was really quite fun.

“Hey, aren’t you worried someone is going to steal all your information with that thing?”  I responded, “Actually, it uses an encrypted token, not my info, to complete the transaction.”   

“Like whatever, encrypted nothing, they got you!  That’s dangerous!  Can’t someone just decrypt it?”  I really wanted to start drawing a diagram to explain how it worked, but I knew the rest of the customers in line were not interested in an extended lecture.  I still switched into professor mode, “Sure, but just keep in mind, this isn’t my information directly, it is just a token identifier.”  

“Man, you really are gullible.”   I wasn’t making any progress.  He shook his head but then proceeded to pull out his credit card in plain sight.  I was able to clearly read his name, card number and expiration date printed on the front.  No, I didn’t try to memorize it but was struck by the irony.  He swiped his card with his in-clear-text magnetic stripe, also showing the CVV.  Sigh.  Yes, I guess I’m gullible.

I appreciate my friend’s paranoia, despite his negligence in protecting his own identity.  No system is 100% secure.  We know that.  Several years ago, I had the privilege of teaching a cybersecurity class at USC where we explored the anatomy of an attack.  One particular study was the 2013 Target breach. We examined all the points of vulnerability that existed in the system at that time.  It began with a phishing scheme that equipped the attackers with a contractor’s credentials to log in to the energy management system for the stores.  That led the attackers to a vulnerable Windows PC that just so happened to bridge the HVAC network with the global store network. That network was home to all the point of sale systems for all their stores.  The card readers on those systems only accepted plain text magnetic stripe data. The hackers installed BlackPOS, a malware opensource package that intercepts track data.  It began reading all of that data, sending it off to a server hosted in Russia.  They managed to extract 40 million credit cards before they were discovered.  A year later, the nearly exact same attack occurred at Home Depot, but for 56 million credit cards.

Vigilance is needed.  Reliability engineering is not just about system performance or uptime, it is also about running secure systems.  As we help design, build and run systems, this is a great reminder that we can all help safeguard our customers’ and company’s data.  Have you found a vulnerability?  Are you concerned about some missing measures or designs that should be modernized or addressed?  If so, don’t wait, raise those issues.  Speak up and act.  You can make a difference.  Let’s continue to help make our systems more secure for the good of the kingdom, our guests, businesses and fellow employees.

Posted on

Automate, Accelerate, Optimize, but first, Delete

“I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better. I think that’s the single best piece of advice: constantly think about how you could be doing things better and questioning yourself.” – Elon Musk

The Tesla Model 3 production line was too slow. Demand was high but delivery was low.  The entire line was being delayed by one particular step in the battery production line.  Specifically, it was a step where a fiberglass mat was added between the battery pack and the floor pan.  Elon Musk talks about the focus that was suddenly placed on this choke point.  In an interview he gave, he says he was basically living on that production line until they could get it fixed.

Automate, Accelerate, Optimize. To address the constraint in the system that was choking the throughput, Elon goes on to explain how they focused on the automation.  To make the robot better, they adjusted the programming.  They increased the speed from 20% to 100%, optimized the paths it would take, increased the torque, removed unnecessary motion and reduced the amount of product needed.  Instead of spackling glue on the entire mat, they programmed it to deliver dabs of glue that were just enough to hold it in place, sandwiching it between the battery pack and floor.  These all added up to some minor time savings.

After investing a lot of time into the efficiency improvements, it occurred to Elon that he didn’t even know the reason for these mats.  He asked the battery safety team, “Are these mats for fire protection?”  They answered, “No, they are for noise and vibration.”  He then went to the noise vibration analysis team and asked them, “Are these mats for noise reduction?”  They answered, “No, they are for fire safety.”  

“I’m trapped in something like a Kafkaesque or Dilbert cartoon!”  Elon discovered the mats had no reason to be included.  After verifying with testing, they eliminated the unneeded parts that were choking the Model 3 production line.  Production throughput increased.

How many times have you optimized a bit of code, a process or a system only to finally realize that the best optimization was to simply delete it?  Before we take on some new work, a new project or even an improvement effort, ask yourself and others, “Do we even need this?”  We all have limited time and resources.  Some upfront investment in validating the real need can pay material dividends.  Seek to eliminate waste.  Instead of focusing on improving unneeded processes, let’s focus our efforts on things that deliver real value and outcomes.  

Before automating, ask yourself if the time investment will deliver more value than we put in.  Before accelerating, ask yourself if the haste will actually eliminate waste.  And before improving something, ask yourself if we should just delete it instead.  Challenge assumptions so we can ultimately deliver bold results that matter.

Posted on

Heroes

“True heroism is remarkably sober, very undramatic. It is not the urge to surpass all others at whatever cost, but the urge to serve others at whatever cost.” – Arthur Ashe

It was an early and cool September morning.  I arrived at our small civil engineering office in Tulsa a little after 7:30am.  The survey crew had just left the building to stake out one of our new projects.  Thankfully they left a half pot of brewed coffee for the rest of us.  I logged in to my AutoCAD station with my cup of coffee and started planning my day.  The phone rang.  It was my wife, Jane, on the other end of the line, who added, “You need to turn on the news, a plane just crashed into the World Trade Center!”  Our first thought was our family in New Jersey, especially my brother-in-law who worked in downtown Manhattan at the New York City Rescue Mission.  He took the Path trains and Subways in and often exited at World Trade before walking to the Mission.  I wondered if he was there and if he was safe.

I quickly told the other engineers in the office what I had just heard.  One of the offices had and old CRT style TV and we managed to get it to work.  It looked a bit like a relic from the old Jetsons cartoons, complete with a bizarre looking antenna adorned with aluminum foil balls to somehow coax a better signal.  It rarely worked.  Thankfully however, this morning it picked up our local ABC station, and that was perfect.  The World Trade Towers were full screen, with smoke billowing out of the North Tower.  The entire office had gathered in the room.  We were glued to the screen listening to GMA anchors Diane Sawyer and Charlie Gibson verbalizing the same questions we all had. What was happening?  How could this happen?  

As we watched, a second plane entered the screen and crashed into the South Tower!  “We are under attack!”  That was the general sentiment in the room.  People started making phone calls to friends and family.  Nearly an hour later and we were all still glued to the news.  The Pentagon attack, then The South Tower collapsed, followed by the North Tower.  We watched the unbelievable footage of first responders, firefighters bravely rescuing people from the towers, many sacrificing their own lives.  Citizens were helping other citizens.  All of us TV bystanders felt a tug to reach through the screen and join the effort.   We couldn’t, so we just watched and watched and watched.  For the next several days and even weeks, everyone around us seemed to be plugged in to the news feed.  Offices, homes and even restaurants were all running the 24×7 coverage of the attack. The event had captured the entire conscious of the nation had awakened our hyper focus.

What was your experience on 9/11?  I will never forget that day or the weeks following.  It changed us.  Yes, we were suddenly remined at how vulnerable we can be.  But more importantly, it taught us how good we can be.  In a crisis, heroes emerge.  They enter the scene not for the drama or the recognition, but to answer the call to save and serve others.  Many did so at the cost of their own lives.  Would we do the same?   Not that we will all be asked to run into burning high-rise buildings, but we can all answer the call to serve others, help each other, and carry each other through tough times.  It’s a job that always has openings.  

In case you are wondering, my brother-in-law never made it in to work that day.  He had missed his train to accommodate some visiting relatives.  It was a frustrating inconvenience that turned into a grace.  He spent the next several months helping others, providing food, clothing and shelter from the rescue mission.  He was a hero too.