“I was a fixer, a builder – an inventor – ever since I can remember.” – Tom Scholz
I love this time of year. The mornings are nice and cool (11C) but can still warm up in the afternoon (32C). Our neighbors have decorated their homes with Fall and Halloween decorations. Target has a mile-long line of candy aisles and we do our best to replicate that in our home.
I confess, I have an addiction. No, not the candy, I can meter that fairly well. My real addiction is my obsession to fix things. I can’t stand it when something is broken or a system is down. It drives me crazy. I am compelled to get it working. This past weekend, my obsession was our new Tesla Solar system. It all began with our decision to get solar a year ago…
After research and talking with several solar companies, we decided on a small 8.5kW Tesla Solar plus Powerwall system. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. There were several good offers out there, but we were delighted when Tesla came in with the lowest price. I’m very cheap, I admit, I love a good deal. What we didn’t realize is that the way Tesla beat the market was to remove some of their overhead, mainly in the area of customer service. That translated into a lot of waiting and frustration.
Our system was finally installed last month and during commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun. They tried for hours, upgrading, rebooting and calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains. They explained that Tesla would send out a software patch to fix the Solar panels, most likely.
I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that he would contact the electricians about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the Tesla teams are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the consumer. I escalated and was finally told that they could schedule a service appointment on Dec. 14th. Seriously?!
We had all this gear in place. We had these beautiful solar panels, high tech Powerwalls and all the gear to drive it. But none of it was working. Tesla’s mistake was bringing all of this into my home. I can’t help myself. My addiction set in. I have to fix it. So, I did. I began poking around, measuring voltages, sniffing the system’s communication links, and researching all of the modules that make up the system.
I finally found the issue. I discovered a WiFi based metering system Tesla had used was defective. I reprogrammed and recommissioned the system. It came online and for the first time ever, we had full solar energy powering our house and charging our Powerwalls. Our home had gone green and we were 100% powered by the sun! I love making things work.
Reflecting back on this experience it occurred to me that this is a lot of what we do. In technology, we are often the Sherpas that help our business partners reach their destination. Sometimes it requires research, investigation, designing, hacking, and even reprogramming to arrive at a reliable and operative solution. All along the way, our partners are depending on us to be the experts to creatively solve problems, fix what is broken and deliver a working solution that helps them, our businesses and organizations deliver the best outcomes as we can.
Thank you all for being the fixers. You substantially make a difference to the world!
If you want the gory details on my investigation and fix you can read them here.
After research and talking with several solar companies, we decided on an 8.5kW Tesla Solar plus Powerwall+ system with their new high efficiency 425W panels. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. Other solar companies had good backups systems but we did not find any who would provide whole house backup. And, more importantly, we were delightfully surprised to see that Tesla came in with the best price. Having said that, we would soon discover that they seemed to have significantly reduced overhead by mostly eliminating customer service.
The Install
25 September 2021
Tesla Crew
After ordering, reviewing designs, applying for HOA and City approval, we were finally ready to get the system installed. Two different crews arrived over a span of a week to get the system installed. The panels went on first. The panel install crew was professional and friendly. However, when they left I noticed that the handsome skirts (frames) we were so excited about were only installed on the front of the house (street facing roof). It looked great but I had expected to get them on the back as well. I reported it and in just a few days a technician came out and added the skirts to the back roof panels. He explained that they usually only install the skirts on the front. So, please note, if you want skirts on all your panels, make sure you let them know in advance. Also, the skirts are only put on the left, right and bottom. There are no skirts on the top to allow heat to escape from the panels during the hot summer.
Panel Mount – Torx T30
While installing the skirts for the back, I noticed one of the panel edges was sticking up about 1/2″ higher than the rest. The technician tried to fix it but he didn’t have all the tools. He only had what was needed to install the skirts. He asked me if I had a Torx T30 driver. I didn’t but he explained how I could adjust the panels myself. I picked up a T30 at our Newhall True Value store. I climbed up on the roof and found the adjustment area. I used a vice grip on the screwdriver to get enough leverage (mostly because I’m pretty weak especially when I’m up on the roof). I was able to lower the panel 1/2″ so it was flush. It looked beautiful.
FrontBack
A week later, another crew showed up to wire it in and commission the system. This meant a day without power, but we were prepared for that. I tried not to be a nuisance, but couldn’t help but watch and ask questions. I made sure all of the crew had plenty of bottled water, Gatorade and snacks, including ice cream candy bars since it was so hot. They installed the Powerwalls in our garage and wired in the breaker panels and Tesla Gateway by the utility meter. After everything was installed, they powered it up and began the commissioning process.
The Problem
During commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun which doesn’t even show up in the app. They tried for hours, upgrading, rebooting, calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains (based on 22% charge from the factory). They explained that Tesla would send out a software patch to fix the Solar panels, most likely.
I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that our inspection would be scheduled in 3-4 weeks and he would contact the electrician about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the advisors are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the customer.
I did manage to finally get an update and a promise to further investigate the issue. While I waited, I decided to do some more research on the system to see if I could find the problem myself…
HIGH VOLTAGE WARNING: I need to stop here and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.
The Investigation
You can connect to the Tesla Gateway by scanning the QR code inside the box. It will have your phone connect to the Gateway’s access point. You will need to stay close to the gateway if you connect this way (and browse to https://192.168.91.1). However, keep in mind that it is also connected to your home network and if you know how to find the IP address, you can point your browser to that IP and login as the Installer to see more details about the system. Your browser will likely require that you ignore the security certificate warning (more on this in the observations section below) and you will need to toggle the power switch to one of your Powerwalls but it will let you in. That is essentially their 2nd factor system to ensure you are authorized. Here is what my system looked like after installation on the main screen and on the “System” screen :
The System screen also shows details about the solar generation, Powerwalls and power usage:
Below the above list was a section for “Remote Meter” that would occasionally appear. This was particularly interesting:
Remote Meter (Vxxxxxxxxxxxxxxx) CT 1 (Solar): —W
That seemed odd. Also when I clicked on the solar panel icon on the main screen, it would say “Stale Meter Data” – that had me wondering if the solar meter was the real issue.
The Fix?
I first disabled the Solar Assembly by clicking “Disable” on the System screen.
I opened up the Inverter, the box above the first Powerall. There is a small latch on the bottom that will unlock and let the panel swing up. I found a wooden dowel to prop it open so it would bang on my head the whole time I was investigating.
I noticed that there was a box on the right that had a “n” LED flashing. The code on the box was the same code that was listed as the “Remote meter” in the system’s display (the one showing no power). Some more research and I discovered that this module is a Neurio W2-Tesla WiFi based current reader that sends the solar power data to the Gateway.
Neurio was recently purchased by Generac but you can still find manuals and some models for sale online. This particular model, W2, has been customized for Tesla. It is designed to connect to the access point of the Tesla Gateway and send the solar power data.
CT-1 Amp Probe Wire
The Neurio has a wire plugged in to the top in the CT-1 (current transformer) port. I traced it over to the solar inverter where a clamp is wrapped around the solar inverter output AC line to measure the amperage. I re-seated that connector.
Antenna
I then noticed that there was an antenna jammed below it that was tucked to the left, under the massive metal inverter shield. I turned the antenna to the right, in the open unshield space.
As soon as I did these two things, the LED “n” on the box began to change and a tune started coming out of the box. It sounded like “I am connected now”. The flashing “n” became a solid blue light.
Eureka!
I went back to the System screen and re-activated the Solar Assembly by clicking “Enabled”. This takes several minutes and you will see the system go through and activate the solar arrays, test relays and impedance before the assembly comes online.
As soon as the Solar Assembly came online, I started seeing kW of power show up on the Systems screen. 5800W of power was coming in, fully powering the house and charging the Powerwalls!
Not so fast…
Sadly, just two hours later my elation was destroyed. The solar energy dropped back to zero.
I checked the inverter. Sure enough, the Neurio was flashing again. I attempt the above process again, several times, but no joy this time. It would chime and go green, but then started flashing again. Based on my research, the Neurio connects to the Tesla Gateway WiFi only. The beeps and flashes indicate that it is unable to connect to the Gateway WiFi.
One thought I had was to reach out to Neurio (which is now owned by Generac) to see if they could provide the API, pinout or schematics for the W2 device so I could troubleshoot at the firmware and component level. When I contacted them, Generac replied that the serial number for my device contains proprietary firmware by Tesla that they cannot support. They recommend that I contact Tesla at: 888-518-3752. Oh well, it was worth a shot.
The Workaround
I love a challenge. In fact, when something isn’t working, it is almost an addiction to me. I have to figure it out and fix it! So, I had two thoughts at this point. First, I wanted to see what the Neurio was actually doing. I thought about setting up an ESP8266 to be an WiFi access point to intercept the Neurio’s communication attempts with the Gateway. But before I do that, it occured to me, I wonder what would happen if the system didn’t have a solar meter at all. In my investigation, I discovered that the solar power meter feature is often an add-on or post-install enhancement. Maybe this was more of an add-on feature than a requirement?
At the minimum, I wanted to see if there are alternatives to the Neurio in the Tesla configuration. Unfortunately, there isn’t an easy way to edit this data. I discovered that settings could only be set during the initial setup time. That would require running the setup wizard again. I decided to be bold and fire up the installation Wizard. At the bottom of the system portal is the “Run Wizard” link. Of course, I clicked it.
WARNING: I’m fairly confident that you can completely break your Tesla Solar setup using the Wizard, maybe even disable power to your house permanently. It is intended for installers. I’m taking the risk, but you should consider this first and be cautious about proceeding. I’m also fairly confident I’m going to void something in the process, but if you put something in my house, fair game, I must hack.
The wizard is straightforward. It requires you to Stop the system, but the settings are mostly intuitive. When I arrived at the Meter screen, it had 3 different meters displayed. I apologize, I did not take screenshots but will update this blog if I capture them in the future but the screens are very basic.
Two of the sensors were for the built-in CT’s used to measure the power in the Tesla Backup Gateway (you can see them on the main bus if you open the Gateway panel – which I did).
Tesla Backup Gateway DANGER: High voltage – Seriously!
These tested extremely fast (subsecond) were working correctly and tested “good”. The third meter, a WiFi meter, was the Solar Meter (Neurio) and it’s status was Error, unable to connect. I clicked the connect button which reported it would take 3 minutes to configure the WiFi sensor. No shock, it didn’t work. I tried it 3 more times. The “Advanced” drop down allows you to add MAC address and IP, but this didn’t help. There was a “Delete” button. I thought it might be worth a try to delete and re-add. At the bottom were options to add “WiFi” or “Wired” CTs. I tried to add the Neurio (WiFi) again, multiple times, rebooting the Neurio occasionally to see if that would help. Nothing.
Here is where something interesting happened. The Wizard would NOT let me advance because the WiFi sensor was not healthy (connected). Hum… Well, I figured I would just have to delete it to see what other screens I could find in the Wizard. I deleted the Neurio. I advanced to the next screen and was presented with a “Warning – you do not have a solar sensor selected.” Naturally, I ignored that and continued.
Commissioned! I completed the Wizard setup and the system came back online. Surprisingly, the system screen looked basically the same but the dynamic flow diagram was actually working. There were no sensor errors or warnings. Power was flowing from the Grid to the House. It was the middle of the night so I signed off and went to bed.
I know what you are thinking. This is dangerous, right? I mean, we seemed to have removed solar power observability from the platform. Will the Gateway and Inverter sill know what to do? Well, it turns out… it does!
The Power of the Sun
Next morning, I woke to discover solar generation was charging the powerwalls and our house was completely powered by the sun! I still want Telsa to fix the Neurio or, better yet, provide some hardwire CT to monitor Solar power generation. I’m assuming that the display below means that the Gateway is computing the the solar generation based on other CTs. In any case, my workaround is in place and we now have a working system again.
As I’m looking at my phone, I realize… I’m holding the power of the sun in the palm of my hand. Yes, that is a geeky Doc Ock reference. We are now powering our home with an ancient but reliable and self-regulating, thermonuclear fusion reactor… our sun.
The Return of Tesla
1 November 2021
I gave up on waiting on Tesla to respond to me about the Neurio. I figured it didn’t matter since I had a working system. A month after the install and I still didn’t have an inspection date. Then it happened. I received a text message and email from Tesla that my inspection was scheduled. There was NO DATE or TIME given. Instead of asking, I figured it didn’t matter. We would see what would happen.
The day of inspection had arrived. A surprise knock on the door and there was Ishmael from Tesla. He explained he was there to meet with the City inspector for the final inspection. I showed him the gear, the Powerwalls, the gateway and the breaker panels. He looked at me and asked, “Did the install crew not put on the warning labels?” Nope.
This was something I had noticed after the installers left. In the Tesla plans are specific instructions on where to place the red warning labels on all of the gear. It includes a helpful diagram for anyone wanting to know how to kill all power in case of emergency. I had raised this issue with my project advisor a few times, but as usual, told me he would look into it and of course, nothing happened. I explained this to Ishmael who rolled his eyes and expressed apologies and said he would need to call to get the labels or it would not pass inspection. He would wait for the delivery and get them installed and ready for the City.
Shortly after meeting Ishmael, another Tesla vehicle pulled up. I figured it was the inspection stickers, but instead, it was Rocío, a Quality Assurance technician. She told me that her job was to make sure everything was installed correctly and running. I almost hugged her! I expressed my delight and appreciation that she would check on us. I explained everything that happened including how the installers said it must have been a Tesla software bug and gave up after trying for hours to get it work. I also told her about the Neurio hack I had done to get it working. She was shocked, sympathetic and determined to fix the issue.
Naturally, Rocío attempted to reset the Neurio and discovered the same thing that I did, with the exception that she was able to get the Neurio to work if she held the connector, pressing on it in a certain way. “There is clearly something wrong with the hardware and it needs to be replaced,” she concluded. I hate to be cynical, but I was definitely thinking this new chapter in my Tesla adventure would turn into an multi-week RMA, repair order and a return visit that may get scheduled sometime next year, if I’m lucky.
To my delight, Rocío looked straight at me and said, “And we’re going to get this fixed today!” She was right! She made a phone call and 30 minutes later another Tesla van showed up with the replacement Neurio!
Rocío got it working. Less than 30 minutes later she had the entire system back online and working correctly. “That’s amazing!” I told her. She clearly saw my astonishment and said, “I used to be an installer, I know what’s needed.” Well, that was completely accurate. She didn’t stop there. She examined all the gear and climbed up on the roof to ensure all the panels were in good order.
Shortly after the good news, the warning labels arrived and were attached to the new gear, ready for the official inspection. I started passing out my sincere appreciation, candy bars, water and Gatorade to these brilliant Tesla soldiers that had come to save the day. After bidding farewell to our new friends, Rocío drove off on her shiney white stallion… uh, I mean Tesla van.
About 30 minutes later, the City Inspector arrived and after a quick survey of the installed gear with Ishmael, signed his approval. Now we are on to the Permission to Operate (PTO) by Southern California Edison.
Power Gremlins
3 November 2021
I should definitely learn to be more guarded in my optimism about this Tesla adventure. After two days of having the new Neurio re-installed, I started noticing something odd. After solar production when I would expect the Powerwalls to kick in and power the house, I would see grid power start to show up and the Powerwalls drop to zero. It would only last for a few minutes then return to normal operations. Looked at the Powerwall Dashboard I set up and can even see the grid power spiking during the day when solar production was more than enough to power the house.
Powerwall Dashboard: Notice the purple (grid power) spikes throughout the day. These just started showing up after the new Neurio was installed.
The grid power spikes did not exist before the new Neurio. I went out to look at the inverter. The Neurio’s purple light was mostly solid but would “flicker” blue. It was random, like a candle flame not like the error condition of the previous Neurio. It was happening constantly as I watched. When the flickering would get bad, I would see the powerwall drop to zero and grid power surge. There seemed to be a correlation. At any rate, I wasn’t going to let the flickering continue.
HIGH VOLTAGE WARNING: I need to stop here again and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.
I powered off the Neurio by unplugging the power next to the antenna at the bottom. I noticed the antenna was once again tucked under the massive shield. I guess that was the typical install. I changed it so it was pointing away from the inverter shield and reworked the cables to plug it back in. The Neurio went through the startup (flashing, then solid green, then blue and then purple). I watched it for a while and noticed it stayed solid purple, no flickering.
I don’t know if this was a fix or a sign of things to come. Other people have reported similar problems with the Neurio, including a YouTube video on how to reset it the way I did. It is rather shocking how unreliable this little box is. I understand it is a “revenue grade” meter which is likely why Tesla is using it, allowing them to report “Solar Renewable Energy Credits” (SRECs). The Inverter itself seems to have a decent meter without the Neurio which is why my workaround hack worked while waiting for the Neurio replacement. If the reset doesn’t work, I will likely revisit my “fix.”
5 January 2022
The “fix” was temporary. It appears to be a resource leak that requires the Neurio to be restarted. The good news is that Tesla finally recognized the instability and sent out a 21.44 firmware update that fixed the Powerwall from disabling solar when the Neurio goes into a bad state. Finally! I was planning on ripping the Neurio out after PTO, but now I don’t have to do that. I’m currently on firmware 21.44.1 and just heard from the community that others are seeing an upgrade to 22.1 that also upgrades the Neurio from firmware “1.6.1-Tesla” to “1.7.1-Tesla” (you have to access the vitals API and decode the protobuf binary payload to see this – see here). Hopefully that helps with stability.
Permission to Operate
2 February 2022
PTO, finally! Our utility company, Southern California Edison (SCE) granted permission to operate. It took Tesla several tries to get the PTO request submitted correctly. SCE was notifying us of all the transactions but we were not able to see the full application or help. Believe me, I tried! In any case, our installation adventure has finally come to an end. It has been seven months since we started this epic journey. It is good to finally have a fully operational system.
To be fair, we have had a working solar system with Powerwall backup since October, but without PTO. PTO means that our system is no longer in self-consumption test mode. We can now push excess solar production to the grid for a credit. For the first time ever, we see grid graph going negative!
24 hour graph of energy usage including Powerwall (green) charging and pushing to the grid (purple).
Conclusion
The Tesla Solar system has been an adventure for us. I don’t regret going with Tesla even though they have improvement areas, especially related to consumer experience. We love the look of the panels and the equipment. In typical Tesla fashion, the design is stunning and feels like quality. If you do select Tesla Solar, my advice is to plan on being the project manager. Stay on top of the details to keep things moving and make sure items are not dropped.
Regardless of who you go with, I recommend you set some non-negotiables to help you filter. Here were my non-negotiables:
Aesthetics – We wanted something that looked high-tech, neat, clean and symmetrical on our roof. I wanted the dark panels (no white lines) with a clean looking frame.
Whole House Backup – We wanted batteries capable of running our house over 24 hours in the event of a power outage and a system that would charge the batteries during the day even if power was out for extended days.
Home Automation and Monitoring APIs – I wanted a system I could hack, use tools to monitor, dashboard, trend and even make decision on home automation components to optimize our energy usage.
Off the Grid – We wanted the system sized to allow us to be self sustaining with no need to use the grid even at night, fully expecting that at some point Net Energy Metering disappears or becomes less attractive, the system will still provide us with a zero grid usage option.
This helped considerably. It eliminated the list down to a handful and the lowest cost on our list was Tesla, who also had the best aesthetics IMHO. Now to be clear, as I mentioned above, Tesla is seeing explosive demand for their option and their customer service struggles a lot. I also had the opportunity to work with several incredible Tesla technicians who helped us.
I am extremely happy with our Tesla system and would recommend it to anyone, despite the bumps along the way. Tesla hit on all my non-negotiables and is elegant, fun and powerful. It has become a delightful hobby as well as a powerful utility for our green energy mission.
While a similar adventure may not be for everyone, if you are in the market for a Solar system, I still highly recommend checking out Tesla’s options. Use this link and you can save $300 if you do order and I get a reward too: http://ts.la/jason50054
I have to confess. I love toys. To me, this new Tesla Powerwall+ Energy systems is a gigantic (and expensive) toy. I have thoroughly enjoyed tinkering with the system and building electronic accessories and software to manage it. As you have seen in this post, I wrote my own python API library (pyPowerwall) and created a Powerwall Dashboard to better see what the system is doing over time (credit to other open source projects I mention below).
One thing that the Tesla is missing is a good instrument panel display. Sure, you can run the app all the time, but I wanted something that would show the solar production and other details like a physical dashboard but without opening an app. I built one. It is hanging next to the Powerwalls in our garage. Much to my wife’s initial trepidation, I also built one and hung it in our kitchen. It turns out that I’m not the only one to appreciate it… well, after a while anyway. 🙂
Here are some of my toys that I gladly share with you. Please reach out and let me know if you find these useful:
pyPowerwall Dashboard with String Data
PowerwallWeb Dashboard – The Tesla App and web based portal present great animations showing the solar generation and usage. However, the information is very limited and not design for visualizing the energy data in multiple ways. I wanted to see a year at a glance as well as the string data (how much power each group of solar panels on the roof are producing). I found this Grafana based dashboard and made some minor changes including the addition of my own python based Powerwall API Proxy. Here is a simple python API module pyPowerwall to pull data from the Powerwall Gateway using your “customer” credentials: https://github.com/jasonacox/pypowerwall. If you are wanting your own Powerwall Dashboard, it is fairly easy to set up with the instructions here using Docker Compose: https://github.com/jasonacox/Powerwall-Dashboard#powerwall-dashboard
Powerwall Digital Display
Powerwall Wall Mounted Display – I really wanted to see the current solar generation and state of the Powerwall on a simple LED digital display. I 3D printed a Tesla themed case and installed the displays to show solar, house, battery, and grid power data. The display show the solar production power at the top. House, Powerwall and Grid power data rotate through the middle display and the battery level of the Powerwalls is at the bottom (89% in this picture). You can see a video of the display running below.
The display uses a WiFi enabled systems-on-chip (SoC) ESP8266 WeMos controller and three simple TM1637 7-segment LED display modules. Naturally I used my own Arduino API library (TM1637TinyDisplay) for those and the pyPowerwall proxy to display the results. It would have been nice if Tesla had built a wall monitor to show vitals like this. I’m sure it would have been a nice animated OLED display of some sorts. But this was fun. I needed to build another toy and I love my retro-LED display. If you want to build your own, I have open sourced the design and code and uploaded here for you to use: https://github.com/jasonacox/Powerwall-Display
I took note of several areas of concern and improvement during my investigation and problem solving. I have recorded them here.
WPA TKIP Command Access Point – The Tesla Gateway uses this weaker method to host its WiFi access point. As I discovered the Neurio uses this same access point to send Solar Power data (if it works). WPA TKIP has been dropped due for security reasons and more modern access points use WPA2 and AES encryption (WPA2-AES).
HTTPS Security Certificate – The HTTPS certificate the Gateway uses will create a browser warning (or error) when you go to the system control portal, either via your home network or via the access point at https://192.168.91.1.
Second Factor – For setup, the user is required to toggle the switch on a Powerwall as a 2nd factor to prove authorization, which is a good thing. That works well for me since my Powerwalls are locked in my garage, but if your Powerwalls are outside next to the Gateway, an attacker on-location could easily join and toggle without you even knowing.
IoT Sensors – The main problem on my system was the Neurio W2 WiFi based sensor. This IoT device sends back power data it measure to the Gateway controller. Generally, this is an elegant way to handle transmitting sensor data between systems without having to wire things. The irony is that the Gateway and Inverter already have several wires and control signal between them. Why not add another wire and avoid any WiFi communication outages? Hopefully I will be able to replace my Neurio with a wired solution.
Solar System Plan – I asked the Tesla Advisor to provide me with the design plans developed for the City Permit. They do not provide this without asking. I am glad I asked. The plans have all the schematics for the wiring as well as the layout. I discovered several things that I wanted changed and was able to get them to update before they came onsite. If you wait until they come onsite, they may not have the materials to make the adjustment and, worse, could charge you for any changes.
I found the following github projects, references and diagrams during my investigation into my Tesla Solar Adventure. I’m pasting them all here to be helpful for anyone else experiencing the same problems. The information may not be directly related but could provide a clue.
“We were born to be free, to expand our horizons by going where we have never gone before, and not to hang out in the relative comfort and safety of the nest, the known. There is a place within us that is courageous beyond our human understanding; it yearns to explore beyond the boundaries of our daily life.” – Dennis Merritt Jones
My mom was a teacher. She taught 3rd grade most of her career. Growing up, my friends thought it was great that my mom was a teacher, but it wasn’t always great for me. Most kids got sent to the principal’s office for acting up or misbehaving. Not me! I got sent to my mother’s classroom. Corporal punishment was still in play and the principal proudly hung his “board of education” paddle on the wall of his office as a deterrent. I would have preferred a meeting with that “board” any day, if I could have avoided being sent to my mom’s classroom. I can still picture the horror on her face as the teacher explained my activity and then hearing the stern, “Just wait till we get home.”
It seemed like most of my “acting up” happened during recess and on the playground. I guess it was an irresistible smorgasbord of trouble waiting for me, but really, I just loved experimenting. I tested Newtonian physics of balls bouncing off all sorts of surfaces, including other kids. I loved exploring fencing techniques with pretend light saber sticks and branches. I even helped other kids prove or disprove their own theories. A cute little girl with glasses and a confident attitude once declared to me an axiom, “You can’t hit a person with glasses.” Much to her dismay, I was able to prove her wrong. I fully expected to be thanked if not awarded some scientific prize for my discovery, but instead, I was granted another visit to my mother’s classroom, one that had particular impact on me, literally.
Recess! What a glorious thing. There are a lot of life lessons that you can learn during recess. You also get a chance to see curious characteristics of our human traits on display. One particularly interesting observation my mom made was the power of fences. Having had the opportunity to work in many schools with different playgrounds, she noticed that when kids went out to play, if there was no fence, the kids would huddle together, close to the building or by the door. Nobody ventured out very far. When there was a fence, the kids would scatter and run all the way out to the edges, running up and down the fence line. Why is that?
For those kids, the fences created a sense of safety, confidence and certainty. With that in place, the kids used the entire space to explore, create adventures and have fun together. I believe there is a good leadership lesson here. We often talk about unleashing the potential of our team by empowering individuals to creatively solve problems, take on responsibility and innovate. A good leader can help us manage the unknowns by bringing clarity, direction, and expectations. They also promote psychological safety by establishing a culture that avoids the blame game, encourages risk and values continuous learning. Those structures help us navigate, explore and create results without fences. But that is only part of the story.
The leadership lesson here also applies to us individually. So much of our life and the world we live in has no fences. It can be intimidating. Our human tendencies, evoking survival instincts at times, will be to huddle close to what we know, where we feel safe, secure, confident and comfortable. But if we do that, we miss out on the greater prospects and the rewards that can be waiting for us. We need to take bold risks. We should gather our courage and set out on an adventure into the unknown. Experimenting, exploring, discovering, creating, solving and ultimately enjoying greater outcomes than we would otherwise.
Spend some time this week in recess! Examine your fence line. Ask yourself, what is helpful and what isn’t. Take a brave step and go beyond, explore outside the fences and enjoy our spectacular world. And, hopefully, you won’t end up in my mom’s classroom.
We are in the midst of a series of critical repairs at the Cox home. The pandemic forced us to postpone many of them, but slowly they are starting back up. This last week involved the removal and upgrade of our failing main electrical breaker box panel. Of course, that meant an extended power outage and the hourly “When will power be back on?” questions from my girls. I must say, we all gained a greater appreciation for our ancestors who navigated the 1800s with gas lights, candles and no air conditioning.
Speaking of appreciating our 21st century lifestyle, I love using Apple Pay! With the team of contractors working hard in the heat to get our power back on, I decided to make a run to our local grocery store to pick up some bottled water, Gatorade and snacks for them. As is my custom, I paid with my Apple watch. The customer behind me was shocked and struck up a conversation.
I grew up in the Midwest where you expected friendly conversations with random fellow human travelers all the time. However, that’s typically not how we do things in California. Here, everyone tends to be more focused on their own business, mostly without even making eye contact. But I find I still revert to my Midwest roots on occasion, much to the embarrassment of my kids, especially when there happens to be a cute baby in line with us. I just can’t help myself. Babies are irresistible. In any case, I happened to run into this concerned citizen in line with me at the grocery store who was seriously worried about my Apple watch. The conversation was really quite fun.
“Hey, aren’t you worried someone is going to steal all your information with that thing?” I responded, “Actually, it uses an encrypted token, not my info, to complete the transaction.”
“Like whatever, encrypted nothing, they got you! That’s dangerous! Can’t someone just decrypt it?” I really wanted to start drawing a diagram to explain how it worked, but I knew the rest of the customers in line were not interested in an extended lecture. I still switched into professor mode, “Sure, but just keep in mind, this isn’t my information directly, it is just a token identifier.”
“Man, you really are gullible.” I wasn’t making any progress. He shook his head but then proceeded to pull out his credit card in plain sight. I was able to clearly read his name, card number and expiration date printed on the front. No, I didn’t try to memorize it but was struck by the irony. He swiped his card with his in-clear-text magnetic stripe, also showing the CVV. Sigh. Yes, I guess I’m gullible.
I appreciate my friend’s paranoia, despite his negligence in protecting his own identity. No system is 100% secure. We know that. Several years ago, I had the privilege of teaching a cybersecurity class at USC where we explored the anatomy of an attack. One particular study was the 2013 Target breach. We examined all the points of vulnerability that existed in the system at that time. It began with a phishing scheme that equipped the attackers with a contractor’s credentials to log in to the energy management system for the stores. That led the attackers to a vulnerable Windows PC that just so happened to bridge the HVAC network with the global store network. That network was home to all the point of sale systems for all their stores. The card readers on those systems only accepted plain text magnetic stripe data. The hackers installed BlackPOS, a malware opensource package that intercepts track data. It began reading all of that data, sending it off to a server hosted in Russia. They managed to extract 40 million credit cards before they were discovered. A year later, the nearly exact same attack occurred at Home Depot, but for 56 million credit cards.
Vigilance is needed. Reliability engineering is not just about system performance or uptime, it is also about running secure systems. As we help design, build and run systems, this is a great reminder that we can all help safeguard our customers’ and company’s data. Have you found a vulnerability? Are you concerned about some missing measures or designs that should be modernized or addressed? If so, don’t wait, raise those issues. Speak up and act. You can make a difference. Let’s continue to help make our systems more secure for the good of the kingdom, our guests, businesses and fellow employees.
“I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better. I think that’s the single best piece of advice: constantly think about how you could be doing things better and questioning yourself.” – Elon Musk
The Tesla Model 3 production line was too slow. Demand was high but delivery was low. The entire line was being delayed by one particular step in the battery production line. Specifically, it was a step where a fiberglass mat was added between the battery pack and the floor pan. Elon Musk talks about the focus that was suddenly placed on this choke point. In an interview he gave, he says he was basically living on that production line until they could get it fixed.
Automate, Accelerate, Optimize. To address the constraint in the system that was choking the throughput, Elon goes on to explain how they focused on the automation. To make the robot better, they adjusted the programming. They increased the speed from 20% to 100%, optimized the paths it would take, increased the torque, removed unnecessary motion and reduced the amount of product needed. Instead of spackling glue on the entire mat, they programmed it to deliver dabs of glue that were just enough to hold it in place, sandwiching it between the battery pack and floor. These all added up to some minor time savings.
After investing a lot of time into the efficiency improvements, it occurred to Elon that he didn’t even know the reason for these mats. He asked the battery safety team, “Are these mats for fire protection?” They answered, “No, they are for noise and vibration.” He then went to the noise vibration analysis team and asked them, “Are these mats for noise reduction?” They answered, “No, they are for fire safety.”
“I’m trapped in something like a Kafkaesque or Dilbert cartoon!” Elon discovered the mats had no reason to be included. After verifying with testing, they eliminated the unneeded parts that were choking the Model 3 production line. Production throughput increased.
How many times have you optimized a bit of code, a process or a system only to finally realize that the best optimization was to simply delete it? Before we take on some new work, a new project or even an improvement effort, ask yourself and others, “Do we even need this?” We all have limited time and resources. Some upfront investment in validating the real need can pay material dividends. Seek to eliminate waste. Instead of focusing on improving unneeded processes, let’s focus our efforts on things that deliver real value and outcomes.
Before automating, ask yourself if the time investment will deliver more value than we put in. Before accelerating, ask yourself if the haste will actually eliminate waste. And before improving something, ask yourself if we should just delete it instead. Challenge assumptions so we can ultimately deliver bold results that matter.
“True heroism is remarkably sober, very undramatic. It is not the urge to surpass all others at whatever cost, but the urge to serve others at whatever cost.” – Arthur Ashe
It was an early and cool September morning. I arrived at our small civil engineering office in Tulsa a little after 7:30am. The survey crew had just left the building to stake out one of our new projects. Thankfully they left a half pot of brewed coffee for the rest of us. I logged in to my AutoCAD station with my cup of coffee and started planning my day. The phone rang. It was my wife, Jane, on the other end of the line, who added, “You need to turn on the news, a plane just crashed into the World Trade Center!” Our first thought was our family in New Jersey, especially my brother-in-law who worked in downtown Manhattan at the New York City Rescue Mission. He took the Path trains and Subways in and often exited at World Trade before walking to the Mission. I wondered if he was there and if he was safe.
I quickly told the other engineers in the office what I had just heard. One of the offices had and old CRT style TV and we managed to get it to work. It looked a bit like a relic from the old Jetsons cartoons, complete with a bizarre looking antenna adorned with aluminum foil balls to somehow coax a better signal. It rarely worked. Thankfully however, this morning it picked up our local ABC station, and that was perfect. The World Trade Towers were full screen, with smoke billowing out of the North Tower. The entire office had gathered in the room. We were glued to the screen listening to GMA anchors Diane Sawyer and Charlie Gibson verbalizing the same questions we all had. What was happening? How could this happen?
As we watched, a second plane entered the screen and crashed into the South Tower! “We are under attack!” That was the general sentiment in the room. People started making phone calls to friends and family. Nearly an hour later and we were all still glued to the news. The Pentagon attack, then The South Tower collapsed, followed by the North Tower. We watched the unbelievable footage of first responders, firefighters bravely rescuing people from the towers, many sacrificing their own lives. Citizens were helping other citizens. All of us TV bystanders felt a tug to reach through the screen and join the effort. We couldn’t, so we just watched and watched and watched. For the next several days and even weeks, everyone around us seemed to be plugged in to the news feed. Offices, homes and even restaurants were all running the 24×7 coverage of the attack. The event had captured the entire conscious of the nation had awakened our hyper focus.
What was your experience on 9/11? I will never forget that day or the weeks following. It changed us. Yes, we were suddenly remined at how vulnerable we can be. But more importantly, it taught us how good we can be. In a crisis, heroes emerge. They enter the scene not for the drama or the recognition, but to answer the call to save and serve others. Many did so at the cost of their own lives. Would we do the same? Not that we will all be asked to run into burning high-rise buildings, but we can all answer the call to serve others, help each other, and carry each other through tough times. It’s a job that always has openings.
In case you are wondering, my brother-in-law never made it in to work that day. He had missed his train to accommodate some visiting relatives. It was a frustrating inconvenience that turned into a grace. He spent the next several months helping others, providing food, clothing and shelter from the rescue mission. He was a hero too.
It was a hot and humid summer day. My hands were sweating and it wasn’t just because of the heat. It was my first job and I was terrified. I had been hired by Calvin, a local farmer, to help mow and bale hay on his fields. He had hundreds of acres, some cattle, horses and a barn. We were standing in a field with tall grass that was nearly up to my armpits. I was listening intently to his instructions. Parked before us was a huge yellow beast. The enormous tractor had towering wheels that were as tall as my scrawny 12-year-old self. I kept thinking, “This is a really bad idea!”
“You can do it! Climb up,” Calvin said enthusiastically with a broad smile. I climbed up the yellow mountain and sat in the well-worn black leather seat. I could hear and feel my heart pounding in my ears. He climbed up next to me. “Now step on the clutch and turn the key.” He pointed at the big metal petal on the left.
My leg was shaking uncontrollably as I stepped on the clutch. The pedal went down with a loud satisfying clunk. My relief was short lived as the pedal pushed back at me with enormous force. It reminded me of one of those spring riders you could sit on as a kid at the McDonalds playground. Just about the time you thought you had wrestled it to the ground, it would whiplash you back up with incredible speed. The same happened with the pedal. It literally shot me out of the seat. Calvin burst out laughing. I grabbed the steering wheel with my sweaty hands. Using that leverage, I was able to mash the pedal down again. It stayed! “Good job!” he exclaimed, “Now turn the key.” I did. The entire beast roared to life. I was somewhat relieved to think that my nervous shaking was suddenly masked by the bone jarring violent rattling of this yellow dragon. Calvin began working a lever on the console that revved up the engine. “You want to set it at this RPM,” he said, pointing at the tachometer. It smoothed out.
“Now, put it in first gear,” Calvin motioned to the gear shift, “and let out the clutch slowly.” My brain refused to accept those instructions. I managed the gear shift, but there was absolutely no way there would be any “slowly” about my ability to let out the clutch. The death grip I had on the steering wheel had turned my knuckles white. “You can do this!” he coaxed. The next part is a bit blurry in my memory. I tried. My foot edged back slowly, shuddering and shaking, until my muscles finally surrendered. Pow! Kaboom! The leviathan leaped forward nearly hurtling Calvin and I off the beast. But thankfully, it stopped as suddenly as it began. I had popped the clutch and killed the engine. I was convinced I had also just emptied my entire bladder in the well-worn leather chair. My face was flush and I didn’t know if I should laugh or cry. I was sure I broke it and Calvin was going to be angry. I looked over at Calvin. He was laughing so hard he couldn’t breathe. I laughed too.
“Well, son, we are going to need to build up the muscles in those skinny legs of yours before we can get much further,” Calvin said with a grin. With a glimmer in his eye, he added, “You will be driving that thing before you know it!” I was given instructions to sit on the yellow beast and just work on pushing down the clutch until I could do it smoothly. “I’ll come back for you at lunch.” Calvin jumped off and headed to the barn.
Needless to say, I was relieved. I followed his instructions until I could execute it without failure. He was right. It wouldn’t be long before I was mowing and baling hay. I worked all summer and even returned to help the following summer. Calvin made a big impression on me. He taught me how to drive a tractor, but more importantly, he taught me about encouragement and trust. I think back to those moments where I was afraid, uncertain and unskilled, yet I had a friendly guide who believed in me. He firmly and patiently encouraged me to go beyond my ability and do something greater, something I didn’t even believe I could do myself. He taught me to face my fear and turn setbacks along the way into humor and teachable moments.
I’ll always remember that big yellow beast, my somewhat soiled pants and those dusty hot summers on the farm. But most of all, I will remember Calvin. He taught me the power of encouragement and trust. Be a Calvin. Encourage someone today!
“Sometimes life hits you in the head with a brick. Don’t lose faith. I’m convinced that the only thing that kept me going was that I loved what I did. You’ve got to find what you love… Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work.” – Steve Jobs
Sunday scaries, anyone? This weekend I happened to come across an article that talked about how the majority of workers and students are often seized by anxiety, dread and sadness that the weekend is ending and Monday is on the way. The uncertainty and the dread of the new week can effectively consume and ruin the whole Sunday. Have you ever experienced that? I have. It can be something as simple as, “Oh no, I need to get ready for that thing on Monday!” Or it can be, “I wonder what new monster or demon is waiting for me in my inbox or possessed some time slot on my calendar?”
Living for the weekend? I have fond memories of a security guard in one of our offices who loved to remind everyone on their way to the elevator, how many days we had until Friday. His sympathy for the first of the week and growing enthusiasm past “hump day” was simply delightful. I always looked forward to seeing his smiling face, regardless of the day. He was channeling the feeling that so many of us felt, counting down the days until the freedom of the weekend. But, if we measure our quality of life by the 2 of 7 days, we are only living 29% of our potential.
I love working for a company that creates happiness. What we do is great work. The magic we ship delights, informs, entertains and inspires people all over the world. More importantly, I love working with our amazing teams. So, why do we face the Sunday scaries? I noticed that there were things I could do to minimize if not remove those scaries. I’m still learning and adjusting, but I wanted to share with you some of the tricks I found to help make the workweek more like the weekend.
Plan. The biggest spike of anxiety for me was a sense of being unprepared for the week. At first, I started logging in on the weekend to prep for the week. This was a bad idea. Not only were the scaries robbing my joy on Sunday, I was suddenly working 7 days a week and setting a terrible example for my team. I changed. I started investing time on Friday, including booking time for myself to go through the following week. I look through and adjust meetings. I create new meetings and add planning time to give myself time to review each day before it gets started. If I need to work on a project the following week, I will create a “placeholder” template document or start a list of “to do” items. I also ensure I create space on my schedule for the following week to start the day with reflection, review and preparation.
Transform dread into anticipation. I try to schedule some “fun” activities each week, and especially on Monday. I also realized I was overbooking myself, not just on the weekdays but on the weekends as well. I had to learn to be compassionate with myself, pacing my activities and treating myself with some leisure time. Those things along with the planning, made an incredible difference for me. I rarely dread Monday now and even have some sense of positive anticipation, wanting to dive into things I am ready to take on. To be fair, there are still moments the scaries return, but they are fewer and easier to defeat.
How do you deal with Sunday scaries? I would love to hear your own experiences and any discoveries you have made. We are part of something special. Let’s keep doing great things, and help vanquish the Sunday scaries along the way!
The London Underground, commonly known as “The Tube” is the oldest rapid transit system in the world. A few years ago, during a summer before the pandemic, my family and I made our way to London for a vacation. Our excursions almost always required at least one ride on the Tube. We absolutely loved it. It was incredibly convenient! With just a little practice and a route guidance app, you can get just about anywhere in London with very little effort.
Anyone who has traveled the Underground quickly becomes familiar with the public service announcement, “Mind the gap!” A loud audio warning is accompanied by visual signs to remind Underground passengers to take caution crossing the gap between the train doors and the station platforms. For some of the older lines and stations this can be a significant gap which can include nefarious vertical steps. Much to my chagrin, somewhere along my journey, my less than graceful physical skills caused me to bump my toe on one of those vertical gaps. I stumbled out onto the platform. A friendly fellow passenger, concerned about my maneuver, smiled and repeated, “mind the gap.” We both laughed and I shook my head walking off with my much-embarrassed family.
Life is full of gaps. Some of those gaps are risks that we must manage for ourselves and others. Sometimes we see them and raise the alarm. Other times, those warning signs are broadcast by experts. Put on your seat belt. Hot, don’t touch! Wear safety goggles and other personal protection equipment (PPE). Get vaccinated. Walk carefully on wet slippery floors. Don’t fly your drone around high voltage power lines.
Another familiar public service announcement on the Underground is, “See it, say it, sorted.” It’s a call to action to all passengers to report unusual activity to keep everyone safe. As technologist, we are often in the front row seat to see systemic problems, reliability issues, security weaknesses, and the like. Be on the lookout for those dangers and gaps. Raise the alarm if you see a problem. Often the issues are not just technical problems, but issues with process, support and usability. Those are important too. Improving inhumane experiences or user frustrating processes is like an ancient magic that unleashes incredible power and potential. It also helps clumsy dads exit the Tube safely without bruising their egos.
As we journey from point to point on this adventure, we will spot gaps. Our attention and expertise are needed to ensure we can continue to ship value… better, faster, safer and happier for everyone. Mind the gap!
The famous Greek mathematician, physicist and astronomer, Archimedes had been given the task to verify that the king’s crown was made of pure gold. The king suspected the goldsmith had somehow cheated him, perhaps by mixing in a cheaper metal like silver. But he had no way of proving that so he asked Archimedes to figure it out. One day Archimedes was contemplating this problem while taking a bath. He happened to notice how the water was being displaced as he stepped into the full tub, spilling out all over the floor. He remembered that silver weighs less than gold by volume. It suddenly dawned on him that if he were to take the same amount of pure gold by weight as the crown, and put it into water, it should displace (spill) the same amount as the crown. Archimedes was so thrilled with this discovery that he immediately hopped out of the bath and ran to tell the king, exclaiming “Eureka!” which means, “I found it!”. In case you are curious, when Archimedes tested the crown, he discovered that it displaced more water than gold, indicating it was less dense (not pure gold). So, indeed the king had been cheated by the goldsmith. You can probably guess what happened to the goldsmith!
Learning is hard. There really isn’t a way around it. If you want to learn something, it’s going to take effort. I often use the excuse that the human brain is optimized to save energy. We build models and synaptic connections to do things “without conscious thinking.” We process huge amount of sensory data every day. We are faced with a plethora of problems we need to solve. It would be overwhelming if it wasn’t for these optimized unconscious neural pathways that allow us to sort, react and perform our tasks without much thought.
Learning builds more capability. At some point in our past, we learned something new and, Eureka! That learning was forged in our brains. It allowed us to perform our duties while we engage our higher brain functions for more important tasks like daydreaming, pondering the next season of Loki, or wondering what’s for dinner. Ok, so maybe those aren’t more important tasks, but you get the idea. By leveraging our learning, we expand our capacity to respond well to incoming tasks, difficult challenges, complex changes and even enjoyable exercises.
Learning builds on learning. I know that sounds a bit meta, but if you examine your own experience, you know that learning builds pathways to future learning. I remember the first time I learned to program a computer. It was hard! I was 12 years old and wanted to make my new computer display a Christmas tree for the holidays. Somewhere in the midst of typing in some code from a Dr. Dobbs Journal article, a Eureka moment hit and I understood the procedural flow that was happening. I had looked at more advanced programming techniques but they were out of reach for me, at least until I hit that Eureka moment. Suddenly that complexity was unlocked. That eventually led me down the path to discover microprocessor design, compiler construction and operating system development.
I recently purchased some dev kits, including a LIDAR kit. This past weekend I decided to learn how to use it to image my room as a stepping stone to my larger robotics navigation project. The funny thing about learning is that it often takes you on roads you didn’t expect to go. LIDARs are basically spinning measuring devices that use a laser to measure distance and send back angle and distance data. I wanted to visualize what the LIDAR was reading but the kit didn’t provide any imaging tools. So, I decided to learn OpenGL to render the output on my Mac. That became an exercise in itself but by the end of the weekend, I had a working project (see https://github.com/jasonacox/OpenGL-LIDAR-Display). It was challenging and frustrating at times. But as with any good learning effort, I had a Eureka moment that unlocked excitement and plans for future learning. I’m looking forward to the next phase!
What are you learning? When was the last time you had a Eureka moment? If you haven’t already, make plans this week to tackle something new to learn. Keep learning!
