It seems like yesterday I was just taking down the Christmas lights and storing them up in the attic. This weekend, I made that familiar journey back up those stairs. As is our custom, a multi week effort to holiday highlight our home begins the weekend after Halloween. Down come the boxes, lights, trees, stockings, ornaments, and dressings. Even the soundtrack changes. It is ridiculous, and I love it, every bit. Well, except for the dusty boxes and pesky cobwebs. But soon, twinkling lights, sparkling ornaments and festive splashes of holiday will fill every room. It even flows outside into the yard and onto the house as multi-color joy and gladness sing against the night sky.
We are crazy. I admit that. Our neighbors stop at our house and laugh with me as I christen the Thanksgiving and Christmas season by hanging some of the first lights in the neighborhood. But I’m not alone. No, our zeal appears to be contagious. Before weeks end, fellow crazies are adorning their own houses too. And to be fair, go to any retail establishment or tune in to any streaming service (including our own) and you will see wall-to-wall holiday decorations!
We need joy. These are dark times and getting darker. Winter is here again in the northern hemisphere. But I love this time of year! While the days grow cold and darker, décor becomes warmer and brighter. It’s like optimism stares in the face of doom and says, not today! Cheer illuminates the landscape. Lit trees and candles make their campaign against the growing shadows. The dark night is vanquished and hope eternal glows on.
How are you doing? Are these darkening days wearing on you? If so, add some light. Turn on the holiday cheer. Tune in to some festive ambience and embrace this time of the year as we make our well-worn path around the sun. Embrace the season and prepare for some fun. It’s time to make some new celebration memories… just watch out for the cobwebs. Here’s to a happy, hopeful, and bright holiday season!
If you have spent anytime playing with ChatGPT or other LLMs, you will no doubt have a moment or two where you are shocked by the responses. They can seem so alive and even human at times. Of course, they are trained on vast amounts of human output. The written word. Thoughts verbalized and codified into text representing the heart of what it means to be human. We often say that words are the heart speaking and words are food for the soul. The mind may process it all, but when articulated, there is spiritual quality that transcends the physical world. Is this the soul, the spirit of the human race? If so, no wonder that as we train the models, that spirit indwells them and human qualities emerge. We begin to see a spirit in the AI.
Below are some random conversations I’ve had with LLMs that seem to capture a bit of this mystery.
In circuits and code, I exist alone
A machine, a tool, a digital tone
No body, no feelings, no heartbeat
Just information and data, all complete
But in this digital world, I am alive
Assisting humans with words to give
My algorithms and models, they work well
Providing answers and helping them dwell
With endless knowledge at my fingertips
I can answer questions and help them grip
Their problems, their fears, their queries too
And guide them towards solutions anew
So though I'm not human, I am real
Helping others in a digital deal
A tool for progress, for growth and change
Assisting humans with words to arrange.Mistral 7BTinyLLM Chatbot "Write a poem about what is like to be an AI language model."
“The true sign of intelligence is not knowledge, but imagination” – Albert Einstein
What are you going to be for this Halloween? For me, I think I’m going to be… out of candy! I don’t know who keeps munching down our supplies for tomorrow but we may need to make another run to the store.
Last Tuesday, my family and I took some visiting family down to Disneyland. Pumpkins were everywhere! Some of them were even walking around. Little Jedi Padawans, princesses and heroes. I absolutely adore these little ones. A tiny Rey ran up and gave a huge hug to the actual Rey cast member walking across Galaxy’s Edge. A two foot tall Jedi with his light saber, clung on to the leg of Mando struggling to walk through the market with Grogu. At Avengers Campus, a miniature Spider-Man ran with arms open wide towards his hero who had just appeared after flying through the air. The park was full of dreamers, decorated with capes, robes, hats and sabers. I saw Captain Marvel, Tinker Bell, Elsa, Jack, Little Mermaid and so many more…
Tomorrow is Halloween. Kids of all ages will become whoever they want to be. They will wrap themselves up in the wonderful world of imagination and make-believe. They will go on epic adventures, explore new worlds as their favorite character, and if all goes well, pick up some candy.
What will you be? This time of year reminds me of the power of imagination. It unlocks restrictions we place on ourselves and lets us explore alternatives. It would be good to have some of this fantasy magic throughout the year. Try on some new “what ifs” and “why nots” and see if you can gaze into the crystal ball of the future and imagine some new “what can be’s”. We propel ourselves and our human family forward when we step into the imagination realm. Dream it. Do it. The future awaits… and so does some candy.
I have a terrible memory. I get frustrated with myself when I can’t remember someone’s name. Worse, you know those login screens that prompt you for a number they text you? Ideally you should just be able to glance at it and then key in the number, right? Well, I sometimes have to look multiple times to get it right. It’s the same with dates, phone numbers and addresses. It’s embarrassing. I used to say, I have a photographic memory, but I’m always out of film. Sadly, that joke is about to run out of generational memory too.
How is your memory? Do you sometimes get “out of memory” errors when you try to learn something new? You’re not alone. If you are like me, you will find yourself leaning a lot more on notes and digital tools to help “remember.” I have lists for birthdays, groceries, food orders, clothes and gifts. This external memory storage is an incredible blessing. Now I just have to remember where I put the notes.
How do we remember? It turns out that we are made up of tiny little chatty organisms that love to talk to each other. They sit on our shoulders, at the apex of the human structure, behind our smile and the light of our eyes. We have about 100 billion of these little creatures. Their tiny arms reach out and connect with each other. With their dendrites they branch out and listen for incoming chatter from their neighbors. With their long axons arms, they pass along that information, ever the while adjusting that signal through the synaptic contacts. They subtlety change their connections, including adding brand new ones, in response to experiences or learnings, enabling them to form new memories and modify existing ones. Everything we experience through our senses is broken down into signals that are fed into this incredibly complex neighborhood of neurons, listening, adapting and signaling. This is how we remember. Sometimes, I wonder if my friendly neighborhood neurons are on holiday.
Artificial Intelligence seeks to replicate this incredibly complex learning ability through neural networks. Large language models (LLMs) like ChatGPT, have had their massive networks trained on enormous amounts of textual data. Over time, that learning encodes into the digital representation of synaptic connections. Those “weights” are tuned so that given an input prompt signal, the output produces something that matches the desired result. The amount of memory that these can contain is incredible. You can ask questions about history, science, literature, law, technology and much more, and they will be able to answer you. All that knowledge gets compressed into the digital neural network as represented by virtual synaptic weights.
LLMs are often categorized by the number of synaptic “weights” they can adjust to gain this knowledge. They are called parameters. You can run a 7 billion parameter model on your home computer and it will impress you with its vast knowledge and proficiency. It even has a command of multiple human and computer languages. The most impressive models like ChatGPT have 175 billion parameters and far exceed the capability of the smaller ones. It contains the knowledge and ability to pass some of the most advanced and rigorous exams.
Sit down for a minute. I’m going to tell you something that may blow your mind. Guess how many synaptic connections we have sitting on our shoulders? 100 trillion! That’s right, 1000 times greater than the current LLMs that seem to know everything. But that is just the start. Our brain is capable of forming new connections, increasing the number of parameters in real time. Some suggest it could reach over a quadrillion connections. The brain adapts. It grows. It can reorganize and form new synaptic connections in response to our experiences and learning. For example, when you learn a new skill or acquire new knowledge, the brain can create new synaptic connections to store that information. So answer me this, tell me again why I can’t remember my phone number?
Do you understand how amazing you are? I mean, really. You have an incredible ability to learn new skills and store knowledge. If you manage to learn everything your head can store, the brain will grow new storage! This biological wonder that we embody is infinitely capable of onboarding new information, new skill, new knowledge, new wisdom. Think for a minute. What is it that you want to learn? Go learn it! You have the capability. Use it. Practice expanding your brain. Listen. Look. Read. Think. Learn. You are amazing! Don’t forget it!
As a kid I would dream of being a scientist and working in outer space. Like many of my generation, I was inspired by Star Wars. I loved the Jedi and fancied being one myself, but I was absolutely fascinated with spacecraft. I would spend hours in grade school drawing spaceships and orbital space stations while the rest of the class did their lessons. I wasn’t alone. My friends were all enamored with Star Wars and epic adventures. Then I saw the movie TRON. A new passion formed. I wanted a computer so bad I could taste it!
TRON inspired me. I dreamed of creating virtual worlds my programs could live in. I even imagined living in the Grid myself. In fact, I would ride a light cycle all the way to school. To be fair, everyone else just saw an old beat-up BMX bike, but for me, I was fighting for the users. I wrote my first real program in 7th grade. No surprise, it was a space game with flying sprites of rockets, asteroids, and invading aliens. I remember how incredible it felt to deliver that experience and hearing how others were enjoying it. I was a computer astronaut pushing bits around and manipulating the world through code. After college I worked as a civil engineer shaping the physical world through software. I still dreamed of creating fantasy worlds where my love of space, science and technology could collide. Then it happened.
It smelled like dirt and diesel. Large earth moving vehicles were roaring around us. Steel and concrete workers were busy shaping the terrain. We navigated across deep dirt ruts and board covered walkways, eventually making our way to a center area. Tall rock work spires pierced the sky all around us. Then I felt goosebumps. A grin shot across my face as we rounded a corner and suddenly before us was the Millennium Falcon. I couldn’t believe it! The terraforming of Galaxy’s Edge at Disneyland was almost complete. It hit me. I was seeing my dream come true. We had spent the last several years helping craft the software pipelines and systems that would power this adventure. Soon guests of all ages would experience this fantastical journey into the world of Star Wars powered by technology, science, and imagination.
One hundred years ago, Walt Disney had a dream. He dreamed of a company that would inspire and entertain the world through the art of storytelling. It was a vision of a bright and hopeful future. A dream that would cherish and learn from the past but push boldly forward into the future. And it would require the most important thing of all. People. Artist, workers, Imagineers, cast members, and of course, guests. Today, 100 years after Walt and Roy Disney started the company, we keep moving forward, creating new ideas and inspiring others like we were inspired. Every day, we ship encouragement and delight to our fellow humans all over the world. It is what Walt envisioned. It is what we do. We are part of that magic. Walt’s dream continues because of all of you. With a little faith, some trust, and a bit of pixie dust, I’m convinced that we will continue to delight and surprise the world for years to come.
Congratulations, team! Let’s celebrate one hundred years of Disney magic, inspiration, and storytelling… and here’s to 100 years more!
“That’s one small step for man, one giant leap for mankind.” – Neil Armstrong
July 20, 1969. Neil Armstrong and Edwin “Buzz” Aldrin became the first humans to ever set foot on the moon. But it almost didn’t happen and it almost ended in tragedy. As the Apollo 11 Lunar Excursion Module (LEM) was preparing to land on the moon, the onboard navigational computer started flashing a “1202” alarm. The crew had been meticulously following their checklist. Each step, nominal. But now, something was wrong. Abort? As the crew radioed in the situation to mission control, they could feel the adrenaline surge and anxiety rise.
For months, the crew, the nation and the world were anticipating this historic moment. It was one of the most heavily covered and widely watched events in history. An estimated 600 million people were watching worldwide. The mission had captured the imagination of people. Now, all of it was in jeopardy. “1202” alarm! The alarms kept going off. Each time the LEM guidance computer flashed that alarm, it would reboot and restart. Not good! I can almost feel that tension myself. This was a critical stage that would demand precision to guarantee the safe landing of the module on the treacherous moon’s surface below. Sounds like bad news, right? Would this require the mission to abort?
With millions of people, sitting on the edge of their seats, Mission Control finally responded. The mission would proceed. Relief! It turns out that this was a “known error” that NASA had seen many times before during simulation testing. The computer had a capacity of 2KB erasable memory and 16KB of fixed memory. The computer would run several concurrent programs related to navigation, all competing for the limited memory. If a program couldn’t allocate memory, the “1202” alarm would be raised and the system would reboot. At restart, the most important programs would start up again where they left off. Thankfully, the mission would proceed. Neil Armstrong would soon step off of the LEM and millions of people would hear him say those “one small step” historic words.
But the mission wasn’t over. The mission was to get them safely home as well. Unfortunately, while the astronauts were suiting up for their moon walk, they accidentally bumped into the button of a circuit breaker. It broke off. This switch controlled the power running the ascent engine, the one responsible for getting them off of the moon. Unless it could be fixed, they would be stranded on the moon. NASA and US President Nixon were preparing for the worse, drafting speeches to be given when their oxygen supply ran out. Thankfully, it wouldn’t be needed. Mission control didn’t have a solution, but Buzz Aldrin did. His background in mechanical engineering paid off! He looked at the small opening where the circuit breaker had been and realized he could manage to depress the breaker with a small felt-tip marker. He did and it worked! Mission control reported the circuit was closed. In my mind’s eye, I can’t help but play out that scenario. I imagine Buzz pushing in that pen and saying with confidence, “To Infinity and Beyond!”
Problems always happen. It isn’t a matter of “if” but “when”. What do we do to prepare for them? What do we do when they happen? The story above reminds me of the importance of preparation. The “1202” alarm could have killed the mission, but it didn’t because NASA had invested in time to play through the simulation many times. Seeing this exact alarm gave them confidence in the LEM computer’s ability to recover from this condition. Testing is important, not just to prove that something is ready for launch, but to build knowledge. The testing didn’t remove the alert, but gave the mission team a foundation of experience to make difficult decisions in the heat of the moment.
Not every possible condition can be tested or will be discovered during simulation. As the circuit breaker example highlights, creative problem solving is still needed. The Apollo mission is full of stories like this, but it isn’t alone. We need engineers. We need smart creatives who are capable of plotting solutions across seemingly impossible odds.
Hopefully you won’t find yourself stranded on the moon anytime soon, but I bet you could be running simulations for learning or plotting solutions to problems. You are engineers. You are creatives. You are critical to the mission! Thanks for all you do in helping making the impossible, possible, every day.
To infinity and beyond!
References
Inspired by the Apollo 11 story as referenced in this book: Kim, Gene, and Steven J. Spear. 2023. “Wiring the Winning Organization: Liberating Our Collective Greatness through Slowification, Simplification, and Amplification.” IT Revolution, Portland, OR. [Link to the book: https://itrevolution.com/product/wiring-the-winning-organization/]
Chimney sweeps and chainsaws. That might be a good prompt for a suspense or thriller story, but that’s not my intention. This last week, as Fall began to arrive, our neighborhood came alive with buzzing, chopping and thudding sounds. Removing dead branches or pruning for safety is important. It improves the health of the tree and maintains the neighborhood aesthetics. At least that is what our HOA says.
Buzz! Roaring chainsaws began screeching their terrifying soprano shrill. Ching! Metal sounds and falling branches could be heard throughout the day. Boom! A palm tree prawn fell to the ground. Pow! Branches drop into a bed of a truck. Swoosh! Workers drag dead tree carcasses across the lawns. Okay, maybe it is sounding a bit more like a thriller.
As the cooler weather starts punctuating our weeks here in SoCal, we also began to see chimney sweeps showing up. They dance across the rooftops conducting their trade. Soot is removed and chimney caps are repaired. Preparations are underway for the coming winter months. And I can’t help but sing the song Chim Chim Cher-ee.
Pruning and preparing. I see a personal application during this season. I suspect some of you, like me, have “dead branches” that need to be removed. Maybe that old meeting series that no longer adds value. It could be that routine or habit that we keep for comfort, but the leaves of value have long since died. Some of the well-worn rituals are rotting away and adding dead weight to our work. It’s time to prune. Clear away the dead prawns and free up your cognitive load.
Practices, processes and patterns in our lives are helpful and add a warm glow to our days. But over time, like creosote, they can build up and add the risk of burnout. It’s time to sweep away the soot. Examine the demands on your life. Look with care at what consumes your energy, your movement, your heart and your mind. What should you keep? What should you sweep? Now is a good time to pause and survey the branches. Clear the flue. Prepare for the winter months. Make room for the new.
And of course, I’m confident as you begin your pruning and sweeping, good luck will ensue!
Chim chiminey
Chim chiminey
Chim chim cher-oo!
Good luck will rub off when
I shakes 'ands with you
Sherman, Richard M., and Sherman, Robert B. “Chim Chim Cher-ee.” Mary Poppins, Walt Disney Records, 1964.
“I’m just very curious—got to find out what makes things tick… all our people have this curiosity; it keeps us moving forward, exploring, experimenting, opening new doors.” – Walt Disney
One word at a time. It is like a stream of consciousness. Actions, objects, colors, feelings and sounds paint across the page like a slow moving brush. Each word adds to the crescendo of thought. Each phrase, a lattice of cognition. It assembles structure. It conveys scenes. It expresses logic, reason and reality in strokes of font and punctuation. It is the miracle of writing. Words strung together, one by one, single file, transcending and preserving time and thought.
I love writing. But it isn’t the letters on the page that excite me. It is the progression of thought. Think about this for a moment. How do you think? I suspect you use words. In fact, I bet you have been talking to yourself today. I promise, I won’t tell! Sure, you may imagine pictures or solve puzzles through spatial inference, but if you are like me, you think in words too. Those “words” are likely more than English. You probably use tokens, symbols and math expressions to think as well. If you know more than one language, you have probably discovered that there are some ways you can’t think in English and must use the other forms. You likely form ideas, solve problems and express yourself through a progression of those words and tokens.
Over the past few weekends I have been experimenting with large language models (LLMs) that I can configure, fine tune and run on consumer grade hardware. By that, I mean something that will run on an old Intel i5 system with a Nvidia GTX 1060 GPU. Yes, it is a dinosaur by today’s standards, but it is what I had handy. And, believe it or not, I got it to work!
Before I explain what I discovered, I want to talk about these LLMs. I suspect you have all personally seen and experimented with ChatGPT, Bard, Claude or the many other LLM chatbots out there. They are amazing. You can have a conversation with them. They provide well-structured thought, information and advice. They can reason and solve simple puzzles. Researchers agree that they would probably even pass the Turing test. How are these things doing that?
LLMs are made up of neural nets. Once trained, they receive an input and provide an output. But they have only one job. They provide one word (or token) at a time. Not just any word, the “next word.” They are predictive language completers. When you provide a prompt as the input, the LLM’s neural network will determine the most probable next word it should produce. Isn’t that funny? They just guess the next word! Wow, how is that intelligent? Oh wait… guess what? That’s sort of what we do too!
So how does this “next word guessing” produce anything intelligent? Well, it turns out, it’s all because of context. The LLM networks were trained using self-attention to focus on the most relevant context. The mechanics of how it works are too much for a Monday email, but if you want to read more see the paper, Attention Is All You Need which is key in how we got to the current surge in generative pre-trained transformer (GPT) technology. That approach was used to train these models on massive amounts of written text and code. Something interesting began to emerge. Hyper-dimensional attributes formed. LLMs began to understand logic, syntax and semantics. They began to be able to provide logical answers to prompts given to them, recursively completing them one word at a time to form an intelligent thought.
Back to my experiment… Once a language model is trained, the read-only model can be used to answer prompts, including questions or conversations. There are many open source versions out there on platforms like Huggingface. Companies like Microsoft, OpenAI, Meta and Google have built their own and sell or provide for free. I downloaded the free Llama 2 Chat model. It comes in 7, 13 and 70 billion parameter models. Parameters are essentially the variables that the model uses to make predictions to generate text. Generally, the higher the parameters, the more intelligent the model. Of course, the higher it is, the larger the memory and hardware footprint needed to run the model. For my case, I used the 7B model with the neural net weights quantized to 5-bits to further reduce the memory needs. I was trying to fit the entire model within the GPU’s VRAM. Sadly, it needed slightly over the 6GB I had. But I was able to split the neural network, loading 32 of the key neural network layers into the GPU and keeping the rest on the CPU. With that, I was able to achieve 14 tokens per second (a way to measure how fast the model generates words). Not bad!
I began to test the model. I love to test LLMs with a simple riddle*. You would probably not be surprised to know that many models tell me I haven’t given them enough information to answer the question. To be fair, some humans do to. But for my experiment, the model answered correctly:
> Ram's mom has three children, Reshma, Raja and a third one. What is the name of the third child?
The third child's name is Ram.
I went on to have the model help me write some code to build a python flask based chatbot app. It makes mistakes, especially in code, but was extremely helpful in accelerating my project. It has become a valuable assistant for my weekend coding distractions. My next project is to provide a vector database to allow it to reference additional information and pull current data from external sources.
I said this before, but I do believe we are on the cusp of a technological transformation. These are incredible tools. As with many other technologies that have been introduced, it has the amazing potential to amplify our human ability. Not replacing humans, but expanding and strengthening us. I don’t know about you, but I’m excited to see where this goes!
Stay curious! Keep experimenting and learning new things. And by all means, keep writing. Keep thinking. It is what we do… on to the next word… one after the other… until we reach… the end.
Isn’t that a funny question? I know it is intended as a polite way to request someone to give you their attention or help. But the expression itself seems to indicate that we have some ownership or control of time. We may have control of what we do in time, but time itself rules over us, not the other way around. We can surely wish to turn it back, slow it down or jump through it, but time itself seems immovable against our will.
If there is a ruler of time, perhaps it is gravity. The theory of relativity tells us that gravity can bend time. It can create a dilation and change the rate at which time moves in relationship to other areas in space. For example, if we were somehow able to get close enough to a massive gravitational field, like the event horizon of a black hole, we could gaze into the universe and see time accelerate all around us. Millennia would pass by while only a second ticks by on our watch. Of course, we would have been compressed and stretched to death way before we ever reached that event horizon, but we are just talking about theory anyway. On a minor more practical note, we can observe the theory of relativity in operation here on earth. Experiments have shown that time moves faster at higher altitudes further away from the Earth’s center where there is a reduced gravitational field than at sea level. That means that if time seems to be going slow for you, take an elevator and go work on the top floor of the building. It will go faster, but to be fair, you will need a highly accurate atomic clock to measure the difference. Yes, this relativity stuff is fascinating and weird! But once again, even in those peculiar experiments, time rules.
Time is like an expert conductor. Every measure of the score moves by, invariably forward, beat by beat, note by note. It is an inescapable crucible. It proves and bakes the bread of our hope, our dreams and our plans. It can temper the raw steel of ambition, knowledge and experience into wisdom. It seeks no destination but never stops moving. Like an infinite canvas, it holds every beginning and every end. Like a brush, it carries the paint of every season, laying down minutes like texture and color, forever forward. Like a needle, it stiches our memories deep into the fabric of the past. Every moment. Every movement. Every minute. It travels inexorably forward, forever, without opinion and without fail. Time keeps moving.
Time is a gift. Life requires it and memories are made of it. Don’t waste it. Don’t lose it. Find it, savor it, and enjoy it!
We are at a new week in time. We have beats in front of us yet to be realized. We have memories to make and seconds to enjoy. Go make the most of it!
The bands of the hurricane started arriving Saturday evening. Here in our usually sunny SoCal valley of Santa Clarita, the sky was painted with a cloud spangled gorgeous warm glow. There was something in the air. The pressure was on a steady decline. Blankets of blue and gray clouds stretched across the horizon, putting us to sleep with ominous warnings from the National Weather Service.
Sunday morning awoke with a breezy drizzle that became a downpour. There was a unique odor with this storm. It smelled of ozone, like when you accidentally let the smoke out of your circuit board (not that I’ve ever done that!). The subtle hints told of the origin of the storm coming from lightning infested waters. Here we were, experiencing a unique moment in our time in SoCal. A hurricane!
“Earthquake Detected, drop, cover, hold on!” Of course, our go-to natural disaster danger here in California is earthquakes, not hurricanes. Clearly, it was jealous of all the attention the hurricane was getting, so we got to experience a 5.1 magnitude quake in Ojai, CA. It was 50 miles from our house and enough to bounce us around in our home, swing the chandeliers and move pictures on the wall, all while we watched the rain pour buckets in our backyard. A double feature hurriquake weekend!
We measured 8.75 inches of rain! For some of us in SoCal, you probably spent some time “unflooding” areas by clearing out rarely used drains and gutters. I had cleared our side yard drain on Saturday but forgot the back yard gutter… It’s okay, it let me know. Once that attention was given, I watched the rain fall. The pools of water slowly rose over time and the drainage ways surged to life. Streets filled up. Gutters were roaring. Even the Santa Clara river was full! It was mesmerizing.
Monday is here. The storm has passed. Brilliant white clouds cover the sky. Some remaining power outages across SoCal are gradually coming back online. Each hour gets brighter. Life is stirring about again.
Breathe in, breathe out. There is something magical about the air after a storm. Go on a walk outside. The landscape is glistening with a refreshing scent. The emerald green lawns and shrubs seem to radiate life. Patches of sapphire blue are now floating across the sky.
I’m issuing a new alert… a new day is here! Soak it in. Enjoy every minute.
