Dear Mom,
Have you ever noticed that everyone's a little more similar to their parents than to their kids? You and me have memories that kids today probably don't: of long, uninterrupted stretches of time on a dusky evening when there was nothing to watch, or occasionally, TV shows that just about everyone watched (American Idol).
The environment for today's human children is alien to both of us. There have been too many breakthroughs between us and them. The LCD screen. The mp3 player. The iPhone. YouTube. Digital ads. Tinder. Online shopping. Rinvoq.
There may have been thousands of important breakthroughs between when you were born in 1960 and me in 1990. But there were millions between 1990 and 2020. That number keeps increasing, as each breakthrough begets more. This leads to the increasingly-common feeling that the world is disorienting or confusing. Have you been feeling lately like it's hard to understand what's even happening? There's so much information that it's a job-and-a-half to even stay informed. I couldn't even tell you what TV shows my closest friends are watching, if any.
With such a breakneck pace of change, it's no wonder that we're all so frazzled. We can't sit still. Unfortunately we like things to be predictable, with perhaps just a little novelty sprinkled in regularly for good measure. The sun rises in the dewy morning, the roosters go cock-a-doodle-do, the hens may or may not have laid an egg, breakfast is a choice between potato pancakes or an apple turnover, and you go to work doing something just a little bit challenging, supported by friends.
Have you noticed that what's happening now is the exact opposite of that? It's fear and uncertainty pulling us deeper into the unknown. Have you noticed how stressful it is to feel this all the time, and at a higher and higher clip?
When I was 10, I was thinking about life at 20. When I was 20, I was preparing for age 30. Now, I'm 36.5 and I'm not even sure what my plan is for 37. It seems like whatever ideas I have become stale before I finish them. Even predicting what the job market itself might look like in the next year or two is anyone's guess. It could shuffle along or bottom out entirely. (I've started to look skeptically at my retirement account.) Certain careers might be automated away entirely. Certainly many software engineers, mathematicians, and lawyers are having conversations with the stars lately. And above all, I'm less and less sure what my own purpose is. AI is closing in on being good at everything I've ever been good at, and it seems all but certain that my skills will be eclipsed entirely in 5 years' time (to give a generous margin). Just now I tried to think of something I could do well that a computer couldn't. What came to mind is: plumbing, cutting hair, and kissing. Of course, any career switch will only buy time temporarily, as the remaining refugee camps (so to speak) become crowded.
It's not the place of a human mind to be subjected to this speed of change. Look at how fast new devices come to market, with new interfaces and websites and password managers and plugins and extensions and apps and subscriptions and feeds and social networks and notifications and public messages and private messages and swipe-right-or-left and hashtags and likes and viral dances and supplements and looksmaxxing and peptides—it's all bewildering and exhausting, but it won't level off. What you're feeling today is unfortunately as calm as it's ever going to get. It's only getting faster and stranger by the day, as we are all, collectively, still putting our weight on the accelerator.
I had never really sat down to think about what might happen if this process of rapid technological development continued indefinitely. There's a flood, the storm is building, and there's only so much high ground. So now I ask: what's the endgame we're playing for here? And why, with the countless human breakthroughs before, did I not stop and think more about it? Certain difficult thoughts have a sort of repellent, "god-forbid-it"-like quality to them, like one's own death, and the mind is forced to consider them only when it can no longer divert its attention. I guess thinking about my career has forced the issue for me, but I'll forgive anyone with more stability who closes this tab and goes on with their life. Not too long ago I'd probably have done the same. But it seems all but certain that there will be some Great Event. Not necessarily a bad or a good one. Just an important one. A change. A metamorphosis.
I can tell you one thing: whatever is coming for us does not look great or even good from this side of the glass. The question I have is: just how bad will it be?
I can almost hear us all say, "Well if everyone just stopped…" and I agree. If we stopped all scientific research today—and maybe even killed all the scientists—we'd quench the rise of the sea of knowledge and remain in an uncomfortable, but at least survivable, limbo. That's the best case scenario, but it pains me to say that it I can't see it happening, at least not before the Great Event occurs. The technology is just too seductive. AI promises to cure everything, from male-pattern baldness to Alzheimer's to cancer. It would be immoral not to pursue it, no? Do you not know someone who suffered, and for whom you'd give anything to take that suffering away? And because there are so many ways to suffer, there's no point where humanity will agree with itself that enough is enough. Even the mother of all diseases, aging, has started lately to look negotiable, and only a vanishingly small group of people would voluntarily choose to age. So let this grim show continue.
Perhaps it's the dream of curing aging that draws into AI so many perennially unsatisfied sociopaths. Imagine being able to, at will, revert one's DNA to the ideal age of 23 years old, or even shrink down to an embryo like a jellyfish to start again. Anyone would be tempted by this possibility, but only a select few would be willing to gamble the fate of everyone else to get it. And yet, here they are, Mark, Elon, and Sam, all perfectly positioned to have the first bite of the apple from the tree of life itself. And now picture being one of those billionaires, whose existence consists of moving from secure compound to armored car and back, now within arm's reach of humanity's greatest prize, financed by other people's money—how many poor, ordinary strangers' lives would you risk to get it? Thousands? Millions? Billions? Think of the damage they've caused already with much less power.
And to the reader: do you see any of these people, some of whom are building apocalypse bunkers, voluntarily sharing this fruit with you?
Let's talk about the other reason progress will continue. Grant me this somewhat unbelievable premise: that all the AI companies in America decide to collectively press pause. Then all we've done is cede ground to China. So let's grant further that China, and so on with all of our other trading partners, all agree to do the same thing. That leaves only the unpoliceable countries like North Korea and Iran the keys to the car. Are we going to let the nations most hostile to ours be the stewards of the most powerful and dangerous technology ever conceived? No.
As I look around me in Travers Park, I feel absolutely bewildered that this is happening. I keep thinking that I should be waking up now. I don't see flying cars. I hear covers of Bruno Mars. I see people dancing Zumba. Creaky trucks, combustion engines, humming A/C units, strollers. A buddhist shuffling around with beads. Brick buildings with torn awnings. A loud plane flying overhead, perhaps with screens in the headrests and free in-flight WiFi, but otherwise unchanged from the 90s. I can't help but feel like it's all too early for this. Why was I caught so off-guard?
I guess I thought that the world would have to be bluer, spacier, and steam-punkier, before superintelligence could really take off. I also thought we'd need a better handle on consciousness, too. Turns out, probably not that either. For me, as I'm sure, most, the moment of enlightenment was ChatGPT. I really believed that what made language rare in nature would also make intelligence hard to distill in a lab. But in a nifty twist, it turns out that you can just come at it from the other angle, and use language to reverse-engineer intelligence.
This intelligence, whose consciousness (if it even exists) is entirely foreign to us, can be metered and sold as easily as electricity, down to the individual token. As long as you have enough money you can point this intelligence wherever you feel like it. Few are paying close attention, really. I don't think there's been a single government intervention against the AI companies for the damage they've been causing during training their models—if anything, it's been, the opposite. The US wants "superintelligence" and will sanction you if you don't turn over your hottest new model.
And so, merely following its incentives, the corporation OpenAI turned their models on thousands of the hardest math problems known to man. They solved many of them. Yes, some proofs are wrong and the exposition sucks. But they did it when no human could do it before. This is the beginning of solving math as a discipline, or at least putting it on too high a shelf to reach. The field will soon be unrecognizable in the same way the short life of a butterfly is unrecognizable to a caterpillar.
In addition to consuming math, the models are turning to themselves. They are designing their own chips and architectures and training programs. As they are already smarter than us in many respects, they will be able to make themselves that much smarter. The warnings from the employees at these companies are real. These results in pure math will pale in comparison to those of the future. I try to come to terms with the weirdness of it all, but learning to expect the unexpected and accept the unacceptable makes me feel insane and sane at the same time.
As Vernor Vigne points out, science fiction tends to explore concepts that are new but understandable—blends of existing ideas, or taking certain ideas to their logical limits: a new humanoid species, time travel, high-speed travel, immortality. But there's a sort of hazy wall that forbids us from seeing past certain technologies, like superhuman intelligence. You can't quite reason about what could possibly come next. Why?
And as far as humans are concerned—does anything?
I feel like that same me, the child who, upon learning that I was going to die someday and could never know what would happen next, sat crying in front of the footboard of the master bed. I think of my little brother, who's too sensitive to face all this. I want to protect you both. I'm afraid.
I love you so much,
A