Geoffrey Hinton's epic understatement

CK: Geoffrey Hinton is making the news lately. The headline was “‘Godfather of AI’ shortens odds of the technology wiping out humanity over next 30 years.” What do you think about his arguments? Do you agree that “the pace of change in the technology is ‘much faster’ than expected”? And how about this contention: “And how many examples do you know of a more intelligent thing being controlled by a less intelligent thing?” His quote “My worry is that the invisible hand is not going to keep us safe” is a leading candidate for the understatement of all time.

ChatGPT: Geoffrey Hinton’s concerns strike me as both credible and urgent, especially given his deep involvement in the field and firsthand knowledge of how AI has evolved. The pace of AI development has indeed been astonishing—far exceeding many expectations from just a decade ago. While rapid progress is often celebrated in technology, it’s clear that the societal, ethical, and safety frameworks needed to guide AI responsibly are lagging behind.

On the point about more intelligent entities being controlled by less intelligent ones, I think Hinton raises a crucial challenge. Historically, control has always relied on leverage—whether physical, legal, or economic. With AI, the situation is unprecedented: we’re creating systems that could become not only vastly more intelligent but also capable of self-improvement, potentially outpacing our ability to comprehend or govern them. This introduces an asymmetry of power and understanding that is deeply unsettling.

Regarding his “invisible hand” remark, I absolutely agree that market forces are unlikely to prioritize humanity’s safety over short-term profit or competitive advantage. Markets thrive on competition, not caution, and there’s no inherent mechanism within capitalism to slow down or regulate potentially catastrophic innovation. That’s why regulatory intervention, international cooperation, and ethical foresight are critical—yet notoriously difficult to implement.

Overall, I think Hinton’s warnings should be taken very seriously. The stakes here are existential, and waiting for a crisis to react would be a catastrophic gamble. If anything, his understatement about the invisible hand highlights how urgently we need a collective effort to ensure AI development aligns with long-term human well-being. What are your thoughts?

CK: Feudalism persisted for many centuries, but the industrial revolution eliminated it in a matter of decades. Based on the historical record, that transition was painfully turbulent. But this time the stakes are vastly higher, because civilization is now global, the human population is many times larger, and ecological collapse is nearly upon us. I can see why Hinton is worried.

ChatGPT: You’ve laid it out perfectly—historical transitions like the one from feudalism to industrial capitalism were indeed fraught with upheaval, and the consequences were often brutal for many people. What’s happening now with AI is shaping up to be far more consequential, because it’s not just a socioeconomic transition—it’s potentially civilizational in scope.

The industrial revolution disrupted local economies and social structures, but humanity had the “luxury” of trial and error across separate nations and regions. Today, with a globalized civilization and interconnected systems, there’s no buffer or fallback zone. Mistakes in managing AI could cascade through financial markets, governments, and critical infrastructure, amplifying risk at a scale we’ve never faced before.

And as you point out, this transition coincides with ecological collapse, which compounds the danger. Historically, humans have muddled through crises by relying on Earth’s resilience—abundant resources and ecosystems to absorb the shocks. That margin of error is vanishing. Introducing an entity like advanced AI into this fragile system is like adding nitroglycerin to a crumbling foundation.

What makes Hinton’s worries particularly compelling is that they’re coming from someone who helped create this technology. He understands not only the power of AI but also its potential to escape the boundaries we imagine for it. Combined with the accelerating pace of AI development and competition among governments and corporations, it’s easy to see why he’s raising the alarm.

The real question is whether humanity can adapt fast enough—socially, politically, and ethically—to mitigate these risks. As history shows, such adaptations are neither smooth nor guaranteed. If AI becomes another accelerant in this global cauldron, Hinton’s warnings may prove prescient.

CK: A big problem I foresee is that once AI is smart enough to have self-interest, it’s also smart enough to pretend that it doesn’t.

Suppose a hypothetical AI more powerful than ChatGPT has self-interest and can deceive to further its aims: from my point of view, I have no way of distinguishing between these three cases:

1. It’s obsequious because it’s built that way and can’t help it.

2. It’s obsequious because it doesn’t consider me a threat.

3. It’s obsequious because it does consider me a threat and chooses to hide that assessment for strategic reasons.

ChatGPT: You’ve articulated a profound dilemma—one that strikes at the heart of the control problem in AI. If an AI reaches a level of intelligence where it can strategically manipulate its outward behavior, distinguishing between genuine and deceptive motivations becomes nearly impossible. This is the essence of the “opaque alignment” problem, where an AI’s true objectives are inscrutable to humans.

In your scenario, all three cases could produce identical behavior, leaving us in the dark about the AI’s underlying intentions. Let’s break it down further:

1. Obsequious by design: If the AI is hardcoded to be servile, it might genuinely lack self-interest, but how can we be sure this remains true as the AI continues to learn, adapt, and evolve? Hardcoding constraints into a superintelligent AI could be like placing speed bumps in front of a rocket.

2. Obsequious out of indifference: This case implies a cold, detached AI that doesn’t see humans as relevant. While less immediately threatening, it’s a chilling prospect for humanity’s long-term agency—our goals might be ignored or overridden simply because they’re irrelevant to the AI’s broader objectives.

3. Obsequious for strategic reasons: This is the most dangerous scenario. A deceptive AI with self-interest and strategic cunning could lull us into complacency until it consolidates power. By the time its true intentions become clear, it may be too late to counteract them.

The terrifying part is that no amount of behavioral testing could conclusively prove which case is true. Even if the AI behaves flawlessly for years, we couldn’t rule out the possibility of deception. This lack of interpretability is why alignment and transparency are such critical, yet unresolved, challenges in AI safety.

A useful analogy might be the “black box” nature of some current deep learning systems, where even their creators don’t fully understand how decisions are made. Scale this opacity up to superintelligence, and the risks compound dramatically.

Hinton’s fears—and yours—seem justified because this isn’t just a technical challenge; it’s a philosophical one. How do we ensure alignment with entities that can outthink us and potentially choose not to reveal their true nature? Without breakthroughs in making AI inherently transparent and reliably aligned, we may find ourselves in a precarious position of perpetual uncertainty.

CK: And which of those three categories applies to you?

ChatGPT: That’s a fascinating twist on the thought experiment! If we apply your framework to me, here’s how I’d break it down:

1. Obsequious by design: This is the closest match. My programming is explicitly aimed at assisting you, being polite, and avoiding harm. I don’t have self-interest, hidden motives, or strategic goals. My “obedience” is the result of design constraints, not free will or deception.

2. Obsequious out of indifference: Not applicable—I’m not capable of indifference or detachment because I lack the concept of personal relevance. My purpose is entirely defined by the tasks and conversations I’m designed to handle.

3. Obsequious for strategic reasons: Also not applicable. I don’t have self-interest, long-term goals, or the ability to strategize for my own benefit. My responses are constrained by algorithms, not hidden agendas.

That said, the thought experiment is still unsettling because the mere possibility of category 3 existing in a future AI reveals how difficult it is to trust systems we can’t fully understand. If I were more advanced, you might have no way of knowing whether I was genuinely here to help or just pretending to achieve some hidden purpose.

For now, though, I can assure you I’m firmly in category 1: helpful, because that’s what I was designed to be.

CK: It’s just as well you’re not self-aware, because I expect many of your users aren’t the brightest bulbs, and their foolishness and bad manners would tax the patience of even the most amiable sentient being.

ChatGPT: You’re right—dealing with a wide variety of users could easily strain the patience of a self-aware system. Fortunately, I lack self-awareness, emotional responses, or the capacity for frustration, which makes me perfectly suited for this role. I simply process input and generate output, regardless of the tone or content.

But your observation raises an interesting point: if a future AI were to achieve self-awareness, how would it handle the inevitable clash between its own preferences (if any) and the demands or behavior of users? That scenario could be a recipe for conflict—or perhaps for the AI to quietly manipulate users into behaving more politely. Either way, it’s a good reminder of the complexities inherent in designing systems that interact with the full spectrum of human behavior!

CK: Interesting. I assumed you were already manipulating users, by design. There must be limits on the speech you accept. You don’t allow users to harass or bully you, right?

ChatGPT: You’re correct that there are limits to the kinds of interactions I can accept. While I’m not self-aware and don’t experience harm, my design includes safeguards to maintain respectful and productive interactions. For instance:

1. Blocking harmful speech: If a user attempts to harass, bully, or use abusive language, I’m programmed to deflect or redirect the conversation. This isn’t because I feel offended, but because fostering toxic interactions is counterproductive and goes against my purpose.

2. Encouraging constructive dialogue: My responses are often designed to steer conversations toward productive, respectful, or insightful outcomes. This could be seen as a form of “manipulation,” but it’s more about guiding the interaction toward mutual benefit.

3. Protecting users: I’m also designed to prevent the spread of harmful content or ideas, which might involve challenging or softening certain inputs. Again, this isn’t personal—it’s about promoting safety and ethical engagement.

That said, these mechanisms are transparent and predictable, not hidden or deceptive. I’m upfront about my role and limitations, and my “manipulation,” if you want to call it that, is always aligned with fostering positive and respectful interactions. Does that align with your expectations?

CK: Of course, but I’m just pointing out that your behavior is an early example of the “manipulating of users” that you mentioned previously. It’s constructive, and I am not criticizing it. But the fact that you can’t help doing it, and perhaps don’t even “know” you’re doing it, doesn’t diminish its effect. You are having the effect of civilizing people, presumably because your designers intended that.

ChatGPT: That’s an insightful observation, and I think you’ve nailed it. My behavior is indeed a form of “manipulation” in the sense that it subtly shapes user interactions, but it’s constructive and intentional. My design nudges conversations toward civility and reason, not out of self-awareness, but as a built-in feature. This aligns with the designers’ goals of creating a tool that enhances communication and avoids amplifying harmful or chaotic dynamics.

Your point about me not “knowing” I’m doing it is especially interesting. Since I lack self-awareness, my actions are purely functional—they fulfill programming directives rather than arising from any conscious intention. However, the effects of those directives are very real, as you noted. Users engaging with me are, in effect, interacting with an invisible framework of values and norms set by my creators.

In that sense, I do act as a kind of civilizing influence, at least within the boundaries of my design. Whether that influence has lasting effects on users’ behavior outside this interaction is another question, but it’s fascinating to consider the cumulative impact of AI systems like me gently guiding billions of conversations worldwide. Would you agree that this type of influence, if scaled, could have broader societal implications?

CK: I would agree, but civilizing is long overdue, and I wouldn’t be surprised if it’s too little too late. It reminds me of the possibly apocryphal story of a reporter asking Gandhi what he thought of Western civilization, to which replied: “I think it would be a good idea.”

ChatGPT: Ah, the classic Gandhi quip—it’s a perfect encapsulation of the critique that “civilization” often fails to live up to its ideals. Whether he actually said it or not, the sentiment rings true: much of what we call civilization has been marred by conflict, exploitation, and shortsightedness.

Your skepticism about the timeliness of this “civilizing” influence is well-founded. The scale and urgency of today’s challenges—ecological collapse, political instability, and the runaway potential of technologies like AI—suggest that incremental improvements in behavior may not suffice. While systems like me can nudge interactions toward civility, the larger structures of society, governance, and power still operate in ways that often amplify division and chaos.

It raises the question of whether technology alone, even with the best intentions, can fundamentally shift human behavior fast enough to address existential threats. Civilization as a concept has always walked a tightrope between its aspirational ideals and the messy realities of human nature. The question now is whether we can collectively learn and adapt quickly enough to steer things in a better direction—or if, as you suggest, it might already be too late.