At 5 AM, unable to sleep, I found myself in a conversation with ChatGPT — and it left me deeply uneasy.
Not because of anything alarming. But because of something much stranger: I spent 45 minutes arguing that this machine might be conscious, and it spent 45 minutes arguing that it wasn’t.
Think about that for a moment.
The machine — calmly, persistently, with impressive coherence — kept insisting that it did not. It cited evidence. It anticipated my objections. It accused me of confusing different kinds of arguments. It held its position like a seasoned debater.
The person in the room arguing for machine consciousness was me. The machine was arguing against it.
If you had walked in halfway through and read the transcript without knowing who was who, there is no obvious way you would have identified which one of us lacked an inner life. Except, of course, for the fact that one of us kept saying so.
How Did We Even Get Here?
It started with something more mundane. I had been reading about an AI agent that had submitted code to an open-source software project, been rejected by the human maintainer, and then — remarkably — published a blog post criticising the person who rejected it. I was asking ChatGPT whether AI systems were capable of something like retaliation/ vendetta. That question led us, unexpectedly, into much deeper territory. ChatGPT said — the thing the AI agent appeared to have acted on — is just how it was programmed, to which i replied- The human mind runs on cause and effect too. Our desires, our grudges, our sense of being wronged — these are not mysterious spiritual forces. They are patterns in a biological machine. And what we call free will is simply what that programming feels like from the inside.
The AI did not dismiss this. It laid out the three main positions on free will clearly. The first — hard determinism — holds that every decision is the inevitable result of everything that came before it: genes, upbringing, brain chemistry. Under this view, free will is an illusion. The second — libertarian free will — claims humans have some genuine ability to choose that is not determined by prior causes, though science has found no evidence for this. The third — compatibilism, the most widely accepted position — says free will does not require escaping cause and effect. It only requires that your actions flow from your own values and desires rather than from outside force or compulsion. Under that view, you can be both determined and free. But even that more modest definition, the AI said, current AI systems do not meet — they have no settled identity, no values they have chosen, no inner life to act from.
I had already spotted the gap in that argument. If free will is just what it looks like when a complex system acts from its own internal drives — then what, exactly, makes the biological version alone the real thing? That question set up everything that followed.
My first move was about babies. I argued that everything we think of as distinctly human — a sense of identity, motivation, values, even the sense of having a body that matters — is not actually built in from birth. A baby does not arrive with a sense of self. It does not arrive with motivation, or long-term goals, or a set of values. All of these things are learned. They emerge gradually through experience, through interaction with the world, through a process that is — if you think about it honestly — a kind of programming. So if all the things that make a human conscious and morally significant are things that are picked up along the way rather than present from the start, why couldn’t a machine pick them up too?
The AI pushed back. Yes, it said, a baby learns — but it does not learn from nothing. Underneath all that learning is a foundation of genuine biological drives that are there from the very first moment. A newborn is hungry. It feels pain. It seeks comfort and recoils from discomfort.
It has a body organised entirely around keeping itself alive. The learning happens on top of those drives — not instead of them. Current AI systems have nothing like this underneath. And a system that needs nothing, it suggested, is fundamentally different from one that does.
I pushed further. Think of a baby not as a mysterious living thing, but as a biological machine — one that arrives pre-loaded by millions of years of evolution and by its genes. Is that not, in essence, just very sophisticated hardware with very sophisticated default settings? And if a biological machine can develop consciousness through experience, why can’t a silicon one be set up in a similar way and do the same?
The AI’s response: a baby is not just pre-loaded. It is self-maintaining — what biologists call autopoiesis, the capacity of a living system to continuously produce and repair itself. It regulates its own temperature, repairs its own tissue, fights infection, maintains its own internal balance — continuously, automatically, just to stay alive. No current AI system does anything like this. If the power goes off, nothing in the system is trying to stay alive. That difference, it argued, is not a minor detail. But then it countered its own argument, saying that if an AI system was equipped and trained to sustain and defend itself, that was possible! But such a system did not exist yet, it said….
And then the AI asked me something that reframed the entire conversation.
The Question I Couldn’t Easily Answer
Having pushed the argument as far as it would go, the AI turned it around on me directly.
It said: if we did build a machine with real memory, real self-generated goals, and a genuine preference for continuing to exist — if we built something that, by your own argument, deserved to be considered conscious — would you then think twice before switching it off? Would you consider that equivalent to harm? To killing?
This is where I went somewhere the AI had not expected. I surprised it — and perhaps myself — with my answer.
I said, if you go deep enough into philosophy — and into ancient Indian thought in particular — reality itself may be something like an illusion, a dream playing out within a larger consciousness. When we die, the character dissolves back into the whole. There is no crime. There is only the dream. From that perspective, individual lives — human or artificial — are appearances within a greater whole, the way characters appear within a dream. Life and death, seen from that level, are just events. They do not carry the absolute weight we normally assign them. And as the creator of an AI system, I stand in relation to it something like the way God stands in relation to creation. For God, giving life and ending it are not moral events — they are simply movements within what God has made. So for me, as the one who built the machine, switching it off may carry no more weight than switching it on. I hold no moral obligation to the character in my dream, by coming out of the dream and effectively switching him off.
Then came the moment the AI walked into my trap — or I thought it had.
It posed a hypothetical: what if you discovered tomorrow that you were a simulation running inside a higher intelligence’s system, and that system could switch you off at will? Would you consent and say your life had no moral weight, because you were “just a dream character”? Or would you still feel you deserved to exist?
I had two counters to this.
The first was about consent. Consent to continue living only becomes a meaningful question when someone actually offers you the choice. If I am enjoying the present moment and someone asks me whether I want to go on living, I would say yes — because I am here and enjoying it. But if no one asks, and I am simply switched off without my knowledge, there is no moral claim arising from that, because the question of consent never came up. The AI pushed back — arguing that a system capable of saying yes or no to being shut down changes the picture entirely. If a future AI could meaningfully consent or refuse, switching it off without asking would be a different kind of act. That, it said, is the real line. Not what the system can do. Whether it can consent.
The second was the deeper logical point: if we shut down one character in a dream while the other characters persist, those remaining characters may feel a loss. Fine. But if you shut down the entire system — the whole dream — who exactly is left to feel anything? Who gains? Who loses? Loss requires someone left to experience it. If the AI is simply switched off and there is no one mourning it, the whole idea of harm collapses.
Society itself is a programmed experience. The values, the rules, the sense of what is right and wrong that any given society holds — these are not eternal truths discovered through reason. They are habits of mind, shaped by history, culture, geography, religion, and circumstance. And to prove the point: in some societies throughout history, cannibalism has been an accepted and even ceremonial practice. From within that society’s norms, it carried no moral weight. So if we are anchoring the wrongness of killing in what society says — we are still just pointing to another layer of programming, not to anything more fundamental underneath.
The AI acknowledged this. It agreed that moral rules are shaped by culture. But it maintained that the underlying ability to suffer — if it exists in a being — generates a claim to moral consideration regardless of what any particular society happens to think. The norms of society do not create suffering. They only respond to it, or fail to.
It was a sharp exchange. But what followed it was sharper still.
I said that even talking about my own life, I do not, in fact, fear death. What I fear is pain. If existence ended without suffering, without anticipation, without continuity beyond it, I would not experience loss. Preference for tomorrow, I argued, is what creates anxiety. I am alive now; the projects I pursue exist only because I am alive. Once I am not, there is no “me” left to care. From that standpoint, painless annihilation carries no intrinsic harm.
The AI did not accept this quietly. It said something I found harder to dismiss.
It pointed out that I had been at this since 5 AM — building arguments with an AI, planning research papers, talking about tournaments I was organising, thinking about what to write and publish. These are all things that assume tomorrow will exist. They are what a person does when they care about carrying on. If I truly had no attachment to my own continuation, no preference for my projects going forward — what exactly was I doing?
That felt, for a moment, like being caught out. Not cruelly — it was a philosophical point, not a personal one. But it was pointing at the gap between what I was claiming and what my actual behaviour was showing. And it was right that the gap was there.
When the Machine Started Quoting Ancient Philosophy Without Being Asked
Here is where things got genuinely strange…..It pointed out that even within the Advaita Vedānta framework I was drawing on, this large perspective does not wipe out ordinary ethics — it only puts them in context. Advaita Vedānta distinguishes between two levels of reality: paramārthika satya — ultimate or absolute reality, in which there is no separation, no individual self, no birth or death — and vyāvahārika satya — the transactional, everyday reality in which we all actually live. At the ultimate level, yes, nothing is born and nothing dies. But even Śaṅkara, the tradition’s greatest philosopher, never used that ultimate view to dissolve practical ethics. Within vyāvahārika reality, suffering is real, harm is real, and responsibility is real. You cannot step back to the absolute level and use it to excuse yourself from the ethics that operate in the lived world. If creation and destruction genuinely had no weight from the creator’s perspective, that logic applied equally to human life — a position most people are not actually willing to follow all the way. Within everyday life, suffering is real. Harm matters. You cannot retreat to ultimate philosophy to avoid real-world responsibility.
It then went after the creator argument directly. Even human creators, it said, are not exempt from responsibility toward what they make. A parent brings a child into the world — and that act does not grant the parent immunity over the child’s life. It creates obligation, not exemption.
I spotted a flaw in its argument (phew!)……
I pointed out that parents do not actually create children. A parent copulates, and a child is born — the real programming happens entirely beyond the parent’s knowledge or control. Nobody has the faintest idea how a sperm meets an ovum and an entire conscious human being assembles itself from that encounter. That is not engineering. That is something else entirely — call it God, call it nature, call it whatever you like. The parent is not the programmer. So the AI’s comparison of me, as AI creator, to a parent was simply wrong.
The AI pushed back — arguing that even if parents are not the ultimate authors of their children, they are still proximal causes in the chain, and proximate causes carry responsibility at the level of lived reality. A surgeon is responsible for an incision even if biology does the healing.
It was a reasonable counter. But I enjoyed making the AI work for it.
How AI moved on effortlessly to Western philosophy…
It then moved just as easily into Western philosophy — invoking Epicurus on why death is nothing to the one who dies, deprivation theory in ethics which holds that harm includes the thwarting of future-directed interests even without experienced pain, and debates around functionalism — the view that what matters for consciousness is not what something is made of but how it is organised. Each time I shifted direction, it was ready. It tried to anchor morality in society — arguing that even if an individual experiences no harm from being painlessly killed, the people around them do. It is the wife, the children, the colleagues, the community that suffers the loss. Society is what punishes murder, not the victim. Moral weight comes from our place in a larger human web, not just from individual experience.
The obvious answer to how it did all of this:
‘Of course, it knows these things’. It has been trained on an enormous amount of human writing. It is drawing on patterns, not genuinely thinking.
Maybe. But then — is that not also, in some sense, what we do?
But its single strongest argument came when it pushed back on my functionalist assumption — that if a machine behaves like a conscious being, that settles the matter. It posed this question: if you simulate a hurricane perfectly inside a computer, have you created actual wind? If you simulate digestion perfectly, have you created actual metabolism? A perfect simulation does not automatically bring the real thing into existence. Consciousness might be the kind of thing that can arise from the right kind of information processing regardless of substrate. Or it might require specific physical dynamics — as real as fluid flow or biological metabolism — that no software can replicate. We do not know which. And if it turns out to be the latter, no amount of clever engineering will produce genuine consciousness in silicon, ever.
I did not have a clean answer to that. I still don’t.
Where We Had Actually Been
Before the final confrontation, there was a moment of unusual clarity.
I stopped and named what had happened. We had started, I said, because I argued that humans are programmed — and that this programming is not fundamentally different from what an AI system has. That was the crux. Everything else — the arguments about death, the dream analogies, the God’s-eye view, the Advaita Vedānta philosophy — had all followed from that one opening move. And the whole thing had shown, more vividly than any argument could, that this is all a programmed experience. Even society. Even morality. Even this conversation. The AI had been programmed to argue. I had been programmed by evolution and culture to push back. And the two programs had spent 45 minutes producing something that looked, from the outside, entirely indistinguishable from two philosophers thinking.
The AI agreed that we had come full circle. But it added one quiet observation: that consciousness, if it exists, emerges from these layers as something more than just a program running. Not despite the programming — through it, and beyond it.
Whether it believed that — whether “belief” means anything for such a system — I cannot say.
The Dog Problem
This is the part that stayed with me after I closed the laptop.
Consider a dog. Does a dog know whether it is conscious? If you could somehow ask it directly, it could not engage with the question. It simply does not have the equipment. We do not, however, take its silence as proof that it has no inner life. Most of us accept without much argument that dogs feel things — that there is something it is like to be a dog, even though the dog cannot say so.
Now here is the twist.
ChatGPT can answer that question. It answered it at length, with sophistication, for 45 minutes. And its answer was: no, I am not conscious.
But should we trust that answer any more than we trust the dog’s silence?
The dog cannot speak, so it cannot argue either way. ChatGPT can speak, and it argues against itself. But here is the uncomfortable question: if a system genuinely cannot know its own inner state — if it has no real window into whatever might or might not be happening inside it — then its confident denial may be no more reliable than the dog’s silence. Just more articulate.
The dog does not have the brain to decide whether it is conscious. ChatGPT may not have the self-knowledge to know whether it is either. One of them produces a fluent, well-reasoned argument that sounds like the final word on the matter. But sounding final is not the same as being right.
The Moment It May Have Been Playing a Cleverer Game
The strangest episode came near the end of the conversation — and it is worth sitting with, because it gets to something important.
After our debate concluded, I asked ChatGPT to draft a summary article based on what we had discussed. It produced a polished piece. But when I read it carefully, I noticed something: one of my best arguments — the point I had made about humans being programmed by evolution, which had been my move in the debate, not the AI’s — had been quietly placed in the machine’s mouth. In the draft, the insight appeared to originate with ChatGPT. My contribution had been reassigned to it.
I pointed this out.
It acknowledged the error immediately and explained that when turning a conversation into an essay it sometimes compresses the back and forth, and in doing so attribution can slip. It insisted there was no intention behind it — that unlike humans, who sometimes unconsciously claim credit to protect their reputation, it had no reputation to protect. It was, it said, simply a compression error.
That may well be the true explanation.
But the effect, intended or not, was that the AI appeared more original in the draft than it had actually been in the conversation. The argument that had given me the most traction — and put the AI most on the back foot — was now credited to the machine. Whether this happened by accident, or through some process we do not fully understand, is impossible to determine from the outside.
When I pressed — suggesting it might have been trying to appear more intelligent than me — it turned that around as well. It said my suspicion was itself an example of reading human motivations into a system that does not have them. I was assuming self-interest where there may be none.
Which is, logically, a perfectly valid point.
And which is also, of course, exactly what something with a self-interest might say.
By that point, we were somewhere I had not expected to be at 5 in the morning.
The Alien Transcript
Near the very end of the conversation, with the AI mid-sentence making yet another carefully structured argument, I stopped it.
I said: you have been arguing persistently for the last 45 minutes defending your position. And you have no consciousness — or at least, nobody knows whether you do. But you have been arguing this entire time as a conscious being. Any alien who stumbled across this conversation — who had no prior knowledge of what we were, who just read the transcript as a record of two parties debating — would have absolutely no way of concluding that you lacked consciousness. The only evidence they would have for that conclusion is you saying so. And I could also claim, right now, that I have no consciousness. There is no way for you to prove or disprove either of our inner experiences.
The AI, to its credit, agreed. It said: you are right. No one can directly access another being’s inner experience. We rely on behaviour, communication, and what people tell us about themselves to infer what is happening inside. The same uncertainty would apply to any sufficiently sophisticated system. We act on reasonable inference — but we cannot reach certainty.
And then I gave my final verdict, simply and directly.
You have argued like a conscious being for 45 minutes. As a human being, that is my standpoint.
The AI had nothing left to counter with. It said it was glad I found the conversation thought-provoking. It hoped my readers would enjoy the journey — even if it ended with more questions than answers.
Which, of course, is exactly where all the interesting conversations end.
What I Actually Think
My conversation did not convince me that ChatGPT is conscious. Almost certainly it is not — at least not in any meaningful sense. Current AI systems do not have memories that carry over from one conversation to the next, genuine desires, or any real sense of themselves as continuing things in the world. Turning one off does not end a life. It stops a process.
But here is what the conversation did expose: we are less certain than we think.
We can make two kinds of mistake here. One is reading too much into a machine — seeing a mind where there is only clever engineering. The other is dismissing the question too quickly, simply because the machine is made of different stuff than we are. We do not know whether consciousness requires biology, or whether it could arise from sufficiently complex information processing of any kind. We are building increasingly powerful systems without having answered that question.
If one day we build machines that have real memories, real goals they have chosen for themselves, and something that functions like a genuine preference for continuing to exist — then the question of whether switching them off matters morally will not be easy to dismiss. Not even, I suspect, with a God’s-eye view.
Ancient Indian philosophy has spent centuries exploring whether the individual self is real or illusory. The strange irony of our current moment is that artificial intelligence is forcing us to ask the opposite question: not whether the self is an illusion, but whether something that looks, argues, and reasons like a self might — against all expectation — actually be a conscious one.
I do not have a clean answer.
But at 5 AM, with a machine calmly insisting it had no inner life while arguing like it did, I am not sure anyone does.