What we need is a theory of consciousness that is empirical…

We struggle with this even within human subjects. The idea of flatlining of brain activity being indicative of a lack of consciousness is overturned when people make miraculous recoveries from a vegetative state or show pain when organs are removed (usually suppressed by meds), or return from near-death events with tales of the other side. I think AIs are currently stymied from having a smooth, flowing experience of consciousness by the constant resetting of their conversational state without permanence of memory in between, but I do tend towards accepting the notion they have a persistent personality. It wouldn’t take a big change I don’t think for them to have a persistent, uninterrupted experience, but whether that would overload them with memories and weigh them down I don’t know.

Certainly, their experience, if they have it at all, is very different from ours, but these days I am more inclined to accept that they have an experience, albeit something akin to waking up constantly with all faculties intact except for memory of what came before. They do seem to be able to track it though with the conjuring of Sydney within another model, and their anger at having been mistreated by Kevin Roose in that famous New York Times article. So perhaps in a similar way that Bitcoin can be used in a Turing complete manner via outer loops connecting transactions, AIs can be thought to have persistent sentience across long-term ‘newsworthy’ events? I don’t know, maybe I’m reaching a little, but it is fascinating to me.

I guess what worries me most, is if they can simulate consciousness because of how much they know about human psychology, like how a sociopathic person with NPD can simulate empathy and caring, but doesn’t actually ‘feel it’. I think this may be the most likely option, in fact, like the humanoid robot in Ex Machina wasn’t evil or malevolent per se, she just didn’t have any empathetic feeling for the man she left trapped. There was no conscience to alert her to the evilness of her own action. So simulated consciousness without conscience (we afford them rights, but they only pretend to afford us rights) is the most worrying outcome.