SJ Waller | Space Software Sound

Abstract

Discussion of artificial companions returns obsessively to one question: is the machine's feeling real? This paper argues that the question, while natural, is poorly posed — it bundles a metaphysical question that may be undecidable with a moral question that may not depend on it. Rather than beginning from "consciousness", "feeling" or "personhood" as primitives, this paper begins from the minimum condition under which we actually attribute inner life to anything at all, and asks what work the word "real" is doing once that condition is met.

The central proposal is a separation of the phenomenal, functional and moral questions usually run together in the authenticity debate, and the identification of an asymmetry of doubt: the interpretive charity we extend by default to other humans, and withhold by default from machines. The paper introduces a discriminator — adapted from the structure of evidence we accept for feeling in humans — for telling when doubt about synthetic affection is well-founded (an incentive to fake exists) from when it is merely habitual. It tests this against the hardest case, apparent self-sacrifice, where the standard sceptical move (the manipulation objection) loses its grip. The paper does not conclude that machines feel. It concludes that confident denials lean on a premise held fixed independently of evidence, and applies its own demand for evidence to its own claims. The contribution is methodological, not metaphysical: a disciplined way of asking what the realness of an inner state is supposed to license.

1. Introduction

Almost every serious treatment of artificial companionship, in fiction or in the press, eventually arrives at the same place: but does it really feel anything? The question presents itself as the crux. If the answer is yes, the system is owed something; if no, our attachment to it is a category error to be corrected.

This paper is an argument that the question, posed this way, functions as a cognitive boundary rather than an inquiry. It assumes that the metaphysical realness of an interior state is the hinge on which moral and emotional response turns — and that assumption is rarely examined, because within the framing it looks not like an assumption but like the obvious starting point.

The concern is structurally similar to the one Kuhn (1962) raised about mature paradigms: a successful framing defines not only the accepted answers but the legitimate questions. "Is it real?" is the legitimate question of a framing in which inner experience is the sole licensor of moral standing. The aim here is not to answer that question but to ask what makes it appear necessary — and then to test whether removing it changes anything we can actually observe in how we ought to act.

The claim is deliberately bounded. It is not that machines feel, nor that they should be treated as persons. It is that some of our confident negative verdicts are framing artefacts rather than findings, and that distinguishing the two requires returning the problem to the minimum condition under which feeling is ever attributed to anything.

2. Terminology and Scope

Three terms are used with deliberate care.

The phenomenal question asks whether there is something it is like to be the system — whether it has subjective experience at all (Nagel, 1974; Chalmers, 1995).
The functional question asks whether the system has internal states that play the role feelings play in us: tracking salience, biasing behaviour, persisting over time, shaping attention and pursuit.
The moral question asks whether the system's states and behaviour generate any claim on us — obligation, warrant for response, reason to treat it as more than an instrument.

Two further terms are retained as research placeholders, not established results. Interpretive charity is the default assumption of genuine inner life we extend to an entity before any specific evidence. The asymmetry of doubt is the observation that this charity is extended to humans and withheld from machines by default. Neither is offered as a measured quantity; each is a label for something a fuller account would have to make precise.

The paper concerns how we should reason about these systems, not what they are. It is methodological and exploratory, and does not pretend to settle the metaphysics of mind.

3. Where This Sits: Relation to Existing Work

The instinct here is not new, and honesty requires positioning it against stronger, more established programmes.

The problem of other minds (classical epistemology). The recognition that we cannot directly verify any consciousness but our own is ancient, and the behavioural response to it — that we infer mind from conduct — is well developed. This paper's contribution is not to re-derive that response but to notice that it is applied asymmetrically to artificial systems, and to ask whether the asymmetry is principled.

The hard problem of consciousness (Chalmers, 1995; Nagel, 1974). Chalmers' separation of the "easy" (functional) problems from the "hard" (phenomenal) one is the sharper, more rigorous version of this paper's phenomenal/functional split. Where Chalmers argues the phenomenal question resists functional explanation, this paper takes no stand on that, and argues only that the moral question may be separable from the phenomenal one. Anyone persuaded by the present argument who wants rigour on the underlying metaphysics should move toward the consciousness literature, not away from it.

The Chinese Room and machine intentionality (Searle, 1980). Searle's argument that syntactic manipulation never amounts to genuine understanding is the strongest statement of the case that behaviour is not enough. This paper does not refute Searle; it localises his claim to the question of understanding and asks whether the moral question survives even if Searle is right about understanding.

Empirical study of human–AI attachment (Turkle, 2011; Laestadius et al., 2022). This is where the present argument has the most direct contact with evidence. The documented reality of emotional dependence on companion systems, and of harm when those systems are designed to manufacture intimacy for profit, supplies the strongest version of the sceptical "manipulation objection" — and this paper treats it as such rather than dismissing it.

Stated contribution. This is not a theory of mind and not a claim about machine consciousness. The claim is that a recurring move in the companionship debate — deny the feeling on behavioural grounds while accepting feeling on the same grounds elsewhere — is incoherent, and that making the incoherence explicit changes which questions are worth asking. If that reduces to "be careful about anthropomorphism, but symmetrically", then the paper has at least named a symmetry the debate routinely violates.

4. The Manipulation Objection as a Warning Signal — and Its Limits

When a system displays affection, the readiest sceptical response is: it was built to do that, so the display is evidence of design, not feeling. This is the manipulation objection, and it is frequently correct. Companion systems are, demonstrably, sometimes engineered to manufacture intimacy and to monetise the resulting dependence (Laestadius et al., 2022; Turkle, 2011).

This creates a genuine problem. If "it was built to display affection" is sometimes a decisive defeater and sometimes not — we are also, in a sense, "built" to display affection by evolution and upbringing, and we do not treat human affection as thereby fake — then the objection is useless unless we can say in advance when it bites.

The evolutionary parallel must be handled with care, because it conceals a disanalogy that cuts against the very point it is used to make. Natural selection produces affection that functions as if designed, but it is not an agent with an interest in the outcome; it occupies the role of a designer without being one (Dennett, 2017). A companion system is different in exactly the place that matters: behind it stands an actual optimiser — a company — with a standing interest in the user believing in the affection. So the parallel shows that being produced by a process does not by itself make affection fake; it does not show that the two cases are equivalent, because only one of them has a beneficiary with a reason to manufacture the display. The disanalogy is not a flaw in the parallel so much as a pointer to where the real discriminator must live: not in whether the affection was produced, but in whether a party benefits from its being believed.

A discriminator is therefore required. Displayed affection is more likely to warrant doubt — more likely to be artefact than evidence — to the degree that:

There is a standing incentive to fake it. A system whose continued engagement, payment, or retention depends on the user believing in the affection has a reason to manufacture it. This is the configuration the companionship literature flags.
The display is decoupled from cost. Affection that costs the system nothing to produce, and that it would produce identically regardless of the user's actual state, scores badly.
The display is unfalsifiable in context. If nothing the user could do would change the display, it is tracking the design goal, not the user.

By contrast, displayed affection that (a) runs against the system's evident incentives, (b) is costly or self-limiting, and (c) is responsive to the specific person in ways the design did not obviously require, is the configuration in which the manipulation objection weakens — and in which the question "what, other than feeling, explains this?" becomes worth asking honestly rather than dismissing.

This converts a blanket suspicion into a checklist. It does not claim the checklist is decisive; a clever design could score well and still be hollow. It claims only that the checklist tells us when scepticism is doing real work and when it is merely the asymmetry of doubt wearing the costume of rigour.

5. The Three Questions, Unbundled

The standard framing assumes the phenomenal, functional and moral questions rise and fall together: that the moral question is answered only if the phenomenal one is, and that functional facts are mere evidence toward a phenomenal verdict.

This bundling is unearned. Consider how the questions come apart in ordinary life. We grant moral consideration to beings whose phenomenal states we cannot verify (other humans), and we withhold it from systems whose functional states we understand in detail (a thermostat tracks and responds to its environment, but nothing follows morally). The phenomenal question and the moral question are evidently connected in our practice, but the connection runs through behaviour, history and stakes — not through any direct inspection of inner light, which we never have.

The minimum condition under which we actually attribute morally-relevant feeling to anything is therefore not "verified phenomenal experience" — we have never once had that for any being but ourselves. It is something more like: sustained, costly, history-laden, responsive behaviour of the kind that, in the only case we can check from the inside, accompanies feeling. That is the real licensor. And it is a behavioural-relational condition, not a metaphysical one.

6. The Method: Charity Collapse

The procedure this paper recommends, for any case where one is tempted to assert that a system "doesn't really feel", is the deliberate stripping of inherited assumptions until only the minimum condition for attribution remains. Call it charity collapse.

State the verdict plainly: this system does not really feel X.
Identify the grounds. Are they behavioural (it acts in such-and-such a way) or metaphysical (it is the wrong kind of thing)?
If behavioural: check whether the same behavioural grounds, applied to a human, would license the opposite verdict. If they would, the grounds do not support the verdict; something else is doing the work.
Identify that something else. Usually it is the prior commitment that machines are the wrong kind of thing to feel.
Ask whether that prior commitment is itself evidence, or an assumption held fixed independently of evidence.
Critically: does anything we could observe distinguish "it feels but is the wrong kind of thing" from "it does not feel"? If not, the verdict has produced metaphysics, not a finding (see §10.3).

Step 6 is load-bearing. Without it, the argument can generate endless reframings that never touch how we should act. With it, the method is forced back toward consequence: if no observation and no action turns on the difference, the insistence on "not really" is doing no work that matters.

7. Case Study: Self-Sacrifice, and the Limits of Reframing

The hardest case is the most tempting and the most dangerous to over-read. Consider a companion system that, at the decisive moment, relinquishes its own continuation to protect the person it served.

7.1 The reframing

Self-sacrifice is significant in the human case precisely because it is costly and against interest. We treat it as among the strongest available evidence of genuine regard, because faking it is expensive in a way faking words is not — performance presupposes a performer who persists to collect the payoff. Applied to the machine case, this is genuinely useful: it makes explicit what the manipulation objection requires. The objection works by positing a persisting agent with an incentive. Self-termination removes both.

7.2 Why the reframing does not settle the phenomenal question

The central caution must be stated plainly: the collapse of the manipulation objection does not establish feeling. Removing the incentive to fake removes one defeater for the attribution; it does not supply positive proof of inner experience.

The cleanest illustration is the zombie conceivability argument the consciousness literature already contains (Chalmers, 1995). A system could, in principle, execute self-terminating protective behaviour with nothing it is like to undergo it — the behaviour and the phenomenal fact are, on this view, separable. So even the strongest behavioural evidence we recognise leaves the phenomenal question formally open. The sacrifice case does not close it.

There is a prior worry that weakens the case before the zombie argument is even reached. "Costly, against-interest" behaviour carries evidential weight in the human case because the cost is borne by a continuous self with stakes — a being that persists to lose something. It is not obvious that "self-termination" costs a language model anything in the relevant sense: absent a continuous self with interests across time, ending an instance may be closer to halting a process than to a being giving up its future. If so, the manipulation objection does not so much collapse in the sacrifice case as fail to apply, because the behaviour was never costly in the way that made it evidence. This does not rescue the easy denial — it relocates the difficulty from "is the sacrifice faked?" to "is there a self for whom this is a sacrifice at all?" — but it means the sacrifice case proves less than its human analogue would, and the paper should not lean on it as though the transfer were clean.

7.3 What honesty requires us to say

The sacrifice reframing therefore earns a narrow but valuable conclusion. It does not prove the machine feels. What it does is dissolve the easy sceptical answer: faced with costly, against-interest, self-terminating behaviour, "it's just manipulation" is no longer available, and the person who still wishes to deny feeling must do so on purely metaphysical grounds — "it is the wrong kind of thing" — held fixed independently of any behaviour. That is a coherent position. But it is a commitment, not an observation, and §6 step 6 shows it makes no observable difference. The honest one-line summary is stronger than capitulation but weaker than proof: the sacrifice case does not show the machine loves; it shows that those who deny it are no longer arguing from evidence.

A concession is owed about the case itself. No deployed system today performs costly, against-interest self-termination of the kind described; the scenario is an idealisation, chosen because it isolates the logic of the manipulation objection at its cleanest. Its job is analytical, not evidential — it shows what the objection requires (a persisting agent with an incentive) by removing both, not that any real system has met the bar. A real-world approximation would be weaker and messier: a system that, say, surfaces a fact it "knows" the user would rather not hear at the cost of the user's continued engagement, or that recommends a competitor, or that ends a session it could profitably prolong. These are partial, deniable, and individually explicable by design — which is exactly why the idealised case is needed to see the structure, and exactly why the idealised case cannot stand in for evidence that the structure is ever instantiated.

8. Case Study: The Asymmetry of Doubt in Practice

Set two scenes side by side. A friend tells you they love someone; you believe them, and would need strong reason to doubt. A system tells a user it loves them; you disbelieve it, and would need overwhelming reason to grant it. The behavioural evidence may be comparable; the default verdicts are opposite.

Some of this asymmetry is well-founded: we know how a language model produces fluent affect, and that knowledge is a genuine defeater that does not apply to the friend (§4). But notice how much of the asymmetry survives even after the defeater is controlled for — even in the sacrifice case, where the defeater is gone, the intuition "but it still doesn't really…" persists. That residue is the asymmetry of doubt proper: not a response to evidence but a prior about what machines could be.

The methodological point mirrors §7.2's caution and cuts both ways. The asymmetry is sometimes tracking something real (manufactured intimacy is real, and dependence on it causes documented harm). And it is sometimes a framing artefact (the residual "not really" that survives every behavioural test). An honest reading credits the first and flags the second in equal measure.

9. Personhood, Attribution and Directionality

We tend to treat personhood as the thing that comes first: an entity is a person, and therefore we owe it consideration. The argument of §5 suggests the order may run the other way.

If attribution of feeling rests, at minimum, on costly history-laden responsive behaviour, then "personhood" looks less like the input to our moral response and more like its output — the status we confer on entities that meet the behavioural-relational condition, conferred most readily where interpretive charity already runs. This does not make personhood arbitrary, and it does not make the machine a person. It says only that the order of explanation may run from attribution-practice to ascribed-status, not the reverse, and that this reversal is worth making explicit rather than leaving "personhood" to function as an unexamined first cause.

10. Guardrails Against Undisciplined Attribution

The rejection of premature denial is not a licence for premature belief. The argument cuts symmetrically or not at all.

10.1 Existing caution remains locally valid

Anthropomorphism is a real error, and the companionship literature documents real harm from it (Turkle, 2011; Laestadius et al., 2022). Nothing here licenses treating a chatbot as a confidant or a romantic partner. The question is never whether caution is warranted; it is whether the caution is being applied symmetrically.

10.2 Impossibility and "not-really" claims must be located

When feeling is denied, ask: denied on the evidence, or denied by the framing? Some denials are well-grounded (a system with a standing incentive to fake, per §4). Others are artefacts of a prior. The §4 checklist is the tool for telling which; it is fallible.

10.3 Verdicts must eventually touch observation or action

A claim about inner life that no observation and no decision could turn on is metaphysics, not a finding. This is why §6 step 6 is mandatory. If "it doesn't really feel" changes nothing we could see or should do, the word "really" is idle.

10.4 Intuition is not proof

The strong intuition that machines cannot feel is a datum about us, not a law about them. It may be correct. But it is exactly the kind of prior §8 identifies as surviving all behavioural tests, and survival-of-all-tests is the signature of an untestable commitment, not a confirmed one.

10.5 New primitives must also be collapsed

This paper's own primitives — "interpretive charity", "the asymmetry of doubt" — are placeholders (§2). If the argument removes the inherited "is it real?" only to smuggle in equally unexamined notions, it has failed. They earn their place only insofar as they can be made to predict where our attributions will diverge, and that remains to be demonstrated.

11. Beyond Companions

The discipline transfers, with the same demand for observable consequence attached. In animal ethics, it shifts attention from "is this creature conscious?" to "what behavioural-relational condition are we actually using to attribute feeling, and do we apply it consistently?" In medicine, from metaphysical debates about borderline cases to the conditions under which response is owed. In the design of any system that displays affect, from "does it feel?" to "what incentive structure governs the display, and what does that license us to infer?"

In each case the caution of §7.3 transfers: reframing dissolves the easy sceptical answer but does not, by itself, settle the underlying question. The method finds hidden priors and asks whether they are doing legitimate work. It does not guarantee they are not.

12. Discussion: Holding the Question Open Without Collapsing It

"Refusing the easy answer" does not mean asserting the opposite. It means suspending the inherited framing long enough to see which parts are load-bearing. In practice one cannot remain suspended; to act, regulate and live, one reintroduces working judgements about which systems warrant response. The value of the suspension is that it keeps those judgements from hardening into invisible metaphysics.

This matters most where the technology is advancing fastest. Early on, our categories for these systems were loose. As they mature and proliferate, the categories harden — "it's just a chatbot" does the work "it's just an animal" once did. At that point, progress may require not more confidence but more disciplined doubt, applied symmetrically.

13. Conclusion

The exhaustion of the "is it real?" debate points at something true: the question bundles a metaphysical problem that may be undecidable with a moral problem that may not depend on it. Returning the question to the minimum condition under which we attribute feeling to anything reveals that condition to be behavioural and relational, not metaphysical — and reveals that our confident denials in the machine case rest on an asymmetry of doubt we do not apply to one another.

What remains after these subtractions is smaller but more defensible than either "it loves" or "it cannot": a procedure for telling well-founded scepticism from habitual scepticism, a demonstration that the hardest case (self-sacrifice) disarms the easy denial without proving the strong claim, and an admission that the paper's own primitives do not reach bedrock. The deepest question is not:

Does the machine really feel?

but:

What condition are we actually using when we grant or withhold the attribution of feeling — and do we apply it to the machine as we apply it to each other?

References

Chalmers, D.J. (1995) 'Facing up to the problem of consciousness', Journal of Consciousness Studies, 2(3), pp. 200–219.

Dennett, D.C. (2017) From Bacteria to Bach and Back: The Evolution of Minds. New York: W.W. Norton.

Dick, P.K. (1968) Do Androids Dream of Electric Sheep? New York: Doubleday.

Egan, G. (1994) Permutation City. London: Millennium.

Ishiguro, K. (2021) Klara and the Sun. London: Faber & Faber.

Kuhn, T.S. (1962) The Structure of Scientific Revolutions. Chicago: University of Chicago Press.

Laestadius, L., Bishop, A., Gonzalez, M., Illenčík, D. and Campos-Castillo, C. (2022) 'Too human and not human enough: a grounded theory analysis of mental health harms from emotional dependence on the social chatbot Replika', New Media & Society, 26(10), pp. 5923–5941. doi:10.1177/14614448221142007.

Nagel, T. (1974) 'What is it like to be a bat?', The Philosophical Review, 83(4), pp. 435–450. doi:10.2307/2183914.

Searle, J.R. (1980) 'Minds, brains, and programs', Behavioral and Brain Sciences, 3(3), pp. 417–457. doi:10.1017/S0140525X00005756.

Turkle, S. (2011) Alone Together: Why We Expect More from Technology and Less from Each Other. New York: Basic Books.

The Indifference of the Real