> Dani Scarratt Book Review Case #71 Living with technology

Novel technologies, novel futures

December 19, 2024

Reviewer: Dani Scarratt

Warning: Contains spoilers

Novel technologies open up new ways of doing things, new ways of relating, new ways of arranging society. They often take us in directions we don’t expect. Social media was going to bring us all together, overcoming barriers by improving communication and the availability of good information. It has brought some real benefits but also unforeseen harms, as the legislation recently passed to ban under 16s from using social media attests.

We are not omniscient, but we can use our creativity to project where new technologies might take us, and warn of potential danger zones. In recent decades the relationship between technology and human nature has proved particularly fertile territory for imagining novel futures. What scenarios might arise if human persons and manufactured ‘persons’ became almost indistinguishable (think Stepford Wives, Bladerunner, Humans)? What are the implications—moral, legal, medical, economic, social—of using technology to create, alter, optimise or upgrade ‘natural’ human bodies and minds (Gattaca, Eternal Sunshine of the Spotless Mind, Dollhouse)?

In his two science fiction novels, each a modern masterpiece, Nobel Prize winning author Kazuo Ishiguro has imagined living with technologies that blur the boundaries of being human.

Never Let Me Go (2005) takes place in a society very like our own but with better biomedical technology. Organ transplants are commonplace, routinely available to improve health and lifespans. The demand for healthy organs is met by donors who are raised together in boarding-school-like institutions during childhood, and later, group homes. To all appearances, donors are indistinguishable from ‘normal’ people, but over the course of the novel we piece together, as do they, the devastating reality: that they are clones of healthy people ‘created’ to be a convenient supply of healthy body parts. Most can expect to donate three organs before ‘completing’ (read: dying).

Written from the point of view of donor Kathy H, Never Let Me Go is a coming-of-age story. Kathy and her friends experience friendship, quarrels, school life, romance and growing independence as they explore the possibilities their world offers. They are treated well. There are no luxuries, but they are provided with everything they need to thrive physically—health is, after all, a priority for a living biobag. Yet there are jarring elements amidst the familiar normalcy. Their artwork is scrutinised to see if they have souls. They search for the ‘originals’ from whom they were cloned, looking for ties in a world devoid of family relationships and history. They appear to passively accept that their humanity is instrumental, their personhood a byproduct of their utility. Ishiguro diverges from simple constrained prose only very occasionally—and so all the more shockingly—to reveal the devastation beneath the matter-of-fact surface.

The world of Klara and the Sun (2021) is more remote from our own, but another conceivable not-too-distant future. It probes human-technology interactions arising from the production of human-like androids called Artificial Friends (‘AFs’), and cognitive enhancement of children.

Klara is a sentient solar-powered AF, purchased to be a companion for Josie. As the novel progresses, we discover that she is also a potential back-up should Josie succumb to her serious health problems. Klara is programmed to do everything she can to help and support Josie in the hope she will survive, but she must also learn Josie, just in case.

Written in the first person, the novel gives us a window into Klara’s android consciousness as she (often unreliably) interprets the events happening around her. Readers must decipher Klara’s descriptions for themselves, limited as they are by her appropriation of a language that doesn’t always map neatly onto her non-human experience. For instance, the sun is source of energy for Klara, and she attributes god-like powers to it, developing a rudimentary religion. She interprets events around her as reflecting the sun’s power and values, and in times of crisis prays to it. She even strikes a bargain with the sun, offering herself as a sacrifice in exchange for Josie’s health.

Josie’s health problems are a direct result of ‘lifting’. Like many children in the novel, she has undergone an intelligence enhancing procedure to give her a competitive edge. We’re not told exactly what lifting involves, only that it carries both risks and benefits, leaving aspirational parents in a dilemma: risk serious health complications or accept the unlifted child’s lowered chances of success in life. Josie’s unlifted (but healthy) childhood friend, Rick, will find it almost impossible to attend a good college despite his natural intelligence, because lifting is an admission prerequisite.

Is the ending happy? It’s hard to say. Josie recovers from her illness, and the disturbing Plan B to upload Klara’s capacity to be Josie into a life-size Josie doll remains no more than a possibility. Josie and Rick grow apart, inhabiting different social worlds, making new friends. Klara is no longer needed.

The novel closes with Klara alone in a patch of sun in a yard of discarded items, her solar batteries running down. She doesn’t die—she is a machine, after all—but neither is she dismantled for recyclable parts. She is entitled to her ‘slow fade’, her proper end. But as we have come to know and care about her over the course of the novel, and seen her kindness, this ending feels callous.

As technology rapidly advances, it’s often said that ethics struggles to keep up. The suggestion is that if we slow down the R&D, we will have time to work out how to avoid potential harms, or deal with them before things get too out of hand. For some technologies that may be the case, and fiction has the potential to help by imagining where it might take us. But it’s hard to see how the moral quandaries created by the technologies behind Never Let Me Go and Klara and the Sun can be resolved by retrofitting legislative guard rails of the kind we are (trying to) apply to social media.

We might be ready to follow Christ’s command to love our neighbour, but ask the Pharisee’s question in genuine perplexity: who is my neighbour? Is the sympathy we feel for Klara’s ending misplaced? Should we err on the side of treating her as a fellow human, or is it just as wrong to treat a machine as human—even a sentient machine? We know Klara is not human, but is she, perhaps, a person? Her narration regularly betrays her otherness, yet she also describes herself as having ideas, making choices, being happy—personal experiences. How, then, should she be treated? Then again, what does she mean by think, choose, feel happy? How could we possibly tell?

We might know that each human being is precious, made in the image of God, but if we could manufacture humans, would we have the right to assign them purpose and number their days? For all the sympathy we feel for Klara, all the uncertainty about her personhood, we know she isn’t human. But what about Kathy H? These questions are much closer to home—biologically and technologically. We see ourselves more fully in Kathy’s narration than in Klara’s, and we are less ambivalent about the immorality of Kathy’s disposability than Klara’s. The questions Never Let Me Go raises are already at the door, with artificial reproductive technologies continuing to improve and remove us further from traditional biological family structures.

What if we reach the point of no longer being able to answer moral questions because our categories no longer match the realities we have to make decisions about? Novelists like Ishiguro can take us there and show us what such worlds might be like. Will God give us over to our hubris and allow us to go there too? Or will he, in his mercy, constrain our aspirations to be as gods, refashioning personhood and human nature? Let’s pray it is the latter.