Jenny had a problem. She was asked to write code to tell what the car should do if it had to decide who to hit. It was her boss, Dr Martinez with the cotton panels of his shirt just barely hugging his rotund tummy, several buttons screaming, “We can’t hold these in anymore!” who wobbled into her sight to tell her so.
“Jenny, LM9X has access to the population data now. It’s been approved as a pilot program. Therefore, we now have to decide who gets hit and who doesn’t. You know, the old trolley problem? Yes, yes, like that Jenny. Now that we have access to the government’s data bank…” Dr Martinez droned on, not caring whether Jenny was still paying attention or not.
Jenny knew instantly that this was going to be a problem. Although she was one of the best programmers in the whole country, she did fail her Philosophy 201B. Her mind was just not wired that way. So how do you get a machine to learn Ethics? Pattern recognition, yes. Financial valuation of companies, easy-peasy, but valuing lives? Jenny might as well quit right now. How much was the number in that Savings Account again? Jenny wasn’t great with money either. She was only great with code, just not the ethical kind.
Soon, word got out that Jenny was writing a very special program. The animal people were the first to contact her. “Hello?” She placed the call on speaker. “We are HTFAAAK and we’re calling to remind you that animals are people too.” Of course, what they meant was that animals have souls and therefore should not be disregarded. The animal people liked to name their organisations with a chain of funny long letters, really! She was not sure why, when animals don’t even know the alphabet. Fine. She might add LemurFaceID to the model. Might.
LemurFaceID started out as an ID bank of lemur faces to track the creatures unobtrusively via facial recognition but the catalogue quickly expanded to include other species. The original name stuck however, and today, you can find a variety of animals from elephants to mousedeers to tigers registered there. Some creatures, due to logistics, stay off the databank, such as ants, bees and fish, although Jenny thought that these too would eventually be catalogued by swarms, colonies and schools. Ridiculous the lengths they go to these days, Jenny thought. Cats on YouTube are no longer merely cute nameless characters – google their ID and a site detailing the brand of cat food they preferred popped up should you wish to show your appreciation and “donate” in Bitcoin. You can blame Iceland for that, when they began a reality show with cat stars called “Keeping Up with the Kattarshians” in 2017. India was very proud of their cow databank and even categorised them into six castes or jatis. A few cows became so popular that fan clubs were formed in their honour.
The very same day, Mrs Higgle popped in from next door just before tea. She too, wanted to remind Jenny that whilst she needed dentures in her mouth to eat and a cane to walk, she had not even one foot put in the grave yet, young lady. In fact, she had been getting the best time with the widower Mr Collier across the street. What? Did you think old folks don’t get jiggy, Mrs Higgle chuckled. Oh my lovely, just wait, better than when you were in your twenties! And she walked out the door leaving Jenny with a somewhat dazed expression.
Dr Martinez materialised again beside her desk the day after, and boomed, “Jenny, remember to put in your code that should you ever be a potential, the car must never ever choose you.” He noticed a piece of skin edging out of the nail on his thumb and tried to get at it with his teeth. “Uhh, you don’t want the programmer to be killed, do you, then we will never be able to debug the bloody code if one is found in the future!”
Jenny didn’t think of that, but he did make sense and she dutifully placed a line halfway down in Xython, which roughly translated in normal words – “Do not hit Jenny Huang: ID 73529G”. It was very easy for Jenny to code whom not to hit, but to instruct the car to hit someone instead of another? That’s a toughie. You know what? Perhaps she should just write what seemed like endless lines of code and hide from Dr Martinez that the result would be a random one anyway. Except for a select few that must not be potentials for any reason whatsoever, of course.
Jenny was beginning to have an idea how to code this. Using matrices and correlating each individual’s dataset to a host of desirable values to society. Score and rank them – this was going to be easy after all, thought Jenny. Productive members against the ZMP workers, the working age versus the pensioners, QALY, what else… Jenny was tempted to include Nana in the special do-not-ever-hit list because she would otherwise come at the bottom of the rank. Hmm… should she? Who would know, right? Dr Martinez wasn’t one to comb through the code searching for anomalies. A little bit of power does corrupt, thought Jenny, but this is Nana, the woman who brought her up!
Two years ago, Jenny was the brainchild behind the absurdly famous Womb car, a self-driving car that made the occups felt like they were in a womb. Safely protected, comfortable – the car continuously monitors the group’s body temperature and adjusts the interior’s climate accordingly. No more leather seats, occups sink in lequede that molds around the body but quickly transforms into protective shells around each occups should the vehicle crash. Jenny even programmed the car to be the perfect traveling companion, knowing when to conduct small talk and when to stay silent, selecting the right personalised auditory and visual stimuli by sensing the mood and neurowave of the occups. It wasn’t Jenny’s fault she designed Womb so well some occups became addicted to the well-being feeling of sensory completion Womb provided. So much so, that they remained in the car even when they weren’t traveling! The most severely addicted occups refused to leave the car but for the most basic of needs, which lead to the firm Jenny used to work for receiving a huge fine from the International Asilomar Court. Womb II was in mid-production when it was halted. Jenny became the sacrificial lamb and was fired.
The International Asilomar Court was the third iteration in an internationally organised attempt to reign in the development of AI. Always seemed though, that developers were always five and a half steps ahead of AI judges and lawyers. The body of Acts covering AI had long grown into a behemoth and still the judiciary system had to constantly play catch-up with the things Jenny and people like her came up with on a daily basis. It was no longer the pen mightier than the sword, but rather the code mightier than… Jenny didn’t know mightier than what, she just knew that she could do many wondrous things by being able to code. Thinking about it, Jenny wondered why she had never thought of developing the ultimate legal mind that can outwit any human or legal system. Well, she might just if she was brought to the court again…
Unlike the pharmaceutical industry with its drugs regulated to the teeth, the thriving AI industry had been given a lot of leeway. No long term trials needed before an intelligence product is released, heck, not even a committee to get approvals from. As long as what you built has not harmed or killed anyone intentionally, the sky’s the limit! Literally, thought Jenny, as she wryly noticed a few company drones monitoring her walk down the lawn of the complex. Stagnation in the economy too, had forced many government bodies to look away in the hope of any little shoots of growth. There were merits to how seemingly non-invasive the introduction of AI to the consumers was, underestimating how deeply consequential the little tweaks Jenny made could impact the lives of the users. For that Jenny was glad, because as an Artisan Coderm (sic!), she would feel her creativity stifled should she be told no to every new suggestion.
Once, her cousin Maya made a mistake of confusing her for that other breed, those idealess, clunky, Soylent-bland machine learning scientists, but Jenny forgave her. Maya wasn’t so bright so she became a doctor and guess who co-designed the heart bypass HB599 that Maya’s profession loved so much? Jenny of course, in her final year at Imperial College. Ah, doctors have it easy, just push a few buttons and the job’s done while she had to spend months teaching the technical and practical medical knowledge into the machine. That reminded Jenny – she should look up her old supervisor Dr G to discuss the hyperparameter tuning of the model over at GI. What did he say many years ago? Oh yes, “Combine ruled-based learning, deep learning and stats learning into an ensemble and you’re good to go kid! Combine, combine, combine!” Dr G… Jenny shook her head. Funny fellow. He used to be at MIT before the country kicked all the Iranians out years ago. Hah, that country’s loss was Britain’s gain, mused Jenny to herself, as Dr G was the leading expert in Gaussian Probability at the time. He was also a great faux-father to her.
Not that Jenny didn’t have one, a father that is. Just that she couldn’t joke around with her real father like she could with her faux one and discuss neural networks. In fact, her real father absolutely hated robots and machsAIs with a vengeance for stealing his job. He was a financial advisor before FinTech came along. Since then, he wouldn’t even use anything that was obviously run by AI. The ones not so obviously AI-managed were still something he sniffed at but he would still use them. In some cases, her father was more ‘practical’ before ‘principled’, fortunately though, that was one of the fifteen things she liked about him.
A few days passed and Jenny sat slumped in her office chair. After a while, she turned right to face the Prince who sat next to her, day in and day out. He wasn’t really a prince, she just called him that for the many times he’s saved her life. “Prince, you know Buridan’s Ass?”
“Uh?” The Prince was obviously too deep into his code.
“Prince, do.you.know. Buridan’s Ass?”
This time, he turned towards her, his hands still deftly moving in the air, adding more lines to his code. “You mean where the donkey couldn’t decide whether to choose the food on his right or his left and ended up starving to death?” Jenny admired the Prince’s multi-tasking skill. No one should dare talk to her while she was coding, on the pain of death. They needed to wait until she came up for air or took a pause to unwrap her chocolate bar.
“Yes, that one. Jean Buridan said, ‘should two courses be judged equal, then the will cannot break the deadlock, all it can do is to suspend judgement until the circumstances change, and the right course of action is clear.’”
“Why are you asking?” The Prince was also not the kind to entertain superfluous information.
“Nothing… I’ve got a kind of meta-stability situation here. Sometimes the two options have the same score and the self-drive can’t decide which to hit. A glitch for other things is ok, but in an accident, milliseconds count. You cannot, as Buridan said, ‘suspend judgement’. Any idea how I could resolve this?”
The Prince took just the slightest pause in his air-typing to think and said, “Toss a coin.”
“Pick randomly?”
“Yes.” The Prince wasn’t much of a talker either.
Jenny squeezed her eyes shut to follow the logic through and see the merit of what he had said. He was right, that would be the only resolve. Then again, if equal scores resulted in a random selection, wouldn’t it be unfair that two people having different scores are not randomly selected as well? Should fairness be a consideration? But fairness in what context? For the thousandth time, she cursed this assignment with a long string of Malay expletives. The Prince was used to her cursing and thus didn’t pay her much attention, his eyes glued back to the screen like it usually was.
After work, Jenny made her way to GI on her bicycle, just a few minutes ride from her own South Kensington office. She checked the pressure on her tyres, pausing a moment to run a finger on the discrete rubber tag on the front spoke. She designed that, she thought proudly. It was one of the early products she worked on when she began in the self-drive industry. At the time, self-drives were having trouble identifying bicycles and motorbikes on the road, and Jenny offered a low tech solution where all the two-wheelers had to be tagged by law to emit a signal. Two tags must be placed, one on the frame of the bike, and the other was placed on the front wheel so that orientation and direction could also be determined. The tags were then charged by the kinetic energy of the wheel rotation. Now that almost all vehicles were electric more and more people were cycling as air quality and safety had improved so much.
Dr G’s office was situated in the basement, because according to him, he had no time to gawk at some scenery outside and secondly, he wanted to be far, far away from the admin. His computer screen was angled such that visitors wouldn’t be able to see what his screen was displaying.
“What do you mean, random?” For a skinny guy, Dr G sure could bellow. Jenny leaned back against the far table stacked with several opened Amazon boxes and packaging. His office was a masterclass on organised clutter.
“Yes, random. Why shouldn’t it be random?” Jenny felt a little bit annoyed at that lack of agreement when it was so obvious why.
Dr G began to pace and Jenny knew what that meant – he wasn’t just annoyed, he was fuming mad with her. “Very careless Jenny. To not choose is very, very careless…”
“So you want me to play God?” Jenny challenged him. The angrier Jenny got the more still she became, and Dr G knew that as well as the little things Jenny knew about him.
“OK. OK. Calm down…” Dr G held both palms up to pause the conversation and used the moment to take a deep breath. They shouldn’t be shouting at each other when just outside the door was a busy stream of graduate students.
“Imagine Jenny, a choice between a man – Peter, whose medical record says that he will not last the week and a young man, let’s name him David, neh? David, who according to his profile is a valuable member of the cancer research team whose lab is just down this block. Which one would you choose, eh Jenny? This team is about to have a breakthrough that will save the whole world from cancer. Tell me again whether random is moral. Random is careless. Apathetic. Even cruel!”
Jenny looked up, noted the small curry stain on his lapel. “Society’s values change all the time, what they appreciated last century might not be what they appreciate today. If there was a choice between Socrates and the village bard back then, who would you choose? Socrates, yes? Because of the influence, the impact. Today, Elon Musk or Beyoncé – which one would society choose? Which one would you choose?” Jenny gave him a side smirk, knowing how crazy he was still for the now decidedly middle-aged Beyoncé.
Jenny squatted down to take her drive out from her bag and set it by the photo frame of Dr G’s family so that he wouldn’t forget to go over her code later.
“Random is fair, Dr G. Random is how life operates. The natural way to select for accidents beyond our control is by not selecting.”
“Oh come on Jenny, even when we had drivers back then it wasn’t fully random. Each driver had his or her latent biases that they used to make the choice. Plus, what happens may be out of our control but not the how. Perhaps with self-driving cars, it is God’s way of giving us back a little say in how life goes. A sliver of grace so that society wouldn’t lose so much.” Dr G combed his fingers across what’s left of his thinning hair. “You can’t just pretend that the access to dataID does not matter, it matters, and you know it deep down Jenny.”
She replied, “Even if you have a high certainty of who you should be choosing, the outcome of the accident is still highly uncertain. I still don’t think it’s as straightforward as you make it out to be – it is an illusion of control. Calculations cannot include every probabilities – look, I’m sorry if you disagree with me. Please don’t be angry…” Jenny scooped her leather jacket off the chair and left.
Perhaps she didn’t need to solve the trolley problem after all and look at this from a different angle. Since both a self-driving car and a trolley are a form of transportation, maybe people naturally assumed that one puzzle could be solved by looking through the window of the other… Could it be that she was barking up the wrong tree? Her anger at Dr G forgotten, Jenny scratched the back of her head. Her brown hair was getting long and needed cutting.
Several years ago, a spate of self-driving cars were hijacked for terrorists’ purposes. She was then urgently asked to program for all self-drives to have an automatic kill-switch should the same thing happen again. The last straw for the Berlin MachAIs Symposium was when in Istanbul, a car was made to swerve into a crowd of tourists. Jenny heard that it might have been the Buyukada Liberation Front who hired the hackers to ram the car into the morning queue outside of Yerebatan Sarayi.
Although the autonomous breaking system was standard by the time, the outdated system was easily bypassed. She wished the Symposium had come to an agreement on the kill-switch guideline before so many lives had been lost, but there were concerns about letting an outside agency, be it the police or other government bodies being able to overtake a self-drive’s control. A self-drive was not only a convenience but could also become a missile, Jenny thought. As long as both were possible, programming self-drives should cover the possibility that it could be used for unintended purposes.
The Berlin MachAIs Symposium was a gathering of all self-driving technology developers as well as many government bodies. At first, they proposed that to reduce accidents, all self-drive traffic should be tracked as how the planes were, but someone reminded them that for the planes, flight plans were pre-submitted which were not so for car journeys. More importantly, privacy of the occups would also be breached, a big deal especially to the privacy-loving Germans hosts. Finally, all the carmakers agreed that the cars would interact with each other when coming within a certain range. This reduced the need for the car to rely solely on sensors and radio to detect other vehicles. Car to car communication was not a new idea, but the old technology that relied on transmitters and wireless was riddled with problems, especially when trying to calculate signals in a vehicle-crowded environment.
The information each self-drive shared with each other was not only things like position and speed, but also the ID of all occups, although this ID information was only kept for the duration the cars were within the specified range – in case of accidents with one another. That, and coupled with the fact that in 2021, Ford had removed the steering wheel, brake and gas pedal from their cars with other carmakers following suit, car insurance became a thing of the past and liabilities were now held by the carmakers.
***
“I overheard something from Dr Martinez that may be of interest to you…” The Prince was Cambridge-educated and it showed in his speech.
“What about?” Jenny loved gossip.
“Well, apparently he was contacted by the Jewish Something-or-rather Organisation who said they were not comfortable with you playing God and to let God play God. And if you were not going to randomise the decision, then you should make some number of self-drives that do – some kind of kosher car, I think.”
“What!” Jenny snorted. She came from a family of different denominations, including Christian, Jewish and Muslim, so she was almost sure that God exists, she just wasn’t sure whose.
“But, I’m not finished,” the Prince said, “The Muslim Something-or-rather Society heard about it and said if you were making kosher cars then they wanted you to make halal cars as well.” He looked at her pityingly.
Hearing it, Jenny wasn’t sure whether she should say a string of prayers or for the thousandth time let out a set of expletives. She chose instead to call it a day and head home. All this, she thought, can wait until tomorrow.
At the door, Jenny dumped her bag by the stairs and called out, “Nana, Nana!”
Nothing like coming home to the smell of Nana’s cooking after a hard day. Jenny’s mom was killed by a drunk driver when she was eight and Nana had put on both mantle of mother and grandmother. “Nana, I’ve placed your ID on the list.” Jenny scooped the steaming rice onto her plate. Her grandmother was just coming down the stairs with a big welcoming smile on her face.
“What did you say?” Nana took the plate from her and added the grilled aubergine and fried fish coated with turmeric onto it, Nana was always taking care of her.
“The do-not-hit list Nana, I’ve put you on it. You know, the thing I’m working on…”
“Don’t, Jenny! Take me off! It’s not fair to the others.” Nana had eaten earlier, she usually just watched and shook her head in a mixture of admonishment and amusement at how ravenously hungry Jenny was at the end of a work day. Today though, Nana did not look pleased.
“What is fair then, Nana?”
“To be fair is…” Nana stopped in mid-sentence, “fairness is…” Nana tried again, but then shrugged her shoulders and urged her to take a second helping, which Jenny didn’t refuse.
“Fairness is not a tidy word you can describe in a few sentences and say, yes, that’s it. I’m an old lady and my attention span is now too short to answer brutal questions, but if you ask me, it’s simple – something that is fair is what doesn’t turn people’s stomach when hearing about it.”
Nana closed the rice pot and took a seat across her. “I’ve lived too long to pretend I don’t know what is right and what is wrong. And raised you well enough to know that you are a human being first, before you are a programmer. If, Jenny, you ever had to put something in that code of yours that made your stomach turn, don’t do it. Models don’t substitute for morality. Underestimate your responsibility, and many people will get hurt by this. How do you think history will remember you? Take me off your list, Jenny!”
Jenny looked carefully at the aged face who had pampered her with so much love, when had she grown so old without her noticing? She gave a resigned sigh and said, “Yes, Nana. I’ll do it tomorrow.”
In bed, her head still too active to sink quietly into sleep, Jenny kept remembering what Nana had said, about being a ‘human being’ first, but what did Nana mean by that? She grabbed the notepad from her night table. She had always believed that lists and diagrams cleared the mind and helped one to think. Outside, the wind was howling. Storm Doris, Jenny thought it was called. If only they would name storms after the dwarves in The Hobbit, Storm Thorin would have been cool. Jenny began the rough list:
- Randomise, only maximise the number of lives (utilitarian)
- Randomise, maximise the number of lives, with conditions (utilitarian with values)
- Not randomise (maximise good for society):
- with only an exclusion list (protecting most valuable members of society)
- prioritisation without exclusion list (all members are ranked according to contribution and importance to society, but without name-by-name no-hit-list)
- with exclusion list plus prioritisation (elitist)
First thing that Jenny noticed were the two overarching objectives: one, to maximise the number of lives not put in danger and two, to prevent a huge loss to society. It was clear that there would be cases where the two motives stood in conflict with one another. She was almost certain that ‘maximising number of lives’ should always be heavily weighted, but in some cases could be overwhelmed by other objectives.
The first on the list assumed that no filter was placed and that the only objective was to maximise the number, regardless of who was in any of the groups. The granularity increased with the second point, where the decision was still randomised, but with conditions that loosely reflected the jumbled-up and unsorted container of society’s moral values – perhaps women and children are prioritised, or that a pregnant woman is counted as two lives. Something, anything, as long as it was an obvious procedure to demonstrate that morality was being practised without really being chained to it. Hopefully, this would persuade some of the opinionated public from labelling the decision process as careless, inhuman, unfeeling or barbaric. There should be a distinct impression that the distribution of risk over the total population is somewhat egalitarian. Perhaps minimising the single largest harm across all victims, a Rawlsian Maximin solution.
Jenny got off the bed and stretched. She walked to the kitchen and turned the kettle on, Earl Grey, she thought. Jenny looked out to the back of her garden and saw that the storm had tipped the empty black garbage bin over. Like a joyous child freed from his overbearing nanny, it rolled around the lawn on its side, to the left and to the right, and sometimes even spun around, tossed aimlessly as the wind saw fit. Jenny hoped the storm wouldn’t take any lives tonight, or fell a tree onto someone’s property. Act of God indeed, Jenny mused, and sat down at the kitchen table to continue working.
Certainly, maximising good at a time when society needed every little help it could get seemed too important an aim to dismiss. In the list, 3.1, a group of people would be selected to be protected at all cost, with those off the list to be randomly picked. This reminded Jenny of the government agents who put their lives at risk to protect the Prime Minister, except that in this case, the consent from the population was needed to protect the select few deemed too important to lose. The existence of this list however, would encourage corruption of the list-makers by those powerful and wealthy enough to want to be included. Who gets to edit the list, and what would be the criteria for inclusion and expulsion? Unless the list is kept secret, there would be a lot of public outcry. The idea had merit, but too vulnerable to manipulation and hence, turn the stomach, as Nana would say.
Absent of the no-hit-list, 3.2 would be based on a score and rank system. There was a time when inequality meant inequality of income, or wealth, but Jenny guessed that inequality could also feature as a risk distribution, with the weakest and the least able members of society bearing the most of it.
For 3.3, there would be a no-hit-list, as well as ranking for those not on the list. In Jenny’s view, this must be the most elitist thing she had ever come across. Didn’t Pope John Paul II say, “A society will be judged on the basis of how it treats its weakest members and among the most vulnerable are surely the dying.” Churchill as well, said something similar, “You measure the degree of civilisation of a society by how it treats its weakest members.”
There needs to be a universal code of ethics used by all the carmakers, of course. It wouldn’t work if each developer came up with their own version of ‘what is right’. One self-drive’s decision would compel another self-drive in the vicinity to have to make an ethics call as well, and these calls must co-ordinate, not be at odds with one another. Thankfully, it had been decided that self-drive accident algorithm cannot be patented, otherwise, this would have ended up as a patent war similar to the CRISPR 2021 fiasco.
Jenny took a sip of her Earl Grey tea. She should get in touch with friends still living in California. A long time ago, people like her used to go to California as if it was the Promised Land, and now, only the old techs who were too comfortable in their homes remained. Shenzhen instead, had been wearing the crown for more than a decade. There, in China’s version of the Silicon Valley, Graphene Ghetto was an especially attractive place for young entrepreneurs to build their start-ups. She will call on her friends, some of the ones who were actuaries wouldn’t hurt either, and try to find out if they knew what Socrates meant when he said that understanding a question is half the answer.
Jenny thought back to when Dr G compared a dying man to a cancer research scientist. She looked through the window once more, and there was the black bin, its roundness dimly illuminated by the light from her kitchen. She decided not to brave the outside to put it away and let the bin continue its merry dance in the rain, the sound of its empty rolling becoming a part of the storm’s midnight orchestra.
EightAteEight 