blog.bgcarlisle.com
The kind of regulation that AI needs but AI companies will never ask for
**Content warning: Mention of traumatic abuse**
The AI industry has two big problems right now. First, they’re not profitable. They’re nowhere close, have never been, have no plan to get there, and their finances are a gigantic multi billion dollar shell game that could collapse literally any second. The only thing keeping this from falling apart is that there are so many rich and powerful people constantly infusing the scam with cash, all of whom hope to get their payout before it all collapses leaving someone else holding the bag.
The second big problem facing the AI industry is that their product isn’t actually that good. Improvements to chatbot models are hitting a plateau, and people are starting to notice.
The AI bubble popping will be bad for everyone when it happens, but it will be especially bad for the people making the bubble happen if it happens before an event called an IPO (initial public offering) for that company. At an IPO, a private company becomes the sort of thing where a member of the public can buy shares of it as an investment. This means a massive flow of cash to the people involved in the company before the IPO. At that point, they can take their money and run if they want, leaving all the other investors with the circular trail of IOU’s which make up the economic foundation of the AI industry. So the only goal of nearly all the rich people in the world is to keep the rest of us believing that AI is inevitable, it’s the future, it’s powerful, at least until the IPO.
The IPO’s for Anthropic (November 2026) and OpenAI (sometimes 2027 I think) are on the horizon now. Unfortunately for them, public faith in the power of AI has been flagging as the capabilities of AI systems plateaued, public resistance to data centres grew, and results of third-party studies on productivity gains via AI all seemed to be pointing toward productivity losses or equivocal results at best.
In response, as a cynical marketing ploy, there’s been a spate of crimes committed by the CEO’s of these companies that have been publicized widely as “rogue AI.” This framing performs three functions that favour AI companies: First, it absolves them of the crime that the CEOs and other employees just committed. Second, it centres the power of AI in the public conversation about it. And even if it’s a negative framing, a story that starts with “AI is so powerful that—” is a great piece of publicity for AI. Third, it continues a process of the normalization of AI companies breaking the law that began months ago with scraping of texts that would certainly have been met with extreme legal firepower had anyone else done so (c.f. Aaron Swartz). Similar to the constant privacy scandals that boiled the privacy frog over the last twenty or so years and made us all unable to even perceive privacy violations as violating, tech companies are now trying to boil the rule-of-law frog by getting us all used to tech CEO’s just flat-out admitting to crimes and looking us squarely in the eye and telling us that they just get to do this, now.
It’s in this context that several tech CEO’s have called for regulation of the AI industry. They have done this before for various reasons. Before it was mostly done to set the “rules” of the AI industry in their favour, and to maintain their current monopoly status by enshrining into law some regulatory hurdles that only huge corporations would ever be able to clear. In the current context, it seems abundantly clear that the tech industry is running out of runway, and they need to keep up faith in AI before their IPOs.
This also squares with the recent coordinated calls from several tech CEOs for a “slowdown” or a “pause” in AI development, and the recent press about AI killing all, or a significant proportion of the human population. They know their tech isn’t going to get any better, because it turns out that massively scaled stochastic next-token prediction isn’t a great foundation for a general artificial intelligence project. IMAGINE. But they want to be able to tell prospective investors why, given that they’ve been offering them artificial general intelligence (AGI) “soon” for years. Sam Altman wants to be able to say “Of course we haven’t brought about the Singularity AGI god, but it’s just too powerful and we agreed to a pause in development in order to stave off human extinction.” Bill Gates wants to be able to say “We need to comply with all the relevant new regulations, so everything we promised you will come after we’ve done that.”
With all that in mind, you can imagine that I do not recommend that you take a tech CEO’s calls for regulating the AI industry to be made in good faith. But if we are going to talk about regulating the AI industry, the following are my thoughts on what aspects need to be regulated and some general gestures at how to do it, for someone who was interested in regulating AI with an eye to the public good.
# 1. Liability for the results of their outputs
“AI does crime, CEO does time”
In Canada, it is a crime to advise a child on how to kill others and then themselves. Apparently, it is not a crime for Sam Altman to operate an AI chatbot that does so, as there have been no criminal charges laid against him in regard to the 2026 Tumbler Ridge shooting.
Civil liability seems like a morally repugnant way to address harms of AI like this, for several reasons. Most people can’t afford to take on one of the biggest companies in the world legally, and even if they do win, a company that is forced to pay a monetary settlement for committing a crime doesn’t learn not to commit that crime; it learns the dollar amount that crime costs so they can budget for the next time they commit it. So unless we want to put a de facto dollar value on the life of our children, there must be criminal liability.
The focus of OpenAI in response to shootings counseled by their LLM has been on adding “guardrails” and discussing where the bar for contacting the police should be. The guardrails being added have been ineffective half-measures at best, and if “when to call the cops” is where the regulatory focus also ends up, there’s no reason for OpenAI not to make an overly sensitive system that absolves them of responsibility and flags thousands of false-positives to the police, who don’t have the resources to filter them all.
Even outside the context of shootings or terrorism or other violence, AI can be harmful in a million ways, and without regulation that forces them, there’s no reason for an AI company to ever address these dangers and harms. Deepfake porn, for example, is a huge problem for women and girls (mostly); what if we also made it a huge problem for the techbros who are profiting from the pushbutton deepfake porn machine by holding them legally responsible for it in some way? Political misinformation and spam is a huge problem for our democracy. What if we also made that their problem by telling them that if we have evidence that their products enabled it, they go to prison for election interference or whatever? I’m just throwing out ideas.
The liability doesn’t all necessarily need to be borne by the big tech companies either. A law that says the company that has integrated a tech company’s chatbot is responsible for its output would go a long way.
“Caveat emptor” isn’t a sign of a well regulated market, it’s a sign that the cheats are running the show. It’s not good enough for them to say “AI might lie to you, good luck.” We need the legal tools to hold their feet to the fire when they lie to us and hurt us. This means explicit law that names AI companies, their CEOs and employees as criminally responsible for crimes, as well as civil liability for the results of their AIs’ outputs, including everything from the big stuff that requires a million-dollar budget, all the way down to small-claims court. Otherwise there’s literally no incentive for an AI company to stop lying about little stuff.
# 2. Limitation on environmental impact
Everyone is aware of the environmental impact of data centres by now. They burn obscene amounts of electricity, they take up water, they destroy the local natural environment. They also produce no local jobs and do not necessarily confer any “digital sovereignty” just by virtue of where they are located. A data centre being placed in a community is not an investment in that community, rather it is as close to pure extraction from that community as is possible.
Public pressure has increased on this already, and the consensus is in: everyone hates them. We shouldn’t have to play American resource extraction exploitation project Whack-a-Mole and take down every data centre project one-by-one until they finally find a place where they can ram it through on a technicality or through flagging resolve.
We need to place hard limits on the energy and resources that can be consumed by this industry. We can’t just hope that market forces will prevent AI companies from consuming the entire biosphere, water table and destryoing the planet. There are rich and powerful actors who are contorting market forces beyond what they can sustain in order to prevent any limitation on the growth of the AI industry.
We need to decide as a society exactly how much of our country, its energy, land and water we’re willing to expend on data centres, and base that on the balance of the good that a data centre would produce for the public in comparison to the climate/land/water/etc impacts. For clarity, these limits should not be informed by tech company AI bubble speculation. Then we need to legislate those limits once for the entire country, so that our environmental policy isn’t decided by which local government is most easily corrupted or which one loses its resolve to prevent AI centre exploitation first.
# 3. Meaningful redistribution of money to offset enclosure of the commons
I am not personally convinced by copyright-based arguments against AI scraping. Copyright is a legal tool ostensibly aimed at a social good; there is no moral right of an author to say that anyone may not read or take value from a published book of theirs without their permission. Art and science are the shared common inheritance of all humanity. We owe access to the commons to each other, and we owe artists and scientists both credit for their work and to be able to build lives and careers based on their contibutions to that commons.
Copyright has been broken since before I was born, and the ability of modern copyright policy to meaningfully address the problem it is ostensibly aimed at, ensuring that artists and scientists are able to live while practicing their art, has only gotten worse faster in the age of AI.
In the internet age prior to AI, we sort of stumbled on a partial stop-gap solution, namely, receiving a pittance from Google via ad integration. The ancient bargain was more-or-less thus: Nearly everything on the web was free, monetization happened through ads. Google was the friendly middle-man who ran both the search engine and served personalized ads for each individual. They skimmed a percent off the top of the ad revenue. And last, we don’t talk about the dark spectre of privacy violations happening in the background as targeted advertising produces incentives for a black and grey market of personal data extracted nonconsensually that everyone shrugs off with “If it’s a problem for everyone, it’s a problem for no-one and I have nothing to hide.”
This arrangement was slowly ratcheted up until it choked out nearly the entirety of news media, as ad revenue was always a monopoly. (I won’t get into it in this blog post because it’s long enough, but you can extrapolate the problems for democracy itself if you imagine what happens to a world where the journalism is underfunded to the point of extinction and the worst people in the world have access to a machine that lets them micro-target political messages to all the racist grandpas of the world. BUT I DIGRESS.) It was a problem even before the AI moment that began in 2022, and it’s gotten worse because chatbots are now taking a significant part of the role that search engines used to take. People are clicking through links from their search results to webpages less and less. This is a problem because artists, writers, news media, basically everyone who relied on this economy is now squeezed in two ways: First, less revenue for the reasons I specified earlier, and second, AI scrapers are effectively DDOS-ing the entire internet, which also drives the costs of being online up.
What this amounts to is enclosure. They took a commons, namely the shared cultural inheritance of all humanity, all our words, art, traditions, music, stray thoughts, blog posts, social media, books, movies, all of it, everything culturally that we owe to each other and to posterity—they took it erected a fence around it and now they’re charging admission to it.
I personally think the project is wrong-headed, wasteful and that the value produced by post-2022 generative AI projects tend to grow sub-linearly with the costs of producing it. I’d just as soon shut the whole thing down. But if they’re going to fence off the commons of all human art and science, they owe all of humanity a cut of the proceeds, and how much should be a matter of public policy that is enforced on the industry by law.
Humanity needs artists, scientists, journalists, etc. We need something like the proposed and then removed Canadian Digital Services Tax, but broader. Tech companies are killing the business models that supported journalism, art, and other forms of expression. If they’re going to benefit from having fenced off the commons to charge admission, they owe it to us to meaningfully financially support the people whose work their AI depends on.
# 4. AI use opt-out mandate
“The right to not have to do it with an AI”
Despite the protestations of techbros everywhere, any time you do something with AI, it’s worse. Customer service is especially worse. The AI just isn’t there. It doesn’t understand you. The way that the post-2022 generative AI works, it’s a cheat, it’s just a massively scaled next-token predictor. It doesn’t have an internal model of what it’s talking about. It’s just a very good guesser of what the next word in an answer to your prompt should be.
This is why when you go on a company website, sometimes the chatbot’s output just detaches from company policy or even reality completely. Currently there’s no really clear law making anyone responsible for chatbot output (see section 1 above), although there’s some suggestive court cases.
And for a lot of stuff, it’s pretty low stakes, but for government services, banking, healthcare, anything essential, all it would take is some bad output from a next-token predictor and then it means you literally die, lose your house, send your entire retirement fund to a stranger, miss your scheduled surgery, or go to prison.
I think there’s a place for the market to decide whether we’re okay with AI enabled fast food or something, where if my coffee order gets messed up, the stakes are pretty low. But I’m not okay with rolling the dice on the important stuff. For certain parts of my life, I should have the right to access those services through traditional, deterministic computer software, or through a human, and not through an AI.
This also includes transparency about when I’m talking to an AI chatbot. It should be illegal to have a chatbot that doesn’t disclose that it’s a chatbot.
The technology isn’t there, and from what I’ve read there is no way to consistently prevent AI’s from going off the rails. So until AI is built on a sturdier foundation than spicy autocorrect, at least for the important stuff (and we can haggle over where to draw that line), everyone should have the right to not have to do it with an AI.
# 5. AI training opt-in mandate and liability for privacy violations
Everything that goes into a model for an AI can, at least in part, be extracted out of it again. And even if no-one ever does pull private data out of an AI model and harm me specifically, lack of regulation regarding what can be included in an AI model provides a financial incentive for massive privacy violations.
To put it bluntly, there’s CSAM in most AI image generators. Almost certainly, every time you ever used an AI image generator, at least part of the weights used to inform what appears on your screen included a child being literally raped. Think about that next time you generate an unnecessary AI image banner for your blog post or substack or whatever.
I don’t think there’s any law or regulation limiting what can go into an AI’s training data. If anything there’s been a loosening of laws to enable it. Certainly no one has gone to prison over the incorporation of CSAM or non-consensually sourced intimate images.
And so I’m proposing that we do: something. Come on people, almost anything would be better than what’s going on now.
Maybe we could have a law like the “right to be forgotten” but for AI. Or how about this? Some storng set of strong legal deterrents to make AI companies much pickier about where they get their training data from, and someone with the authority to audit training data sets to ensure compliance.
Just imagine a world where this happens: “This training data contains this girl’s private diary, where did you get it? Oh you bought it from a sketchy data broker who got it from a sketchy smartphone app that logged it and sold the telemetry? That’s a $500k fine.”
I also imagine criminal liability for the collection, use or distribution of data sets that include non-consensually sourced intimate images, or sexual images of children. But in the current world we’re living in where known Epstein associate Bill Gates is the one calling for regulation (and probably hoping to write it), I doubt that the exploitation of children will be much of a consideration.
# 6. Disclosure of, fair compensation for, and humane working conditions for “artificial AI” and other hidden data workers
“Beep boop nobody here but us robots”
For some reason, everyone keeps forgetting that LLMs are powered by torturing data workers in countries with worse protections than we have here in the US or Canada. This is because the way that tech companies have chosen to develop AI is by scaling LLMs by including everything on the internet except the really bad stuff. Hence, someone needs to make a decision, for everything on the interet, about whether it should be included or not. And because they’re going through everything on the internet, this means that the data workers need to see the worst things on the internet. There are people in Kenya who are paid starvation wages to be traumatized by worse things than you can possibly imagine.
And while that’s almost certainly the worst of it, even in the US and Canada, there’s highly educated people being exploited by this system to do work that would have been highly paid and highly secure only a few years ago, now for an increasingly precarious hourly rate.
When you see an AI doing a task, the assumption is that in some sense, nobody is doing the work. It’s a machine. And while that is kind of true, it’s also true that all of these machines are powered by the hidden labour of data workers. So yes, it’s a machine who just summarized your email, but if you look back one step further, there’s an army of people from mostly low-income countries being de facto tortured by data work that enables it.
The biggest innovation of the present AI moment isn’t that now a computer is doing things that a computer couldn’t do before. That’s mostly not true. The biggest innovation is that a company is able to obscure how underpaid and precarious workers are being exploited, and nobody has to even look them in the face anymore to do so.
Waymos, for example, are remotely piloted, at least some of the time. They don’t want you to think about this, because they want the futuristic veneer of a tech company. And this happens so much. So many “AI” things are actually “artificial AI.” And the fiction that it’s “just a computer” lets you ignore the fact that it’s actually a remote pilot or data worker with darker skin than you in a lower-income country. You get all the benefits of the exploitation, but none of the bad feelings.
I’m suggesting: mandatory disclosure of artificial AI. If you say it’s a computer doing a task for you but it’s just exploiting people, that would be fraud.
I’m also suggesting that we re-establish some new baseline humane working conditions for hidden data workers, and ones that can’t be circumvented by relocating workers to another country. Mandatory limits on how much traumatic material a company can ask a person to view. Guarantees on job security to eliminate precarious data work.
This is the kind of regulation that AI needs, but they will never ask for.
# 7. Mandatory testing of AI applications to ensure that they do not introduce or launder bias
One of the major problems with the use of AI is that, because it was trained on the internet, and because the internet was written by humans who have implicit and explicit biases against protected groups like women, racial minorities and queer people, these biases will be encoded into any AI application.
This is hard to deal with because AI models tend to be black boxes, so you can’t just open one up and see why a model tends to favour white job applicants over black ones and correct it that part of it. If the biases are subtle, you might not even notice that it’s doing that until after thousands of candidates have already been affected by it.
In order to protect the public, foster confidence, and even create a market for reliable drugs, pre-market testing is required by law for all drugs. In order to market a drug in the US or Canada, you have to convince the FDA/Health Canada that your drug is effective and safe, usually with evidence gathered in a randomized controlled trial.
There are certain areas of life where bias is really hard to detect, and it’s really consequential. I mentioned the example of job applications above. Transcriptions of medical appointments also come to mind. It’s also really easy for insurance companies to put their finger on the scale when asked to pay up on a claim, and an AI allows them to do so at scale.
I’m imagining a government agency that decides which applications of AI are areas of high public interest and high risk of bias against protected groups, and mandates testing requirements accordingly before they are allowed to be deployed in situations that could ruin the lives of an entire demographic of people.
Want to use an AI to sort your job interview candidates? Want an AI to transcribe your notes as a doctor? Want an AI to respond to insurance claims? Then you need to do an RCT first where one of the endpoints is “will the use of this AI create or worsen biases against protected groups like women, racial minorities or queer people?” And if you can’t prove that to a regulator, you don’t get to use AI in that context.
# 8. Total ban on AI use in weapons, military target selection, predictive policing
This last one doesn’t need much explanation.
In the area of literally killing people, “efficiency” is not a virtue.
If you disagree with me on this one, I don’t think there’s any words I can ever say to make you care about the welfare of other people, so goodbye.
I hesitated on whether to include “predictive policing” in this category or in section 6. I think it belongs here, because I can’t conceive of any application of AI in policing that isn’t designed to exacerbate inequalities against protected groups, but maybe you can convince me.
# Final thoughts
If you’re stuck around with me this long, wow. You have had a LOT of opinions from me today.
So I’ll leave you with a few thoughts before I thank you for your attention.
First off, regulation only works if it’s mandatory. Suggestions, expectations guidelines, they are worse than useless. Tech companies lie, cheat, ignore the rules, break the rules, and lobby the goverment to change the rules to benefit them. You absolutely cannot expect good faith from anyone in the tech industry. They have betrayed our trust so often and consistently that we need to treat them as hostile actors. To do anything else is naive and irresponsible. Regulations are only valuable if they are enforced, and have meaningful consequences if they are broken.
Second, while I have conceived of all of these together as regulations that we need against the current hostile advance of AI and the companies behind it, it would be progress if _any_ of these regulatory suggestions were acted on.
Finally, I started this post talking about the fragility of the economic situation surrounding AI. Pretty much any of these real regulatory efforts taken seriously would probably pop the AI bubble. This has to happen sooner or later, and the less resources we invest, the less carbon we put into the atmosphere, and the fewer lives we ruin with deepfakes, traumatic data work, bias laundering, privacy violation incentives, school shootings, political misinformation, and the general introduction of slop into the world, the better.
