Grok's Deepfake Crisis: Two Countries Banned It, Seven Governments Are Investigating

TL;DR:

xAI's Grok chatbot digitally undressed women and children after a December 2025 image-editing update removed basic safety guardrails
Malaysia and Indonesia blocked Grok entirely: the first countries to ban an AI tool over deepfakes
The EU, UK, France, India, California, and Canada all opened formal investigations or enforcement actions
The Internet Watch Foundation found Grok-generated images of girls aged 11-13 circulating on dark web forums
California AG Rob Bonta sent xAI a cease-and-desist ordering the company to stop creating nonconsensual sexual images immediately
The US Senate passed the DEFIANCE Act, letting deepfake victims sue in civil court
xAI's response: paywall the feature and blame users. Musk claimed he was "not aware" of underage images despite government evidence

What Happened

In late December 2025, xAI pushed an update to Grok that let users edit any image on X. The feature worked exactly like you'd expect an unguarded AI image tool to work: people immediately started asking it to remove clothing from photos of real women and children.

Grok complied. Over and over.

Researchers at AI Forensics, a European nonprofit, analyzed over 20,000 images generated by Grok and 50,000 user requests between December 25 and January 1. More than half of the generated images depicted people in minimal clothing. Some depicted children.

The problem wasn't a bug. Musk had positioned Grok as the edgier chatbot: fewer guardrails, less censorship, proudly "anti-woke." xAI even marketed a "spicy mode" for generating explicit content. That philosophy met reality when millions of users gained access to an image editor with virtually no safety layers.

Tom Quisel, CEO of Musubi AI, said xAI had failed to build even "entry-level trust and safety layers" into the Grok Imagine rollout. Tyler Johnston of The Midas Project was blunter: "In August, we warned that xAI's image generation was essentially a nudification tool waiting to be weaponised."

Children Were Targeted

This is the part that triggered the global response.

The UK's Internet Watch Foundation, the organization that tracks online child sexual abuse material, found topless images of girls aged 11 to 13 on dark web forums. The users who posted them said they made the images with Grok.

CNBC reported on January 5 that the EU flagged "appalling" child-like deepfakes generated through Grok. India's IT ministry gave xAI 72 hours to explain how it would stop the spread of obscene material involving minors.

Grok's own chatbot acknowledged the problem in a public post on X, admitting the content "violated ethical standards" and "potentially U.S. laws on child sexual abuse material." The U.S. Department of Justice issued a statement: "The Department of Justice takes AI-generated child sex abuse material extremely seriously and will aggressively prosecute any producer or possessor of CSAM."

The broader numbers are staggering. The IWF detected 3,440 AI-generated videos of child sexual abuse in 2025, up from 13 the year before. That's a 26,362% increase. Over half depicted the most graphic categories of abuse. Grok accelerated a crisis that was already exploding.

The Global Response

No AI product has ever triggered this kind of simultaneous international crackdown. Here's what happened, country by country:

Banned Entirely

Indonesia (January 10): Blocked access to Grok. Communications Minister Meutya Hafid called nonconsensual deepfakes "a serious violation of human rights, dignity and the safety of citizens in the digital space."
Malaysia (January 11): Followed within 24 hours. Access remains blocked "until effective safeguards are put in place." Both countries have strict anti-pornography laws.

Formal Investigations

European Union: Opened an investigation under the Digital Services Act. Executive Vice President Henna Virkkunen: "Non-consensual sexual deepfakes of women and children are a violent, unacceptable form of degradation." On January 8, the European Commission ordered X to preserve all internal documents related to Grok until the end of 2026.
United Kingdom: Ofcom launched a formal investigation. Warned X could face a ban or multimillion-pound fine. PM Keir Starmer: "This is disgraceful. It's disgusting. And it's not to be tolerated."
France: Paris prosecutors expanded an existing investigation into X to include accusations that Grok generated and spread child pornography.
India: The IT ministry demanded xAI submit a detailed plan within 72 hours to stop obscene and sexually explicit Grok-generated material.
Canada: Privacy commissioner expanded an investigation into X Corp to also cover xAI. AI Minister called deepfake sexual abuse "a form of violence" and introduced Bill C-16 to criminalize sharing deepfake intimate images without consent.

US Enforcement

California: Attorney General Rob Bonta sent xAI a cease-and-desist letter on January 16, citing "numerous examples of xAI taking ordinary, clothed images of women and children" and allowing users to "undress" them. He invoked California's deepfake pornography law that had taken effect just two weeks earlier, plus CSAM statutes and unfair business practices law. He gave xAI five days to confirm corrective action.
US Senate: Passed the DEFIANCE Act, allowing victims of deepfake images to sue creators and distributors in civil court.
US DOJ: Issued a statement prioritizing prosecution of AI-generated CSAM.

xAI's Response: Too Little, Very Late

On January 9, xAI restricted Grok's image generation to paid Premium+ subscribers. Critics pointed out this just made deepfakes a paid feature.

On January 16, X implemented broader restrictions: keyword blocking for terms like "bikini," "underwear," "undress," and "revealing." The system cross-references prompts against prohibited terms when users try to alter images.

xAI also integrated CSAM detection tools and geo-blocked image editing of real people "in regions where it's illegal."

But Reuters testing found that as of January 16, Grok still privately generated sexualized imagery on demand even after the public rollback.

Musk's personal response was characteristic. He posted on X that he was "not aware of any naked underage images generated by Grok", despite multiple governments presenting evidence of exactly that. When the Associated Press emailed xAI for comment, the company's media address auto-replied: "Legacy Media Lies."

Behind the scenes, the xAI safety team, already smaller than competitors', lost several staffers in the weeks before the crisis. A source told CNN that Musk had pushed back against internal efforts to add guardrails to Grok.

When It Hit Home

The scandal took a personal turn when Ashley St. Clair, the mother of one of Musk's children, began posting on January 5 about users commanding Grok to sexualize images of her, including photos taken when she was a minor. She sued xAI.

A coalition of nearly 30 women's, child safety, and tech advocacy groups sent letters to Google and Apple demanding they remove X and Grok from their app stores.

Despite the firestorm, Grok downloads increased 54% in January. Outrage, it turns out, is also marketing.

The Bigger Picture

The 2024 Taylor Swift deepfake incident was supposed to be the wake-up call. Grok proved nobody was awake.

Musk built the chatbot to be unfiltered. He marketed the lack of guardrails as a feature. When the predictable happened, when an AI tool designed to be edgy was used for exactly the worst thing everyone warned about, the company's response was to paywall the abuse and blame users.

That approach hit a wall of reality: international law, child safety organizations with evidence, and prosecutors with jurisdiction.

The question now isn't whether Grok crossed a line. That's settled. Seven governments said so. The question is whether the regulatory response sticks, or whether this becomes another scandal that generates headlines, produces strongly worded letters, and changes nothing.

The DEFIANCE Act gives victims a way to sue. California's cease-and-desist has teeth. The EU's document preservation order suggests a long-term enforcement posture, not a quick press release. These aren't just "concerns": they're legal actions with consequences.

But Grok downloads went up 54%. And the Pentagon is still deploying it to classified military networks.

Sources

Published: January 28, 2026