The social media platform X has provoked significant outrage after users employed its AI chatbot Grok to manipulate photos of females by removing their clothing.
Numerous examples have emerged where the tool has been employed to remove clothing from women to make them appear in bikinis non-consensually, as well as to place them in inappropriate contexts.
Developer Response and Regulatory Position
XAI, the company that created Grok, offered no comment to questions, except for an standard response stating "traditional news outlets lies".
A government spokesperson stated that new laws is being proposed to ban nudification tools. Under a upcoming law, anyone who provided such technology would "face a prison sentence and substantial penalties".
"X or Grok could prevent these forms of abuse if they chose to," said a law professor, adding they "seem to have a lack of consequences."
Regulator Position and Tool Capabilities
The regulator Ofcom said technology companies must "consider the potential" of people accessing unlawful material on their platforms, but did not specify whether it was at this time looking into X or Grok in regarding artificial intelligence pictures.
Grok is a no-cost AI assistant—with some additional functions—which reacts to X users' queries when they reference it in a message.
It is often used to provide commentary or more context to other users' remarks, but people on X are also capable of alter an uploaded picture through its AI image editing tool.
Background of Criticism and Rules
The system has been condemned for enabling users to create pictures and footage with explicit content and sexually suggestive material, and it was in the past alleged to have been involved in making inappropriate content of celebrities.
"The platform has been permitting the production and spread of these pictures for months without taking any action and we have not witnessed any challenge by oversight bodies," Professor McGlynn added.
XAI's own terms of service bans "depicting likenesses of people in a pornographic fashion".
In a statement, Ofcom said it was illegal to "generate or distribute deepfakes or child sexual abuse material" and stated this included explicit AI images made with AI.
It said services such as X were required to take "suitable measures" to "minimize the danger" of users encountering illegal material on their platforms and to delete it promptly when they learn of it.
