Messaging app Telegram is under pressure to crack down on an AI bot that has generated tens of thousands of non-consensual images of women on its platform.
Law enforcement bodies are looking into the activity of a deepfake bot, which is believed to have been used to produce explicit images of underage girls. Data protection regulators in Italy have also opened an investigation into its use, and access to the bot has been restricted on Apple’s iOS.
The scrutiny of Telegram comes as multiple investigations into the use of the messaging service have also discovered private groups sharing non-consensual “revenge porn” photos and videos of women that were not generated by AI. Reports from America, Italy, South Korea, and Israel have all detailed how Telegram has been used to share abusive images over the past year.
Researchers claim Telegram has failed to act against the deepfake bot, which automates the creation of nude images of women. The bot uses a version of the DeepNude AI tool, which was originally created in 2019, to remove clothes from photos of women and generate their body parts. Anyone can easily use the bot to generate images. More than 100,000 such images have been publicly shared by the bot in several Telegram chat channels associated with it. These channels contained tens of thousands of members each.
When researchers at the security firm Sensity discovered the bot on Telegram at the start of this year, they reported it to the messaging app. The hope was that the chat platform would eject the bot and put a stop to the way women were being abused with technology. However, this hasn’t happened.
Since Sensity revealed the bot’s existence in October, the groups that used it to share images have hidden it. “It is actually harder now to actually get to the bot,” Giorgio Patrini, CEO and chief scientist at Sensity says. “The groups that were advertising the bots on Telegram have essentially gone silent.”
A number of groups that used the bot have changed their names so they can avoid being identified. Many of the channels now share other content related to deepfake technology in general, and a public gallery of “nude” images created by the bot was wiped by its owner. And some of the channels have vanished completely.
Although the groups around the bot are not currently posting about it, it still exists and continues to work. “The bot has never been taken down by anybody,” Patrini says. “Since we went public the bot is still operational and still is today.” In one instance the bot’s creator said the bot will keep operating under the radar. The creator, whose identity is unknown, did not respond to a request for comment.
At the end of October, the Telegram bot became inaccessible on iPhone and iPads and showed a message saying it violates section 1.1 of Apple’s developer guidelines. Apple’s rules say “overtly sexual or pornographic material” is not allowed in apps accessible through the App Store. That message within Telegram has since been replaced with a generic warning that says the bot cannot be displayed.