How can action against illegal online content become more effective and targeted? Through careful analysis and innovative methods. That is why we developed an AI tool capable of detecting potential online legal violations more efficiently than ever before.

Protecting human dignity is at the heart of our mission. But how can this be achieved in a complex and vast online environment? Part of the answer lies in the responsible use of artificial intelligence (AI). Since 2020, we have used an AI-based tool to support our regulatory work by identifying potential legal violations online more efficiently than would be possible through manual review alone. It has also strengthened cooperation between authorities, helped reduce the burden on employees, and enabled faster responses to emerging developments online.

“Protecting freedom of expression requires that we take a firm stand against hate and incitement. With the help of AI, we can do this better than ever before.”

Dr. Tobias Schmid, Director of the Media Authority of North Rhine-Westphalia

This tool is called KIVI—a combination of the german term for “AI” ("KI") and “vigilare,” which is Latin for “to be vigilant.”

By automatically monitoring more than 10,000 public social media profiles and websites a day, KIVI enables us to identify potential legal violations and prepare them for review by trained staff.

Today, each of the 14 media authorities in Germany uses the AI tool developed on our behalf. In this way, we are working together nationwide to combat violations of human dignity, hate speech, depictions of violence, pornography, and other content harmful to minors. This has also allowed us to further professionalize our cooperation with law enforcement agencies. In 2019, we commissioned a feasibility study on the use of digital tools in media regulation. In 2020, Condat AG in Berlin began developing the tool. Within half a year, the prototype was up and running, after which the tool could be rolled out channel by channel. From Twitter and YouTube to platforms like Telegram and VK, the tool now scans more than 10,000 pages daily.

KIVI improves collaboration among the German media authorities

The tool detects potential legal violations. For example, it identifies the international channels through which users are being addressed in German. The AI tool can detect suspected violations that are relevant in North Rhine-Westphalia—or that concern media regulatory authorities in other federal states or European countries. Today, all media organizations in Germany use KIVI. This allows us to work together—beyond state borders and our respective jurisdictions—to address the large number of violations where we do not know the origin of a hate post or violent video.

More protection, less burden

KIVI improves the detection of potential violations—and enhances the protection of our teams. Before content is opened, KIVI indicates which violation category it likely falls under. For example, KIVI identifies whether the detected violation involves depictions of violence, hate speech, or pornographic content. Employees can specify which potentially disturbing content should initially be blurred—which can reduce psychological stress, particularly in the case of depictions of violence.

KIVI's Role in the Oversight Process

The AI tool is a resource. Any suspicion flagged by the tool is reviewed during monitoring. If the suspicion is confirmed, this is noted in the tool and the case is forwarded to our legal team. Only then are further steps taken against potential offenders. Unconfirmed cases are returned to the tool. This allows KIVI to learn from its own mistakes.