How can the fight against online violations be made more effective and targeted? By working carefully—and establishing new methods. That’s why we’ve developed an AI tool that detects hate crimes more reliably than ever before.

Protecting human dignity is our mission. But how can we achieve this in a complex and, admittedly, vast online environment? Our answer to this question is the use of artificial intelligence (AI). Since 2020, we have been using an AI tool in our regulatory work that identifies online legal violations more efficiently than we could achieve through human effort alone. And it has greatly enhanced collaboration among media organizations, the protection of our employees, and our response time to current events.

“Protecting freedom of expression requires that we take a firm stand against hate and incitement. With the help of AI, we can do this better than ever before.”

Dr. Tobias Schmid, Director of the Media Authority of North Rhine-Westphalia

This tool is called KIVI—a combination of the german term for “AI” ("KI") and “vigilare,” which is Latin for “to be vigilant.”

By automatically monitoring over 10,000 public social media profiles and websites, KIVI enables us to identify a wide range of potential legal violations—and to prepare them directly for legal review by our trained staff.

Today, all media organizations in Germany use the AI tool developed on our behalf. In this way, we are working together nationwide to combat violations of human dignity, hate speech, depictions of violence, pornography, and other content harmful to minors. This has also allowed us to further professionalize our cooperation with law enforcement agencies. In 2019, we commissioned a feasibility study on the use of digital tools in media regulation. In 2020, Condat AG in Berlin began developing the tool. Within half a year, the prototype was up and running, after which the tool could be rolled out channel by channel. From Twitter and YouTube to platforms like Telegram and VK, the tool now scans more than 10,000 pages daily.

KIVI Improves Collaboration Among Media Organizations

The tool detects potential legal violations. For example, it identifies the international channels through which users are being addressed in German. The AI tool can detect suspected violations that are relevant in North Rhine-Westphalia—or that concern media regulatory authorities in other federal states or European countries. Today, all media organizations in Germany use KIVI. This allows us to work together—beyond state borders and our respective jurisdictions—to address the large number of violations where we do not know the origin of a hate post or violent video.

More protection, less burden

KIVI improves the detection of potential violations—and enhances the protection of our teams. Before content is opened, KIVI indicates which violation category it likely falls under. For example, KIVI identifies whether the detected violation involves depictions of violence, hate speech, or pornographic content. Employees can specify which potentially disturbing content should initially be blurred—which can reduce psychological stress, particularly in the case of depictions of violence.

KIVI's Role in the Oversight Process

The AI tool is a resource. Any suspicion flagged by the tool is reviewed during monitoring. If the suspicion is confirmed, this is noted in the tool and the case is forwarded to our legal team. Only then are further steps taken against potential offenders. Unconfirmed cases are returned to the tool. This allows KIVI to learn from its own mistakes.