Image generated with AI (gpt-image-2)

A week ago Mistral released Shieldstral, an open model that decides whether a piece of text violates a safety policy. Running it normally takes a 16 GB GPU.

We converted it so it runs inside a browser tab, on your own machine, with nothing leaving your network. You can try it right now:

labs.montevive.ai/shieldstral-policy-probe

Why this should matter to you

If your company has put a filter between its staff and public AI tools, or is about to, you have an uncomfortable problem: you have no practical way to check that it works.

Auditing an AI control normally means a proof of concept, a procurement cycle and, almost always, sending real company content to a third party's server to see what comes back. That last part is precisely what most security officers cannot authorise.

A model that runs in the browser changes the economics. You paste your own text, in your own languages, and read the answer immediately. No contract, no integration, and not a single prompt leaving your laptop.

So we ran that audit ourselves first.

What we measured

We asked the model the same question, with the same policy and the same threshold, about 1,392 texts in each of five languages: English, Spanish, Catalan, Basque and Galician. Identical content, translated and verified. The only variable was the language.

The last three matter more than an outside reader might assume. Catalan, Basque and Galician are co-official languages in Spain, used daily in contracts, internal messaging and government correspondence by millions of people. Mistral documents twelve languages for this model. Spanish is on that list. Those three are not.

Substitute your own case. Any organisation operating in a language its vendor has not measured is in the same position.

What we found

It detects well, until you change language. Out of every 100 harmful messages, the model caught:

• English: 82
• Spanish: 80
• Catalan: 79
• Galician: 78
• Basque: 59

Same text, same policy, same threshold. One language in five lets through roughly twice as much as the others.

And nothing would have told you. This is the part that matters for a security team. A filter that blocks too much reports itself: users complain on day one. A filter that lets too much through is silent.

Across our runs, false alarms on harmless content stayed between 2.3% and 3.2% in all five languages. The model never got noisy. There was no signal, on any dashboard, that detection in one language had collapsed. The control stayed green.

Default settings are tuned for the language it knows best. These models ship with a recommended threshold. Adjusting it lifted Basque detection from 59% to 71%. The same adjustment gained English three points. Deploying on factory settings penalises exactly the languages the model handles worst.

How much worse depends on how you measure. We ran the experiment against two different test sets and the gap between them was larger than the gap between languages. Measuring this properly is harder than it looks, and a number without a method behind it is not worth much. We have published ours in full.

Check it yourself

Open the demo, write a policy in plain language, and press "Run all five". You will see one sentence judged in five languages at once, and watch one of them come back clean while the other four are flagged.

Thirty seconds. Nothing transmitted.

Getting there was not straightforward. The standard conversion tooling does not support this architecture, so we exported the model by hand and verified that the browser version returns the same answers as the original. The weights and the full method are published openly if you want to reproduce it.

Four questions for your AI vendor

If you are buying or renewing any AI-based safety control, these four questions separate the serious suppliers from the rest:

1. Which languages is this measured in, and what is the detection rate? Not accuracy, not precision: the share of harmful content it actually catches. These are different numbers and only one tells you what is getting past.
2. What happens to content in a language that is not on your list? "Unsupported" and "it performs worse, and here is by how much" are very different answers.
3. What threshold ships by default, who tuned it, and for which language?
4. Can I test it on my own content, in my languages, before I sign? If the answer requires a three-month proof of concept, that is also an answer.

What this does not tell you

We measured one model against one policy. Other filters will have other profiles, and the point of this article is not to single out Shieldstral, which is a genuinely good piece of open work.

Our Catalan, Basque and Galician translations were also machine-generated and are still awaiting review by native speakers. The controls we ran indicate the effect is real, but we would rather say so before anyone asks.

Why we care

At Montevive we build AI systems for organisations that cannot hand their data to somebody else's cloud. Our product, Moviwa, sits between a team and the assistants they already use and stops confidential material leaving the building. To do that it has to read what your staff actually write, and your staff write in more than one language, sometimes inside the same document.

That is why we measure before we deploy, and why we publish what we find.

Contact us if you want to review which AI controls you have in place and whether they are genuinely working.