“Users and Deployers Increasingly Expect Credible Third-Party Evidence Before Accepting Important Claims at Face Value”

The market for AI evaluation, audits, and Europe's room to act: an interview with Dinah Rabe, EMEA Lead of BABL AI

Does Europe seize its chance as a middle power to play a significant role in the global AI market? Monika Schnitzer, Clemens Fuest, Daniel Privitera and others demand in their "Transformative AI Strategy for Europe": Europe must become the global leader in AI assurance and shape the market. I could not agree more! And so do many large companies who deploy new products and services based on Frontier AI models. "Users and Deployers Increasingly Expect Credible Third-Party Evidence Before Accepting Important Claims at Face Value," says Dinah Rabe, an independent AI evaluator with the Scale-Up BABL AI, and our Beyond Beta guest this week.

The opportunity for Europe to create a market in AI assurance is currently served on a silver platter: Dario Amodei's essay "We must pace the frontier" has put embedded evaluators into the limelight. AI Evaluators, "who have employee-like access to verify safety practices and report incidents", as Amodei envisions, are one of his core proposals to prevent incidents like the OpenAI-Hugging Face agent swarm attack.

Dinah's boss Shea Brown, a former professor at the University of Iowa, founded his startup BABL AI in 2018, almost a decade ago, when algorithm audit was barely a phrase. BABL AI has just co-founded the first trade organization for AI assurance, PACT AI. It is one voice of AI safety organizations. Others are, for example, METR. METR's CEO and founder is Beth Barnes, a former alignment researcher at OpenAI who left in 2022 and founded METR in December 2023. Others, like Apollo Research, Fathom AI, or Transluce have also been a vocal part of the debate of the past weeks on AI safety.

With Dinah, we discuss what the current debate is missing, what AI testing and evaluating must focus on, and what Europe's chances are in this emerging field.

Kirsten Rulf: You have been professionally evaluating AI systems for years now, for large corporate clients both in the US and in Europe. What is your perspective on the current debate?

Dinah Rabe: There are all sorts of current discussions, some prompted by the recent "Hugging Face incident", surrounding AI agents having gone rogue. There are two points I find fascinating:

For one thing, I find it odd that it is being framed as a shocking surprise. We have known for months, if not years, that these models can display deceitful behaviour. The scale may come as a surprise, but the underlying behavior is simply a product of how these models were trained. This is not emergent behavior. These models were trained not to give up and to keep searching for solutions. Those are deliberate design decisions, not superintelligent machines that have suddenly decided to be "evil".

For another, what strikes me is how quickly people map the Hugging Face incident to their personal or business contexts. At this stage, this is an incident that could only really have happened in a very particular environment: inside the experimental setup of one of the big frontier AI labs. The idea that any AI agent could quietly burn through millions of tokens, link up with hundreds of other agents and simply "go rogue" for weeks or months is absurd. This did not just happen by accident. I understand why it causes alarm, but it takes rather particular conditions for AI agents to behave this way at all.

That said, I think it is healthy that incidents like this are pushing the debate further onto the political stage.

Kirsten Rulf: Then what would be a more productive turn of the debate from your point of view?

Dinah Rabe: What is largely missing from the debate is that, for example in the Hugging Face incident, the lab itself set many of the conditions that made this attack possible in the first place. Look at the details: running 1,200 agents for weeks takes an enormous amount of resources. Anyone who runs AI agents at regular scale and checks the bill for the tokens they consume would notice if those agents suddenly started working their way through thousands of euros' worth of tokens.

So, I do not think the failure was misjudging the model's capabilities. Rather, the organizational safeguards did not measure up to the model's capabilities. It is an experimental setting after all. What could have been done differently in its governance and socio-technical set-up to ensure safety? That is a far more interesting question than whether this incident has revealed a superintelligent system.

Should Frontier AI Labs Embed Independent Evaluators, and What Would That Look Like?

Kirsten Rulf: Almost all the Frontier AI Lab CEOs now call for independent evaluators like yourself to come on board and attest to the safety of their models. What do you think about that?

Dinah Rabe: The big question right now is what it would actually mean to embed evaluators inside these labs.

In my view, there is far too much focus on capability evaluations themselves. Those are not really the missing piece. We have had capability benchmarks for a long time, and they are quite good at exposing problematic behavior such as gaming and deception. Those evaluations have consistently shown that frontier models are getting better at these behaviours.

The harder questions about embedded evaluators are these: What do they have access to? Are they truly independent from the labs they are evaluating? And what exactly are they evaluating? The answers to all of these questions are still up in the air, including in the AI assurance industry itself.

The AI Evaluators Forum, a newly founded organization with members such as METR and Transluce, issued a statement stressing the importance of independence, which we as BABL AI generally support. But views differ on what "truly independent" means. For me, independence begins with funding: an evaluator should not ultimately depend financially on the organization they are supposed to scrutinize. It also requires some form of enforceability. An evaluator that can identify a serious problem but has no mechanism to compel action is independent in name only.

Then there is the question of what is actually at stake for us as the evaluators. If we do not do our job properly, what are the consequences? And if the labs do not let us do our job, are there consequences then for them? None of this is clear and what is really needed right now is a broader group of stakeholders coming together to design an AI assurance structure with real influence.

Otherwise, there is a genuine risk that a few outside evaluators are embedded in the labs as token safety measure, and that is the end of it. At worst, these embedded evaluators could even end up sharing the blame and legal liability the next time something goes wrong. If there are no consequences or full access, then calling them "independent evaluators" achieves very little.

And there is still the most fundamental question: what happens if we as evaluators do find something? That, too, is unclear and it needs ideally international rules.

Kirsten Rulf: What skill profile does an AI evaluator need? What does an AI system or AI model audit look like on the ground?

Dinah Rabe: One thing is certain: this is a team sport. No single person has all the necessary skills, because the work touches on so many different areas. You obviously need deeply technical people, machine learning experts who understand technical evaluations and how a model is built. However, to audit a model is not to build a model. The kind of expertise needed to push the frontier may be extremely scarce, but that is not the same as the expertise required to understand and evaluate these systems rigorously. And of course, AI itself is increasingly helping us, too. I am not worried about Europe's sovereignty when it comes to technical talent. Europe also has excellent universities with outstanding people.

But technical expertise alone is not enough. A serious evaluation or audit is built on a strong understanding of the law, the norms of society, and the broader context in which a system will operate. When we conduct risk assessments and ask whom we need to consider, we find that purely technical people sometimes struggle to put themselves in the system's context: Who might be affected by it, from the company itself to groups across society? Thinking about risk requires the most interdisciplinary group possible; otherwise you will always have a blind spot.

And for highly specific risk categories, such as biological risk, you always need dedicated subject-matter experts, too. No one person can cover all of that.

That is why, in our own risk assessments, we try to bring together as broad a group of experts and stakeholders as possible. And we encourage companies with in-house evaluation teams to do the same. The central questions are not only technical: Which risks do we need to consider? How should they be assessed? Which are genuinely high risks and deserve priority? Which can be set aside?

And once you have identified the relevant risks, the next step is to define metrics to evaluate them. But the answer is not always a quantitative technical analysis or a test. Some risks simply cannot be quantified.

Does AI Sovereignty Mean Increased AI Safety for Deployers?

Kirsten Rulf: The frontier AI crowd long treated compliance as a box-ticking exercise that did not really apply to them. They were the saviors, their models were at the frontier, and they were moving the world forward. Almost four years on, how do you see it?

Dinah Rabe: The situation is starting to shift. We can see it because people increasingly recognize the risks and problems of the closed-source frontier model market. Many organizations that have so far relied mainly on API access to the big frontier models, or built them into their own AI systems, are worried about vendor lock-in: about the model being swapped out in the background without them noticing, as some clients report, or about prices being changed arbitrarily.

At the same time, open-weights models, where you have a degree of control over the weights and can run them yourself, keep getting better. And for a great many use cases, that might be good enough. There are certainly individual use cases where the frontier labs remain far ahead. But capability evaluations repeatedly show that open models lag closed-source models by somewhere between three and seven months. So it is worth asking whether that might be sufficient for some of your use cases. And evaluation then becomes much easier, too.

Of course, the added sovereignty and control you gain with an open-weights model comes at a price: you have to run it yourself, secure it yourself and so on. To some extent, you take on more responsibility. How much risk a company is willing to accept, and what matters most to it, is a very individual decision. And then it becomes much easier to test these systems for your own use cases.

What Does the AI Assurance Market Look Like?

Kirsten Rulf: Speaking of the options or challenges to evaluate models, what does the market for AI evaluators look like? Most players in the field are based in the US or the UK, yours included. Many are non-profits, or for-profit but independent. But there is nowhere near as much money in this market as in frontier AI.

Dinah Rabe: The key is that AI evaluation and assurance does not require us to invent an entirely new profession from scratch. We work extensively with established audit and assurance frameworks, including standards such as ISAE 3000. Many of the fundamentals already exist: how you structure an assurance engagement, how you gather and verify evidence, how you document findings, and how you establish confidence in a claim. AI introduces new technical questions, but a lot of the underlying assurance discipline can be reused.

What is changing now is demand. Customers, regulators and citizens are asking more questions: Has anyone other than the vendor ever tested this system? In the vast majority of cases, when an AI provider says its model performs exceptionally well, is safe, or is fair, that claim still rests largely on its own evaluation. Sometimes there is rigorous internal testing behind it, but from the outside it is usually very difficult to verify. At best, some internal testing was done; otherwise it is closer to marketing.

That is really a question of market maturity. The same dynamic is now starting to reach frontier models: Are you just marking your own homework, or has someone else checked it? That is how a market takes shape: not because every company suddenly wants a formal audit, but because buyers and deployers increasingly expect credible third-party evidence before accepting important claims at face value.

The US is particularly interesting because the pressure there tends to come more from the market than from regulation. We have just co-founded the first industry organisation around AI assurance, with a key focus on standardisation, quality criteria and professionalisation of the field. That is an important development, because the long-term credibility of this market will depend not simply on having more evaluators, but on having clearer regulatory standards for what good evaluation actually looks like.

What Is the Role of AI Safety Institutes?

Kirsten Rulf: Speaking of regulation, what role should the state-backed AI safety institutes now springing up everywhere play?

Dinah Rabe: It is a very welcome development in principle, but a lot depends on what they choose to focus on. I do not believe every country needs a safety institute running capability evaluations on frontier labs. We do not need ten institutions doing essentially the same thing. That would be an inefficient use of expertise and public money. But more importantly, I think it would be a missed opportunity if these institutes only focus on AI safety, because there are other dimensions that matter enormously at the national level and deserve attention.

For Germany and Europe, for example, there is the big question of how much room for manoeuvre we actually have when so much technological development is taking place in the United States and China. Perhaps I am influenced by my time in the German public sector, but I believe we have considerably more agency and capacity than we sometimes give ourselves credit for. We do not simply have to regard ourselves as exposed to whatever direction AI development takes elsewhere.

I would love to see these institutes, perhaps under a different name than "AI safety institute," focus as well on questions like: What do we want the future of AI in Germany or Europe to look like, and what will it take to get there? Which societal debates do we need to have now in order to make those choices deliberately?

Take the labour market, healthcare or schools. What role do we actually want AI to play in each of those areas? We may decide that certain uses of AI are technologically possible but nevertheless undesirable. That is precisely the point: technological possibility does not remove political and social choice.

In Europe, of course, the question is often framed at the European level. But even starting these conversations nationally would be extremely valuable, and would help make our debates increasingly well informed. A lot depends on how confident people feel discussing these issues, and on how much knowledge, or lack of it, there is in the field.

What Should AI Assurance Test?

Kirsten Rulf: When it comes to actual evaluation, we have established that there are enough standards and benchmarks for capabilities, and that the labs and think tanks do a fairly good job there. Where else in the lifecycle should evaluation take place, for frontier models or elsewhere?

Dinah Rabe: The big difference between general-purpose frontier models and everything else, to draw a rough distinction, is context.

With a general-purpose large language model, you do not necessarily know how it will ultimately be used. That makes broad evaluations sensible: you test for general capabilities because the eventual application is still unclear.

As soon as that model is embedded in a particular system or used for a specific purpose, however, the picture changes. You know much more about the users, whether decisions are being made based on the output, the consequences of failure. That allows you to evaluate much more precisely.

The relevant questions become much more concrete. Who is using the system? Are consequential decisions being made on the basis of its outputs? Is the model operating in a process where the occasional error is tolerable, or one where it is not? What other controls sit around it? The sociotechnical context determines not only which risks matter, but also how rigorously particular parts of the system need to be evaluated.

You therefore have to look at the entire system around the model: What inputs flow into the model or system? What decisions are being made? What actions can it take, and, in agentic systems, how many steps can it take autonomously? What outputs does it produce, who receives them, and what happens next? In principle, you could evaluate every one of these points if there is a meaningful risk associated with it.

That is also where organisations can quickly become overwhelmed: Where do I even start? I could evaluate everything. This is where I think a risk-based approach, which is also central to the EU AI Act, is so important. Because resources are always limited. Look at the whole system systematically, decide which risks are most relevant and significant, and start your evaluation there. A company may have hundreds of use cases; start with the most critical ones. Start with the most consequential systems, the areas where failure could cause the greatest harm, and work outward from there.

At its core, that is what good AI assurance should look like: it is contextual and proportionate to risk.

European AI Sovereignty: A Pipe Dream?

Kirsten Rulf: You said Europe does not have to feel so helpless. Yet that has been the prevailing mood for the past two years: Europe has nothing except perhaps the EU AI Act, and the EU AI Office cannot build up evaluation capacity fast enough. We supposedly have nothing to counter with: no frontier model, no cloud provider, not even the infrastructure to host these systems properly. So we cannot really be sovereign. You see it differently.

Dinah Rabe: I do. I think we sometimes treat sovereignty as too binary: either Europe has its own frontier model, cloud, chips and everything underneath them, or we have no sovereignty at all. The reality is more nuanced.

There are certainly areas where those dependencies matter enormously. But not every AI application requires the latest frontier model that can only be hosted in the cloud. There are countless use cases where I do not need OpenAI's or Anthropic's or Google's latest model, but rather an open-weights model I can simply run myself. Smaller models keep demonstrating this.

So the question should really be: Which dependencies are we comfortable with, and where do we consider control strategically important?

Kirsten Rulf: A decision for the CEO.

Dinah Rabe: Exactly. At company level, it becomes a strategic risk decision. What capability do I actually need? What am I giving up in return for it? How dependent am I willing to become on a particular model or cloud provider, and how easily could I move if that relationship changed?

The important thing is to make that decision consciously. Using the most powerful model available by default is not necessarily the same thing as making the best technological or safety choice.

And I think the same logic applies to Europe. Sovereignty should not mean trying to recreate the entire global technology stack inside our borders. It should mean identifying the areas where dependency would genuinely limit our ability to act, making sure we have alternatives there, and accepting interdependence where the risk is manageable.

The Role of Regulation Like the EU AI Act

Kirsten Rulf: We both have been part of the making of the EU AI Act in our time in the government, and we have already agreed earlier that it is fundamentally a sound approach. Now it is being implemented. For one, it would not even have covered some of these cases, nor would the Code of Practice. And implementation across Europe is still patchy. So what is the role of the AI Act now, and of those implementing it, from the EU AI Office to national authorities such as Germany's Federal Network Agency ("Bundesnetzagentur")?

Dinah Rabe: It remains important to make explicit what our values in Europe are and what we consider important, such as fundamental rights. It will not capture every problematic case, and neither will the Code of Practice, but that does not make the framework irrelevant. It gives regulators, companies and standards bodies a common structure for thinking about what good AI governance should look like.

You can already see that influence in the amount of standardisation work now under way. CEN-CENELEC, ETSI, IEEE and others are doing very serious work in this area. Those standards will not be perfect in every detail, but without the AI Act I doubt we would have seen the same level of attention or urgency around developing them.

The real test, of course, will be implementation and enforcement, and how well that works, given that different national authorities will approach it very differently. The national authorities will determine how much of this actually reaches companies in practice. These entities need to carefully distinguish between genuine risk reduction and compliance that exists only on paper. Of course, there is a legitimate concern about burden, particularly for smaller companies. Anyone who has been through an ISO certification process knows how quickly compliance can turn into a paper battle. And documentation alone is never proof that good practice is actually happening.

To add a much more basic point: At the core, the AI Act asks companies to improve the quality of products and systems overall. Having a functioning quality-management system, carrying out a serious risk assessment, documenting how a system works and actually testing it before deployment, all of that is good for the product.

So for me, the success of the AI Act will depend on whether the institutions implementing it can operationalise its principles. If they do that well, the Act can become part of the infrastructure for more trustworthy AI in Europe.

Kirsten Rulf: Creating verifiable evidence on "Our agent does not attack other systems," something along those lines.

Dinah Rabe: Exactly, basic safety and quality management requirements. And it is funny: we regularly talk to people in heavily regulated sectors such as medical technology, and they laugh at how worked up everyone gets about the AI Act. In their world, the regulator can walk in unannounced on any given day and inspect everything. For them that is completely normal. They look at the AI Act and think: Well, that is just the basics.

Of course, the AI Act now affects many sectors and companies that have never been this regulated before. That is undoubtedly hard at first, and it surely costs resources. So the question is what the AI Office and national supervisory authorities can do to make sure it is handled as pragmatically as possible, for instance by accepting that a small company's quality management system will look different from that of a corporation with thousands of employees. The key question will be how much of a burden it actually becomes.

Kirsten Rulf: And it is unclear whether market surveillance authorities have the expertise to really understand what companies are doing.

Dinah Rabe: Yes. And once again, they are competing for the same talent everyone else wants. That will be an interesting question.

What Does the Day-to-Day of an AI Evaluator Look Like?

Kirsten Rulf: One final question: What does your day-to-day work as an auditor and evaluator look like?

Dinah Rabe: The work varies a great deal.

Sometimes it is very much traditional assurance work: reviewing evidence, test reports, model documentation, governance processes and the claims a company is making about its system. On other days, it is much more hands-on. We might analyse data, design metrics, run evaluations, test specific behaviours or write code. So the work can range from audit and evidence review all the way through to quite technical evaluation.

What makes AI particularly interesting is that there is often no established playbook. We regularly run into questions where the system we need to test has never really been tested in that way, or in that particular context, before. Then the work becomes much more research-driven: What would a credible test actually look like? What evidence would be sufficient? How do we design an evaluation that is robust enough for us to stand behind the conclusion professionally?

That last point matters a great deal at BABL. An audit report is not just an internal research note. We have to be able to defend the opinions we give. In the worst case, if a client is challenged in court or by a regulator, we need to be able to explain exactly how we reached our conclusion, what evidence we relied on and where the limits of that conclusion are.

So a large part of the job is really about the boundaries of assurance. What can we genuinely establish? Which claims can we verify with confidence? Where is the evidence too weak? And where do we have to say: we simply cannot provide assurance on this yet?

That is what makes the work so varied. And as we evaluate new systems and contexts at BABL, we are continually testing and extending the boundaries of what credible AI assurance can cover.

Kirsten Rulf: Thank you very much.