How do you keep AI agents under control? 

Christian Owusu explores AgentOps: 

How do you ensure an AI agent remains reliable over time when you change its instructions, tools, underlying model or data? Intern Christian Owusu is investigating this at AMIS Conclusion. His assignment focuses on AgentOps: an approach to developing, testing and deploying AI agents in a controlled way. 

You can adjust an AI agent you use yourself as you work. Once you share it with colleagues and make it part of their workflow, you are effectively introducing business software. Responsibilities, availability, quality monitoring and cost control then need to be clearly defined.

Christian’s research explores the approach needed to support this. How do you track versions? Who reviews changes? And how do you check whether a new version is good enough to put into use?

An AI agent’s behaviour depends on its code, instructions, settings, model choices, tools and data. Christian is investigating which components need to be version-controlled and which readily available tools are suitable for the task.

This aligns with AMIS Conclusion’s goals: developing high-quality software for critical systems where reliability and continuity are essential. We aim to apply the same care when using AI in our own processes and for customers. Improvements are developed, tested and introduced in small, controlled steps, with clear agreements on ownership, change management and cost management. Our professionals remain responsible for how AI is used and the results it delivers.

An AI agent can give different answers to the same question. Christian is therefore investigating automated evaluations, also known as evals. These assess, for example, whether a task has been completed, the correct sources and tools have been used, and the outcome meets safety criteria.

By comparing multiple runs, he examines how consistent the quality is and whether changes cause regressions. The intended outcome is a practical AgentOps framework, supported by a working proof of concept and recommendations for use within AMIS and customer projects.