Senior/Staff AI Engineer, Quality & Evals

Build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality, focusing on evaluation systems that power multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines.

What you'll do

• Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.
• Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
• Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.
• Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents.
• Team up with backend engineers to improve the speed and reliability of our retrieval and reasoning agents.

What you'll need

• Strong Python and/or backend engineering experience.
• Solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics.
• Experience deploying and operating systems in the cloud, ideally on GCP.
• Hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
• Comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.
• Understanding of system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.
• Senior-level engineering judgment to make architectural decisions, communicate trade-offs, and build systems other engineers can extend.

Nice to have

• Designing data pipelines, ETL/ELT workflows, event-processing systems, or feedback loops for production data.
• Building infrastructure around LLM-based products or agentic systems.
• Using workflow orchestration systems such as Temporal or similar.
• Familiarity with audit, finance, compliance, or other high-accuracy domains.

What they offer

• Competitive salary plus significant equity.
• Flexible vacation, team lunches, retreats, central Berlin office.
• Generous coding tools budget.