We build AI agents that do the work of a site reliability engineer: they catch new errors in production early, find the root cause, and fix it automatically, so our customers' systems stay up. We have a wide range of customers today.
We're looking for a Tech Lead who has strong opinions about questions like this, and who will still be the one writing the code that answers them.
Why this role is different
Our backend powers AI agents running inside the production environments of Fortune 500 companies. Those environments emit a constant stream of unstructured events — for example, when our customers have bugs that impact their customers. Each event needs to be investigated and grouped into a small number of unique “Incidents” that our customers should focus on investigating.
This translates to two specific challenges for our backend team:
- How can you use frontier LLMs at scale, despite frontier LLMs being too expensive at scale? The analysis each event needs can only be done by a frontier LLM — cheaper models do not yet provide the accuracy we need — but invoking an LLM per event is prohibitively expensive at our volume. We solve this with a number of creative approaches that we'd like to discuss with you in an interview, and see how you think about the problem.
- How can you leverage LLMs safely in enterprise environments? What security model gives customers the confidence to roll out our agent, while also giving our agent the flexibility it needs to deliver accurate results? There are specific challenges here regarding multi-agent investigations, agent-to-agent communication, and shared memory. Given how fast the space is evolving, you cannot simply rely on ‘best practices’ but must approach the problem from first principles and design a system that satisfies the fundamental constraints.
As at any company operating on the frontier, best practices can change overnight as LLMs improve! Something that did not work on a previous generation of models might suddenly become viable. You need to be comfortable with rapid change, and with regularly invalidating your own previous assumptions.
What you'll work on
You take on the hardest problems on the team: you design the solution, build it yourself, and set the pattern others follow. Most of those problems sit in three areas: keeping the platform fast and cheap at scale, controlling who and what can access what, and keeping our agents safe inside customer environments.
- Build for heavy concurrent load. You design, build, and scale the services and APIs that ingest customer events and run agent investigations in parallel.
- Treat reliability as a design decision. Every layer you design is reliable, scalable, and fast, with extra care where the system is distributed.
- Design for where the platform is going. You design for the scale we'll be running at next year rather than this quarter, and set the standard for performance and security as we grow.
- Design reviews. Other engineers bring their designs to you before they start building, and you're the one who spots what will break at scale.
- Code reviews. Your reviews keep the codebase consistent with where the platform is heading.
What we expect you already know
If you've built high-scale backend systems, you already have most of what this role needs. The hard problems here are classic ones — throughput, cost, isolation, access control — applied to a new kind of workload. The AI-specific parts are learnable on the job.
- 8+ years of backend engineering
- Hands-on experience solving real performance problems at scale — you've profiled a high-throughput system, found the actual bottleneck, and fixed it.
- Breadth across a whole backend — you can reason about the queue, the database, and the service layer in the same conversation
- The ability to explain tradeoffs clearly, including to people who disagree with you
- Extensive experience using coding agents like Claude Code to take on work that used to be too large to attempt
Nice to have
- Experience building products on top of LLM APIs
- Python and Kubernetes
- Experience with observability systems — metrics, logs, traces, and the alerting built on top of them
- Security experience in multi-tenant SaaS, especially software that runs inside customers' own infrastructure
- Open source contributions, particularly to AI agent tooling
If you're a strong engineer and this role excites you, please reach out even if you don't check every box.
To apply
Send a short note to maayan@robusta.dev with your CV.