This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Devansh about why the trickiest legal AI hallucination is a real case used the wrong way, how his team splits their system into three tiers to find failures fast, and why he'd rather optimize for maintenance than for generation.

Devansh is the Founder and Head of AI at Irys. He builds reasoning and verification systems for legal AI, a tough place for hallucinations and confident wrong answers. He also publishes open source AI research on Substack.

We start with why AI for professionals breaks down. Devansh thinks models are getting good at doing a lot of things somewhat well, but they fall short on the last mile to reliable. So his team engineers structure back in instead of hoping a better model fixes it. In legal, fake citations are the failure everyone has heard about. The more interesting one is a real case that doesn't apply. So much of law is interpretation, and a model built to predict the most probable token pulls toward the average.

Then he walks through the three tiers behind Irys One. Ingestion maps messy input, like PDFs with handwritten notes and email chains that refer to them, into a knowledge graph so the AI knows where to look. Reasoning navigates that graph. Delivery applies how a firm or a lawyer likes to work. The payoff is error isolation: if a lawyer says something is missing, two or three targeted queries tell them whether it was ingestion or reasoning. We also get into why chunk-based retrieval misses connections, why going from 10 docs to 10 gigabytes is mostly about finding the few documents that matter, and how "answer contracts" handle "it depends."

Then I take a DevOps angle. Devansh says the hardest part of running AI in production is tracking everything. A fully agentic system might perform 20% better, but you never know how it breaks until it breaks. The first thing his team tracks is whether the system behaves as expected, because labs change models behind the scenes with post-training, quantization and tokenizer changes. We also cover why he calls reinforcement learning the most overhyped technique in AI: it works when the output is truly verifiable, and falls apart when "better" has no real signal.

The big takeaway: you can't fix what you can't find. Engineer for error isolation first, and optimize for maintenance, not generation. Coding agents will churn out heaps of code, but maintenance is where a system lives or dies.

Highlights

• Why a real case applied wrong is worse than a fake citation

• Why AI's pull toward the average struggles with legal judgment

• Ingestion, reasoning and delivery as three separate tiers

• Knowledge graphs vs chunking for finding what matters

• Isolating failures with two or three targeted queries

• Models that change under you, and tracking expected behavior

• Why reinforcement learning is overhyped

• Optimize for maintenance, not generation

Links

Devansh on LinkedIn: https://tsn.io/7Hjfm

Irys: https://tsn.io/ytG1A

Artificial Intelligence Made Simple (Devansh's Substack): https://tsn.io/ZM6eS

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://tsn.io/lAERA

This week on Ship It Weekly, GitHub Actions can now run eligible Linux jobs on AWS as well as Azure, so allowlists and Entra Conditional Access policies that only trust Microsoft ranges can break. The Brotli decoder bug (CVE-2026-111497) landed on CISA's exploited list, and it reaches past Node.js to Python Brotli, Bun, Envoy, Istio, a Go port and .NET. A campaign called Gheist is using stolen GitHub tokens to push ci-sync branches that print Actions secrets past log masking, then delete the run. And attackers are dropping AGENTS.md files into Downloads folders to hijack AI coding agents.

The thread: each one is an inventory question. Which services decompress Brotli, which allowlists assume CI runs on Azure, which OIDC trust policies accept any branch, and which laptops have started an agent in Downloads?

In the lightning round: GitHub Actions workflow lockfiles are generally available, Railway disclosed a breach of its internal systems, Tailscale 1.104.4 fixes four security bulletins, and Datadog moved its US1 status page to its own Status Pages product.

The human closer, from Bill Duncan: "Twenty minutes in, you haven't started diagnosing anything. You've been doing archaeology." Write down who owns it before the pager asks.

Highlights

• GitHub Actions on AWS: why the same job can pass, then fail

• Sync runner IP ranges from GitHub's meta API, not one cloud's

• Brotli: a decoder bug your HTTP stack may run for you

• Gheist: what private repos and OIDC trust let through

• AGENTS.md in Downloads: an instruction file is code now

• Service catalogs and the "what depends on me?" question

Links

GitHub Actions multi-cloud compute: https://tsn.io/fJkLJ

Node.js October security releases (Brotli): https://tsn.io/BvZBU

Brotli CVE-2026-111497, affected libraries (Wiz): https://tsn.io/HJuKS

.NET out-of-band security updates: https://tsn.io/j0Xwf

Gheist steals GitHub Actions secrets (Wiz): https://tsn.io/nunEs

AI agents hijacked via the Downloads folder (Forever Security): https://tsn.io/qnwuc

Actions workflow lockfiles GA: https://tsn.io/M84uA

Railway security disclosure: https://tsn.io/21t0j

Tailscale security bulletins: https://tsn.io/GvZxR

Who Owns This? (Bill Duncan): https://tsn.io/LFrzc

SRE Weekly #540: https://tsn.io/O8tpO

This week's On Call Brief: https://tsn.io/HqfCP

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://tsn.io/lAERA

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Darko Fabijan about what starts breaking after the pull request when AI writes more of the code, what should sit between an AI agent and production, and why his answer is more automation.

Darko is the CEO of SuperPlane and co-founder of Semaphore, the CI/CD platform. At SuperPlane, he's working on how to give AI agents write access to production in a controlled way.

We start with what AI-generated code does to delivery. Pull requests are bigger and more frequent, reviewers can't keep up, and Darko points to a Faros AI report that shows defects and incidents going up. He describes AI as alien technology that dropped from the sky, landing in processes built for people who review code eight hours a day.

His main hypothesis is a deterministic layer between AI agents and production, with scoped access that goes through approval. I bring up the rubber stamping problem with human approvals, and his reply is that humans hallucinate too. So he wants a risk score on every change. Renaming a bucket scores low. Deleting an S3 bucket or an RDS instance scores red.

Then we get into the glued-together ops tooling most teams run on. Darko doesn't think anything is wrong with it. When I ask how SuperPlane differs from Argo Workflows or n8n, he says he wants agents and humans on the same plane, and he'd onboard an agent like a new engineer: give it an identity, let it look around, and show it what it can do. We also cover team-scoped agents, compliance at AI speed, and docs where every line traces back to what produced it.

We finish on where to start. Darko says read-only first: he pretends to be super stupid, asks a lot of dumb questions, and says "onboard me." For the first write operations, he'd pick loops you can measure with a number, like latency, GitHub Actions speed or CVE count, tested on a copy of production or staging first.

The big takeaway: AI didn't break software delivery. It showed how much of it still runs at human speed. Darko's answer is more automation, with a deterministic layer and a risk score between AI and production.

Highlights

• What breaks after the pull request when AI writes more of the code

• A deterministic layer between AI agents and production

• Rubber stamping, and a risk score for every change

• Onboarding an AI agent like a new engineer

• The dam and the river: compliance at AI speed

Links

Darko Fabijan on LinkedIn: https://tsn.io/DySr5

Darko Fabijan on X: https://tsn.io/xJGhN

SuperPlane: https://tsn.io/o08yM

Semaphore: https://tsn.io/jiHmd

Faros AI's Acceleration Whiplash report: https://tsn.io/5ZCR3

Argo Workflows: https://tsn.io/oyTYF

n8n: https://tsn.io/nloFF

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://tsn.io/lAERA