When AWS has a major outage, what actually happens behind the scenes? Ben Hartshorne, a principal engineer at Honeycomb, joins Corey Quinn to discuss a recent AWS outage and how they kept customer data safe even when their systems couldn't fully work. Ben explains why building services that expect things to break is the only way to survive these outages. Ben also shares how Honeycomb used its own tools to cut their AWS Lambda costs in half by tracking five different things in a spreadsheet and making small changes to all of them.

About Ben Hartshorne:

Ben has spent much of his career setting up monitoring systems for startups and now is thrilled to help the industry see a better way. He is always eager to find the right graph to understand a service and will look for every excuse to include a whiteboard in the discussion.

Show highlights:

(02:41)Two Stories About Cost Optimization

(04:20) Cutting Lambda Costs by 50%

(08:01) Surviving the AWS Outage

(09:20) Preserving Customer Data During the Outage

(13:08) Should You Leave AWS After an Outage?

(15:09) Multi-Region Costs 10x More

(18:10) Vendor Dependencies

(22:06) How LaunchDarkly's SDK Handles Outages

(24:40) Rate Limiting Yourself

(29:00) How Much Instrumentation Is Too Much?

(34:28) Where to Find Ben

Links:

Linkedin: https://www.linkedin.com/in/benhartshorne/

GitHub: https://github.com/maplebed

Sponsored by:
duckbillhq.com

R. Tyler Croy, a principal engineer at Scribd, joins Corey Quinn to explain what happens when simple tasks cost $100,000. Checking if files are damaged? $100K. Using newer S3 tools? Way too expensive. Normal solutions don't work anymore. Tyler shares how with this much data, you can't just throw money at the problem, but rather you have to engineer your way out.

About R. Tyler:

R. Tyler Croy leads infrastructure architecture at Scribd and has been an open source developer for over 14 years. His work spans the FreeBSD, Python, Ruby, Puppet, Jenkins, and Delta Lake communities. Under his leadership, Scribd’s Infrastructure Engineering team built Delta Lake for Rust to support a wide variety of high performance data processing systems. That experience led to Tyler developing the next big iteration of storage architecture to power large-scale fulltext compute challenges facing the organization.

Show Highlights:
01:48 Scribd's 18-Year History

04:00 One Document Becomes Billions of Files

05:47 When Normal Physics Stop Working

08:02 Why S3 Metadata Costs Too Much

10:50 How AI Made Old Documents Valuable

13:30 From 100 Billion to 100 Million Objects

15:05 The Curse of Retail Pricing

19:17 How Data Scientists Create Growth

21:18 De-Normalizing Data Problems

25:29 Evolving Old Systems

27:45 Billions Added Since Summer

29:29 Underused S3 Features

31:48 Where to Find Tyler

Links:

Scribd: https://tech.scribd.com
Mastodon: https://hacky.town/@rtyler
GitHub: https://github.com/rtyler

Sponsored by:
duckbillhq.com

Corey Quinn sits down with Avery Pennarun, co-founder and CEO of Tailscale, for a deep dive into how the company is reinventing networking for the modern era. From finally making VPNs behave the way they should to tackling AI security with zero-click authentication, Avery shares candid insights on building infrastructure people actually love using, and love talking about.

They get into everything: surviving 100% year-over-year growth, why running on two tailnets at once is pure chaos, and how Tailscale makes “secure by default” feel effortless. Plus, they dig into why FreeBSD firewalls needed some tough love, the uncomfortable truth behind POCs, and even the surprisingly useful trick of turning your Apple TV into an exit node.

About Avery:
Avery Pennarun is the co-founder and CEO of Tailscale, where he’s redefining secure networking with a simple, Zero Trust approach. A veteran software engineer with experience ranging from startups to Google, he’s known for turning complex systems into approachable, user-friendly tools. His contributions to projects like wvdial, bup, and sshuttle reflect his belief that great technology should be both powerful and easy to use. With a mix of technical depth and dry humor, Avery shares insights on modern networking, internet evolution, and the realities of scaling a startup.

Highlights:
(0:00) Introduction to Tailscale and Security

(00:52) Sponsorship and Personal Experiences

(02:07) Technical Deep Dive into Tail Scale

(06:10) Challenges and Future of Tail Scale

(22:45) Building the Tail Net's API

(23:54) Connecting Cloud Providers with Tailscale

(25:22) Tailscale as a Security Solution

(26:44) Innovations and Future of Tailscale

Sponsored by:
duckbillhq.com