Marcos Sánchez

The Counterexample Is the Test

Sun, 06 Sep 2026 00:00:00 +0000

I’ve spent the week putting a TLA+ model into a side project and wiring it into the checks. Not as a document that says the protocol is sound, but as something that fails when the model and the code drift apart. I understood it about halfway through doing it, which is the wrong order, so this post does it the right way round: a counter, two workers, and one new piece per section until the shape of the real thing appears.

WIP: an RFC for check-based monitoring semantics

Sun, 30 Aug 2026 00:00:00 +0000

Check-based monitoring has run production floors for twenty-five years and nobody ever wrote down what it means. A scheduler runs probes, each probe exits with a code, state machines turn streams of results into confirmed states, and a pager goes off on confirmed transitions. Every tool in the Nagios lineage implements this. No two of them agree at the edges, because there is no spec to agree with: the exit-code convention, the host/service split, UP versus OK, UNREACHABLE, soft and hard states, acknowledgements and downtimes are folklore, documented separately and inconsistently inside each tool that inherited them.

A k9s for Redpanda

Sat, 29 Aug 2026 00:00:00 +0000

The sagacious reader will have noticed this blog’s fondness for Redpanda by now. My homelab keeps everything in its topics (part I), I patched the broker itself to read them by key (part II), and the serialization experiment that became parco was born streaming trades through it. Today the streak continues with something smaller: I’ve released readpanda, a terminal UI for looking inside those topics.

The itch

My inspection loop was rpk topic consume piped through jq. That works until the topic is Avro, where the payload is a magic byte, a schema id and bytes jq won’t touch. Then the loop grows a curl to the schema registry and a decode script, multiplied by however many topics I’m watching. k9s solved this shape of problem for Kubernetes years ago: stop composing the inspection by hand, put the operator inside the cluster. I wanted the same seat for my broker.

The Art of Not Buying Disk, Part II

Fri, 28 Aug 2026 00:00:00 +0000

Part I ended with rpkv: a sidecar that reads compacted Redpanda topics by key while storing only an index, key -> (partition, offset). The values never leave the log. It works, and one of its benchmark numbers is where this story starts. I’m telling it as a timeline because that’s the order I’ll want it back in.

The floor

rpkv answers a read in 11.1 ms p50 over loopback. I went looking for my bug and found the design instead: every lookup is a real Kafka fetch, so it pays TCP, protocol framing, and a fetch request the broker schedules like any other, all to retrieve one record the broker could read from its own disk in microseconds. In other words, the sidecar sits outside the process that owns the bytes, and every read pays admission.

The Allowlist That Runs Anything

Thu, 27 Aug 2026 00:00:00 +0000

I maintain mcp-shell, a small MCP server that lets an LLM run shell commands under a policy you control. It has a “secure mode” that’s supposed to be the safe default: no shell interpretation, a narrow allowlist of executables, a parser that only accepts a single, fully-literal command.

Over a few months I got four security advisories against it. Each one was a different command. Each one slipped past the allowlist. Each one I “fixed” with a patch that closed that case and left the door open for the next. This post is about why those patches kept failing, and the one change that closed the class.

Un lexer para parsear 'now-24h'

Mon, 24 Aug 2026 00:00:00 +0000

De vez en cuando entro en npm a mirar cómo le va a datetoken. Unas mil doscientas descargas a la semana, cero proyectos que dependan de ella. Mi teoría es que son los pipelines de CI de mi antigua empresa, ejecutándose con la puntualidad que nunca tuvieron nuestros informes. Si es así, es el usuario más fiel que he tenido nunca.

El problema, que era de verdad

Datetoken nació en wocu-monitoring.com, donde casi todo lo que pintábamos era una ventana de tiempo relativa: las últimas 24 horas, la última semana. Si el frontend resolvía “las últimas 24 horas” a dos timestamps y los metía en la URL, cada petición era única en el universo y la caché del recurso no servía para nada, porque el “ahora” cambiaba en cada segundo que pasaba.

A Job Board That Remembers

Mon, 17 Aug 2026 00:00:00 +0000

I’ve put jobctl.net online. It’s a job board that keeps the history other job boards throw away. Postings are polled, reconciled against the last look, and every difference is written down: appeared, changed, went missing, closed, came back. It’s small and it’s opinionated. It exists because of two grudges and one curiosity.

Grudge one: they took RSS away

Browsers used to understand feeds. There was an icon in the address bar, you clicked it, and from then on the site came to you. Firefox removed live bookmarks in 2018. Chrome never cared. Safari still has an RSS button, though I don’t think anyone at Apple remembers why. The web didn’t stop publishing feeds. Most of it still does. It just stopped telling you.

The Art of Not Buying Disk

Tue, 11 Aug 2026 00:00:00 +0000

I have released rpkv v0.1.0: key-value reads over Redpanda topics without duplicating values. That is the whole pitch. “Without duplicating values” is not a feature. It is a grudge.

The economics of 2026 hardware

RAM is currently priced like it is mined by hand. NVMe went up with it, out of solidarity. The industry’s advice has not changed: buy more anyway. Storage is cheap, say the articles, all of them written when it was.

Paranoid

Sun, 19 Jul 2026 00:00:00 +0000

Un clon de Arkanoid empezado en 2014 y terminado en 2026. Rompe el muro, caza las cápsulas, persigue la máxima puntuación.

The Repository Graveyard Just Lost a Tenant

Mon, 13 Jul 2026 00:00:00 +0000

Every developer I know keeps a graveyard. Mine lives under ~/src. Until recently its most distinguished resident was an Arkanoid clone I started in 2014, a game that was born dead. It never compiled for anyone but me. It never shipped. It never earned a README. It had a paddle, a ball, four levels, and the smell of a side project abandoned the week reality resumed. Twelve years in the freezer. The first commit of its resurrection says, with more honesty than I usually put in commit messages, “Rescue operation done.”

El aire acondicionado no fabrica frío

Sun, 05 Jul 2026 00:00:00 +0000

Cada verano me pasa lo mismo. Alguien me explica cómo funciona el aire acondicionado –o lo leo, o me lo cuenta un vídeo–, lo entiendo con esa sensación limpia de haberlo entendido de verdad, y unas semanas después ya no sabría reconstruirlo sin ayuda. Llevo años en ese bucle, así que esta vez lo dejo escrito, con diagramas y todo, para el yo del verano que viene. Esto no va de sistemas distribuidos ni de colas de mensajes; va del aparato que tengo colgado en la pared del salón, que sospecho que es la pieza de ingeniería más antiintuitiva de toda mi casa.

Parallelization Index

Fri, 10 Apr 2026 00:00:00 +0000

El otro día me encontré con cuatro terminales abiertas, cada una con un agente trabajando en una parte distinta del sistema: uno refactorizando el módulo de autenticación, otro generando tests de integración para un servicio nuevo, un tercero explorando una migración de base de datos y el cuarto escribiendo el plan de un cambio en la capa de eventos. Cuatro líneas de trabajo en paralelo, todas avanzando, todas requiriendo mi atención intermitente para validar decisiones, corregir el rumbo, aprobar o rechazar. Y me descubrí pensando algo incómodo: esto no se parece en nada a lo que hacía hace dos años, y sin embargo es lo más productivo que he sido nunca.

Plan-Driven Development

Mon, 09 Mar 2026 00:00:00 +0000

Estaba hace unos meses inmerso en un cambio arquitectural de cierta envergadura: sistema de colas, muchas piezas móviles, el tipo de modificación que toca suficientes capas del sistema como para que un error de criterio sea difícil de deshacer. Abrí el modo de planificación del agente y empecé a trabajar con él: que escaneara el código existente, que identificara dependencias, que propusiera opciones y razonara las descartadas. El plan fue creciendo solo, incorporando trade-offs, fragmentos de código, enlaces a funciones concretas. En algún punto me alejé del teclado y lo leí de un tirón, como se lee un documento antes de enviárselo a alguien.

Parco: An Experiment in Go Serialization

Sat, 07 Feb 2026 00:00:00 +0000

At Atani we built a real-time producer that connected to over 30 cryptocurrency exchanges and DEXes. It streamed trades, orderbook updates and market data through Redpanda and NATS to a crowd of consumers. Millions of messages a day. Microseconds mattered.

We used easyjson for serialization. It worked. JSON is readable and debuggable, and easyjson generates marshal code that’s much faster than the standard library. For our throughput and latency it was enough.

About

Thu, 09 Oct 2025 00:00:00 +0000

Who

I’m Marcos Sánchez. I work as a Site Reliability Engineer at Chess.com, from Madrid. Before that I spent six years at Atani, now defunct, building the real-time plumbing of a crypto exchange: order books from thirty venues, billions of messages a day, microseconds that mattered. Most of what I know about systems I learned there, at 03:00, from things that had broken.

Why this site exists

Three reasons, in the order they occurred to me.

Career Timeline

Thu, 09 Oct 2025 00:00:00 +0000

A chronological timeline of my career building distributed systems and leading technical teams.

Site Reliability Engineer at Chess.com

November 2025 - Present | Remote

Reliability and infrastructure work for a high-traffic platform serving millions of players. Focused on keeping production available, observable, and fast.

Key responsibilities:

Own service availability, incident response, and operational readiness
Improve observability and performance across production systems
Automate infrastructure and delivery workflows

Senior Reliability Engineer at Atani

December 2023 - October 2025 | Madrid