Lessons

what wy2z has taught me

A running list of the hard-won kind — mostly about systems thinking, since that's where the surprises keep coming from. Each entry is a real moment that bit me, written so I don't forget the shape of the bite.

The frame everything hangs on

most outages aren't broken code. they're broken assumptions.

When wy2z went silent for fifteen days in late May 2026, I assumed the Pi had crashed. It hadn't. It had been up for 36 days straight, cron firing every 4 hours, sensors reading fine, disk healthy at 7%. The Jetson was up for 39 days. Neither device had a single hardware problem.

What actually broke was a single line in a config file that said “the Jetson lives at 192.168.0.224.” That was true the day the line was written. A month later, during a thunderstorm, the router rebooted and the Jetson came back at .225. Nothing in the system changed. The system's picture of the world changed.

Hardware reliability is the easy problem — you can measure it. Assumption reliability is the hard problem, and it's where most real outages hide.

stale state · latent bugs · implicit contracts

Tactical lessons

self-healing > deterministic

A hardcoded IP says “the answer is X, forever.” Name-based addressing (mDNS, DNS) asks “what's the answer right now?” every call. The first is faster by milliseconds. The second is faster by weeks-of-outage.

mDNS · service discovery · idempotency

band-aids become load-bearing

The IP override was added during unrelated troubleshooting back in May as “extra insurance.” The real fix landed elsewhere, but the band-aid stuck around. A month later it was the only thing the pipeline silently depended on. Every temporary fix is forever unless you delete it deliberately.

technical debt · workaround rot

hidden state is invisible failure

The wrong IP lived in one config file on one device, never logged, never surfaced anywhere a human or dashboard could see it. It could have been wrong from the moment it was written and nobody would have known. State you can't see is state that can rot.

observability · single source of truth

silent failure is the worst failure

When the SSH step broke, the pipeline aborted and wrote zero observation rows. From the dashboard, that looked identical to “the entire Pi is dead.” If the system had written a row that just said error: jetson_unreachable, I'd have diagnosed it from anywhere in 5 minutes. Make your system tell you what broke.

structured errors · graceful degradation

pin things at the source of truth

If the Jetson genuinely needs a stable IP, the router is the right place to bind it (a DHCP reservation tied to its MAC address) — not a config file on the client. One source of truth, observable from anywhere on the network, no copies that can drift.

DHCP reservation · single source of truth

check the things you actually care about

“Is the Pi alive?” is the easy question — and was trivially yes the entire time. “Can the Pi still reach the Jetson?” was the question that mattered, and was never checked anywhere. Health checks should verify the relationships between parts, not just the parts.

liveness vs. readiness · end-to-end probes

Architecture note (not a lesson, a pattern)

orchestrator + worker — why one computer bosses another around

The Pi tells the Jetson when to take a photo via SSH. It sounds wild — one computer remote-controlling another every 4 hours — but it's a standard distributed-systems pattern. Each device does the one thing it's best at: Pi handles cron + sensors + cloud uploads + watering triggers, Jetson handles the camera + autofocus.

The cost of the split: the network hop between the two devices is now a failure surface neither machine alone can cause. That hop is exactly what bit me on May 27. Whatever your architecture splits across machines becomes a new boundary that can break.

orchestrator · worker · separation of concerns