Users of our internal assistant kept repeating the same preferences: their team, report format, time zone. Adding memory sounds simple and is not. How we separated short-term and long-term memory, decided what to store, handled conflicts and stale facts, and gave users control over what the assistant remembers.
A Go ingestion service that forwarded events to a partner API was OOM-killed every time traffic spiked. Each event started a goroutine that waited on the slow partner, so goroutines and their buffers grew without limit. How we found it with pprof and fixed it with a bounded worker pool.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.