Aqua Voice - Geschichte beachten

After the previous deployment problem, we made a ton of upgrades to ensure deployments wouldn't take down the service for people. More redundancy. Blue-green deployments. More and better tests.

That had been working well... but rolling out this batch of changes caused a bunch of database connections to have invalid caches while a migration was applying. That broke auth with a cascade of 'cached plan must not change result type' errors.

Impact
Aqua was totally down for 10 to 15 minutes.

Fixes
No more yolo migrations. Deployment synchronization.

This incident has been resolved.

A migration failed during one of our deployments.