解決済み
重大な障害
開始日時 10か月前持続した 10分
アップデート
- 死後死後
After the previous deployment problem, we made a ton of upgrades to ensure deployments wouldn't take down the service for people. More redundancy. Blue-green deployments. More and better tests.
That had been working well... but rolling out this batch of changes caused a bunch of database connections to have invalid caches while a migration was applying. That broke auth with a cascade of 'cached plan must not change result type' errors.
Impact
Aqua was totally down for 10 to 15 minutes.
Fixes
No more yolo migrations. Deployment synchronization. - 解決済み解決済み
This incident has been resolved.
A migration failed during one of our deployments. - 調査中調査中We are currently investigating this incident.