해결됨
심각한 장애
시작 10개월 전지속됨 10분
업데이트
- 사후 분석사후 분석
After the previous deployment problem, we made a ton of upgrades to ensure deployments wouldn't take down the service for people. More redundancy. Blue-green deployments. More and better tests.
That had been working well... but rolling out this batch of changes caused a bunch of database connections to have invalid caches while a migration was applying. That broke auth with a cascade of 'cached plan must not change result type' errors.
Impact
Aqua was totally down for 10 to 15 minutes.
Fixes
No more yolo migrations. Deployment synchronization. - 해결됨해결됨
This incident has been resolved.
A migration failed during one of our deployments. - 조사 중조사 중We are currently investigating this incident.