Resolved -
All back to normal, main cause for the increased error rate was aws apigway increased error rate for aws lambda invocation, which cased authorization and middleware issues in moving the requests forward into the system.
Jun 8, 21:35 CEST
Monitoring -
A fix has been implemented and we are monitoring the results.
Jun 8, 21:28 CEST
Update -
Escalated via ticket to AWS as our API gateway principal is getting 500 errors on Lambda invoke
Jun 8, 21:22 CEST
Identified -
Current error rate for submission endpoints is around 0.5%, errors connected to aws lambda instance stuck in corrupted state, applying redeploy
Jun 8, 21:02 CEST
Investigating -
Spike in 500 on submissions endpoints connected to vendor issue
Jun 8, 20:51 CEST