What happened?
Between 10:34 AM UTC on July 28th and 14:03 UTC on July 29th, email notifications containing attachments, primarily Incoming Feedback and Invoice emails, were not being sent.
We’ve implemented a fix for this and any queued or failed email notifications will gradually be sent out over the next 48 hours approximately.
Why did this issue occur?
After upgrading the Celery version, the serialization protocol changed, causing some jobs to receive malformed events.
What will we do to prevent this from happening in the future?
We didn't receive alerts for this issue, so we'll look at how we can improve alerting and logging for these types of issues. Also, we’ll be looking at how to detect serialization/deserialization issues with end to end tests for the background jobs.