Disruption to processing of background tasks.

Incident Report for Hotjar

Postmortem

On May 11th, 2021 at 09:57 UTC our engineers identified an issue with processing of Heatmap and Recording data that lasted for several hours.

What happened?

After an update to Hotjar we realised that data processing wasn’t working correctly.

We took steps to fix this immediately, but the outage caused a backlog in our system that overloaded our servers. To reduce load, we had to limit our data collection for all sites.

The system took until 16:00 UTC to fully ramp back up to normal data collection levels again. It will attempt to over-collect to make up for the shortfall.

We are very sorry for inconvenience caused, and thank you for patience on this.

What will we do to prevent this from happening in the future?

We will put in place safeguards against this kind of overload happening again.

Posted May 11, 2021 - 16:28 UTC

Resolved

The issue has been resolved. Heatmap and Recording data collection and processing are back to normal levels. Thank you for your patience.

Posted May 11, 2021 - 16:09 UTC

Monitoring

In order to restore normal service we have intentionally reduced the number of sessions being processed, which means there will be fewer Recordings captured, and Heatmaps will be updated less frequently.

We apologize for the inconvenience. Our engineers are working on it and full service should be restored soon.

Posted May 11, 2021 - 11:43 UTC

Update

We're experiencing a major degradation in processing of Heatmaps and Recordings. Our engineers have identified the problem and are working on a fix ASAP.

Posted May 11, 2021 - 11:13 UTC

Identified

We’ve identified the issue with background task processing and our engineers are working to resolve it.

Posted May 11, 2021 - 10:24 UTC