Processing degradation

Incident Report for Hotjar

Postmortem

On 2021-04-19 at 15:59 UTC our engineers identified an issue with data processing.

What happened?

Our background tasks system became overwhelmed (it ran out of resources). This caused delayed processing and some degradation (some items will not have been processed). All Hotjar features were affected by this incident.

During this time, any operations where we need to process information in the background may have been affected. From your perspective, you may have noticed delays or failures in carrying out these operations, not limited to:

Heatmap generation
Survey response ingress
Heatmap Retaker tool failing

Processing was delayed for approximately one hour.

Why did this issue occur?

We released an update to a component in the background tasks system which incorrectly used more resources than desired. This had a knock-on effect on the rest of our background task processing.

Posted Apr 19, 2021 - 17:22 UTC

Resolved

The issue has been resolved. Thank you for your patience!

Posted Apr 19, 2021 - 16:51 UTC

Monitoring

Our engineers have fixed the issue and are monitoring the situation to make sure everything is working smoothly again.

Posted Apr 19, 2021 - 16:36 UTC

Identified

We’ve identified the issue and our engineers are working to resolve it. You may notice delays in the generation of heatmaps and incoming feedback.

Posted Apr 19, 2021 - 16:19 UTC

Investigating

We’re currently investigating an issue with data processing. Our engineers are working to fix the issue ASAP and will post updates to status.hotjar.com. Thanks for bearing with us!

Posted Apr 19, 2021 - 16:16 UTC