This has now been resolved. Our engineers made a deployment to the live servers which caused an incorrect hostname to be set for our memcached service. The memcached service is used to cache the settings for websites, to ensure they can be retrieved as efficiently as possible. Since the hostname was incorrect, our servers were not able to communicate with the memcached service, resulting in errors when changes were made to the tools used on site.
A policy document for deployments have already been updated to ensure hostnames aren't changed and are retained for all instances. Furthermore, we have created a story for our backlog to have a more secure and robust storage for settings and credentials, ensuring they are enforced at a higher level.
Posted Nov 02, 2016 - 11:44 UTC
Investigating
We're investigating an issue which is causing errors when users try to create or edit items in Hotjar.