Infrastructure upgrades
Incident Report for Status for Fathom Analytics
Resolved
We performed some upgrades to our default Fathom script (cdn.usefathom.com) on the 11th October 2021. We moved the ingest endpoint to a new CDN, with added security and global availability (to improve performance worldwide).

For 1-2 hours during this move, whilst DNS was propagating, we had some incorrect configuration on our new CDN which meant that not all pageviews were being tracked. So for customers who aren't using custom domains, you'll see that your pageviews will have dropped during that time. Unfortunately, we have no way of "back-filling" missing pageviews, as we don't keep any kind of access logs.

Again, customers who are using custom domains were not affected. However, folks using our default Fathom script will notice a slight drop during that period. The reason this issue wasn't caught sooner is because we monitor for downtime, not incorrect configurations, and the response wasn't technically broken. And then the reason why some pageviews were collected and some weren't was down to the fact that global DNS propogation takes time, meaning some of your website visitors were hitting our old infrastructure, whilst some will have hit the new infrastructure.

Following this, we're going to be implementing changes around testing. We'll now be monitoring end to end, ensuring that the pageview is collected and that it appears in our database. Clearly monitoring for uptime alone isn't enough, and we need full, 24/7, minute by minute checks for the full end-to-end process. When testing manually, we had assumed the DNS had propogated, but it was still hitting the old servers. We apologize to everyone affected here.
Posted Oct 11, 2021 - 00:00 PDT