Today’s attack was unique because it was completely unintentional.
There was a problem with a customer’s site, because they had programmed an infinite loop on their event tracking code. So, what would happen is that a visitor would load their page, and then an event would fire itself at a constantly-high rate until the page was closed.
(Making things worse: the page played a popular and very fantastic song that’s 3:08 long, so the page was left open for quite a while by most people.)
Now, we’ve hardened our security a lot since we were first DDoS’ed last year, and our firewall routinely blocks similar attacks every week. However, the issue with this incident is that our security was focused on page collection, not event collection. As of now, we've put additional security in front of event collection to prevent this from happening again.
Fathom did not go offline, but it did create a backlog. Once we isolated and blocked the offending customer’s event (and had them remove the code from their site), our backlog cleared in less than five minutes.
How will this be avoided in the future?
We’re migrating to a new database (finished March 12, 2021) that can easily handle things like this, and it will process backlogs like the above much faster. We’ve now added security checks to event as well. If a similar event happened in the future, our software would automatically block offenders (even if their music tastes are quite acceptable). Let us know if you have any questions. We’re always just an email away.