I am going to post the whole story of what happened here…

On Saturday afternoon, I was watching our scraping machines via an interface that we have that tells us how they’re running. I noticed a problem and I tried to load the bluffmagazine.com website. It timed out. Then I checked another interface that we have to actually remotely look at the desktop of our data gathering machines. All 7 were offline and inaccessible. I’d never seen that before — sometimes one is offline or something due to some issue, but never more than that.

So I called the head developer at Bluff and asked wtf was going on. He obviously didn’t know, so he said he’d investigate and call me back. We talked again after about an hour and he said he didn’t know what the hell was going on. No one was answering his calls and the host itself’s website was down. He later talked to another person who hosts websites at the same facility and he was told that a fiber optic cable was cut leading to an internet outage at the host. We went to bed on Saturday thinking that the cable repair was being done and that the internet would be back up at any moment.

We woke up on Sunday and found out that the fiber optic cable story was BS. I’m not sure where it came from, but it was like 7th hand by the time it came to me. As it turns out, there was a massive router failure in the facility. There were people on-hand bringing cages/shelves in the server farm back online one-by-one. By Sunday, most were back online. For whatever reason, our servers’ connections weren’t fixed when they should have been. We decided at that point to find a new host. By Sunday night, we found one. On Monday, three of us went down there to pick up all of our machines. Today, we installed them at the new facility.

At this point, everything should be back up and running as soon as the servers are configured at the new facility. I expect that to happen tonight. Sometime tomorrow, we should be able to put in all of the data that hasn’t been inserted since the systems were down — just because we kept collecting data doesn’t mean that it has been inserted yet. But pretty soon, it should look like this whole nightmare never happened.

Thanks for your patience.

Related posts:

1. Wow, 10 days fly by
2. Back from Ireland for a few days
3. A few days into CrossFit
4. Recap of last couple days
5. thepokerdb site improvement