March 11th, 2014
Local Domain Name Server (DNS) Resolution Failures
Some markets have continued to report network issues connecting to our servers. We first received reports of this on 3-08-2014. Depending on what you're accessing, we have servers in Dallas, New York, and Chicago. We also have monitoring all over the USA and Europe to help us maintain visibility over our network traffic and up-time.
Our services have been 100% up throughout all reports since last Saturday. See for yourself, here:
The response time has consistently trended at around a quarter of a second. We use that to show that we're responding to requests consistently without interruption.
We've been informed by our hosting provider that Level 3 has reported incidents impacting multiple regions, but we've taken steps to find a workaround for those affected.
For those experiencing issues:
So your internet connection tries to find a path to our servers when you type signal88.inteliguide.net into your browser address bar. This is how all domain names resolve from their names (like google.com) to their machine IP addresses (like 126.96.36.199). Sometimes the details get lost in translation if records are changed too often, or not cached properly, or the network en route from your computer to our servers breaks down somewhere along the line.
That's essentially what's been happening. Local DNS has been unable to connect to our servers, meaning there's a breakdown in communication somewhere between your ISP and our service. This is beyond our control, but there is something we can suggest. Use a national DNS provider!
You will need Admin access to your Toughbook or desktop computer to complete this workaround.
- Go here for Windows: http://answers.microsoft.com/en-us/windows/wiki/windows8_1-networking/how-to-change-ipv4-dns-server-address-to-public/56548e8d-eb9c-4fc5-807c-eefca5d278d0
- Here for Mac: http://www.howtogeek.com/howto/38793/how-to-switch-mac-os-x-to-use-opendns-or-google-dns/
And follow the walkthrough to enter Google as your DNS provider on your active network connection.
This is not just a temporary solution. You can keep it in place to help avoid this sort of issue in the future. In the eight years that we've been providing web services, this is only the second time this has happened, so it's generally been a very rare occurrence, but this workaround will help if/when it happens again.
Note: This is a confirmed workaround, but will only apply in cases where DNS is failing to resolve from your ISP. If there's a broader internet connectivity issue, or you lack any solid connection to the internet, this workaround does not apply.
We received the following from our hosting partner late Saturday night in response to tickets we logged shortly after hearing our first reports of issues:
"I was informed that there have been multiple notifications that the Internet Service Provider 'Level3' (which is a major back-bone provider) has had issues world wide, with severe packet loss to other providers noted last night and earlier today. These backbones providers included Cogent, Sprint, Savvis, ATT, and Verizon."
We're continuing to follow up with these providers to see if we can get an ETA on resolution of these issues, and will post updates here as they come in.
After talking with Verizon and our hosting provider, we discovered that there were multiple points of failure causing these local outages. Packet loss from Level3 temporarily knocked Comcast and other users out over the weekend, and it appears that a DDoS (distributed denial of service) Attack has been responsible for the rest of the remaining issues. We're still working to see if all affected markets are able to consistently access our services again, and have had several report that the issues have been resolved.