June 3rd - Service Interruption

On Friday morning, June 3rd, 2011, Intervals experienced a 40 minute service interruption between the hours of 8:40 AM (PDT) and 9:20 AM (PDT). The problem was due to a network configuration issue and has since been resolved by our system administration team. We sincerely apologize for any inconvenience this may have caused. We would like to reiterate that consistent uptime is our number one priority with Intervals.

What exactly happened?
At approximately 7:10 AM (PDT) on June 3rd, 2011, our network probes and monitoring system reported a problem with network availability. Our system administration team immediately started an investigation and isolated the problem to a configuration issue with the load balancer technology we utilize for high availability. While we were working to resolve the issue, connectivity was intermittent, but appeared mostly stable until around 8:40 AM (PDT). At that time the problem escalated and the servers in the server farm were virtually unable to communicate with one another, which in turn caused the load balancer to fail. At approximately 9:20 AM (PDT) the problem was resolved and all network traffic was fully restored.

How did we fix it?
The source of the problem was a configuration problem between the load balancer technology we utilize and a recently upgraded router in the data center. The problem did not become apparent until after the router was recently upgraded. The router upgrade exposed a configuration glitch in our load balancer settings that resulted in our servers being unable to communicate with one another over the private network. As the saying goes, "it was working, but it wasn't working right." Once we updated the load balancer configuration and network configuration, the private network came back online with no incident.

What did we learn from this issue?
During this interruption we were focused on two things: fixing the problem and responding to customer support inquiries as quickly as possible. In addition, we should have broadcast updates to Facebook and Twitter immediately once we knew of the problem. We want to be sure to maintain transparency and keep our customers in the loop.

Questions?
If you have any questions or concerns please feel free to email our support team at support@myintervals.com. We’d be more than happy to provide you with any more information you might need. We appreciate the patience and understanding extended to us during this interruption in service.