Emergency Status Announcements
Internal Network Problems
From: jdw
Posted: 2 days ago
We are experiencing an unusual problem on our internal network that is affecting services. We are investigating now and should have more information soon.
Update: Sorry for the delay in updating this; we have been working hard to understand the issue. We discovered a bizarre switch partition in our network (literally, a single switch partitioned itself into multiple spanning trees). We're finalizing the process of rebuilding that switch now and should have all services restored shortly.
Update: We are seeing some time synchronization issues as a result of this incident; a handful of servers are about 90 seconds off. They should converge to the correct time soon. Other than that, most problems should be resolved. We will be checking beta pools shortly.
Update: We believe everything is back to normal except the servers with the clock skew.
Update: The clock issue is resolved so everything should be back to normal now. We are sorry for the trouble, and will be checking with the switch manufacturer to find out what could cause that sort of issue.
There are currently no open system problem reports.