Impact: No connectivity to all services
Services: All
Date: Tuesday, 8 December 2009
Duration: 11:00 – 13:40 AEDST
Status: Resolved
Summary:
We are currently experiencing network connectivity issues for all services. Technicians are looking into the issue.
Update, 11:18: We have established that this issue is related to an outage at our data centre provider. We will update again once we have further information.
Update, 11:35: Thanks for your patience. Services are now coming back online progressively.
Update, 11:55: Engineers are bringing services back online carefully and in sequence. Currently, we estimate that all services will be restored within 30 minutes.
Update, 12:30: Some services are back online, including POP/IMAP mail. Technicians are still working to restore other services. We will provide an update within the next 15 minutes.
Update, 12:45: Technicians are still working to restore full service. A number of web services are back online, and we’re working to bring the rest back up. We will provide another update in the next 15 minutes.
Update, 13:05: Most services have been restored to partial or full capacity, including the web sites, Toolkit, Webmail, POP/IMAP and inbound email. Technicians are still working on restoring full capacity to services. We will provide another update in the next 15 minutes.
Update, 13:20: Some services are still operating at partial capacity, and we’re working to restore these to full capacity shortly.
Update, 13:40: All services have been restored to full capacity. Some email sent between 11:00 – 13:00 has been queued and may be delayed for a few hours.
Update, 14:05: We’ve observed some issues with some MySQL databases using InnoDB tables. Our engineers are working to fix this now.