Learning
Earlier today (Sunday) our hosting provider, Rackspace Cloud, had a redundant power cycle failure, which caused ~2 hours of downtime for ManageBac users. We sincerely apologize and we are very unhappy when there is any period of time when you cannot access ManageBac.
We have a duty to ensure high-availability. We had switched to a new server on Saturday without any issues, but a day later, there was a power failure with our new server at Rackspace. Both our backups and our database server were functioning normally throughout this period.
The server upgrade on Saturday was completed in order to expand storage limits and to improve the application response times. Our application response times are stable and normal at ~184 ms.
This is a timeline of what caused the downtime on Sunday morning:
– The host that our server was on went down at 9:04 am CST. We immediately contacted Rackspace and their operations team tried to do a remote power cycling (i.e. to restart power to the server).
– The remote power cycling did not work, so the Rackspace Data Center Team tried to manually restore power to the host.
– This was not effective, so a chassis swap process was initiated, where the hard disks were removed from one server and replaced into a new healthy host.
– The host and servers came back up successfully at 11:05 am.
Again we sincerely apologize for the inconvenience, and we expect the new servers to operate smoothly going forward.
