This incident is now resolved. The data has been restored from backup and the restored data is now live.
Posted 8 months ago. Sep 01, 2018 - 16:15 EDT
This incident applies to 167 people across 18 customer accounts. Once the issue is resolved, we will be proactively issuing these account SLA credits for the email down time.
"Early this morning, a RackSpace technician who was removing an unused disk from this shared enterprise class server also removed and deleted a separate live disk. This was human error issue and we have escalated the issue with them to find our more about how it could have happened, etc. The net result is that the storage disk for some or all in 18 customer accounts went offline about 2:15am ET today. We are in the process of restoring this data from our latest daily backup snapshot, which is from about 10am ET on Friday, August 31st.
At the current rate of copy, we anticipate that the restore should be done between 2 and 3pm ET today. Once the restore is done, we will make all of the restored data visible to the affected users. Inbound email that has arrived since the disk went offline is being queued and will be delivered after the restored data is made live. Any changes to affected email folders between about 10am ET on Friday and 2:15 AM Saturday would be lost. Affected customers who have our Email Archival service will find copies of sent and received email therein.
We sincerely apologize for this incident."
Posted 8 months ago. Sep 01, 2018 - 11:18 EDT
It appears that we will be restoring the contents of this disk from this morning's backup. This process us underway. We will update this notice with further status.
Posted 8 months ago. Sep 01, 2018 - 10:16 EDT
One hard drive on on one of our shared enterprise class email servers (rs155.luxsci.com which serves email32.luxsci.com and email49.luxsci.com) has failed and is no longer visible to the server. This is causing email for about 170 users to not be visible. Our operations team is working on resolving the issue. We will update this notice with the status.