Issue With LuxSci Shared WebMail Servers
Incident Report for LuxSci
Resolved
The root cause of the degradation in our WebMail acceleration system was a disk drive on a server at Rackspace that began suffering from a "noisy neighbor" issue which decreased the read/write throughput of the disk significantly.

We have resolved this by (1) standing up 3 new servers to replace the 1 server having an issue. We have divided all customers that were sharing the 1 problem acceleration database server between the three new servers. This serves to increase the resources available by 3x and lower the scope of impact of any future issue.

As part of this recovery process, we have had to clear and reset the WebMail acceleration cache for the affected users. This is cache an optimized view of your recently accessed email folders that allows WebMail to function efficiently, especially with very large folders. As customers open folders in WebMail going forward, the system will automatically start recaching their contents in the WebMail acceleration databases and will use the databases and/or IMAP, as available. Customers will very large folders may still experience some slowness when initially using them in WebMail; however, this should clear up relatively quickly as the data is cached ... especially now that we have added 3x the resources and no longer have a slowed disk drive in the mix.

We apologize for any inconvenience this issue may have caused. We will also note that the issue had no impact outside of WebMail; i.e., customers using programs like Outlook or Thunderbird were not affected at all.
Posted Nov 26, 2019 - 01:17 EST
Monitoring
We are currently experiencing an issue with LuxSci Shared WebMail servers which is resulting in slower load times when accessing the LuxSci interface and when viewing email messages, especially in larger folders. The issue was caused by the need for our WebMail Acceleration server to be reset. Once time has been given for the WebMail Acceleration server to "catch up" servers levels will be normal. We will provide an update once this has been fully resolved.
Posted Nov 25, 2019 - 16:44 EST
This incident affected: Shared WebMail.