Apache Web Server Updates This Week

LuxSci will be updating Apache, PHP, and openssl software on its dedicated and shared web servers this week.

This will occur in a rolling fashion during the following maintenance windows:

  • Monday: 5pm-8pm ET
  • Tuesday: 5pm-8pm ET
  • Wednesday: 5pm-8pm ET
  • Friday: 5pm-8pm ET

When your web server is updated, you may expected a few seconds of downtime as the apache server is stopped and then started again.

In addition to updating PHP, the primary goal of this maintenance is to upgrade all web servers from Apache v2.2.x to v2.4.x.

Inbound email delays for some users

One of LuxSci’s email filtering servers has experienced an issue this afternoon and this has resulted in some email for some users being queued. This email is currently being processed from queue and delivered… we expect this process to finish sometime later tonight.

The users affected are those without our McAfee-based Premium Email Filtering, but with an Enterprise-level account. Email for these users uses through a number of redundant servers, only one of which experienced an issue. As such, only some email for affected users would have been delayed.

We have recorded the root cause of the issue and are taking steps to ensure that such an event is automatically remediated should it arise again in the future.

Hailstorm/Snowshoe Spam and Spam Flood Protection

Recently McAfee (our partner for Premium Email Filtering) enabled a new spam filtering feature called “Spam Flood Protection”. This feature was created to protect against SnowShoe and HailStorm spam attacks.

SnowShoe attacks are spam attacks that spread the output of the spam attack across many IPs and Domains. This is done in order to dilute the reputation metrics and filters that most spam filtering services use to detect spam. During a SnowShoe spam attack the spammers will send email from domains that have anonymous or unidentifiable WHOIS information. Some SnowShoe spammers will use tunneled connections from their back-end spam source to the spam egress IP. This removes the backend IP from the spam’s headers making it difficult to determine the source of the spam and block it based volume based spam filters, and DNSbls, and IP address filters.

A HailStorm spam attack is a SnowShoe attack run across smaller netblocks (/25 and /27 observed, but not always contiguous) over a smaller period of time with many simultaneous connections. This can occur in a timeframe of seconds to a few minutes.

McAfee’s “Spam Flood Protection” feature blocks email based on proprietary filters that look for the hallmarks of a SnowShoe/HailStorm attack. This allows them to block email that is due to a SnowShoe/HailStorm attack without affecting normal emails. This feature is already being used by many clients at McAfee.

Email that gets caught by that filter will return an error of “451 Exceeding connection limit: RBLDNSD”.

If legitimate messages are incorrectly caught by this filter, there are work arounds (see below).  LuxSci has worked with McAfee to have the IP addresses of all of its servers white listed (as of Monday) so that messages sent through LuxSci would be exempt from this filter.

To disable this feature within McAfee’s Spam Filtering portal you will need to go to “Email Protection > Policies > Choose the Inbound Policy you want to edit > Spam Tab”. There you’ll see a option at the bottom of that page that has a checkbox next to “Spam Flood Protection”. To disable it simply uncheck the box and save your changes.

The changes will take affect within 20 minutes. Another way to get around the error is to add any domain that is sending to you and getting the error to your domain’s Allow list. This has the effect of getting rid of the 451 errors for the domain you added to your Allow list without disabling the SnowShoe/HailStorm protection that the option provides. Adding the domain (or better yet, the IP of the sender) to your Allow list is the McAfee suggested course of action in the face of false positives.

Scheduled Network Switch Reboot

(Yes — there are two of them.)

Rackspace is performing scheduled maintenance that will affect our Network infrastructure in Dallas, Texas, USA:

Mandatory Switch Reboot scheduled for June 2nd, 2015

As part of an ongoing effort to maintain our data center infrastructure, we will be performing software upgrades on network switches in our DFW data center. The upgrade will require a reboot of the network switches, causing approximately five minutes of network downtime.

The maintenance is scheduled for June 2nd, 2015.

The maintenance window is 12am – 6am Central Time, USA.

The 5 minutes of network downtime can affect access to:

  • WebMail Interface
  • Receiving email
  • Sending email
  • MobilsSync
  • Hosted web sites
  • Hosted MySQL databases
  • SecureForm services

This will not affect:

  • Receipt of inbound email (it should at most be delayed a few minutes)
  • SecureChat
  • Premium Email Filtering
  • Premium Email Archival
  • DNS services

Please accept our apologies for any inconvenience this may cause you. If you have any questions or concerns regarding this maintenance, please do not hesitate to contact our support staff.

Scheduled Network Switch Reboot

Rackspace is performing scheduled maintenance that will affect our Network infrastructure in Dallas, Texas, USA:

Mandatory Switch Reboot scheduled for June 6th, 2015

As part of an ongoing effort to maintain our data center infrastructure, we will be performing software upgrades on network switches in our DFW data center. The upgrade will require a reboot of the network switches, causing approximately five minutes of network downtime.

The maintenance is scheduled for June 6th, 2015.

The maintenance window is 12am – 6am Central Time, USA.

The 5 minutes of network downtime can affect access to:

  • WebMail Interface
  • Receiving email
  • Sending email
  • MobilsSync
  • Hosted web sites
  • Hosted MySQL databases
  • SecureForm services

This will not affect:

  • Receipt of inbound email (it should at most be delayed a few minutes)
  • SecureChat
  • Premium Email Filtering
  • Premium Email Archival
  • DNS services

Please accept our apologies for any inconvenience this may cause you. If you have any questions or concerns regarding this maintenance, please do not hesitate to contact our support staff.