-Ryan
-----Original Message-----
From: CITES Communications to UIUC tech support staff On Behalf Of Allan Tuchman
Sent: Sunday, October 29, 2006 10:42 PM
Subject: [CITES-TECHSUPPORT] Email delays, spam, and discussion
Colleagues,
Some campus email users are experiencing delays in email delivery.
The main campus email relays are suffering from the large influx of spam email. This increased email volume is challenging the filtering capacity of CITES Spam Control (CSC), and causing email delivery delays to users and departmental email servers. The limits in the amount of email that we can accept may exhibit itself as long delays in email delivery. Other email servers which are not able to immediately transfer email to our relays may not retry the exchange again for minutes or hours. The effect is that our campus community experiences delays in receiving their email. We are working to alleviate the problem in several ways which I'll describe here.
We plan to purchase new, faster hardware for spam filtering and have it installed over the Winter break. This new hardware will increase our processing capacity significantly and allow additional room for expansion when we need it. In the meantime, we are approaching the problem in other ways: filtering, blocking, whitelisting, and compliance checking.
For several months we have filtered SMTP (email) traffic, limiting access from hosts which have sent a high percentage of spam traffic in recent hours. We have now changed this rate limiting to use more aggressive settings. We have instituted IP blocks at the "sendmail" level to prevent some hosts from even contacting our email relays at all. These are computers determined to be flooding us with email spam. Some domains will be allowed to pass email into campus, checking for virus but omitting the more costly spam check. Currently this includes only "facebook"
(an internet service), which accounts for 2.5% of our inbound email. Finally, we have implemented sendmail's "greet pause"
feature to deny access to email "slammers" who spew data at us without regard to proper protocols.
Each of these processes carries some risk and increases human resources to monitor this risk. For example, gmail (Google's email) and others can also be affected by the "greet pause" feature, and we do not want to stop legitimate email. But by tuning these processes we hope to keep email reliable and with as little delay as possible until the hardware upgrade can occur over Winter break. I expect that we will continue to see some delays though, as spam volume increases even more. Most of the standard techniques for improving our situation have already been attempted. The large scale of our mail infrastructure presents unique challenges that are not encountered in smaller installations
The growth of spam appears limitless and unpredictable (see details below). Our current email relays and Spam Control hardware were designed to handle four times the campus email volume experienced just over one year ago. It has since been scaled up with additional hardware yet is barely meeting today's load. UIUC is not alone in feeling this pain. Discussions with our spam control vendor and colleagues at other institutions show this to be a widespread increase. In fact, we see an increase of email sent from UIUC being delayed (and queued on our servers) as other servers defer receipt of email from UIUC.
CITES Spam Control has protected the campus from widespread virus outbreaks and has largely given us back control of the email we receive. CITES recognizes its importance and is directing appropriate resources to attack the increased traffic. We appreciate your positive response to our Spam Control solution and your patience as we adjust to the unexpected growth of email traffic.
Additional technical details
----------------------------
-- Statistics --
We have passed the 3 million messages per day mark. Our email relays and CITES Spam Control (CSC) processed 3.23 million messages on October 14, 2006. 77.2% of the incoming email is tagged as spam (data from representative sample Oct 25-27). The CSC spam quarantine has grown from 6 million to 13 million messages since late summer.
I've put a graph in a public NetFiles folder which illustrates the incredible growth of spam during this calendar year:
https://netfiles.uiuc.edu/tuchman/public/ytd_spam_and_ham.jpg
The spam level began its ascent in May (upper, or red curve) and has accelerated ever since.
-- Short Term Solutions --
1. SMTP rate limiting -- Specific hosts are self-selected for
rate limiting based on the percentage of spam being sent.
Machines sending 70%+ spam are lightly rate limited. Sending
hosts encounter increasingly stricter rate limits as they hit
80%+, 90%+, and 99%+ spam percentage levels.
2. IP blocks -- again, the IP addresses and/or domains of machines
causing heavy spam loads change continuously. This weekend 145
class C subnets have been blocked. All the network ranges we've
blocked have historical CSC scores exceeding 99. (The majority
are actually averaging 100.) We don't have a good way to
make the list of blocked hosts generally available. But you may
contact CITES (the best way today is to open a ticket via
net-trouble) if you need information on blocked hosts. This
technique has been somewhat successful. Our quarantine has
reduced considerably and nightly spam digest generation time has
dropped.
3. Whitelisting -- Only facebook is whitelisted today. It accounts
for 2.5% of our email and with an average spam score of only
0.05 (out of 100). A future consideration is to whitelist any
email originating with CITES Express Email or originating on the
UIUC campus. We do not whitelist UIUC email today as there
is more likelihood of spam getting through. The average
spam score from Express email is 5.8. Individuals may whitelist
any domain, however, when CITES whitelists a domain an individual
may not override this. So whitelisting is not a good long-term
solution.
4. Sendmail greet_pause -- Last week we stopped over 73,000 hosts,
some thousands of times, by using greet_pause. However, many
large sites (e.g., gmail, hotmail) are not behaving in an
RFC-compliant manner with regards to mail standards so that utilizing
standard anti-spam strategies may delay or even prevent delivery of
legitimate mail.
5. We have tried backup MX servers (ones that just say "try again"),
but these had a bad effect on other campus services such as
Exchange and Listserv.
-Allan
--
-----------------------------------------------------------------
Allan Tuchman Manager, Production Applications Group
tuchman@uiuc.edu Campus Information Technologies and Educational Services
217-244-0048 University of Illinois at Urbana-Champaign