Big mail solutions. How did you do it?

[fjutt]

Not directly related to gentoo but my guess is that a gentoo-based solution could work so posting it here.

What I want is some material how you did your big mail solutions. What I mean with big is around 50000 users where most will mail pretty much (not sure how much, ill check and see if i can find any numbers).

What problems do we want to solve?

a) Accessability. The users should be able to read and send mail where ever they are.
b) Securtiy. The users mail will be safe. Backuped and so on.
c) Spam and viruses. The user should be able to get spam filetering and virus scanning if he/she wants. Probably most users will use this service.

So, nothing fancy but a stable, fast mail solution that takes care of spam and viruses as well.

What cant be changed? Well, the users are all in an ldap db. So authentification have to be against ldap. Thats really it. No other decisions exist so all ideas are welcome.

Ive thought about this a bit and the the users should be able to read/send mail via a webmail service (sqwebmail, squirrelmail, etc), use stand alone mail clients via ssled imap. And smtp of course got to be usable. Thats about it. No shell access so no mutt/pine.

Would love some ideas/solutions. How did you solve it? What software/hardware did you use. How did you build it? Problems you notices and so on. Any information is very welcome.

Thanks!

[kashani]

This can be accomplished in the price range of $2k-200k. I helped build Netzero's mail system which is 30 machines, 2 hardware load balancers, and 17TB of the backend so some of the issues I mention you may never run into. Like the directory structure of your file system being 200GB.

1. The requirements
a. pop-auth via ldap
b. smtp-auth via ldap
c. imap or pop?
d. web access
e. ssl/tls?
f. spam filtering
h. virus filtering
i. data integrity

2. Number of servers.
Seems odd for #2 on our lust of things to do. Imagine though that each of the services mentioned above will evenutually need their own cluster. This may or may not happen. You might run several services on one cluster. But making the decision NOW to run Redundant Arrays of Inexpensive Servers will save your ass down the line. And make rolling out new servers simpler. Not to mention simplifying maintenance.

3. Data
We went NFS. Other people went SAN. Depends on your budget. NFS will be cheaper and 50k users should push your NFS server too hard. I'd ignore doing any sort of backups and spend my money on a multiheaded NetApp type system. Recovering 5 million mail files onto new disks takes forever. Also depending on throughput go with faster disks vs larger disks. 15k 18GB would be preferable to 10k 36GB drives.

4. Split your services, SPLIT them!
smtp relay is going to take the least resounces. inbound email is going to take the most. pop is somewhere in the middle, but closer to smtp than inbound. Plan accordingly. We liked Raid cards in inbound for disk caching, but didn't use them in the other servers.

5. load balancing
go with hardware load balancing. You can get them fairly cheap these days and linux virtual service stuff while getting better still does not cut the mustard.

6. spam and virus filtering
plan on multiplying the number of inbound servers by 4. That's at least what I've been hearing by people trying it. You might want to do virus first and then slowly rollout spam. Part of the problem is that even the spamd daemon with spam assassisin isn't very effcient yet. I'd do virus filtering on the smtp relay boxes but no spam filtering there.

7. Web mail
You need imap and maildir. I'd suggest using maildir anyway for something this large. Add another cluster for the webservers. Make you pop cluster larger for the imap servers. More space on the NFS server. Now you have to start enforcing some sort of quotas.

The Gentoo Virtual Mail guide is a pretty good starting point for building this sort of system. Postfix or qmail would also work though people have made noises that doing spam filtering in qmail is easier right now. Imap/pop is up in the air, I like courier and I don't recall what got used at Netzero when we went to webmail... uw-imap I think.

If you've got any specific questiosns or want to pick my brain send me a PM.

kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape.

[Supermule]

@kashani:

Just wondering: which OS did u use for that implementation?
_________________
regards,
Supermule