| View previous topic :: View next topic |
| Author |
Message |
tecknojunky
Veteran
Joined: 19 Oct 2002
Posts: 1937
Location: Montréal
|
 Posted: Fri Oct 29, 2004 9:04 am Post subject: SpamAssassin as grown to be excessively hard on ressources |
 |
|
I have a mail setup on a P1 that has 128MB ram and as much swap.
Now a day, when I start spamd, it will eat up all the ram and half the swap. When a mail arrives, it will eat the rest of the swap and machine start to trash, programs randomly crash, etc.
Is there some clean up I can perform?
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax. |
|
| Back to top |
|
 |
tecknojunky
Veteran
Joined: 19 Oct 2002
Posts: 1937
Location: Montréal
|
| Posted: Sat Oct 30, 2004 11:21 am Post subject: |
|
|
I can't run a server in this posture: | Code: | Mem: 127116k total, 123712k used, 3404k free, 4592k buffers
Swap: 133048k total, 87788k used, 45260k free, 5884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9674 root 9 0 163m 96m 2148 S 0.0 77.9 3:14.03 spamd
|
Is this normal, or am I missing something?
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax. |
|
| Back to top |
|
|
tecknojunky
Veteran
Joined: 19 Oct 2002
Posts: 1937
Location: Montréal
|
| Posted: Sat Oct 30, 2004 2:52 pm Post subject: |
|
|
This is rocket science?
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax. |
|
| Back to top |
|
|
zerojay
Veteran
Joined: 09 Aug 2003
Posts: 1033
|
| Posted: Sat Oct 30, 2004 3:13 pm Post subject: |
|
|
| What version of SA are you using? I've heard 3.0.x is a lot better on resources. |
|
| Back to top |
|
|
tecknojunky
Veteran
Joined: 19 Oct 2002
Posts: 1937
Location: Montréal
|
| Posted: Sun Oct 31, 2004 2:33 pm Post subject: |
|
|
| DarkStalker wrote: | | What version of SA are you using? I've heard 3.0.x is a lot better on resources. |
It's pre-3.0. I take your comment as a suggestion to upgrade to the 3.0 release  .
I will, because for the time being, I'm always shuting down SA since it will crash my server sooner than later. So since I'm not running it, I was just about to unmerge it. It did not occured to me to try SA3 because I thought it would be even more of a ressource hog then the previous version.
EDIT:
Vini, vidi, vici. I installed it as if it was a first timer, started it and... | Code: | Mem: 127116k total, 124696k used, 2420k free, 1072k buffers
Swap: 133048k total, 48532k used, 84516k free, 7576k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22664 qmaild 19 0 90168 81m 70m R 37.3 65.7 0:01.19 spamd
22649 root 9 0 89660 66m 66m S 0.0 53.9 3:31.59 spamd
22668 root 9 0 89660 65m 64m S 0.0 52.4 0:00.00 spamd
22667 root 9 0 89660 65m 64m S 0.0 52.4 0:00.02 spamd
22666 root 9 0 89660 65m 64m S 0.0 52.4 0:00.01 spamd
22665 root 9 0 89660 64m 64m S 0.0 52.4 0:00.01 spamd |
Fairly satisfying improvments. But I fear that as SA will learn stuffs, it will grow bigger and bigger and I'll be back at square one.
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax. |
|
| Back to top |
|
|
Casper Gasper
Tux's lil' helper
Joined: 07 Sep 2004
Posts: 75
Location: London
|
| Posted: Sun Oct 31, 2004 3:23 pm Post subject: |
|
|
Are you rebuilding your bayesian db? For heavy loads I think it's suggested that you do this as a cron job. I've found the sa mailing list to be very helpful if you ask a question.
Casper. |
|
| Back to top |
|
|
hughescr
n00b
Joined: 26 Apr 2004
Posts: 1
Location: Portola Valley, CA
|
| Posted: Sun Oct 31, 2004 7:50 pm Post subject: |
|
|
| On a machine with little free memory, you can also just run /usr/bin/spamassassin instead of spamd/spamc -- then the spamassassin stuff will only load to process the message, then exit. No resident giant spamd process. |
|
| Back to top |
|
|
georwell
Guru
Joined: 25 Jun 2003
Posts: 430
Location: Uppsala, Sweden
|
| Posted: Sun Oct 31, 2004 9:14 pm Post subject: |
|
|
| Maybe you should try mailscanner. It allows you to scan mail in batches so it isn't as resource intensive for so long... |
|
| Back to top |
|
|
Aleksejs
n00b
Joined: 26 Oct 2004
Posts: 7
Location: Imanta, Riga, Latvia
|
| Posted: Sun Oct 31, 2004 11:43 pm Post subject: |
|
|
| Casper Gasper wrote: | Are you rebuilding your bayesian db? For heavy loads I think it's suggested that you do this as a cron job. I've found the sa mailing list to be very helpful if you ask a question.
Casper. |
Could you elaborate on that, please? |
|
| Back to top |
|
|
j2brown
n00b
Joined: 25 Jun 2003
Posts: 12
|
| Posted: Mon Nov 01, 2004 2:11 pm Post subject: Rulesets |
|
|
Are you using any custom rulesets? A recent upgrade to bigevil brought my server to it's knees.
I'm looking to upgrade to SA3 today (which is what brought me to your post).
I'll try and remember to post some results here. Initial research looks good, though.
jeff
sdg |
|
| Back to top |
|
|
j2brown
n00b
Joined: 25 Jun 2003
Posts: 12
|
| Posted: Mon Nov 01, 2004 8:53 pm Post subject: Followup to SA3.0.1 install |
|
|
It's only been up for about 5 hours, but it seems to be working like a champ on my setup:
P4, 2.6
1G RAM
Just shy of 10K message per day
jeff
sdg |
|
| Back to top |
|
|
robfantini
Tux's lil' helper
Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts
|
| Posted: Sun Nov 07, 2004 9:12 pm Post subject: Re: SpamAssassin as grown to be excessively hard on ressourc |
|
|
Running Spamassassin I ran into memory and cpu problems, and I;ve got 1-gb ram with a Xeon 3.06GHz cpu. this was the output from 'top:'
from top:
| Code: |
top - 15:52:44 up 5:21, 14 users, load average: 37.44, 29.94, 16.09
Tasks: 301 total, 41 running, 250 sleeping, 0 stopped, 10 zombie
Cpu(s): 0.0% user, 100.0% system, 0.0% nice, 0.0% idle
Mem: 1032060k total, 1020532k used, 11528k free, 3772k buffers
Swap: 987988k total, 987988k used, 0k free, 4972k cached
|
To eliminate the problems I did the following:
1- /etc/conf.d/cpamd :
SPAMD_OPTS=" -m 2 -a -L -d "
-L local tests only
-m 2 makes it so only 2mails are checked at one time. with your cpu + mem i'd use -m 1
2- /etc/mail/spamassassin/local.cf
skip_rbl_checks 1
3- I use postfix. I setup postfix to do the rbl checks. rbl checks work better from postfix as
1 check is done per an incoming mail that is destined for multiple recipients.
Spamassassin does the rbl checks once per recipient.
This is from our /etc/postfix/main.cf:
| Code: |
smtpd_recipient_restrictions =
reject_invalid_hostname,
reject_non_fqdn_hostname,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
permit_mynetworks,
reject_unauth_destination,
reject_rbl_client relays.ordb.org,
reject_rbl_client opm.blitzed.org,
reject_rbl_client list.dsbl.org,
reject_rbl_client sbl.spamhaus.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client dul.dnsbl.sorbs.net,
permit
|
Also there is an update to Spamassassin which is supposed to fix memory usage problems. see this: http://article.gmane.org/gmane.mail.spam.spamassassin.general/58379 |
|
| Back to top |
|
|
yassen
Apprentice
Joined: 26 Mar 2004
Posts: 194
|
| Posted: Thu Nov 18, 2004 2:37 pm Post subject: |
|
|
hughescr, anybody else:
| Quote: | | On a machine with little free memory, you can also just run /usr/bin/spamassassin instead of spamd/spamc -- then the spamassassin stuff will only load to process the message, then exit. No resident giant spamd process. |
Sounds nice. Does anybody run it that way with postfix? If so, any config files here for a hint? Thanks in advance!
yassen
_________________
:: Adopt an Unanswered Post :: |
|
| Back to top |
|
|
robfantini
Tux's lil' helper
Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts
|
| Posted: Fri Nov 19, 2004 2:57 am Post subject: |
|
|
We do online spam checks thru postfix.
Here is our /etc/postfix/main.cf file. Note the anti UCE settings section.
| Code: |
## IMPORTANT: /etc/procmailrc sets home_mailbox
# an attempt to allow sending from my home backup sever:
relay_domains = $mydestination, /etc/postfix/relay-domains
# i made change during spamassassin install
# ------------------------------------------ get info here:
# man 5 postconf # man page for this file
# postconf -n >> /root/postconf.txt # list non default settings
# http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt
# http://www.geocities.com/scottlhenderson/spamfilter.html
# ------------------------------------------ check and reload:
# postfix check # to check this file
# postfix reload # to use new settings
#------------------------------------------------------------
queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
mail_owner = postfix
myhostname = my.hostname.com
# ** ALWAYS UNCOMMENT ON MAIN SERVER **
myorigin = $mydomain
## ---------------------------------------------------------------
# fbc:
inet_interfaces = all
# fbc:
mydestination = $myhostname, localhost.$mydomain, $mydomain, mail.$mydomain, www.$mydomain, ftp.$mydomain
# 08-31-04 was 450. 450 = try again, 550 = reject
unknown_local_recipient_reject_code = 550
#mail_spool_directory = /var/spool/mail
mailbox_command = /usr/bin/procmail -a "$EXTENSION"
debug_peer_level = 2
sendmail_path = /usr/sbin/sendmail
newaliases_path = /usr/bin/newaliases
mailq_path = /usr/bin/mailq
setgid_group = postdrop
manpage_directory = /usr/share/man
readme_directory = /usr/share/doc/postfix-2.1.3/readme
# next 2 lines were 2 . change to 1 to make log reading easier:
default_destination_concurrency_limit = 1
local_destination_concurrency_limit = 1
alias_database = hash:/etc/mail/aliases
alias_maps = hash:/etc/mail/aliases
#------------------------------------------------------------------------
# anti UCE settings
# http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt
# http://www.geocities.com/scottlhenderson/spamfilter.html
smtpd_helo_required = yes
disable_vrfy_command = yes
# 2004-09-04. added check_sender_access to allow fbc6 mail in from home.
# 2004-11-09 elim reject_rbl_client dul.dnsbl.sorbs.net, as I was not
# able to send mail from home using Thunderbird
# so deleted: reject_rbl_client dul.dnsbl.sorbs.net,
#
# 2004-11-17: ##from postfix-anti-UCE.txt: moved permit_mynetworks up.
# also try readding reject_rbl_client dul.dnsbl.sorbs.net,
smtpd_recipient_restrictions =
permit_mynetworks,
check_sender_access hash:/etc/postfix/sender_access,
reject_invalid_hostname,
reject_non_fqdn_hostname,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
reject_unauth_destination,
reject_rbl_client relays.ordb.org,
reject_rbl_client opm.blitzed.org,
reject_rbl_client list.dsbl.org,
reject_rbl_client sbl.spamhaus.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client dul.dnsbl.sorbs.net,
permit
smtpd_data_restrictions = reject_unauth_pipelining,
permit
# ----------------------------------------------------- end anti UCE settings
|
Check the links that are listed in the above file.
/etc/procmailrc:
| Code: |
# /etc/procmailrc
# if a user has a .forward file in $HOME, this will not be used.
# hamtrap and spamtrap have a .forward file in order to use ~.procmailrc
# use a system wide log file. keep perms ok using cron script
LOGFILE="/var/log/mail/procmail.log"
VERBOSE=off
# 03-25-2004 this gets mail to imap .maildir
# i used to use .forward in $HOME.
# can;t do that as .procmailrc won't be used.
# .procmailrc needed for SpamAssassin
# do not set mail target anywhere else. not postfix/main.cf not ~.procmailrc ....
DEFAULT=$HOME/.maildir/
MAILDIR=$HOME/.maildir/
#DROPPRIVS=yes forces the rest of the procmailrc to be processed with the
#recipient's default privileges, and is necessary to allow SpamAssassin to
#process individual user preference files. If you are already using
#procmail to handle incoming mail system-wide, make sure any processing
#which must be done as a privileged user (some anti-virus setups
#fall into this category) appears in the system procmailrc
#before DROPPRIVS=yes and SpamAssassin filtering.
DROPPRIVS=yes
# If none of the filters match, it will go to your inbox.
# also see /etc/mail/spamassassin/local.cf
# The condition line ensures that only messages smaller than 250 kB
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam
# isn't bigger than a few k and working with big messages can bring
# SpamAssassin to its knees.
#
# 06-02-04 added. see man formail and
# http://www.ii.com/internet/robots/procmail/qs/#SA
# 08-17-04 was 256000.. uped to 512000
# 08-31-04 down to 100000 ... spamd used all resourses...
# 2004-09-07 backup to 256000. we test just 1 or 2 msgs at a time.
:0 fhw
* > 256000
* ^Subject:\/.*
| formail -I "Subject: {* -BIG- not checked for spam. be careful..*} $MATCH"
# http://www.ii.com/internet/robots/procmail/qs/#SA
# http://article.gmane.org/gmane.mail.procmail/9833
# # ELSE if the previous recipe's conditions did not match (i.e. if the message
# is smaller than or equal to 250 KB), pipe it through spamassassin. The lock
# file ensures that this procmailrc invokes only one instance of spamassassin
# at a time. This is to try to prevent spamassassin from using too much memory.
# If you pipe messages through spamc, you probably do not need to use a lockfile
# and you can remove the string : spamassassin.lock below
:0 E fw
| spamc
# added 06-01-04 see http://wiki.apache.org/spamassassin/UsedViaProcmail
# The next stanza protects against errors. If spamassassin returns an error
# it is passed back to the mail transfer agent and
# improves robustness of mail delivery.
:0e
{
EXITCODE=$?
}
# 100 + hits:
# http://www.ii.com/internet/robots/procmail/qs/#SA
:0
* ^X-spam-status: Yes, hits=[1-9][0-9][0-9]
/dev/null
# 40-99 to /dev/null added 07-05-04.
:0
* ^X-spam-status: Yes, hits=[4-9][0-9]
/dev/null
# 20-99 hits:
# 06-18-04, was 30.
# a cronjob checks this mail and rpeorts to Kenia mon-fri.
# the mail is kept 2 days..
# 2004-11-20 commented out:
#:0
#* ^X-spam-status: Yes, hits=[2-9][0-9]
#/bkup/SpamIn/.--30Plus/
# 2004-11-20 added
# see http://www.ii.com/internet/robots/procmail/qs/#SA
# If the spam score is between 10.00 and 48.99, put it in my --2delete
# mailbox. Once I am assured that only spam is delivered to this box,
# I will have procmail delete it (discussed below)
:0
* ^X-Spam-Level: RRRRRRRRRR
.--2delete/
# If the spam score is between 5.00 and 9.99, put it in my --red box.
:0
* ^X-Spam-Level: RRRRR
.--red/
# All messages that get here are put in my --yellow box. This
# includes messages with a spam score of 2-4.99 or less.
:0
* ^X-Spam-Level: RR
.--yellow/
|
|
|
| Back to top |
|
|
Jimmy Jazz
Guru
Joined: 04 Oct 2004
Posts: 333
Location: Strasbourg
|
| Posted: Sat Nov 27, 2004 8:30 pm Post subject: Re: SpamAssassin as grown to be excessively hard on ressourc |
|
|
| tecknojunky wrote: | I have a mail setup on a P1 that has 128MB ram and as much swap.
Now a day, when I start spamd, it will eat up all the ram and half the swap. When a mail arrives, it will eat the rest of the swap and machine start to trash, programs randomly crash, etc.
Is there some clean up I can perform? |
Hello,
i think you have fell /etc/mail/spamassassin with huge ruleset files  . Try to eliminate some of them and see if spamd becomes less hungry.
Jj |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
