bogofilter+razor filters for kmail spam filtering

[alkan]

I search the forum and internet how to integrate razor into KMail. It looks like the only way is to use a MTA with razor to deliver the mail to a local box and then read the mail from kmail. Since I don't want to run a mail server, I found a workaround so that kmail directly uses razor.

Since razor checks the central database over the internet, hence it is slow, I added bogofilter. KMail filters first check the bogofilter, if the result is negative, only then it tries razor.

Other feature is that razor is used to train the bogofilter, this way you become less and less depended on razor over the time. Statring from an untrained bogofilter database, bogofilter started detecting spam within a few hours.

First we need to install bogofilter, razor, maildrop

[rocketchef]

nice howto, but what if bogofilter considers a mail as ham? Shouldn't it deliver it directly instead of checking razor?
My approach makes use of bogofilters ability to classify mail in tri-state ham, unsure, spam. Only if bogofilter is unsure about the status of the mail it lets razor check for known spam. Additionally my solution adds a whitelist feature for even reduced processing time. I did not manage to configure procmail in the way I wanted it so wrote a little script which grabs the mail from fetchmail and delivers it appropriately.

[alkan]

Thanks for sharing your configiration. It will help further to improve mine.

I will eventually extend it not to double check razor if is marked ham by bogofilter. See, initially, I wasn't sure about the performance of the bogofilter. It's been 3 days since bogofilter training started, currenly bagofilter detects 1/3 of the spams while razor catches remaining 2/3 (eventhought bogofilter says it is a ham for some). But the rate is improving fast for the bogofilter. Until I am satisfied with the bogofilter I will keep double checking with razor since it trains bogofilter.

[rocketchef]

Well, you're using two-state classification with bogofilter, aren't you? I'd suggest using tri-state, so bogofilter can decide if it's ham, spam or whether it is unsure about this. You could stop processing of the mail after bogofilter has made a decent decision about it, but if bogofilter is not sure about the mail you could go on checking with razor.
I don't think bogofilter will mark a spam mail as ham in this way, it is plainly unsure about that, even with a a very small corpus of spam token. If bogofilter is unsure about the mail and razor considers it as spam you can automatically retrain bogofilter. This way you are doing some kind of "train on error" instead of "train on everything" which is supposed to lead to better classification results without bloating your token database.

I am sure you will be satisfied with your combined bogofilter - razor setup, bogofilter does a great job for me, either in terms of speed and in terms of precision. Razor on the other hand had great improvements in the >2.6 versions.

[alkan]

Ok I got your point, I don't want to bloat my bogofilter database.

Do you you know how to get "X-Bogosity: Unsure" header? bogofilter always returns "X-Bogosity:Yes" or "X-Bogosity: No". (sure, I can use the exit code of the bogofilter, but I'd rather do it directly with bogofilter passthrough feature so I can use it in kmail filters)

[rocketchef]

I think it should be sufficient to define ham_cutoff and spam_cutoff in your ~/.bogofilter.cf.
If that does not work review the bogofilter configuration sample in your doc directory. It is well commented and helpful.

[alkan]

thanks, It did the trick.