Published 2004-09-18 21:38:12

Defer unknowns has proved very successful, with success rates of more than 1000 spams per day removed from my email box.

My original code parsed the exim log, and built black,white and greylists by selecting checkboxes ona web page. It was effective, but involved a little maintenaince. (daily review, and mostly blacklisting IP addresses.)

By looking at the general pattern of this, it became clear, that most spammers fire off a large number of hijacked PC's and just run through a big email list. If it fails (eg. defer), they just give up on that machine, and pass it along to the next (often changing the signature). - It's a known trick to do defer greylisting on this. Basically first time that ip contacts you, you respond, defer, try later.. next time, you let it through. (In my new scheme, I only black/grey/white list the ones that tried more than once - which should significantly reduce the amount of maintenance, and makes spotting good IP addresses alot easier.

The other beauty of the new solution is that it doesnt involve parsing logs anymore, it's almost a pure exim/mysql solutions, with my manual categorizing a considerably simpler web page.

The exim config I'm using is available in the extended entry.
or have a look at the simple spam manager interface

I guess if you want to run this on a bigger site, you might want to go to the mysql conference where you can find out reall answers form mysql developers, and experts. (and if you go to the php conference at the same place/time, you can see me talking about php5 and pear.)

# prune logs. - delete records from trylog that are older than 5 days.
warn condition = ${lookup mysql{delete from email_trylog where try < DATE_SUB(NOW(), INTERVAL 5 DAY) }{no}{no}}
log_message = "Something impossible happened";

# totally new ip address (based on iplog)
#- insert into email_trylog
#- insert into email_iplog
#- defer.

#no record what so ever of the IP address:
defer condition = ${lookup mysql{select count(id) from email_iplog \
where ip='${sender_host_address}' having count(id) > 0 \
}{no}{yes}}
condition = ${lookup mysql{insert into email_trylog (ip,title,sender,host,try) values ( \
'${sender_host_address}',SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
'${quote_mysql:$sender_address}', '${quote_mysql:$sender_helo_name}',NOW()) \
}{yes}{yes}}
condition = ${lookup mysql{insert into email_iplog (ip,title,sender,host,try) values ( \
'${sender_host_address}',SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
'${quote_mysql:$sender_address}', '${quote_mysql:$sender_helo_name}',NOW()) \
}{yes}{yes}}
message = "Delivery defered while we check the IP address (new IP)"

#blacklisted.
deny condition = ${lookup mysql{select id from email_iplog where ip='${sender_host_address}' and blacklist > 0}{yes}{no}}
condition = ${lookup mysql{update email_iplog set \
title=SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
sender='${quote_mysql:$sender_address}', \
host='${quote_mysql:$sender_helo_name}', \
try=NOW(), \
efforts = efforts + 1 \
WHERE ip='${sender_host_address}' \
}{yes}{yes}}
condition = ${lookup mysql{insert into email_trylog (ip,title,sender,host,try) values (\
'${sender_host_address}',SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
'${quote_mysql:$sender_address}', '${quote_mysql:$sender_helo_name}',NOW()) \
}{yes}{yes}}
message = "Your IP {$sender_host_address} was blacklisted, please email this message to akbkhome@the-gmail.com to get unlisted"


#not approved or looked at yet..

defer condition = ${lookup mysql{select id from email_iplog where ip='${sender_host_address}' and whitelist < 1 and greylist < 1}{yes}{no}}
condition = ${lookup mysql{update email_iplog set \
title=SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
sender='${quote_mysql:$sender_address}', \
host='${quote_mysql:$sender_helo_name}', \
try=NOW(), \
efforts = efforts + 1 \
WHERE ip='${sender_host_address}' \
}{yes}{yes}}
message = "Delivery defered while we check the IP address (more that 1 effort)"


warn condition = ${lookup mysql{insert into email_trylog (ip,title,sender,host,try) values (\
'${sender_host_address}',SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
'${quote_mysql:$sender_address}', '${quote_mysql:$sender_helo_name}',NOW()) \
}{no}{no}}
log_message = "updating trylog failed.?"

warn condition = ${lookup mysql{update email_iplog set \
title=SUBSTRING('${quote_mysql:$h_subject:}',1,255), \
sender='${quote_mysql:$sender_address}', \
host='${quote_mysql:$sender_helo_name}', \
try=NOW(), \
efforts = efforts + 1 \
sucess = success + 1 \
WHERE ip='${sender_host_address}' \
}{no}{no}}
log_message = "updating iplog failed.?"



This code is mainly concerned with blocking and only allowing approved IP addresses, another site, which doesnt have anyone to maintain it, just takes advantage of the blocking by letting through ip's which have tried a few times.
defer condition  = ${lookup mysql{select count(email_trylog.id) from \
email_trylog LEFT JOIN email_iplog on email_trylog.ip = email_iplog.ip where email_trylog.ip='${sender_host_address}' AND \
email_trylog.title=SUBSTRING('${quote_mysql:$h_subject:}',1,255) AND \
email_trylog.sender='${quote_mysql:$sender_address}' AND \
email_trylog.host='${quote_mysql:$sender_helo_name}' AND \
email_trylog.try > DATE_SUB(NOW(), INTERVAL 2 DAY) AND \
email_iplog.whitelist = 0 AND \
email_iplog.success < 12 \
HAVING count(email_trylog.id) < 1}{yes}{no}}
condition = ${lookup mysql{insert into email_trylog (ip,title,sender,host,try) values ('${sender_host_address}',SUBSTRING('${quote_mysql:$h_subject:}',1,255), '${quote_mysql:$sender_address}', '${quote_mysql:$sender_helo_name}',NOW()) }{yes}{yes}}
message = "Delivery defered while we check the IP address (new message)"
The database tables are pretty simple:
CREATE TABLE `email_trylog` (
`id` int(11) NOT NULL auto_increment,
`ip` varchar(16) default '',
`title` varchar(255) default '',
`sender` varchar(255) default '',
`host` varchar(255) default '',
`try` datetime default NULL,
PRIMARY KEY (`id`)
) TYPE=MyISAM

CREATE TABLE `email_iplog` (
`id` int(11) NOT NULL auto_increment,
`ip` varchar(16) default '',
`title` varchar(255) default '',
`sender` varchar(255) default '',
`host` varchar(255) default '',
`try` datetime default NULL,
`blacklist` int(11) default '0',
`whitelist` int(11) default '0',
`greylist` int(11) default '0',
`efforts` int(11) default '1',
`success` int(11) default '0',
PRIMARY KEY (`id`)
) TYPE=MyISAM
Mentioned By:
cpanelskindepot.com : cpanelskindepot.com :: View topic - Exim log frontend. [ Guest ] (108 referals)
google.com : april (76 referals)
google.com : december (44 referals)
google.com : php spam (28 referals)
cpanelskindepot.com : cpanelskindepot.com :: View topic - Exim log frontend. (19 referals)
cpanelskindepot.com : cpanelskindepot.com :: View topic - Exim log frontend. [ ] (18 referals)
google.com : php date_sub (15 referals)
google.com : "condition check lookup defer" (12 referals)
google.com : date_sub php (12 referals)
google.com : php remove substring (10 referals)
google.com : php delete substring (8 referals)
google.com : exim condition check lookup defer (6 referals)
google.com : mysql update substring (6 referals)
google.com : sender_host_address (6 referals)
google.com : condition check lookup defer (5 referals)
google.com : delete substring php (5 referals)
google.com : exim blacklist (5 referals)
google.com : mysql insert condition (5 referals)
google.com : mysql insert where condition (5 referals)
cpanelskindepot.com : cpanelskindepot.com &bull; View topic - Exim log frontend. (5 referals)

Add Your Comment

Follow us on