|
News About Features Documentation FAQ ChangeLog Mailinglist Download GIT Screenshots RRDtool MRTG WebQ Logwatch Reporter Latest stable: 2.0.2-1 Latest devel: 2.0.3-0.beta5
|
SAGATOR
(c) 2003-2021 Jan ONDREJ (SAL) <ondrejj(at)salstar.sk>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
Contents
1. Introduction
1.1. Features
1.2. Requirements
1.3. Benchmark
2. Installation
2.1. Getting started
2.2. Creating chrooted environment
2.3. Upgrading from previous versions
3. Sagator configuration
3.1. Service configuration
3.2. Scanner configuration
4. Detailed scanner configuration
5. SMTPd configuration
5.1. Postfix configuration
5.2. Sendmail configuration
5.3. Another smtpd configuration
6. Antivir/Antispam installation
6.1. ClamAV installation
7. Staring sagator
8. Testing
9. Scanning mailboxes
10. Retunrning badly quarantined mail to mailq/user mailbox
11. Removing old files from quarantine
12. Reporting summaries to users
13. Graphs
13.1. RRDtool
13.2. MRTG
14. Web quarantine
15. Done
1. Introduction
This program is an email/web antivirus/antispam gateway. It is an interface
to the postfix or sendmail (or any other smtpd), which runs antivirus
and/or spamchecker. It also can be used as http proxy with
filtering abilities. It's modular architecture can use any
combination of antivirus/spamchecker according to configuration.
It currently supports clamav, nod32d, AVGlinux, sophie, trophie,
symantec antivirus scan engine (via ICAP), Bitdefender,
spamassassin, bogofilter and QuickSpamFilter.
It has some internal checkers (string_scanner and regexp_scanner).
Sagator can parse MIME mails and decompress archives, if it is configured so.
---------------------------------------------------------------------
1.1. Features
- simple chroot support
- modular antivirus/spamchecker support
- currently supports clamav, nod, sophie, trophie, AVGlinux,
ICAP protocol (Symantec Antivirus), ...
- any other antivirus/antispam can be implemented
- spamaassassin support
- database support
- SQL logging
- dynamic scanner (antivirus/antispam) configuration
- nice RRDtool or MRTG statistics
- daily reports for users
- web quarantine accessible for all users
- smtp policy service (greylist)
- mailbox/maildir scanning and cleaning
- easy installation and configuration
- SMTP and LMTP protocol support
- LMTP can be used to use different scanners for each recipient
- you can use simple scanners to determine files which you don't need:
- attach_name - define a virus by mime extension
- file_type - define a virus by file type (content is checked)
- max_file_size - define max. attachment or max. file size
- string_scan - define a virus by string
- regexp_scan - define a virus by regular expression
- you don't need any perl modules or any other modules, only python
- you can return any quarantined mail to mailq/user mailbox
---------------------------------------------------------------------
1.2. Requirements
- python (python 2.6 is required for SAGATOR2 due to python 3.x support)
- an supported antivir/antispam (clamav, spamassassin, ...)
- SMTP daemon (one from following list)
- postfix 1.1.11 (or may be later or newer)
- sendmail with milter and python milters
- any other SMTPd with special configuration.
---------------------------------------------------------------------
1.3. Benchmark
Hardware & Software:
Athlon 2000+,
Fedora Core 2
clamav 0.80rc4 (used as libclam)
Pattern:
90000 emails processed in 312 minutes
speed ~= 4.8 emails/second
---------------------------------------------------------------------
2. Installation
You can skip this part if you are using rpm or deb packages.
They can be downloaded from SAGATOR's homepage or via a package
installer (yum, apt-get or similiar).
2.1. Getting started
Unpack the package. Go into main directory and run:
./configure
make
make install
2.2. Creating chrooted environment.
Create user vscan:
useradd vscan -M -s /sbin/nologin
Use script:
./scripts/mkchroot.sh /some/where/new/root/
The directory can be for example: "/var/spool/vscan".
This script creates a small copy of your files, which are required
for antivirus, unpacker, ...
2.3. Upgrading from previous versions
Upgrade information is available only for sagator 0.5.0 and higher.
After upgrade run "updatecfg.py" script from your installation directory.
This script updates your configuration files automatically. Not all changes
can be fixed automatically, but many of them can.
- upgrading to SAGATOR 1.2.0
Recheck, if you are using an obsolete scanner and remove/replace them
with newer ones. For web-quarantine users, please install python-genshi,
if your distribution does not do this for you.
- upgrading to SAGATOR 1.0.0
Please check, if you are using an obsolete scanner or service. All
obsolete scanners/services have been removed. If your configuration has
been not fixed automatically by updatecfg script, do it manually.
- upgrading to SAGATOR 0.8.0
There are many of changes in scanner names. For detailed description see
ChangeLog file, or run updatecfg.py script.
You need to move '/tmp/status' to '/var/lib/sagator/status' in your CHROOT
directory, or define '/tmp/status' manually in collector() service.
Also RRD template has been changed. There is no way to update your
sagator.rrd file now. You must remove or rename this file and a new rrd
file will be created automatically.
- upgrading to SAGATOR 0.5.0
In this version you need to add a SRV array in sagator's configuration.
If you are using COLLECTOR_SERVER and BIND_ADDR variables, you need to add:
SRV=[
collector(COLLECTOR_SERVER[0],COLLECTOR_SERVER[1]),
smtpd(SCANNERS,BIND_ADDR[0],BIND_ADDR[1])
]
3. Sagator configuration
3.1. Service configuration
Services are subprograms to communicate with an SMTP daemon or with other
client. For more help see Services.txt. Use SRV variable in config.py.
SRV varialble is a python array, defined as:
SRV=[ service1, service2, ... ]
Services are defined in separate files, you must include it into
config.py. You can do it with this command:
from srv import *
Examples:
a) smtpd service (good for postfix and useable for any smtpd) with
combination for collector (for statistics):
SRV=[
collector('0.0.0.0',28),
smtpd(SCANNERS,'127.0.0.1',27)
]
b) lmtpd service (if you want different scanners for each recipient)
SRV=[
collector('0.0.0.0', 28),
lmtpd(LMTP_SCANNERS, '0.0.0.0', 27)
]
c) milter service (for sendmail) with combination with statistics
collector:
SRV=[
collector('0.0.0.0',28),
milter(SCANNERS,"sagator","inet:3333@127.0.0.1")
]
d) http sanning service:
SRV=[
collector('0.0.0.0',28),
http_proxy(SCANNERS,'0.0.0.0',3128)
]
3.2. Scanner configuration
Edit config.py. There are some parameters, described in config.py file.
Scanner configuration:
Use SCANNERS variable in config.py. You can use more scanners.
If one of them found a virus, scanning is stopped. If one of them
fails, an error is raised.
SCANNERS variable is a python array, defined as:
SCANNERS=[ scanner1, scanner2, ... ]
Scanners are defined in separate files, you must include it into
config.py. You can do it with:
from scanners import *
Some examples:
a) Use only clamscan binary to scan for viruses:
SCANNERS=[
clamav(['/usr/bin/clamscan','--stdout','--infected',
'--disable-summary','-r','--mbox'])
]
b) Adding spamassassin tester:
SCANNERS=SCANNERS+[ spamassassind(['localhost',783]) ]
c) If you want to add some special stuff, like scan for
names of attachments, and if you don't want any attachments,
use mime_ext scanner (YOU NEED PYTHON 2.2 FOR THIS@):
EXECUTABLES='(\.exe|\.COM|\.com|\.COM|\.pif|\.PIF|\.lnk|\.LNK|\.scr|\.SCR)'
SCANNERS=[ mime_ext(EXECUTABLES) ]
There are also subscanners ... they are scanners, which parses emails
and runs another scanner. For example there are parsemail and
decompress scanners.
Example for parsemail:
I wan't to test parsed files by clamd. Clamd can parse mail,
but I don't want to use its scanner now.
parsemail(clamd(['localhost',3310]))
4. Detailed scanner configuration
There are three types of scanners:
- bufferscanner
- filescanner
- combination (filescanner+bufferscanner)
Scanners are also divided into:
- realscanners
These scanners are real scanners, which scans the buffer/file
and then returns its status.
- interscanners
These scanners are an interface between main scanning engine
and realscanners. You also can attach interscanner to another
interscanner.
By default bufferscanner is used. If you wan't to use filescanner,
there is a interscanner named buffer2file().
If you need to know, which scanner is which, look at this table:
------------------------------------------------------------------
| name | module | inter | buffer | file |
------------------------------------------------------------------
| report() | report | * | * | - |
| report_recipients() | report | * | * | - |
| quarantine() | actions | * | * | - |
| drop() | actions | * | * | - |
| deliver() | actions | * | * | - |
| deliver_to() | actions | * | * | - |
| rename() | actions | * | * | - |
| time_limit() | actions | * | * | - |
| modify_subject() | header | * | * | - |
| modify_header() | header | * | * | - |
| add_header() | header | * | * | - |
| remove_headers() | header | - | * | - |
| file2buffer() | file | * | -/out | in/- |
| buffer2file() | file | * | in(m) | -/out |
| buffer2mbox() | file | * | in/- | -/out |
| match_all() | match | * | in/out | - |
| alternatives() | match | * | in/out | - |
| match_any() | match | * | in/out | - |
| nothing() | match | * | in/out | - |
| recover() | match | * | in/out | - |
| parsemail() | parsemail | * | in/out | - |
| attach_name() | parsemail | - | * | - |
| mimeparse() | mimeparse | * | in/out | - |
| mime_ext() | mimeparse | - | * | - |
| decompress() | decompress | * | - | in/out |
| cache() | cache | * | in/out | - |
| log() | logger | * | in/out | - |
| log_syslog() | logger | * | in/out | - |
| log_sql() | logger | * | in/out | - |
| sql_find() | condition | * | in/out | - |
| regexp_find() | condition | * | in/out | - |
| check_level() | condition | * | in/out | - |
| stat() | stats | * | in/out | - |
------------------------------------------------------------------
| clamscan() | clamav | - | - | * |
| clamd() | clamav | - | * | - |
| libclam() | clamav | - | - | * |
| avgd() | avg | - | - | * |
| bdc() | bdc | - | - | * |
| kav() | kav | - | - | * |
| kavclient() | kav | - | - | * |
| nod2() | nod | - | - | * |
| nod2dazuko() | nod | - | - | * |
| nod2pac() | nod | - | - | * |
| esets() | nod | - | - | * |
| dazuko() | dazuko | - | - | * |
| savse() | icap | - | - | * |
| sophie() | sophie | - | - | * |
| trophie() | sophie | - | - | * |
| const() | basic | - | * | * |
| smtp_comm() | basic | - | * | * |
| regexp_scan() | basic | - | * | - |
| string_scan() | basic | - | * | - |
| sender_regexp() | basic | - | * | - |
| max_file_size() | basic | - | * | * |
| file_type() | basic | - | * | * |
| file_magic() | basic | - | * | * |
| sanitize() | sanitize | - | * | - |
| cmd() | cmd | - | - | * |
| cmd_bdc() | cmd | - | - | * |
| cmd_clamav() | cmd | - | - | * |
| cmd_drweb() | cmd | - | - | * |
| cmd_fprot() | cmd | - | - | * |
| cmd_kavscanner() | cmd | - | - | * |
| cmd_trendmicro() | cmd | - | - | * |
| cmd_uvscan() | cmd | - | - | * |
| cmd_vbuster() | cmd | - | - | * |
| filter() | filter | - | * | - |
------------------------------------------------------------------
| spamassassin() | spamassassin | - | * | - |
| spamassassind() | spamassassin | - | * | - |
| bogofilter() | bogofilter | - | * | - |
| qsf() | qsf | - | * | - |
| dspam() | dspam | - | * | - |
------------------------------------------------------------------
For detailed scanner description see Scanners.txt.
There are some opearators, which you can use over scanners:
------------------------------------------------------------
| operator | "+" | "-" | "*" | "/" | "|" | "&" |
| S1 | S2 | | | | | | |
------------------------------------------------------------
| OK OK | L1+L2 L1-L2 L1*L2 L1/L2 L1 (1) |
| F OK | Fail Fail Fail Fail L2 Fail |
| OK F | Fail Fail Fail Fail L1 Fail |
| F F | Fail Fail Fail Fail Fail Fail |
------------------------------------------------------------
------------------------------------------------------------
| operator | "<" | ">" | "=" | "<=" | ">=" | "!=" |
| S1 | S2 | | | | | | |
------------------------------------------------------------
| L1 < L2 | 1.0 | 0.0 * | 0.0 * | 1.0 | 0.0 * | 1.0 |
| L1 = L2 | 0.0 * | 0.0 * | 1.0 | 1.0 | 1.0 | 0.0 * |
| L1 > L2 | 0.0 * | 1.0 | 0.0 * | 0.0 * | 1.0 | 1.0 |
------------------------------------------------------------
S1 - status of first scanner
S2 - status of secondary scanner
L1 - returned level of first scanner (on left side of operator)
L2 - returned level of second scanner (on right side of operator)
NA - not available, can't occur
(1) - if L1>=1.0: L1*L2; else 0.0
* - virus name is cleaned (set to empty string "")
When you are using interscanners, you can't attach scanners of
wrong type. This means, that you can't attach a scanner with
input of buffer to scanner, which has only file output.
For example you can use:
parsemail(string_scan(...))
but can't:
parsemail(decompress(...))
5. SMTPd configuration
5.1. Postfix configuration
# /etc/postfix/main.cf
mynetworks = 127.0.0.0/8, ...
content_filter = smtp:[127.0.0.1]:27
# for lmtpd() service use: content_filter = lmtp:[127.0.0.1]:27
# /etc/postfix/master.cf
# Local smtpd without scanning
127.0.0.1:26 inet n - n - 30 smtpd
-o content_filter=
-o myhostname=sagator.mydomain.sk
-o local_recipient_maps= -o relay_recipient_maps=
-o mynetworks=127.0.0.0/8 -o mynetworks_style=host
-o smtpd_restriction_classes= -o smtpd_client_restrictions=
-o smtpd_helo_restrictions= -o smtpd_sender_restrictions=
-o smtpd_data_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
-o smtpd_use_tls=no
Myhostname can't be equal with your hostname, otherwise postfix
detects a loop.
Add sagator to startup scripts. In this package there is
a sagator.init scipt, which can be used in /etc/init.d/ .
You can start it also manually.
5.2. Sendmail configuration
As first step, please install required packages:
sendmail with milter support
python milters (http://www.bmsi.com/python/milter.html)
Configure your sendmail to use sagator's milter. Add these line into
/etc/mail/sendmail.cf:
O InputMailFilters=sagator
Xsagator, S=inet:3333@127.0.0.1, F=T
For more information see milter documentation.
Configure sagator to run as milter filter. Add this service to SRV in
sagator.conf:
milter("sagator","inet:3333@127.0.0.1")
Limitations:
Milter can't modify Subject and other headers, only can add some new.
5.3 Another smtpd configuration.
For other smtpd, you can use sagator as main smtpd on port 25.
Configure your smtpd on another port as 25. Then configure
sagator to run on port 25 on your IP.
For example:
Configure smtpd on port 26 on localhost and on your internal ip on port 25.
Allow relaying from your site on internal IP, but don't forget to deny
relaying on localhost:26 or from 127.0.0.0/8.
Use following configuration for sagator:
SMTP_SERVER=('127.0.0.1',26)
SRV = [ smtpd('YourExternalIP',25) ]
6. Antivir/Antispam installation
6.1. ClamAV installation
Download and install clamav - clam antivirus.
URL: http://www.clamav.net/. You will also need some
decompressors, like unrar, unace, unarj, zoo, lha, tar, ...
If you are using clamd, do not forget to start it in chroot,
and set these parameters in chrooted clamav.conf:
DatabaseDirectory /var/lib/clamav
LocalSocket /var/run/clamav/clamd.sock
StreamSaveToDisk # recomended
ScanMail # required
ScanArchive # recomended
7. Staring sagator
There are init scripts in scripts/ directory named sagator.init.rhfc or
sagator.init.suse. Copy one of them into your init.d directory and set it.
For RedHat based system do:
cp scripts/sagator.init.rhfc /etc/init.d/sagator
Then edit file /etc/sysconfig/sagator:
CHROOT="/var/spool/vscan"
Warning: You must have your antivirus installed in chroot.
8. Testing
There some test scripts. Go into test directory and try:
./download_viruses.sh
./smtptest.py -h localhost -s tmp/Eicar
./smtptest.py -h localhost -s spam
...
With these scripts you can also test your sagator daemon,
and you don't need to configure content_filter in postfix.
You can test it without real traffic. I recomend for you to
start sagator as command line program, and see its reports first.
Only if it works really, then use it as daemon.
9. Scanning mailboxes
In some case an virus can pass over sagator (for example, if your
antivirus is not actual). If you detect this, you need to find viruses
in user mailboxes. There is an program named sgscan. If scans over
mailboxes and return finded viruses/spams. You can use:
sgscan /var/spool/mail/*
If you can scan only for viruses, you can use --av-only command
line parameter. Another parameter is --clean, if you want to create
clean mailboxes as /var/spool/mail/*.clean.
For renaming these mailboxes to real names use a shell script:
for i in *.clean; do mv $i `echo $i | sed 's/\.clean$//'`; done
Warning: Don't use this script, if an user of an SMTPd has access
to mailbox!!!
10. Retunrning badly quarantined mail to mailq/user mailbox
If by chance is any mail quarantined, which should not, you can return
to queue or deliver to user mailbox by running command:
sqback quarantined_file1 [quarantined_file2 [...]]
This can be done also via netcat program:
nc localhost SMTP_PORT_WITHOUT_FILTER < .../quarantine/filename
11. Removing old files from quarantine
Use tmpwatch. For example, if you want to delete older files,
than aprox. one month, use crontab command in /etc/crontab:
0 0 * * * vscan /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine
or same line for vscan user's crontab:
0 0 * * * /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine
Replace "/var/spool/vscan/tmp/quarantine" with path to quarantine directory.
For more information read "man tmpwatch" manual page.
12. Reporting summaries to users
Many of emails are simply dropped or rejected and users are not
informed about these actions. If you want send summaries periodically,
add a special log() scanner into your configuration and run
reporter script from cron. Here is an example of SCANNERS configuration:
SCANNERS=[
log(1,log.SUMMARY_REPORT,
... your scanner configuraion follows here ...
)
]
And also add something similiar into /etc/crontab:
0 5 * * * vscan /usr/share/sagator/reporter.py
You can define your own custom report. Do this in reporter() service.
For more information see reporter() service documentation.
Example (all parameters are optional, something similiar is default):
reporter(
begin='''\
Subject: Report mail
This is a report of your undelivered emails.
''',
body='''
$DATETIME Sender: $SENDER [$SENTBY_IP]
Status: $VIRNAME $STATUS [$SCANNER_NAME=$LEVEL]
''',
end='''
End of report
''',
include='@my-domain.com$',
exclude='^root@'
)
13. Graphs
There are two ways, how to use Sagator's statistic graphs.
You can use MRTG or it's newer version RRDtool. It's suggested to use
RRDtool, because it's graphs are very nice, easy to install and can display
more information.
13.1. RRDtool
RRDtool usage is simplest than MRTG. Make /var/www/mrtg/sagator/rrd
directory into your web server configuration and add following lines
into your /etc/crontab:
# SAGATOR's RRD
*/5 * * * * root /usr/share/sagator/stats.py rrdtool /var/www/mrtg/sagator \
> /var/log/sagator/rrd.log 2>&1
You can see your RRDtool graphs at http://localhost/mrtg/sagator/ .
Rrdtool 1.2 is used by default in sagator 0.8.0 and higher, if you are using
rrdtool 1.0, please change "...stats.py rrdtool" to "...stats.py rrdtool10".
13.2. MRTG
Make /var/www/mrtg/sagator directory into your web server configuration.
Copy a file named mrtg.cfg to /etc/mrtg/sagator.cfg and add this line
into your /etc/crontab:
# SAGATOR's MRTG
0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/sagator.cfg
You can see your MRTG graphs at http://localhost/mrtg/sagator/ .
14. Web quarantine
This service can be used to access quarantine files. Each user can see,
if some emails have been quarantined, can see it's content, check it again
with a virus/spam scanner and get it or resend as email.
Requirements: python-genshi-0.4 or higher is required for this service.
webq() service requires to log information about quarantined emails
into a SQL database supported by sagator. You can see supported databases in
Databases.txt documentation file.
If you are using apache on port 80, you can use mod_proxy module to
redirect webq requests to port defined in sagator, for example:
ProxyPass /webq http://localhost:8008
ProxyPassReverse /webq http://localhost:8008
Each user must authenticate to sagator's web quarantine. It is done in
webaccess table. Users must have CRYPT_MD5 password in this table. You
can create admin user by inserting this row:
INSERT INTO webaccess (email, pass, perms, datetime, lang, showrows)
VALUES ('admin', '$1$WA6srD5g$h4XRTVZiH/goIEuNzdegC1',
'A', CURRENT_TIMESTAMP, 'en_US', 50);
This is an admin account with full access to all users! It's default
password is "sagator". If you want regular users, replace 'A' flag by
empty string ''.
15. Done
Congratulations. You have sucessfully installed sagator. :-)
|
