j-chkmail Change Log Sep 25 2009 j-chkmail 2.1.1 See changes in file README-2.1.x Jun 8 2009 j-chkmail 2.1.0 See changes in file README-2.1.x Mar 10 2009 j-chkmail 2.0.0 [FEATURE] Detection of very short messages (very small body length). These short messages can be used to validate the content of addresses databases and can be considered as a privacy violation. [FEATURE] Detection of messages with Delivery-Notification-Request headers. [FEATURE] Some configuration data files are available for download at http://j-chkmail.ensmp.fr/data/conf-data [CHANGE] Global code clean-up. Aug 11 2008 j-chkmail 1.13.0 [BUG] Validation and correction of a race condition on j-greyd. Thanks to Dominique Lalot who helped with his platform. Aug 05 2008 j-chkmail 1.13.0 RC3 [CHANGE] BerkeleyDB version changed back to 4.6.21. Jul 30 2008 j-chkmail 1.13.0 RC2 [BUG] Solved problem with j-greyd when using a "local:" socket to communicate with j-chkmail. Problem pointed out by Dominique Lalot [CHANGE] Default j-greyd socket changed from local to inet. Usually j-greyd will be shared by filters on different computers. This situation needs a inet socket, instead of local socket. Jul 28 2008 j-chkmail 1.13.0 RC1 [CHANGE] DNS RBWLs code was rewritten to provide better handling and configuration or Black/White lists of both IP addresses and URL blacklists, which are now configured at j-tables data file [FEATURE] Added support to POP before SMTP. Based on tests done by Steve Hsieh [CHANGE] Default configuration is no more a null filter but have some features enabled. SPAM_URLBL YES SPAM_REGEX YES SPAM_ORACLE YES CHECK_CONN_RATE YES CHECK_OPEN_CONNECTIONS YES CHECK_BADRCPTS YES CHECK_RCPT_ACCESS YES CHECK_NB_RCPT YES CHECK_MSG_RATE YES CHECK_NB_MSGS YES CHECK_BAD_SENDER_MX YES CHECK_DATE_IN_FUTURE YES GREY_CHECK YES [CHANGE] j-chkmail -M option added allowing to create configuration files : j-chkmail -M null : create null filter j-chkmail -M default : default configuration with some options enabled j-chkmail -M running : a clean configuration file with current options This option replaces old -m and -n options, which become deprecated [CHANGE] In default configuration, URLBL data comes from DNS. [BUG] configure script changed to take into account changes in FreeBSD 7 [FEATURE] IPv6 Support [UPDATE] PCRE library was updated to 7.6 version [UPDATE] Berkeley DB library updated to 4.7.25 [FEATURE] Some scripts included at tools directory and a new Makefile target (upgrade), allowing easier filter upgrade [FEATURE] Basic message archiving feature added Mar 21 2008 j-chkmail 1.12.0 [RELEASE] j-chkmail 1.12.0 stable release Mar 11 2008 j-chkmail 1.12.0 RC3 [UPDATE] j-ndc help command [BUG] greylisting - lifetime for pending entries without coherent DNS resolutions too short (pointed out by Serge Aumont) [CHANGE] Default values were included on j-policy database in order to avoid problems if the mail admin doesn't define them Feb 27 2008 j-chkmail 1.12.0 RC1 [CHANGES] Many changes take a look at README-1.12.X file Dec 11 2007 j-chkmail 1.11.1 Release [BUG] - Bug when handling some kind of uuencoded attached files. Under a particular condition, the filter could fall in an endless loop and finish consumming all CPU resources. Apr 11 2007 j-chkmail 1.11.0 Release [FEATURE] Added handling of "SPAMTRAP" value for "CheckRcptDomain" prefix in j-rcpt database [UPDATE] j-policy.badmx.txt file updated with data from IANA (valid IP networks) : http://www.iana.org/assignments/ipv4-address-space [CHANGE] Configuration option FILE_EXT removed from j-chkmail configuration file. j-xfiles shall be used instead to configure this. [FEATURE] The number of messages detected as SPAM by the bayesian filter is now added to /var/jchkmail/j-stats file, in order to be able to do graphics presentation. [CHANGE] Oracle clean-up. Some tests giving too many false positives or not enough efficient were removed. [CHANGE] Code to limit connections to accept from SMTP clients with bad DNS resolution rewritten. [CHANGE] Greylisting database clean-up is now distributed over time (done in chunks of 1 second size) instead of completely doing it each defined time interval. This is done to avoid blocking databases for too long. Problem pointed out by Tibor Weis SEE README-1.11.x Dec 18 2006 j-chkmail 1.10.1 [BUG] j-policy VirusCheckXXX prefix not taken into account. Problem remarked by Dennis Peterson Nov 03 2006 j-chkmail 1.10.0 Release [BUG] Corrected an error which caused greylisting to "randomly" reject pending connections already ready to be accepted [BUG] Corrected an error on j-regex file handling -> to be able to do pattern matching on the content of SMTP EHLO/HELO command. Oct 16 2006 j-chkmail 1.10.0-RC1 [NAME CHANGE] 1.9.1 is renamed to 1.10.0 as this version contains a major change (bayesian filtering) [BUG] Error evaluating global score of pattern matching (REGEX) on headers. Problem noted by Christian Pelissier [CHANGE] CHECK_FROM_CONTENT configuration option changed to CHECK_ENVFROM_CONTENT, as this check is really applied to envelope not header. [BUG] Error detecting scripts and html code inside headers [UPDATE] Bundled Berkeley database updated to 4.5.20 Sep 28 2006 j-chkmail 1.9.1 - Beta 5 [BUG] Small corrections on text preprocessor to remove non printable characters before passing messages to bayesian filter. Aug 21 2006 j-chkmail 1.9.1 - Beta 4 [CHANGES] Bayes filter tokeniser changes to improve filter efficiency [CHANGES] ContentCheck is disabled for connexions coming from 127.0.0.1 [BUGS] Error logging detected virus name at j-xreport file [CHANGES] j-bayes-tbx now can completely replace perl scripts in the learning chain. Doc to do. Jul 13 2006 j-chkmail 1.9.1 - Beta 3 [CHANGES] bayes database moved from /var/jchkmail to /etc/mail/jchkmail directory [FEATURE] Bayes score is now part of global message score [BUGS] Some bugs and changes applied to bayes-toolbox Makefiles Jun 28 2006 j-chkmail 1.9.1 - Beta 2 [FEATURE] Bayesian filter implemented... Need : - some more test - better integration and evaluation of global message score Apr 2006 j-chkmail 1.9.1 - Beta 1 [INTERNALS] map abstraction for BerkeleyDB hashes implemented and tested [FEATURE] IP addresses to hostname cache implementation. Tested with adding and cleaning up data [INTERNALS] Cyclic tasks handler implemented and being tested. [BUG] Logged messages when rejecting binary messages corrected. [BUG] Under some conditions, text log files weren't correctly opened. [BUG] j-grey-expire log file wasn't included in the list of files to rotate when sending "reopen logfiles" to the filter [FEATURE] Heuristic filter (oracle) checks may now be enabled or disabled by users and scores can be changed. This is done at a new configuration file. [BUG] Some results of j-printstats can't be redirected to some file. Solved with patch from Bruno Verlyck [BUG] Lock missing in the client side of greylist filter. This affects greylist filter only when running in client mode. Apr 24 2006 j-chkmail 1.9.0 RELEASE No change since last Release Candidate Apr 14 2006 j-chkmail 1.9.0 RC2 [FEATURE] Validation of greylisting working in mode CLIENT/SERVER [CHANGE] Added "RESET GREYERRORS" command to j-ndc tool Apr 12 2006 j-chkmail 1.9.0 RC3 [BUG] Corrected some bugs on the handling of compile.conf file Apr 11 2006 j-chkmail 1.9.0 RC2 [CHANGE] MIME extraction code was partially rewritten to be able to handle new kinds of spam. [FEATURE] Detection by the heuristic filter (oracle) of some spams hidden inside gif messages Apr 07 2006 j-chkmail 1.9.0 RC1 [FEATURE] Created file compile.conf to add persistent configuration options. [CHANGE] Removed code to link j-chkmail against some antivirus scanners libraries (sophos and clamav). [BUG] URLEXPR expressions no more consider '?' as the end of an URL. [FEATURE] Content of log files inside /var/jchkmail work directory may now be redirect to syslog or an UDP server. Options defining this files where changed to reflect these new features. [FEATURE] A new application will be installed (j-logserver) : an UDL log server to receive log messages (see feature above) [BUG] Bug when checking policy for some triplets. Problem noted by Tibor Weis. [FEATURE] Add recipient checks database (j-rcpt.db) allowing to control recipient access and be able to handle correctly recipient address existence. [CHANGE] Logging now includes the reply code returned to the SMTP client when applicable. [CHANGE] Spool file is open at the first header or body, instead of at from command. [BUG] Corrected j-regex configuration file handling bug. ANYWHERE tag caused expression to be used also in place of URLSTR and URLEXPR (so, used more than once). [BUG] Bad error handling when the number of open streams is exceeded (when reading notify message file) [CHANGE] CF_LOG_ATTACHMENTS allows now to log all attached files into /var/jchkmail/j-files, instead of syslog. Problem noted asked by Stephane Lentz [FEATURE] URL filtering can now check numeric URLs. Jan 03 2006 j-chkmail 1.8 - First official 1.8 release [UPDATE] Graphic tools (contrib:rrd-jchkmail) updated to draw stats about greylisting Dec 13 2005 j-chkmail 1.8 RC4 [CHANGE] get-urlbl and Makefile.db files were changed to reflect changes on how the URLBL rsync server works. [CHANGE] Bundled BerkeleyDB software upgraded to 4.4.16 Nov 24 2005 j-chkmail 1.8 RC3 [BUG] Compiling j-chkmail error on j-mxcheck.c file. Problem noted by Konstantin Symorot Nov 23 2005 j-chkmail 1.8 RC2 [BUG] Bad compile options with gcc and Solaris 9 and 10, sparc and X86 architecture. [CHANGE] Messages were checked and corrected by Dennis Peterson, mainly about english problems... 8-( j-chkmail developper speaks C better than english. Thanks Dennis. [BUG] Misinterpretation of j-nets defined netmasks when the filter is compiled in 64 bits mode - Solaris. Thanks to Veronique Bouzid. Nov 10 2005 j-chkmail 1.8 RC1 No difference from last Beta version Nov 07 2005 j-chkmail 1.8 - Beta 8 [FEATURE] j-check-message command line tool was added to distribution. This is a command line tool similar to j-scanfile, but to check message content against anti-spam rules : regular expressions, heuristic filter and URLBL. [CHANGE] Scanning of ISO-2022 charset messages was reenabled again. Oct 28 2005 j-chkmail 1.8 - Beta 7 [BUG] Log of rejected messages done before the DATA command wheren't logged as they should be Oct 21 2005 j-chkmail 1.8 - Beta 6 [BUG] Under some conditions, under old Linux kernels, the supervisor could use all available CPU if a child died. [BUG] Some connections rejected at connection time weren't included in the history. [BUG] Within some cases some recipients weren't checked against policy database to check if messages to them shall be checked against content. Oct 14 2005 j-chkmail 1.8 - Beta 5 [CHANGE] CONTENT_CHECK configuration option removed use SPAM_REGEX SPAM_ORACLE and SPAM_URLBL options [CHANGE] When logging connection results, and the event to log is after RCPT comand, one line is generated for each recipient. [BUG] Exception error when the SMTP client hostname is empty : buggy DNS declarations... [FEATURE] FROM_PASS_TOKEN and TO_PASS_TOKEN configuration options added allowing to generate handling exceptions. Ex: if you run a good list server such as "Sympa" and if you declare : TO_PASS_TOKEN -owner@ and at j-policy database you declare : GreyCheckTo:-owner@lists.domain.com NO and bounce addresses of your list are of the form listname-owner@list.domain.com Bounces won't be greylisted. This is a good idea. Asked by Serge Aumont. You can also use this to let some messages with executable files pass through the filter. In this case, you define : TO_PASS_TOKEN +somerandomtoken@domain.com and add to your j-policy database : XFilesCheck:+somerandomtoken@domain.com NO. In this case, messages sent to : user+somerandomtoken@domain.com won't be checked against XFiles Oct 07 2005 j-chkmail 1.8 - Beta 4 [CHANGE] Default log messages is now one line per message. [CHANGE] DNS resolve check is no more applied to clients from known networks [BUG] SPAM_URLBL doesn't work when SPAM_REGEX isn't enabled at the the same time Sep 28 2005 j-chkmail 1.8 - Beta 3 [CHANGE] Bundled PCRE updated from 5.0 to 6.4 version - resulting from vulnerability found in PCRE code [CHANGE] Little changes to run on Tru64 OS [FEATURE] Grey activity counters (GREYMSGS and GREYRCPT) added to stats dump file [FEATURE] Log events changed from multiline to a single line. A configuration option was added to be able to use multiline logs. [FEATURE] LOG_ORACLE_LEVEL configuration option added to control how much information is logged by the heuristic filter. [FEATURE] Two new files are installed at /etc/mail/jchkmail : j-chkmail.cf.default and j-chkmail.cf.running. You can use this files to upgrading your configuration file. j-chkmail.cf.running can probably replace previous configuration file Sep 07 2005 [BUG] - Error when evaluating default Connection Rate limit from policy database. Problem pointed out by Serge Algarotti [FEATURE] - Counters now dumps the number of messages/recipients greylisted. [FEATURE] Checks to be applied to EHLO parameter can now be defined with configuration option BADEHLO_CHECKS. [FEATURE] Equivalent domains on greylist domain check. Added GreyEquivDomain tag to policy database. Ex : GreyEquivDomain:minesparis.fr ensmp.fr [CHANGE] Structure (and size) of history record (/var/jchkmail/j-history) file changed to accomodate some new data. Old file shall be removed when upgrading to this version. [CHANGE] Greylisting reply code changed from 451/4.7.1 to 451/4.3.2 to solve problems with some @_(#@@@{ Interscan mail servers. [CHANGE] GREY_REPLY environnement variable handling added to be able to change greylisting reply code and message. The value of this variable shall be set at /etc/default/jchkmail file, this way : GREY_REPLY="451 4.3.2 Hi Joe ! I'm busy now : come back later !" export GREY_REPLY Aug 11 2005 j-chkmail 1.8 - Beta 1 [BUG] Problem when using spam filter results to expire greylisting database entries. Noted by Dennis Peterson and Serge Algarotti [FEATURE] Bundled BerkeleyDB package upgraded to 4.3.28 patched by Claus Assmann, given a bug on 64 bits OSs. Aug 03 2005 [BUG] Bug refreshing grey whitelist [FEATURE] Add "-m rg" option to j-printstats to show which SMTP clients had been greylisted. Aug 01 2005 - j-chkmail 1.8 - Alpha 21 [CHANGES] Some configuration variables had changed leur name or were added : QUARANTINE_MAX_AGE -> QUARANTINE_LIFETIME GREY_MAX_AGE_NORMAL -> GREY_VALIDLIST_LIFETIME GREY_MAX_AGE_NULLSENDER -> removed GREY_REPLACE_NULLSENDER -> removed GREY_WHITELIST_LIFETIME -> new GREY_BLACKLIST_LIFETIME -> new Jul 16 2005 - j-chkmail 1.8 - Alpha 20 [FEATURE] Possibility of logging all info on a single line. To do this j-chkmail shall configured with tmp/config script tmp/config ONELINELOG This feature is still changing. [CHANGE] CHECK_BAD_HELO - new check added - if helo parameter is an IP address, checks if this address is bounded by brackets. [BUG] Corrected bugs on versions jchkmail-1.8-050621 to jchkmail-1.8-050708 when checking EHLO conformance [CHANGE] j-chkmail returns an temporary reject when external scanner returns an error, instead of accepting the message. Behaviour change asked by Stephane Lentz. [BUG] Erroneous error messages appearing during greylist database purge. Jul 02 2005 - j-chkmail 1.8 - Alpha 19 [BUG] Bug handling policy database when both DEFAULT and domain name conditions are defined. Bug detected by Tibor Weis [FEATURE] j-greylisting database generation activated. Jul 01 2005 - j-chkmail 1.8 - Alpha 18 [FEATURE] Added handling of triplets (Connect/From/To) to policy database. [FEATURE] Added checking of ContentCheck, XFilesCheck and VirusCheck tags to policy database [BUG] Circunvented problem with sendmail which counts greylist rejections as being "unknown users" and increments nb_badrcpts macro value. [BUG] BAD_EHLO check moved from mlfi_ehlo to mlfi_from callback. Problem remarked by Sebastien Cat. [CHANGE] greylist database changed. Jun 10 2005 - j-chkmail 1.8 - Alpha 17 [CHANGE] Default values for connection rate adjusted Apr 25 2005 - j-chkmail 1.8 - Alpha 16 [CHANGE] Default list of unsafe file extensions moved from hardcoded .c file into top/xfiles.def file. Users can edit and change this file before configuring and compiling j-chkmail. [FEATURE] "-x" option added to j-chkmail to show default XFILES extensions. [FEATURE] Added help command on control channel [FEATURE] j-rotate script to rotate log files (once a month ?) [FEATURE] Two environnement variables added PERIODIC_AUTO_RESTART="1d" - to periodically restart j-chkmail HIGH_LOAD_AUTO_RESTART="yes" - automatically restart j-chkmail if server CPU load is greater than 95 % for longer than 5 min. [BUG] Load measurement bug under Debian Linux - problem noted by Serge Algarotti [FEATURE] j-greyd greylisting server developped to be used by more than one filter [FEATURE] Add grey mode (STANDALONE or CLIENT). Client mode works with greyd server [BUG] uuencoded xfiles isn't handled correctly, after some code cleaning. [FEATURE] NetClass, ConnRate and ConnOpen definitions implemented at j-policy database implemented. [FEATURE] DESIGNATED_QUARANTINE implemented as a way to quarantine designated recipients, senders or smtp clients. Mar 03 2005 - j-chkmail 1.8 - Alpha 13 [FEATURE] Policy database being implemented. Begin migration of j-chkmail configuration data into it. Using j-makemap to create j-policy database from j-policy.txt. Using j-policy database can be enabled if _FFR_DBPOLICY macro is defined at compile time. [CHANGE] Reply messages can be configured on policy database. Look at etc/j-policy.txt for examples, and valid tags. [CHANGE] Network Classes (LOCAL, DOMAIN and FRIEND) can be defined at policy database. Also, general names, other than these, can now be used to declare network classes. Host and domain names can be used to declare network clases. [CHANGE] Connection rate can be declared at policy database, per IP addresses, IP networks, network classes and domains. [INTERNAL] XXX rates source code rewrotten to allow more more flexibility on adding new parameters to limit rate. [FEATURE] Bounce rate measurement (per client) code added. [INTERNAL] DNS query code added to allow query for MXs and other DNS contents. [FEATURE] Check MX of sender address added. This feature allow to block senders having known BAD MXs or MXs on local network. [FEATURE] Check of NULL SENDERs sending messages (bounces) to more than one user. [FEATURE] Measure of bounce rate per client IP address [FEATURE] Add size criteria to declare XFILE definition - defined at /etc/mail/jchkmail/j-xfiles [FEATURE] Create /var/jchkmail/j-xreport log file, with detailled data about blocked messages containing XFILES [FEATURE] j-ndc allows to change all configuration values, except file extensions and communication sockets. [FEATURE] j-ndc new command to reopen log files [PORT] Integration of libbind from bind 9.3, in order to provide multithreaded versions of DNS calls (res_n...). [CHANGE] Changing how default set of unwanted charsets are defined : - configure script option : --with-ucharset=LISTE - internal hard coded list if not defined at configuration time [FEATURE] Defined new environnement variable JCHARSET, to override software defined unwanted charsets. If the environnement variable is defined and is empty, only charsets defined at j-oracle configuration file is taken into account. [PORT] Berkeley DB source code included in j-chkmail distribution. To use it, you shall, before configure j-chkmail, build BerkeleyDB library (build, but not install it...) : cd libdb ./Build [BUG] At some situations Base64 decode of some malformed messages could lead to errors under Solaris [FEATURE] Added measurement of message rate and service time rate to short history. [FEATURE] Validate configuration option CHECK_BADEHLO. If set, this connections coming from unknown clients with bad ehlo parameters will be rejected. [FEATURE] Two configuration options added : SCORE_ON_SUBJECT_THRESHOLD SCORE_ON_SUBJECT_TAG These options work with existing SCORE_ON_SUBJECT option. If SCORE_ON_SUBJECT is set, then tag on subject line is added if message score is greater than SCORE_ON_SUBJECT_THRESHOLD. If SCORE_ON_SUBJECT_TAG is defined, the content of this tag will precede original subject. If not, new subject tag will be preceeded by something like [J-XXX], where the count of Xs is the message score. [FEATURE] Greylisting added [INTERNALS] j-policy syntax frozen [FEATURE] Passthrough feature added, allowing to pass through the filter without content checking (xfiles, virus and spam). Jan 06, 2005 - jchkmail - 1.7 release [BUG] Wrong base 64 decode of some malformed messages under Solaris [PORT] Correct some port problems under Tru64 Dec 20 - jchkmail - 1.7-RC5 [BUG] Lock conflit when logging Pattern matching results, may relaunch filter. This problem appeared only on FreeBSD. Dec 02 - jchkmail - 1.7-RC4 [BUG] Disabled a debug flag where messages spending more than 10 secs to be handled remained in quarantine [CHANGE] Quarantined messages containing XFILES have now the suffix ".xfile" appended. This allows to identify files on the quarantine directory when they are there as they have an XFILE attached or if the filter was killed without closing open connections. [FEATURE] Quarantine management script included on the distribution Contribution from Christian Pelissier. [BUG] Address extraction on SPAMTRAPs problem. Corrected. [CHANGE] j-urlbl database is now distributed on compressed format. Oct 14 - 1.7 RC3 [BUG] j-chkmail doesn't compile if target machine doesn't have auto(conf|make) tools installed Sep 30 - 1.7 RC2 [CONFIG] get-urlbl and Makefile.db scripts needed to create URL blacklist are installed at /etc/mail/jchkmail directory. Sep 14 2004 j-chkmail 1.7 RC1 [FEATURE] Environnement variables. Some variables were defined : These variables are to be set usually at default configuration scripts : /etc/default/j-chkmail or /etc/sysconfig/j-chkmail - XFILEs extensions FILE_EXT - libmilter debug level MILTER_DEBUG_LEVEL - filter log level JCHKMAIL_LOG_LEVEL - database cache size DB_CACHE_SIZE [FEATURE] Some scripts created at bin directory. Not installed by default - get-urlbl - update URL database. To be launched by cron - j-ndc - script to connect to control channel and control and query the filter [BUG] Some MIME Content-Disposition/Content-Type could be bad interpreted when some virus add spaces around "=" sign. [CONFIG] Some configuration options added. - timeout waiting for external scanner - max message size to pass to external scanner - Internal options for sophos scanner [FEATURE] get-urlbl script to rsync and install urlbl. [FEATURE] Support for Sophos SAVI - integrate sophos libraries into j-chkmail filter, and a scanner daemon. [FEATURE] sophosd - a daemon doing virus scan and using Sophos savi library. [FEATURE] Block messages coming from NULL SENDER, when the number of recipients is greater than one and the connection comes from an unknown network. [FEATURE] Detect messages which body is empty or near empty. [FEATURE] Check enveloppe addresses syntax [CHANGE] Internal protocol between j-chkmail and the external scanner changed to put allways a return code on each line. [FEATURE] XFILES * Possibility of add TNEF and CLSID files to XFILES class. Validation is done at j-xfiles configuration file. * Detection of message/partial and message/external MIME content type is integrated into j-chkmail code (no more as FFR). Validation of this as a XFILE is done at j-xfiles configuration file. * It's now possible to completely define what XFILES are at j-xfiles configuration file. Nowadays j-chkmail.cf FILE_EXT et FILE_REGEX will be obsoleted sometime in the future. [FEATURE] Policy database... Under construction... [CHANGE] Message content is now checked before calling external virus scanner (the order was inversed) [CHANGE] Integer configuration values of j-chkmail.cf file can be now expressed with units : Time : s,m,h,d default = s Units : (K,M,G) -> (1024, 1024**2, 1024**3) [FEATURE] Adding option to setup BerkeleyDB cache size (configuration file and environnement variable) [UPDATE] Bundled PCRE updated from 4.4 to 4.5 Jun 11 2004 j-chkmail 1.7 Alpha-7 [FEATURE] HELO/EHLO parameter check : numeric vs value [FEATURE] URL Blacklist - database and/or DNS based. [FEATURE] j-makemap tool to create database (j-chkmail equivalent of sendmail makemap). Needed to create URLBL database. [CHANGE] /var/jchkmail/j-regex format changed to include both pattern matching and URL blacklist logging. [CHANGE] j-regex-stat.pl modified to correctly handle new j-regex file format. [FEATURE] - j-dburlbl-stat.pl script added to generate statistics about URL blacklist detection. Changed to j-urlbl-stat with three possible parameters j-urlbl-stat -t DBURLBL j-urlbl-stat -t URLBLSTR j-urlbl-stat -t URLBLEXPR [FEATURE] CPU load measure under FreeBSD [CHANGE] IP address information added to /var/jchkmail/j-files. j-xstat.pl contrib/script modified accordingly. [BUG] Error decoding filename with spaces in a uuencoded attachment [OPTION] "z" command line option. This option makes the daemon set work directory to "/tmp" instead of "/"., in order to allow core dump when debug is needed. Check documentation for details. [FEATURE] New command defined for the control channel which prints the histogram of message scores (# of Xs). STATS SCORES STATS ORASCORE STATS REGSCORE [BUG] Quoted Printable decoder - some tricky qp coded messages could generate an error. [OPTION] Compile time option _FFR_REGEX_ACCESS added to select if j-access address matching is regular expression based. To activate this, j-chkmail shall be configured with : ./configure ...OPTIONS... CFLAGS="-D_FFR_REGEX_ACCESS=1" [OPTION] CLUSTER option added to j-chkmail.cf configuration file. Setting this option to yes will allow using j-chkmail in a cluster sharing resources. This option changes two things : - "Connect from" log line will show the hostname of the mailserver, as defined by the Macro $j. This is necessary when the filter is shared between multiple mail servers - j-chkmail spool filename will created with the mailserver hostname prepended to the connection ID. This is neccessary when the spool is shared by multiple filters. [CHANGE] - Connection rate control and Open connections control are now alse checked at connection time (instead of only at helo callback), to prevent from gateways abusing the delay between CONNECT and EHLO. [CHANGE] AVL binary tree management module was completely rewritten and recursion removed to solve some segmentation fault issues under FreeBSD. [FEATURE] Two new kind of lines created at j-regex configuration file for faster URL pattern matching : URLEXPR and URLSTR. [CHANGE] Under FreeBSD, start-up script was renamed from jchkmail.sh to jchkmail, to best conform with FreeBSD rules. [FEATURE] New configuration option SCORE_ON_SUBJECT. If set, the spam score will be also represented on the Subject Header as a number of Xs in a tag of the kind [J-XXX] [CHANGE] Under sendmail 8.13.0, j-chkmail will close SMTP connection instead of simply sending an error DSN, in certain cases : - too much open connections - connection rate exceeded - global number of file descriptors too high - too much bad recipient errors [CHANGE] Rejecting intranet users from external messages no more checks host address part against class W. You shall define intranet users in one of these three ways : - user@host, user or @host [CHANGE] Results of control channel commands STATS ( SCORES | ORASCORE | REGSCORE ) is now presented in cumulative mode Apr 09 2004 j-chkmail 1.6 RC3 [CHANGE] __TO__ variable defined at j-error-msg configuration file and set to use "To" header. [UPDATE] j-printstats reports also number of virus detected when launched with options "-q -m x". [UPDATE] All references to j-host-access were finally removed. [UPDATE] USE_TCPWRAPPER configuration option removed as TCP Wrapper is no longer needed to control access to control channel [BUG] j-printstats -q doesn't print individual individual connection rate for smtp clients when connection rate control isn't enabled. [BUG] NO_RESOLVE_CHECK wasn't handled correctly. [DOC] Some more doc... Mar 30 2004 j-chkmail 1.6 RC2 [UPDATE] j-xstat.pl script (contrib/scripts) was updated to handle j-files and checking quarantine directory contents using or ClamAV clamdscan, McAfee uvscan or Sophos sweep, to identify blocked virus. [BUG] Imcomplete code at load measure functions, under FreeBSD and TRU64. Code fully implemented only on Solaris and Linux platforms. [DOC] Some more lines inserted at FAQ. Mar 22 2004 j-chkmail 1.6 RC1 [FEATURE] PCRE was integrated to the package - no more need to install PCRE as an external package and configure it. [BUG] Some installation issues regarding rpm packaging and configuration files not being installed at default places. Thanks to Robert Schmitt. [TEST] Using NO_RESOLVE_CHECK was validated - no more need to use the j-host-access configuration file. Entries at this file shall be replaced by entries of the following kind at j-access configuration file : Connect:IP_ADDRESS NO_RESOLVE_CHECK [TEST] Validation of saving quarantined messages at UNIX text format instead of SMTP text format. [BUG] Corrected bug at j-xfiles configuration file handling. This bug could generate bad interpretation of normal files. Mar 04 2004 j-chkmail 1.5-Alpha-3 [FEATURE] Added file /var/jchkmail/j-virus, which is the equivalent of j-files, and allows to get stats from virus. Each virus is logged at this file, with msg id and timestamp and smtp client ip address. [BUG] Found a bug (possibility of buffer overflow) on the statistical evaluation of message word length (oracle). [CHECKING] External virus scanner tested with Trendmicro. Thanks to Cove Schneider, who adapted the perl script and did the tests. [DOC] Added some installation doc to the rrdtool-jchkmail directory. [CHANGE] Add checking to existence of initialisation scripts at /etc/sysconfig/jchkmail and /etc/default/jchkmail. As startup script is overwritten after each software update, it's interesting to have way to save persistent startup options. Mar 01 2004 j-chkmail 1.5-Alpha-2 [FEATURE] Added support for ClamAV, using libclamav (experimental). j-chkmail shall be configured and linked with clamav. Shall be configured as : ./configure --with-clamav=/path/to/clamav/root j-chkmail.cf shall be configured as follows : CLAMAV_ACTION WARN CLAMAV_DBDIR /opt/clamav/db CLAMAV_SAVE YES CLAMAV_MAXSIZE 0 Feb 26 2004 j-chkmail 1.5-Alpha-1 [FEATURE] Added support for ClamAV, using clamd. No need to link with libclamav. (experimental). Configure j-chkmail.cf as following to use clamd : SCANNER_ACTION WARN SCANNER_SOCK inet:3310@localhost SCANNER_PROTOCOL CLAMAV SCANNER_SAVE YES Feb 23 2004 j-chkmail 1.5-040223 - AKA j-chkmail-1.5-Alpha-1 [CHANGE] Scripts to handle new format of j-stats file were finished. [BUG] Boundaries detection when some special characteres appear at boundary string. [CHANGE] A new configuration file was created (j-xfiles) allowing one to adding new options do xfiles definition, based at the same time on the mime type and a regular expression for the filename. Feb 12 2004 j-chkmail 1.5-040212 [BUG] Corrected external antivirus interface problem. Checked under Solaris Feb 09 2004 [BUG] Error when closing connection generating spurious process under Linux. [FEATURE] STATS CONNOPEN command added to the control channel. Feb 02 2004 j-chkmail 1.5-040202 [DOC] A FAQ begins to be constituted at doc directory. [BUG/CHANGE] configure script was modified to better detect PCRE and Berkeley DB libraries. [PORT] First port to Hewlett-Packard HPUX 11.11. TODO - minor changes to be able to run under HPUX 11.00. Thanks to Emmanuel Collignon from Universite de la Rochelle. [FEATURE] Adding support to RBL check at the oracle (experimental) [FEATURE] Adding support to SpamTraps (experimental) [CHANGE] all dependencies of old libcompat were removed. [CHANGE] /var/jchkmail/j-stats file format has changed. Scripts on contrib directory are being changed to handle new file format. Jan 21 2004 j-chkmail 1.5 040121 [CHANGE] Specific build directory created for j-makemap. j-makemap will be the equivalent of sendmail makemap tool, used to create and dump database files. [CHANGE] Changes at the oracle - the weight of some tests were changed, and some new statistical tests were created and had a first validation. [CHANGE] j-checkspam tool was created allowing to check message files by hand. The interest is to help developpement and maybe be a first prototype of what could be a postfix version of j-chkmail. [LICENSE] Changed from usual GPL open source model to a closer model. Redistribution is no more allowed, and access to j-chkmail is given only to to registered users Jan 01 2004 j-chkmail 1.5 0101 [FEATURE] Support being added to use Berkeley DB with some configuration files. [CHANGE] Code cleaning after validation of old features. [CHANGE] Specific build directories created for j-scanfile and j-printstats tools. Dec 15 2003 - Jose Marcio Martins da Cruz j-chkmail 1.5 [FEATURE] -m and -n options added to j-chkmail to generate commented current runtime configuration and default configuration files. [FEATURE] one thread was create on the filter server to be able to handle commands and queries from an external source - See CTRL_* configuration options. At the moment, the following commands work : STATS ORACLE STATS THROTTLE STATS CONNRATE STATS CONNOPEN STATS COUNTERS RESET STATS RELOAD TABLES RECONFIG RESTART A simple perl client (j-ndc) was created (look for it at bin directory) to contact the control channel. [CHANGE] configuration options names of variables used to check message contents were changed to become more understandeable. [FEATURE] A simple client was added to the filter, in order to replace features related to external scanner. For the while, the external scanner may be launched by inetd - see example at contrib/ufilter subtree. Some day on the future, the client will be able to talk to sophie and other external scanners. See SCANNER_* configuration options. Experimental. [DOC] INSTALL file received some useful information... Read it for some basic instructions on how to install/upgrade j-chkmail from previous versions. MMM DD 2003 - Jose Marcio Martins da Cruz j-chkmail 1.4 [FEATURE] -m option added to j-chkmail to create configuration file from runtime configuration [BUG] Checking X-Files of base 64 encoded file names [CHANGE] Remove useless caracteres (e.g. spaces, points...) at end of file before checking if it's a XFile. Only one charactere was removed before. [BUG] Bug checking filenames against regular expressions. [FEATURE] - sendmail message id is now logged with milter logs. [FEATURE] - Content checking engine was completely rewritten allowing checking inside coded mime-parts [FEATURE] - Content checking now supports also taging messages instead of simply reject. Moreover, there are now two spam levels of content checking : - greater than CONTENT_CHECK_SCORE - greater than 0 and lower than CONTENT_CHECK_SCORE You can define what to do for both cases : reject, discard or simply add a X-j-chkmail-Score header. [FEATURE] - Add somme fuzzy verifications called "ORACLE". Activate it with SPAM_ORACLE configuration switch [FEATURE] - Preliminar score definition combining results from content checking score and oracle score. Final score is presented as the number of 'X's on the X-j-chkmail-Score header. [FEATURE] - Added support to use PCRE instead of native REGEX libraries - engine goes two times faster. It really rocks... Tested with pcre 4.3 version pcre can be obtained at : ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/ [FEATURE] - script to do statistics of content checking. Working in progress. Check it in the contrib tree [FEATURE] - some more j-printstats query options [FEATURE] - checking more gateway behaviour - dictionnary attacks and too much open connections [CHANGE] - Process specific state files (socket and pid) default directories were changed from /var/jchkmail to /var/run, as this last directory is cleaned up at boot time under some OSs (Solaris, at least) [FEATURE] - Begin using j-user-access file, to allow change filtering behavior for some particular sender addresses of smtp clients. [PORT] - Ported to FreeBSD, Tru64 and NetBSD - Thanks to Emmanuel Dreyfus and Michel Bidaud [BUGS] - Corrected some bugs on checking subject and sender content. Pb. signaled by Russell Berry. [CHANGE] - Configuration options UNIX_SOCK and INET_SOCK removed as they were causing confusion Nov 03 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1d Release [BUG] Bug checking filenames against regular expressions. Sep 25 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1c Release [PORT] j-chkmail now works under FreeBSD [BUG] Option NO_FROM_HEADERS, rejecting messages from NULLSENDER (<>) - corrected [BUG] Content checking of Headers doesn't works corrected Sep 08 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1b Release [BUG] Corrected problem with WARN_SENDER disabled. Jul 03 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1a Release [BUG] Corrected problem when defining multiple FILE_EXT or FILE_REGEX lines at configuration file Mar 18 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1 Release [BUG] Check to see if configured external scanner exists and if j-chkmail has execution access. Mar 07 2003 - Jose Marcio Martins da Cruz j-chkmail 1.3.1RC5 [FEATURE] - -q option added to j-printstats. Allows query about gateway activity during some past time period Try j-printstats -q j-printstats -q -l 86400 j-printstats -q -l 6h hostname j-printstats -q hostname [FEATURE] start-up script modified to be compatible with LSB (Linux Standard Base Specification) - contributed by Michel Gaudet [BUGS/CHECKING] Some little bugs corrected, mainly, [FEATURE] - Better DoS protection against [FEATURE] - Some configuration options added. See j-chkmail.cf configuration file [BUGS] - Some minor bugs corrected : - libsm check when configuring - Check deleted as this check is no more needed - Pb noted by Serge Algarotti - Linux snprintf bug causes bad interpretation of FILE_EXT configuration option - Pb noted by Michel Gaudet - socklen_t not defined under Solaris 6 - Problem remarked by Pascal Cabaud - wrong error message when rejecting connections with high recipient throttle Dec 05 2002 - Jose Marcio Martins da Cruz j-chkmail 1.3 Release [BUG/CHECKING] - Global checking of new features and some code cleaning. Very little bugs between last RC and final release [CHANGE] - Default configuration directory changed from /etc/mail to /etc/mail/jchkmail, as suggested by J. D. Bronson. [BUG] - TAGS inside comment lines inside j-error-msg file were taken into account. Oct 15 2002 - Jose Marcio Martins da Cruz j-chkmail 1.3RC7 [BUG/CHECKING] - Body content checking testing done with the help of Ayamura.Kikuchi [FEATURE] - Content checking of HELO command. Suggested by Ayamura Kikuchi. [UPDATE] - Code for content checking of all headers finished. [FEATURE] - script to generate weekly statistics (look for it at contrib/scripts directory) [FEATURE] - Configuration parameters of periodic cleaning up of old quarantined files : CLEANUP_INTERVAL and QUARANTINE_MAX_AGE. Oct 08 2002 - Jose Marcio Martins da Cruz j-chkmail 1.3RC6 [FEATURE] - Content checking has two new parameters and the other one has changed CONTENT_CHECK_SIZE - to check message contents only if message size is below specified parameter CONTENT_CHECK_ORIGIN - to check message contents comes from anywhere or only if message comes from unknown IP networks. CONTENT_REGEX_MATCHES changed to CONTENT_CHECK_SCORE. Score needed to reject a message is no more the number of matches, but the sum of weights (each regular expression may have a different weight on the decision to reject the message).In order to realize this, a new weight parameter was added to j-regex file. [FEATURE] - now checks contents of base 64 encoded message body. [FEATURE] - added __MSGID__ parameter to j-error-file, to indicate the file name of quarantined message. [FEATURE] - added periodic cleaning up of spool directories with cleaning periodicity and max age configurable. [BUG] - j-chkmail stops logging when it receives a HUP signal to reload configuration (/etc/init.d/jchkmail reload). Problem noted by Jeff. D. Bronson [BUG] - jchkmail start-up script. Now verify and delete pid file before launching j-chkmail if there are no daemons running. Thanks to Patrick Gatt who remarked the problem and proposed a solution. Oct 04 2002 - Jose Marcio Martins da Cruz j-chkmail 1.3RC5 [UPDATE] Some documentation updates, some checks... Sep 30 2002 - Jose Marcio Martins da Cruz [FEATURE] - Content check code is completed. Body and body content may be verified agains a list of regular expressions. If the number of matches is greater than the threshold, message is rejected. Experimental feature. Too much time consumming. [FEATURE] - Messages which body is completely encoded in base64 or quoted-printable to avoid content checking may be refused. I'm not talking about attachment encoding, but entire message body encoding. Experimental feature - don't use it as it may block legitimate messages. [BUG] Problems when configuring j-chkmail to use tables in the database format. Corrected, but this is allways an experimental feature. Sep 24 2002 - Jose Marcio Martins da Cruz j-chkmail 1.3RC1 [FEATURE] - this minor address the vulnerability generated by sending fragmented messages (message/partial MIME type). Problem reported by the alert http://www.securiteam.com/securitynews/5YP0A0K8CM.html [FEATURE] - some minor changes to verify the vulnerability of sending attached files by reference (message/external-body MIME type). At the moment, this vulnerability wasn't announced anywhere, in my knowledge. [FEATURE] External (antivirus or spam) scanners are now launched by a pre-forked server instead of a concurrent server. [FEATURE] Interface to user defined mail scanner. [FEATURE] Configuration tables storage format selectable between text files or database files (db, ndbm or gdbm) formats. This is experimental. Not all tables are already ported to database formats. [FEATURE] Mail filtering according to mail gateway DNS resolution (failure or forged) [FEATURE] Access (black/white list), allowing to override bad DNS resolution results. [FEATURE] Temporal quota management of number of connection allowed to gateways without or forged DNS declarations. [FEATURE] More options add to j-printstats (see j-printstats -h) [PORT] j-chkmail was tested under Solaris 9. Thanks to J.B. Bronson from Aurora Health Care Center [BUG] - Some configure values - mainly directories - were not taken into account [UPDATE] Validation of j-chkmail under FreeBSD platforms. Thanks to Stephane Lentz and Egon Niederacher. [UPDATE] As allways, code related to experimental features was replaced by cleaner and optimised code as soon as experimental features becomes validated. [FEATURE] - validation of experimental features of previous versions Sep 18 2002 - Jose Marcio Martins da Cruz j-chkmail 1.2p1 [FEATURE] - this minor address the vulnerability generated by sending fragmented messages (message/partial MIME type). Problem reported by the alert http://www.securiteam.com/securitynews/5YP0A0K8CM.html Jul 01 2002 - Jose Marcio Martins da Cruz j-chkmail 1.2 [BUG] - Antivirus interface does not handle SIGINT signals correctly. Jun 20 2002 - Jose Marcio Martins da Cruz j-chkmail 1.1 [FEATURE] - Possibility of mail filtering based on the connection rate, the recipient rate of peer and the number of recipients of the message. This is an experimental feature. [FEATURE] - option to change the SUBJECT header when sending warning messages This is an experimental feature [FEATURE] - Reject messages if headers contains or