fdm ============================================================================ *** Introduction fdm is a program to fetch mail and deliver it in various ways depending on a user-supplied ruleset. Mail may be fetched from stdin, IMAP or POP3 servers, or from local maildirs, and filtered based on whether it matches a regexp, its size or age, or the output of a shell command. It can be rewritten by an external process, dropped, left on the server or delivered into maildirs, mboxes, to a file or pipe, or any combination. fdm is designed to be lightweight but powerful, with a compact but clear configuration syntax. It is primarily designed for single-user uses but may also be configured to deliver mail in a multi-user setup. In this case, it uses privilege separation to minimise the amount of code running as the root user. *** Table of contents ## Installation ## Quick start ## The configuration file %% Including other files %% Macros %% Testing macros %% Shell commands ## Invoking fdm %% Temporary files %% Command line arguments %% Running from cron %% The lock file %% Testing and debugging ## Fetching mail %% Mail tags %% POP3 and POP3S %% SSL certificate verification %% The .netrc file %% IMAP and IMAPS %% IMAP or POP3 over a pipe or ssh %% stdin and local mail %% From maildirs and mboxes %% Using NNTP and NNTPS %% New or old mail only ## Defining actions %% Drop and keep %% Maildirs %% Mboxes %% IMAP and IMAPS %% SMTP %% LMTP %% Write, pipe, exec and append %% stdout %% Rewriting mail %% Adding or removing headers %% Tagging %% Compound actions %% Chained actions ## Filtering mail %% Nesting rules %% Lambda actions %% The all condition %% Matched and unmatched %% Matching by account %% Matching a regexp %% Matching bracket expressions %% Matching by age or size %% Using a shell command %% Attachments %% Matching tags %% Using caches %% Cache commands ## Setting options ## Archiving and searching mail ## Using fdm behind a proxy ## Bug reports and queries ## Frequently asked questions ### Installation fdm depends on the Trivial Database library (TDB), available at: https://tdb.samba.org Ensure it is installed, then download the source tarball and build fdm with: $ tar -zxvf fdm-?.?.tar.gz $ cd fdm-?.? $ ./configure && make Then run 'make install' to install fdm to the default location under /usr/local. The --prefix argument may be set to specify an alternative installation location: $ ./configure --prefix=/opt/fdm && make $ sudo make install If being run as root, fdm requires a user named "_fdm" to exist. It will drop privileges to this user and its primary group. The user may be added on OpenBSD with, for example: # useradd -u 999 -s /bin/nologin -d /var/empty -g=uid _fdm It is not necessary to add a user if fdm is always started by a non-root user. fdm can be built to use PCRE rather than standard regexps. To do so, add -DPCRE to the make command: $ make -DPCRE Or PCRE=1 if using GNU make: $ make PCRE=1 ### Quick start A simple ~/.fdm.conf file for a single user fetching from POP3, POP3S and IMAP accounts and delivering to one maildir may look similar to: # Set the maximum size of mail. set maximum-size 128M # An action to save to the maildir ~/mail/inbox. action "inbox" maildir "%h/mail/inbox" # Accounts: POP3, POP3S and IMAP. Note the double escaping of the '\' # character in the password. If the port is omitted, the default # ("pop3", "pop3s", "imap" or "imaps" in the services(5) db) is used. account "pop3" pop3 server "my.pop3.server" user "my-username" pass "my-password-with-a-\\-in-it" account "pop3s" pop3s server "pop.googlemail.com" port 995 user "my-account@gmail.com" pass "my-password" # If the 'folder "my-folder"' argument is omitted, fdm will fetch mail # from the inbox. account "imap" imap server "my.imap.server" user "my-username" pass "my-password" folder "my-folder" # Discard mail from Bob Idiot. Note that the regexp is an extended # regexp, and case-insensitive by default. This action is a "lambda" or # unnamed action, it is defined inline as part of the match rule. match "^From:.*bob@idiot\\.net" in headers action drop # Match all other mail and deliver using the 'inbox' action. match all action "inbox" A simple initial configuration file without filtering, perhaps to replace fetchmail or getmail delivering to maildrop, may look similar to: # Set the maximum size of mail. set maximum-size 128M # Action to pipe directly to maildrop. action "maildrop" pipe "/usr/local/bin/maildrop" # Account definitions. account .... # Send all mail to maildrop. match all action "maildrop" To run fdm every half hour from cron, add something like this: */30 * * * * /usr/local/bin/fdm -l fetch See the fdm.conf(5) man page or the rest of this manual for more detail of the configuration file format. ### The configuration file fdm is controlled by its configuration file. It first searches for a .fdm.conf file in the invoking user's home directory. If that fails, fdm attempts to use /etc/fdm.conf. The configuration file may also be specified using the '-f' command line option, see the section on that subject below. This section gives an overview of the configuration file syntax. Further details of syntax, and specific keywords, are covered in later sections. The configuration file has the following general rules: - Keywords are specified as unadorned lowercase words: match, action, all. - Strings are enclosed in double quotes (") or single quotes ('). In double quoted strings, double quotes may be included by escaping them using the backslash character (\). Backslashes must also be escaped ("\\") - this applies to all such strings, including regexps and passwords. The special sequence '\t' is replaced by a tab character. In single quoted strings no escaping is necessary, but it is not possible to include a literal ' or a tab character. - Comments are prefixed by the hash character (#) and continue to the end of the line. - Whitespace is largely ignored. Lines may generally be split, concatenated or indented as preferred. - Lists are usually specified as 'singular item' or 'plural { item item }', for example: 'user "nicholas"', 'users { "nicholas" "bob" }'. The singular/plural distinction is not required, it is recommended only to aid readability: 'user { "nicholas "bob" }' is also accepted. - Regexps are specified as normal strings without additional adornment other than the "s (not wrapped in /s). All regexps are extended regexps. They are case insensitive by default but may be prefixed with the 'case' keyword to indicate case sensitivity is required. - Strings may be concatenated using plus: "a" + "b" is the same as "ab". This is most useful to wrap strings across multiple lines. Definition/option lines generally follow the following basic form: Example lines that may appear in a configuration file are: # This is a comment. set lock-types flock account "stdin" disabled stdin action "strip-full-disclosure" rewrite "sed 's/^\\(Subject:.*\\)\\[Full-disclosure\\] /\\1/'" match "^X-Mailing-List:.*linux-kernel@vger.kernel.org" in headers or "^(To:|Cc:):.*@vger.kernel.org" in headers action "linux-kernel" %%% Including other files The fdm configuration may be split into several files. Additional files may be referenced using the 'include' keyword: include "my-include-file.conf" include "/etc/fdm.d/shared-conf-1.conf" %%% Macros Macros may be defined and used in the configuration file. fdm makes a distinction between macros which may hold a number (numeric macros) and those that hold a string (string macros). Numeric macros are prefixed with the percentage sign (%) and string by the dollar sign ($). Macros are defined using the equals operator (=): %nummacro = 123 $strmacro = "a string" Macros may then be referenced in either a standalone fashion anywhere a string or number is expected, depending on the type of macro: $myfile = "a-file" include $myfile %theage = 12 match age < %theage action "old-mail" Or embedded in a string by enclosing the macro name in {}s: $myfile2 = "a-file2" include "/etc/${myfile2}" %anum = 57 include "/etc/file-number-%{anum}" Macros are not substituted in strings specified using single-quotes. %%% Testing macros The 'ifdef', 'ifndef' and 'endif' keywords may be used to include or omit sections of the configuration file depending on whether a macro is defined. An 'ifdef' is followed by a macro name (including $ or % type specifier) and if that macro exists, all following statements up until the next endif are evaluated (accounts created, rules added, and so on), otherwise they are skipped. 'ifndef' is the inverse: if the macro exists, the statements are skipped, otherwise they are included. An example is: ifdef $dropeverything match all action drop endif These keywords are particularly useful in conjunction with the '-D' command line option. Any statements between 'ifdef'/'ifndef' and 'endif' must still be valid syntax. %%% Shell commands The value of a shell command may be used at any point in the configuration file where fdm expects a string or number. Shell commands are invoked by enclosing them in $() or %(). They are executed when the configuration file is parsed and if $() is used, any output to stdout is treated as a literal string (as if the output was inserted directly in the file enclosed in double quotes); %() attempts to convert the output to a number. For example: $mytz = $(date +%Z) %two = %(expr 1 + 1) $astring = "abc" + $(echo def) Parts of the command within double quotes (") are subject to tag and macro replacement as normal (so it is necessary to use %% if a literal % is required, see the section on tags below); parts outside double quotes or inside single quotes are not. ### Invoking fdm fdm accepts a number of command line arguments and may be invoked as needed from the command line or by a mail transfer agent, such as sendmail, or at regular times using a program such as cron(8). %%% Temporary files As each mail is being processed, it is stored in a temporary file in /tmp, or if the TMPDIR environment variable exists in the directory it points to. fdm tries to queue a number of mails simultaneously, so that older can be delivered while waiting for the server to provide the next. The maximum length of the queue for each account is set by the 'queue-high' option (the default is two) and the maximum mail size accepted by the 'maximum-size' option (the default is 32 MB). In addition, the 'rewrite' action requires an additional temporary mail. Although fdm will fail rather than dropping mail if the disk becomes full, users should bear in mind the possibility and set the size of the temporary directory and the fdm options according to their needs. %%% Command line arguments The fdm command has the following synopsis: fdm [-klmnqv] [-f conffile] [-u user] [-a account] [-x account] [-D name=value] [fetch | poll | cache ...] The meaning of the flags are covered in the fdm(1) man page, but a brief description is given below. The flags are also mentioned at relevant points in the rest of this document. Flag Meaning -k Keep all mail (do not delete it from the server). This is useful for testing delivery rules without risking mail ending up permanently in the wrong place. -l Log to syslog(3) using the 'mail' facility rather than outputting to stderr. -m Ignore the lock file. -n Run a syntax check on the configuration file and exit without fetching any mail. -q Quiet mode. Don't print anything except errors. -v Print verbose debugging output. This option may be specified multiple times for increasing levels of verbosity. Useful levels are -vv to display the result of parsing the configuration file, and -vvvv to copy all traffic to and from POP3 or IMAP servers to stdout (note that -l disables this behaviour). -f conffile Specify the path of the configuration file. -u user Use 'user' as the default user for delivering mail when started as root. -a account Process only accounts with a name matching the given pattern. Note that fnmatch(3) wildcards may be used to match multiple accounts with one option, and that the option may be specified multiple times. -x account Process all accounts except those that match the given pattern. Again, fnmatch(3) wildcards may be used, and the -x option may be specified multiple times. -D name=value Define a macro. The macro name must be prefixed with '$' or '%' to indicate if it is a string or numeric macro. Macros defined on the command line override any macros with the same name defined in the configuration file. If -n is not specified, the flags must be followed by one of the keywords 'fetch' or 'poll' or 'cache'. The 'fetch' keyword will fetch and deliver mail, the 'poll' keyword print an indication of how many mails are present in each account, and the 'cache' keyword is followed by one of a set of cache commands used to manipulate caches from the command-line (see the sections on caches below). 'fetch' or 'poll' or 'cache' may be abbreviated. Examples: $ fdm -v poll $ fdm -vvnf /etc/my-fdm.conf $ fdm -lm -a pop3\* fetch $ fdm -x stdinacct fetch # fdm -u nicholas -vv f %%% Running from cron To fetch mail regularly, fdm must be run from cron. This line in a crontab(5) will run fdm every 30 minutes: */30 * * * * /usr/local/bin/fdm -l fetch The '-l' option sends fdm's output to syslog(3) rather than having cron mail it. To keep a closer eye, adding '-v' options and removing '-l' will have debugging output mailed by cron, or, using a line such as: */30 * * * * fdm -vvvv fetch >>/home/user/.fdm.log 2>&1 Will append extremely verbose fdm output to the ~/.fdm.log file. Note that this log file can become pretty large, so another cronjob may be required to remove it occasionally! %%% The lock file fdm makes use of a lock file to prevent two instances running simultaneously. By default, this lock file is .fdm.lock in the home directory of the user who runs fdm, or /var/db/fdm.lock for root. This default may be overridden in the configuration file with the 'set lock-file' command: set lock-file "/path/to/my/lock-file" Or disabled altogether by being set to the empty string: set lock-file "" The '-m' command line option may be used to force fdm to ignore the lock file and run regardless of its existence and without attempting to create it. %%% Testing and debugging fdm has some features to assist with testing and debugging a ruleset: The '-n' command line option. This is particularly useful in conjunction with '-vv', for example: $ cat test.conf account "pop3" pop3 server "s" user "u" pass "p" action "rw" rewrite "sed 's/\\(Subject:.*\\)\\[XYZ\\]/\1/'" action "mbox" mbox "%h/INBOX" match all actions { "rw" "mbox" } $ fdm -vvnf test.conf version is: fdm 0.6 (20061204-1433) starting at: Tue Dec 5 15:45:41 2006 user is: nicholas, home is: /home2/nicholas loading configuration from test.conf added account: name=pop3 fetch=pop3 server "s" port pop3 user "u" added action: name=rw deliver=rewrite "sed 's/\(Subject:.*\)\[XYZ\]/1/'" added action: name=mbox deliver=mbox "%h/INBOX" finished file test.conf added rule: actions="rw" "mbox" matches=all configuration loaded locking using: flock headers are: "to" "cc" domains are: "yelena" using tmp directory: /tmp Looking at the output, the parsed strings used by fdm can be seen, and it is possible to spot that an escape character has been missed in the command. If '-vvvv' is used, fdm will print all data sent to and received from remote servers to stdout. Note that this is disabled if the '-l' option is given, and includes passwords, usernames and hostnames unmodified. The 'fdm-sanitize' script provided with fdm may be used to remove passwords and usernames from this output, either while it is being collected: fdm -vvvv -a testacct f 2>&1|./fdm-sanitize|tee my-output Or afterwards: ./fdm-sanitize my-output Since fdm fetches multiple accounts simultaneously, which may intersperse debugging output, it is recommended to fetch each account seperately if running the output through fdm-sanitize. If this is not done, it may not be able to detect all usernames or passwords. The '-k' command line option (and the 'keep' keywords on actions and accounts, covered later) prevent fdm from deleting any mail after delivery. This may be used to perform any number of test deliveries without risk of losing mail. ### Fetching mail fdm fetches mail from a set of 'accounts', defined using the 'account' keyword. Each account has a name, a type, a number of account specific parameters and a couple of optional flags. The general form is: account [] [disabled] [] [keep] The item is a string by which the account is referred in filtering rules, log output and for the '-a' and '-x' command line options. The portion specifies the default users to use when delivering mail fetched from this account as root. It has the same syntax as discussed in detail in the section below on defining actions. If the optional 'disabled' keyword is present, fdm ignores the account unless it is specified on the command line using the '-a' flag. The optional 'keep' keyword instructs fdm to keep all mail from this account (not delete it from the server) regardless of the result of the filtering rules. The item may be one of: 'pop3', 'pop3s', 'imap', 'imaps', 'stdin', 'maildir' or 'maildirs'. %%% Mail tags As mail is processed by fdm, it is tagged with a number of name/value pairs. Some tags are added automatically, and mail may also be tagged explicitly by the user (see the later tagging section). Tags may be inserted in strings in a similar manner to macros, except tags are processed when the string is used rather than always as the configuration file is parsed. A tag's value is inserted by wrapping its name in %[], for example: match string "%[account]" to "myacct" action "myacctact" Most of the default tags have a single-letter shorthand which removes the needs for the []s: match string "%a" to "myacct" action "myacctact" Including a nonexistent tag in a string is equivalent to including a tag with an empty value, so "abc%[nonexistent]def" will be translated to "abcdef". The automatically added tags are: Name Shorthand Replaced with account %a The name of the account from which the mail was fetched. home %h The delivery user's home directory. uid %n The delivery user's uid. action %t The name of the action the mail has matched. user %u The delivery user's username. hour %H The current hour (00-23). minute %M The current minute (00-59). second %S The current second (00-59). day %d The current day of the month (00-31). month %m The current month (01-12). year %y The current year as four digits. year2 The current year as two digits. dayofweek %W The current day of the week (0-6, Sunday is 0). dayofyear %Y The current day of the year (000-365). quarter %Q The current quarter (1-4). rfc822date The current time in RFC822 date format. mail_hour The hour from the mail's date header, converted to local time, if it exists and is valid, otherwise the current time. mail_minute The minute from the mail's date header. mail_second The second from the mail's date header. mail_day The day from the mail's date header. mail_month The month from the mail's date header. mail_year The year from the mail's date header as four digits. mail_year2 The same as two digits. mail_rfc822date The mail date in RFC822 format. hostname The local hostname. folder The name of the IMAP folder that the mail originated from. In addition, the shorthand %% is replaced with a literal %, and %1 to %9 are replaced with the result of any bracket expressions in the last regexp (see later section on regexps). A leading ~ or ~user is expanded in strings where a path or command is expected. Some accounts add additional tags, discussed below. Tags are replaced in almost all strings (including those in single-quotes!), some when the configuration file is parsed and some when the string is used. %%% POP3 and POP3S Mail may be fetched from a POP3 account. A POP3 account is defined by specifying the following parameters: the server host and optionally port, and optionally the user name and password. If the port is not specified, the default port ('pop3' in the services(5) database) is used. If the user name, password, or both is omitted, fdm attempts to look it up the .netrc file, see the next section for details. Optionally fdm can read the password from a command line program, see below for details. Examples of a POP3 account definition are: account "pop3acct" pop3 server "pop.isp.com" user "bob" pass "pass" account "gmx" pop3 server "pop.gmx.net" port 110 user "jim" pass "pass" account "acct" pop3 server "10.0.0.1" port "pop3" user "nicholas" keep account "lycos" disabled pop3 server $localserver port 10110 pass "password" Note that the server string is enclosed in double quotes even if it is an IP, and don't forget to escape any " and \ characters in passwords! fdm will attempt to use APOP to obscure the password, if the server offers it. If the server advertises itself as supporting APOP but subsequently refuses to accept it, fdm will not retry with a cleartext password. Use of APOP can be disabled for an account using the 'no-apop' flag, for example: account "acct" pop3 server "server" user "bob" pass "pass" no-apop The 'starttls' keyword may be added to a POP3 account to attemp STARTTLS after connection. POP3S is specified in exactly the same way, except using the 'pop3s' keyword for the type, and the default port is 'pop3s' rather than 'pop3': account "pop3sacct" pop3s server "pop.isp.com" user "bob" pass "pass" POP3 accounts automatically tag mail with 'server' and 'port' tags, with the value of the server and port attributes exactly as specified in the account definition. A 'server_uid' tag is also added with the server unique id (UIDL). POP3 adds 'lines', 'body_lines' and 'header_lines' tags with the number of lines in the complete mail and its body and header. These tags are not updated to reflect any changes made to the mail by fdm rules. %%% SSL certificate verification fdm can verify SSL certificates before collecting mail from an SSL server. This is enabled globally with the 'verify-certificates' option: set verify-certificates And may be disabled per-account using the 'no-verify' keyword (this applies to both POP3S and IMAPS accounts): account "pop3sacct" pop3s server "pop.isp.com" no-verify For an introduction to SSL, see: http://httpd.apache.org/docs/2.0/ssl/ssl_intro.html A cert bundle is required to verify SSL certificate chains. For more information see: http://lynx.isc.org/current/README.sslcerts A pregenerated bundle is available courtesy of the MirOS project: http://cvs.mirbsd.de/src/etc/ssl.certs.shar %%% The .netrc file If the user name or password is omitted in POP3 or IMAP account definitions, fdm will attempt to look it up in the .netrc file in the invoking user's home directory. The .netrc file format is shared with ftp(1) and some other programs. It consists of a number of 'machine' sections and optionally one 'default' section containing a username ('login') and password for that host. fdm accepts entries only if the machine name matches the POP3 or IMAP server string exactly. If no matches are found and a 'default' section exists, it is used. An example .netrc file is: machine "my.mail-server.com" login "nicholas" password "abcdef" machine "pop.googlemail.com" password "pass1" default login "bob" password "moo" fdm will abort if the .netrc file is world-writable or world-readable. %%% Passwords from a command fdm can read the password from a command by using command substitution with $(). For example: user "fdm@example.com" pass $(gpg --quiet --decrypt ~/.password.gpg) %%% IMAP and IMAPS IMAP and IMAPS accounts are defined using exactly the same syntax as for POP3 and POP3S, aside from using the 'imap' or 'imaps' keywords and that the default port is 'imap' or 'imaps'. There is also an additional, optional 'folders' option to specify the folders from which mail should be fetched. If omitted, fdm defaults to the inbox. When the 'folders' tag is used the recieved mail will have a 'folder' tag with the origin folder name. Note that with IMAP and IMAPS, mail is still removed from the server unless the 'keep' option is given, or the '-k' command line option used. Examples of IMAP and IMAPS accounts include: account "imapacct" imap server "imap.server.ca" user "bob" pass "pass" account "oldimap" disabled imaps server "192.168.0.1" port 10993 user "nicholas" pass "pass" folders { "Saved" "MyStuff" } account "workspam" disabled imap server "my-work.ath.cx" user "Nicholas" folder "Junk" By default, fdm prefers the CRAM-MD5 authentication method, since no passwords are sent in the clear. If the server does not advertise CRAM-MD5 capability, and 'oauthbearer' option is not passed the older LOGIN method is used. For IMAPS connections (which use SSL), the LOGIN method is just as secure. Either of these methods may be disabled with the 'no-cram-md5' and 'no-login' options. If the server advertises OAUTHBEARER capability, the 'oauthbearer' option will use OAuth 2.0 bearer tokens - passed via the 'pass' keyword - as authentication method. The 'starttls' keyword may be added to an IMAP account to attemp STARTTLS after connection. As with POP3, IMAP adds the 'server', 'port', 'server_uid' and the three line count tags to mail. %%% IMAP or POP3 over a pipe or ssh Mail may be fetched using IMAP or POP3 via a pipe. This is particularly useful for fetching mail over ssh using public keys. For IMAP, a user and password may be supplied, but fdm will only use them if the server asks. If the connection is preauthenticated, the user and password are unnecessary. For POP3, a user and password must be supplied as usual: due to the lack of server name, it cannot be read from the .netrc file. Communication takes place via the pipe program's stdin and stdout. If any output is found on stderr, fdm will print it (or log it with '-l'). Examples are: account "imapssh" imap pipe "ssh jim@myhost /usr/local/libexec/imapd" account "imapssh2" imap pipe "/usr/bin/whatever" user "bob" pass "bah" account "pop3local" pop3 pipe "/usr/local/bin/ipop3d" user "me" pass "foo" %%% stdin and local mail fdm may be configured to fetch mail from stdin, by specifying an account of type 'stdin', for example: account "stdin" disabled stdin This is most useful to have fdm behave as a mail delivery agent. To configure it for single-user use with sendmail, the simplest method it to add: "|/usr/local/bin/fdm -m -a stdin fetch" To the user's ~/.forward file (including the double quotes). Note the use of '-m' to prevent stdin delivery from interfering with any normal cronjob, and '-a' to specify that only the disabled "stdin" account should be fetched. stdin accounts add the three line count tags described in the POP3 section. %%% From maildirs and mboxes Fetching from maildirs allows fdm to be used to filter mail on the local machine. This is covered more detail in the later section on archiving and searching. Maildir accounts are specified as follows: account "mymaildir" maildir "/path/to/dir" account "mymaildirs" maildirs { "/path/to/dir1" "/path/to/dir2" } Shell glob wildcards may be included in the path names to match multiple maildirs, but every directory found must be a valid maildir. Maildir accounts tag mail with a 'maildir' tag which is the basename of the maildir. Fetching from mboxes is similar: account "mybox" mbox "/path/to/mbox" account "mymboxes" mboxes { "/path/to/mbox1" "/path/to/mbox2" } Note that if an mbox is modified (mail is dropped from it), sufficient disk space is required to create a temporary copy of the entire mbox. %%% Using NNTP and NNTPS fdm can fetch news messages from a news server using NNTP or NNTPS. News accounts are specified like so: account "news1" nntp server "news.server.sk" port 119 group "comp.unix.bsd.openbsd.misc" cache "%h/.fdm.cache/%[group]" account "mynews" nntps server "ssl.news.server" port "nntps" user "myuser" pass "mypass" groups { "alt.test" "alt.humor.best-of-usenet" } cache "%h/.fdm.cache" The cache is a file used to store details of the last article fetched. If only one group is supplied in the account definition, %[group] tags are replaced by the name of the group in the cache path. If multiple groups are provided, %[group] is removed. Note that whether a message is kept or deleted is irrelevent to NNTP, articles are always left on the server. The index and message-id of the last article is recorded in the cache file so that older articles are skipped when the a newsgroup is again fetched. This happens regardless of any 'keep' keywords or the '-k' command line option. As with POP3 and IMAP, NNTP accounts add the 'server' and 'port' tags to mail. In addition, a 'group' tag is added with the group name. This can ensure articles are matched purely on the group they are fetched from (trying to do this using headers is unreliable with cross-posted articles). For example: match account "news" { match string "%[group]" to "comp.lang.c" action "news-%[group]" match string "%[group]" to "comp.std.c" action "news-%[group]" match all action drop } %%% New or old mail only With POP3 and IMAP, fdm can be set up to fetch only new or old mail. For POP3 this is achieved by recording the current state of the server in a cache file, which is updated as each mail is fetched. For IMAP it makes use of the 'seen' server flag which is updated by the server after each mail is fetched. These options are specified as in the following examples. For POP3: account "name" pop3 server "blah" new-only cache "~/.fdm-pop3-cache" account "acct" pop3s server "my-server" user "bob" new-only cache "my-server-pop3-cache" no-apop And for IMAP: account "imap" imap server "blah" new-only account "sslimap" imaps server "imaps.somewhere" user "user" pass "pass" old-only no-verify Note that currently, when using this with IMAP, the server is permitted to flag the mail as 'seen' before fdm has successfully delivered it, so there is no guarantee that mail so marked has been delivered, only that it has been fetched. ### Defining actions An action is a particular command to execute on a mail when it matches a filtering rule (see the next section on filtering mail). Actions are named, similar to accounts, and have a similar form: action [] The item may be either: - the keyword 'user' followed by a single username string or uid, such as: user "nicholas" user "1000" - the keyword 'users' followed by a list of users in {}s, for example: users { "1001" "nicholas" } If users are specified, the action will be run once for each user, with fdm changing to that user before executing the action. Note that fdm will execute the action once for each user even when not started as root, but will not be able to change to the user. The user keyword is primarily of use in multiuser configurations. If users are present on an action, they override any specified by the account definition. If running as root and no user is specified on either the action or on the filtering rule (see the section on filtering below), the default user is used, see the '-u' command line option and the 'default-user' option in the setting options section %%% Drop and keep The simplest actions are the 'drop' and 'keep' actions. They have no parameters and are specified like this: action "mydropaction" drop action "mykeepaction" keep The 'drop' action arranges for mail to be dropped when rule evaluation is complete. Note that using 'drop' does not stop further evaluation if the filtering rule contains a 'continue' keyword, and it may be overridden by a 'keep' option on the account or by the '-k' flag on the command line. The 'keep' action is similar to 'drop', but it arranges for the mail to be kept once rule evaluation is complete, rather than dropped. %%% Maildirs Mails may be saved to a maildir through a 'maildir' action, defined like so: action "mymaildiraction" maildir "/path/to/maildir" If any component of the maildir path does not exist, it is created, unless the no-create option is specified. Mails saved to a maildir are tagged with a 'mail_file' tag containing the full path to the file in which they were saved. %%% Mboxes An action to deliver to an mbox is defined in the same way as for a maildir: action "mymboxaction" mbox "/path/to/mbox" The same % tokens are replaced in the path. If the mbox does not exist, it is created. Mboxes may optionally be gzip compressed by adding the 'compress' keyword: action "mymboxaction" mbox "/path/to/mbox" compress fdm will append .gz to the mbox path (if it is not already present) and append compressed data. If the mbox exists but is not already compressed, uncompressed data will be appended. As with maildirs, if any component of the mbox path does not exist, it is created, unless the no-create option is set. Mails saved to an mbox are tagged with an 'mbox_file' tag with the path of the mbox. %%% IMAP and IMAPS An action may be defined to store mail in an IMAP folder. The specification is similar to the IMAP account definition. A server host and optionally port (default 'imap' or 'imaps') must be specified. A username and password may be supplied; if they are omitted, fdm will attempt to find a .netrc entry. Examples include: action "myimapaction" imap server "imap.server" action "myimapaction" imaps server "imap.server" port "8993" user "user" pass "pass" folder "folder" action "myimapaction" imaps server "imap.server" user "user" pass "pass" no-verify no-login %%% SMTP An action may be defined to pass mail on over SMTP. The server host must be specified and optionally the port and string to pass to the server with the RCPT TO and MAIL FROM commands. If the port is not specified it defaults to "smtp". Examples include: action "mysmtpaction" smtp server "smtp.server" action "mysmtpaction" smtp server "smtp.server" port 587 action "mysmtpaction" smtp server "smtp.server" port "submission" from "bob@server.com" action "mysmtpaction" smtp server "smtp.server" to "me@somewhere" %%% LMTP An action may be defined to store mail in a LMTP server. UNIX and TCP sockets are supported. Examples include: action "mylmtpaction" lmtp server "/path/to/socket" to "user@domain" action "mylmtpaction" lmtp server "hostname" port 24 to "user@domain" action "mylmtpaction" lmtp server "10.1.1.7" port 24 to "user@domain" %%% Write, pipe, exec and append Actions may be defined to write or append a mail to a file, to pipe it to a shell command, or merely to execute a shell command. The append action appends to and write overwrites the file. % tokens are replaced in the file or command as for maildir and mbox actions. Examples are: action "mywriteaction" write "/tmp/file" action "myappendaction" append "/tmp/file" action "mypipeaction" pipe "cat > /dev/null" action "domaildirexec" exec "~/.fdm.d/my-special-script %[mail_file]" Pipe and exec commands are run as the command user (by default the user who invoked fdm). %%% stdout fdm can write mails directly to stdout, using the 'stdout' action: action "so" stdout %%% Rewriting mail Mail may be altered by passing it to a rewrite action. This is similar to the pipe action, but the output of the shell command to stdout is reread by fdm and saved as a new mail. This is useful for such things as passing mail through a spam filter or removing or altering headers with sed. Note that rewrite only makes sense on filtering rules where the continue keyword is specified, or where multiple actions are used (see the next section for details of this). Possible rewrite action definitions are: action "myspamaction" rewrite "bmf -p" action "mysedaction" rewrite "sed 's/x/y/'" %%% Adding or removing headers Simple actions are provided to add a header to a mail: action "lines" add-header "Lines" value "%[lines]" Or to remove all instances of a header from mail: action "del-ua" remove-header "user-agent" action "rmhdr" remove-header "x-stupid-header" action "remove-headers" remove-headers { "X-*" "Another-Header" } %%% Tagging Mails may be assigned one of more tags manually using the tag action type. For example, match account "my*" action tag "myaccts" match "^User-Agent:[ \t]*(.*)" action tag "user-agent" value "%1" The tag is attached to the mail with the specified value, or no value if none is provided. %%% Compound actions Compound actions may be defined which perform multiple single actions. They are similar to standard single actions but multiple actions are provided using {}. For example, action "multiple" { add-header "X-My-Header" value "Yay!" mbox "mbox2" } action "myaction" users { "bob" "jim" } { rewrite "rev" maildir "%h/%u's maildir" } Compound action are executed from top-to-bottom, once for each user. Note that the effects are cumulative: the second example above would deliver a mail rewritten once to 'bob' and rewritten again (ie, twice) to 'jim'. If this is not desired, seperate actions must be used. %%% Chained actions An action may call other named actions by reusing the 'action' keyword: action "abc" action "def" action "an_action" { rewrite "rev" action "another_action" action "yet_more_actions" } There is a hard limit of five chained actions in a sequence to prevent infinite loops. ### Filtering mail Mail is filtered by defining a set of filtering rules. These rules tie together mail fetched from an account and passed to one or more actions. Rules are evaluated from top-to-bottom of the file, and evaluation stops at the first matching rule (unless the continue keyword is specified). The general form of a filtering rule is: match [] [continue] The optional item is specified as for an action definition. If users are specified on a filtering (match) rule, they override any specified on the action or account. The item is set of conditions against which the match may be specified, each condition returns true or false. Conditions are described in the next few sections. Aside from the 'all' condition, which is a special case, conditions may be chained as an expression using 'and' and 'or', in which case they are evaluated from left to right at the same precedence, or prepended with 'not' to invert their outcome. The item is a list of actions to execute when this rule matches. It is in the same list format: 'action "name"' or 'actions { "name1" "name2" }'. It may also be a lambda (inline) action, see the section below. If a rule with the 'continue' keyword matches, evaluation does not stop after the actions are executed, instead subsequent rules are matched. %%% Nesting rules Filtering rules may be nested by using the special form: match [] { match ... } If the conditions on the outer rule match, the inner rules are evaluated. If none of the inner rules match (or they all specify the 'continue' keyword) evaluation continues outside to rules following the nested rule, otherwise it stops. %%% Lambda actions Lambda actions are unnamed actions included inline as part of the filtering rule. This can be convenient for actions which do not need to be used multiple times. Lambda actions are specified as a combination of the rule and an action definition. For example: match all action maildir "mymaildir" match all actions { rewrite "rev" tag "reversed" } continue %%% The all condition The all condition matches all mail. Examples include: match all action "default" match all rewrite "rewaction" continue %%% Matched and unmatched The matched and unmatched conditions are used to match mail that has matched or has not matched previous rules and been passed on with the 'continue' keyword. For example, match "myregexp" action "act1" continue # This rule will match only mails that also matched the first. match matched action "act2" # This rule will match only mails that matched neither of the first two. match unmatched action "act3" %%% Matching by account The account condition matches a list of accounts from which the mail was fetched. It is specified as either a single account ('account "name"') or a list of accounts ('accounts { "name1" "name2" }'). fnmatch(3) wildcards may also be used. Examples include: match "blah" accounts { "pop3" "imap" } action "go!" match matched and account "myacc" action drop %%% Matching a regexp Matching against a regexp is the most common form of condition. It takes the following syntax: [case] [in headers|in body] The 'case' keyword instructs fdm to match the regexp case sensitively rather than the default of case insensitivity. The 'in headers' or 'in body' keywords make fdm search only the headers or body of each mail, the default is to match the regexp against the entire mail. Any multiline headers are unwrapped onto a single line before matching takes place and the process reversed afterwards. The regexp itself is an extended regexp specified as a simple string, but care must be taken to escape \s and "s properly. Examples include: match "^From:.*bob@bobland\\.bob" in headers and account "pop3" action "act" match ".*YAHOO.*BOOTER.*" in body action "junk" %%% Matching bracket expressions The results of any bracket expressions within the last regexp match are remembered, and may be made use of using the 'string' condition, or used to construct an action name, maildir or mbox path, etc. The bracket expressions may be substituted using the %0 to %9 tokens. For example, match "^From:.*[ \t]([a-z]*)@domain" in headers action "all" continue match string "%1" to "bob.*" action "bobmail" match "^From:.*[ \t]([a-z]*)@domain" in headers action "all" continue match all action "%1mail" This is particularly useful in combination with nested rules (see later): bracket expressions in a regexp on the outer rule may be compared on inner rules. Note that %0 to %9 are used only for 'regexp' rules. Regexps that are part of 'command' rules use the 'command0' to 'command9' tags. %%% Matching by age or size Mail may be matched based on its age or size. An age condition is specified as follows: age [<|>] [hours|minutes|seconds|days|months|years] If '<' is used, mail is matched if it is younger than the specified age. If '>', if it is older. The item may be a simple number of seconds, or suffixed with a unit. Examples are: match age < 3 months actions { "act1" "act2" } match age > 100 hours action "tooold" The age is extracted from the 'Date' header, if possible. To match mails for which the header was invalid, the following form may be used: match age invalid action "baddate" The size condition is similar: size [<|>] [K|KB|kilobytes...] Where is a simple number in bytes, or suffixed with 'K', 'M' or 'G' to specify a size in kilobytes, megabytes or gigabytes, such as: match size < 1K action "small" match size > 2G action "whoa" %%% Using a shell command Mail may be matched using the result of a shell command. This condition follows the form: [exec|pipe] returns (, [case] ) If 'exec' is used, the command is executed. If 'pipe', the mail is piped to the command's stdin. The is a simple string. % tokens are replaced as normal. Any of the or or both may be specified. The is a simple number which is compared against the return code from the command, the is a regexp that is matched case insensitively against each line output by the command on stdout. The result of any bracket expressions in the stdout regexp are saved as 'command0' to 'command9' tags on the mail. Any output on stderr is logged by fdm, so 2>&1 must be included in the command in order to apply the regexp to it. Examples: match exec "true" (0, ) action "act" match not pipe "grep Bob" (1, ) action "act" match pipe "myprogram" (, "failed") actions { "act1" "act2" } match exec "blah" (12, "^Out") action "meep" %%% Attachments There are five conditions available to filter based on the size, quantity, type and name of attachments. They are all prefixed with the 'attachment' keyword. Two compare the overall number of attachments: The 'attachment count' conditions matches if the number of attachments is equal to, not equal to, less than or greater than the specified number: match attachment count == 0 action "action" match attachment count != 10 action "action" match attachment count < 2 action "action" match attachment count > 7 action "action" The 'attachment total-size' condition is similar, but compares the total size of all the attachments in a mail: match attachment total-size < 4096 kilobytes action "action" match attachment total-size > 1M action "action" There are also three conditions which matches if any individual attachment fulfils the condition: 'any-size' to match if any attachment is less than or greater than the given size, and 'any-type' and 'any-name' which compare the attachment MIME type and name attribute (if any) using fnmatch(3): match attachment any-size < 2K action "action" match attachment any-type "*/pdf" action "action" match attachment any-name "*.doc" action "action" %%% Matching tags The existence of a tag may be tested for using the 'tagged' condition: match tagged "mytag" action "a" match tagged "ohno" and size >1K action drop Or the tags value matched using the 'string' match type (in a similar way to matching bracket expressions): match string "%[group]" to "comp.lang.c" action "clc" match string "%u" to "bob" action "bob" %%% Using caches fdm has builtin support for maintaining a cache of string keys, including appending to a cache, checking if a key is present in a cache, and expiring keys from a cache once they reach a certain age. These caches should not be confused with the NNTP cache file. Key caches are referenced by filename and must be declared before use: cache "%h/path/to/cache" cache "~/.fdm.db" expire 1 month If the expiry time is not specified, items are never expired from the cache. Once declared, keys may be added to the cache with the 'add-to-cache' action: match all action add-to-cache "~/my-cache" key "%[message_id]" Or removed with the 'remove-from-cache' action: match all action remove-from-cache "~/my-cache" key "%[message_id]" And the existence of a key in the cache may be tested for using the 'in-cache' condition: match in-cache "~/my-cache" key "%[message_id]" action "foundincache" Any string may be used as key, but the message-id is most often useful. Note that the key may not be empty, so care must be taken with messages without message-id (such as news posts fetched with NNTP). Caches may be used to elimate duplicate messages using rules similar to those above: $db = "~/.fdm-duplicates.db" $key = "%[message_id]" cache $db expire 2 weeks match not string $key to "" { match in-cache $db key $key action maildir "%h/mail/duplicates" match all action add-to-cache $db key $key continue } %%% Cache commands fdm includes a number of commands to manipulate caches from the command-line. These are invoked with the 'cache' keyword followed by a command. The following commands are supported: cache add cache remove These add or remove as a key in the cache . cache list [] This lists the number of keys in a cache, or in all caches declared in the configuration file if is omitted. cache dump This dumps the contents of the cache to stdout. Each key is printed followed by a space and the timestamp as Unix time. cache clear Delete all keys from a cache. Examples: $ fdm cache list /export/home/nicholas/.fdm.d/duplicates: 4206 keys $ touch my-cache $ fdm cache dump my-cache $ fdm cache add my-cache test $ fdm cache dump my-cache test 1195072403 ### Setting options fdm has a number of options that control its behaviour. These are defined using the set command: set