The DIR_CONFIG file

The AMG expects a configuration by default in the file $AFD_HOME_DIR/etc/DIR_CONFIG. This default can be changed in $AFD_HOME_DIR/etc/AFD_CONFIG with the parameter DIR_CONFIG_NAME. By inserting multiple DIR_CONFIG_NAME lines in AFD_CONFIG, you split up the content of the configuration over several files. The content should conform to a certain set of rules. All entries are grouped in sections, a section begins with the line [section name] and continues until the next section starts. Supported sections names and the corresponding entries are listed below and must be set in this order:

Comments start with # and may be placed anywhere except directly behind the files and destination identifiers.

The directory entry [directory]

The directory in which the AMG has to search for files which have to be send. Each directory must begin with / or a ~ and this directory must exists or the AMG must have read and write access so it can create the directory recursively. Otherwise it ignores the complete entry for this directory. The number of directory entries in the DIR_CONFIG is unlimited. However duplicate directory entries will be ignored. If you wish to give the directory an alias name you can do so by putting the name directly behind the directory entry: [directory] <dir alias>. AFD is also able to fetch files from a remote host, the directory must then be specified in URL format as follows: <scheme>://[[<user>][;fingerprint=<SSH fingerprint>][:<password>]@]<host>[:<port>][/<url-path>][;type=<i|a|d|f|n>][;protocol=<protocol number>][;auth=<basic|digest|aws4-hmac-sha256>][;region=<region name>][;service=s3]

The items are separated by special characters :, /, @ and ;. If these characters appear within one of the items, they must be masked with a \.

See recipient for a more detailed description on how to use the fingerprint parameter and others.

There are currently five schemes for fetching files supported:

Scheme Description Example
ftp Normal FTP transfers. ftp://donald:secret@ducktown/directory
ftp[sS] FTP transfers over secure sockets (TLS/SSL). When ftps is used only the control connection is encrypted. With ftpS both control and data connection are encrypted. ftps://donald:secret@ducktown/directory
sftp SFTP, file transfer via SSH. sftp://donald:secret@ducktown/directory
http Normal HTTP transfers. http://donald@ducktown/directory
https HTTP transfers over secure sockets (TLS/SSL). https://donald@ducktown/directory
exec Execute some external command to retrieve data. exec://ext_fetch/fetch_command

For remote FTP, SFTP and HTTP directories it is possible to insert a date or current hostname with the following modifiers:

Feature Description Example [directory] entry Resulting directory name
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
ftp://daisy@ducktown/diary/%tY/%tm/%tH
 ftp://daisy@ducktown/diary/2006/04/01
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
ftp://goofy@hollywood/%td_%T+1d%td_%T0%td
 ftp://goofy@hollywood/28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
ftp://donald@daisy/archive/%h/todo
 ftp://donald@daisy/archive/ducktown/todo

If you wish to monitor many directories with the same set of recipients, you can use groups of directory names. A group can be specified with the sign & and a group name in brackets, either {} or []. In case the square brackets [] are used the group name will be looked up in $AFD_WORK_DIR/etc/group.list and will be inserted. Curly brackets make it look into the file with the name $AFD_WORK_DIR/etc/groups/directory/group_name. One can also use this group name to only insert part of the directory name. For example a directory entry specified as follows: 

      [directory]
      ftp://donald@daisytown/&[dirs_in_daisytown]

And with a corresponding entry in the $AFD_WORK_DIR/etc/group.list: 

      [dirs_in_daisytown]
      /daisy/archive
      /micky/data/house
      ~donald/backup

Will have the same result if the user had specified each directory on its own section [dir options], [files], [destination], [recipient] and [options]: 

      [directory]
      ftp://donald@daisytown//daisy/archive

         [dir options]
         do not report unknown files

         [files]
         *.txt

            [destination]

               [recipient]
               ftp://goofy@ducktown//data/messages
               mailto://dagobert@moneyhouse

               [options]
               exec compress %s

      [directory]
      ftp://donald@daisytown//micky/data/house

         [dir options]
         do not report unknown files

         [files]
         *.txt

            [destination]

               [recipient]
               ftp://goofy@ducktown//data/messages
               mailto://dagobert@moneyhouse

               [options]
               exec compress %s

      [directory]
      ftp://donald@daisytown/~donald/backup

         [dir options]
         do not report unknown files

         [files]
         *.txt

            [destination]

               [recipient]
               ftp://goofy@ducktown//data/messages
               mailto://dagobert@moneyhouse

               [options]
               exec compress %s

You will agree that the following via dir groups is much shorter: 

      [directory]
      ftp://donald@daisytown/&[dirs_in_daisytown]

         [dir options]
         do not report unknown files

         [files]
         *.txt

            [destination]

               [recipient]
               ftp://goofy@ducktown//data/messages
               mailto://dagobert@moneyhouse

               [options]
               exec compress %s

NOTE: The option [directory] must always start at the beginning of a line. There may not be any tabs or spaces in front.

The directory options [dir options]

With the options in the table below, it is possible to specify what the AFD should for example do with files that are in the directory but are not being distributed, how files are to be retrieved from a remote hosts, etc. [dir options] may only be specified once for each [directory] option. A full list of options for this optional section and the defaults is shown in the table below:

If none of the above is specified, it assumes 'yes'.
Option Description Local Dir Remote Dir Default
inotify <bit(s)> This option uses a feature of the operating system that notifies the application when some change occurs in the monitored directory. It is much faster than scanning the directory for new files. The following flags can be set:
Bit Name Description
1 Rename to For rename there are two events 'rename from' and 'rename to'. This just catches the 'rename to' inotify event. So this catches if a file is renamed from .name to name.
2 Close Catches the inotify close event. If the process closes the file will be caught by this event. Note that this is not save when files are being given to AFD, since a close event does not mean the process has finished writing. It could have crashed or the network connection went down. Only a rename event and in some cases a delete event are save.
4 Create Catches the inotify create event. The create event can be useful for files that are created instantaneously for example via touch.
8 Delete Catches the inotify delete event. The delete event is handled specially in AFD. It only triggers on files that start with a leading dot. This is useful where some programs do not use the atomic rename() function to rename a file.
16 Attrib Catches the inotify attrib event. This is useful to catch events such as if permission, time, number of links, owner/group and/or extended attributes change.
One can also combine the events. The following table lists some useful combinations:
Bit Name Description
3 Rename + Close Catches both files being renamed and files being created.
5 Rename + Create Catches both files being renamed and files being created.
Do not use this option together with: time, force reread, wait for, accumulate and accumulate size		 X    
delete unknown files [time in hours] Delete all files that are older then the number of hours specified with the <old file time> option and are not to be distributed. If the optional time in hours is specified it will override the value given in "old file time". If you specify a value of 0, it will be deleted after 120 (DEFAULT_TRANSFER_TIMEOUT) seconds. This can be overwritten by setting -1. Then the file will be deleted immediately. Only use the -1 if you really know what you are doing. X X  
do not delete unknown files The opposite of the above option. X X X
delete queued files [time in hours] Delete all files queued on input that are older then the number of hours specified with the <old file time> option and are to be distributed. If the optional time in hours is specified it will override the value given in "old file time". X    
delete old locked files <time in hours> Delete all old locked files on input that are older then the number of hours specified. X X not set
delete unreadable files [time in hours] Delete all unreadable files that are older then the number of hours specified with the <old file time> option. If the optional time in hours is specified it will override the value given in "old file time". If you specify a value of 0, it will be deleted immediately.   X  
report unknown files Any files found in this directory that are not being distributed and are older then the number of hours specified in the <old file time> option will be reported in the system log. X   X
do not report unknown files The opposite of the above option. X    
old file time <value in hours> The age of a file in hours before a file is being reported in the system log or being deleted. X   24
end character <decimal number> Sometimes some application may not be able to send files with a leading dot and then rename these files after they have send all data. With this option AFD will wait until the character specified with <decimal number> will appear as the last character in the file before it will grab the file. X    
create source dir When set and the directory does not exist, it will be created. This will override option CREATE_SOURCE_DIR in AFD_CONFIG. X X  
do not create source dir When set the source directory will not be created if it does not exist. This will override option CREATE_SOURCE_DIR in AFD_CONFIG. X    
important dir This marks the directory as important. This is useful when you have lots of directories that are to be monitored by the AMG and/or you have only a slow disk. The process dir_check will now check all important directories every DIR_CHECK_TIMEOUT (default: 30) second. This ensures that the important directories will at least be checked every DIR_CHECK_TIMEOUT second and not until all directories have been checked, which could take very long. X    
Option Description Local Dir Remote Dir Default
time <min hour dom month dow> With this option it is possible at what interval the directory is to be scanned. See time under options for a more detailed description of what each field means. There may be up to 12 (MAX_FRA_TIME_ENTRIES) time entries. X X  
timezone <zone> Here one can set the timezone for the above time option. When not set it will take the default timezone from the system. Valid timezone names are system dependent and can be found in /usr/share/zoneinfo, /usr/lib/zoneinfo, /usr/share/lib/zoneinfo, ... X X  
time external Setting this will allow one to trigger the scan event externally via the command 'afdcmd -R'. X X  
get dir list[ yes|no|href] This option is only for retrieving files via HTTP and can have one of the following parameters:
Parameter Description
yes AFD will try to get a directory listing and then automatically try detect the listing type. Note, it only knows some well know listing types from nginx and apache. If it is unable to detect it, it will switch to href automatically.
no This will disable the directory scanning for this directory. This can be used when directory listing are not possible or disabled because of security reasons. This also means that under [files] the full filenames without wildcards must be listed.
href There are many different types of HTML directory listings and AFD only knows a few. With href it will get nearly all href entries from the page, so one needs to set a correct filter in [files] to download the correct files. Note that it will not be able to differentiate between files or directories (or other special files). Only those href entries which reference the same server and protocol will be shown. To facilitate the creation of the correct [files] filter, there is a program
  X  
do not get dir list Setting this option will disable the directory scanning for this directory. This can be used when directory listing are not possible or disabled because of security reasons. Currently this only works for HTTP. This also means that under [files] the full filenames without wildcards must be listed, since the protocol does not resolve any wildcards.   X  
url creates file name This is an option for HTTP where the GET request can generate a new file name via the Content-Disposition Field. The difference to the above 'do not get dir list' option is that here the 'file name' or better request needs to be set as part of the directory information. Under [files] the user can then filter what he wants to distribute locally.   X  
url with index file name When the HTTP server serves a non standard index file name one can set this option and add the name at the end of the directory. For example in https://ducktown.org/dagobert/finance_index.html finance_index.html is this non standard index file with a directory listing.   X  
one process just scanning When a process scans a directory and finds files to retrieve it also will download the files. In some situation this is not optimal when you download large files on very slow connections and you want to download many files in parallel. This option tells fd to have one gf_xxx process for just checking if there are new files. Ensure that you have specified enough parallel connections, otherwise you will just have process scanning the remote directories but none downloading the actual files.   X  
do not remove By default when AMG will remove all files that it finds in the directory being scanned. With this option set it will NOT remove these files. X X  
store retrieve list [not exact|once|once not exact|append] When the above option <do not remove> is set, it can happen that always the same files are being collected by the AMG/FD. To avoid this situation this option can be set. Then AMG/FD will keep a list of files and their modification time and there size of all files found during the last scan. So only when a file has changed will it be retrieved by the AMG/FD. The following additional parameters are possible:
Extension Description
not exact Some protocols do not deliver the exact size and/or date when giving a list of file names. By exact we mean for size the unit byte and by time the unit second. For example many web servers just deliver time accuracy in minutes. In such a case AFD will try to send an additional command (HEAD (HTTP), MDTM (FTP), etc.) to get this value. However with lots of files in a listing and high latency networks, it can consume a lot of time before all the exact time (in second) and size (in bytes) are collected. In some cases this may not be necessary. When the file name itself contains enough redundant information, directory contend is cleaned, etc. In such a case with the 'not exact' extension to 'store retrieve list', AFD will not send these additional commands.
once With this option files will be fetched only once even if the size and/or date of the file changes.
once not exact Same as above only with the 'not exact' so no additional commands are send, to try and get the exact size (in byte) and date (in seconds).
append With append, it only fetches the new appended data. This can be useful if you want to retrieve for example log data that is always appended. It will only retrieve the new appended data and not the whole file.
X (not append) X  
ls data filename <name> By default the ls data filename will be that of the directory alias name. In some situations it may be useful if several directories use the same file. One such situation is if the directory has the %T modifier set in the directory path. If there are several directories with the same path but different times. X X  
Option Description Local Dir Remote Dir Default
priority <value> With this you can give the directory a priority. This option is only useful for remote directories, ie. from where the files have to be retrieved. 0 is highest priority and 9 is the lowest. X X 9
force reread [local|remote] Before AFD scans the directory for files it checks if the time has changed since the last scan. With this options set it will always read the directory. This can be useful for some broken filesystems that do not update this information correctly. Needless to say that this will create a higher system load. For remote directories this also disables inotify for the local remote directory. If you just want to force reread for the remote directory one can add the additional parameter remote. If neither local or remote is set it means for all directories, local and remote. X X not set
force copy Normally AFD tries to move files from the source directory into AFD's local pool directory. Sometimes this needs to be set on a global filesystem that does not handle atomic operations. In this case, one can set this option which forces AFD to copy the files into the pool directory. Normally it is not necessary to set this option, since copying has a much higher overhead. X   not set
max process <value> Here you may limit the number of process that the AMG may fork to handle files for this directory. AMG will only fork a process when using the exec option, or some other option that can use a lot of time to process. X X 10
max errors <value> The number of errors that may occur before this source/directory will become red in dir_ctrl dialog or if an action is configured this will activate this action. X X 10
do not parallelize Normally when AFD fetches files from a remote site it tries to parallelize this. However, if you have many directories on a remote host that are all fetched via the same alias, it can take very long before AFD fetches files from the other directories. If you set this option it does not parallelize the fetching of one directory/source.   X not set
ignore size <[=|>|<] value> If the files in the directory have the given size they will be ignored. A greater or less then sign may also be used to ignore files which have a size greater or less then the given value. The unit entered is in bytes. X X  
ignore file time <[=|>|<] value> If the files in the directory have the given difference of the current time and the files modification time, then it will be ignored. The unit is in seconds. A greater or less then sign may also be used. For example specifying 
             ignore file time < 60
means it will ignore any file whose modification time is less then 60 second. This option is best used together with 'force reread', because the directory is only scanned when something happens (rename, delete or added a new file) in the directory.		 X X  
max files <value> The maximum number of files that are copied before they are send to the FD. This value is set so that the AFD starts sending files as soon as possible even when there are thousands of files waiting in the source directory to be send. Otherwise it might take to long to process all the files at once and time is lost that could be used to transmit data. X X 100
max size <value> The maximum size in Kilobytes that are copied before they are send to the FD. X X 100 MB
accept dot files Setting this option will enable AFD to retrieve files starting with a dot from a remote host. When downloaded the dot will be automatically removed. Use the trans_rename option if you want to keep the dot.   X  
Option Description Local Dir Remote Dir Default
wait for <file pattern> With this option it waits for the given file before it sends all the other files from this directory. X    
warn time <time in seconds> When no data has arrived in the directory for the given time it will warn in the RECEIVE_LOG. X X 0 (disabled)
info time <time in seconds> When no data has arrived in the directory for the given time it will inform in the RECEIVE_LOG. X X 0 (disabled)
accumulate <number of files> Waits for the given number of files before it accepts files from this directory. X    
accumulate size <number of bytes> Waits for the given number of bytes before it accepts files from this directory. X    
dupcheck[ <timeout in secs>[ <check type>[ <action>[ <CRC type>]]]] Performs a duplicate on files that come into this directory. The options have the following meaning:
Option Description Default
timeout Time in seconds when this CRC value is to be discarded from the database. 3600
check type What type of check is to be performed, the following values are possible:
1 On filename only.
2 File content.
3 Filename and file content.
4 On filename without last suffix.
5 On filename and size.
1
action What action is to be taken when we find a duplicate. The following values are possible:
24 Delete the file.
25 Store the duplicate file in the following directory: $AFD_WORK_DIR/files/store/<id>. Where <id> is the directory id.
26 Only warn in SYSTEM_LOG.
33 Delete and warn.
34 Store and warn.
24
CRC type What type of CRC check is to be performed. The following values are possible:
16 CRC32
17 CRC32C. This CRC algorithm computes better checksums and if your CPU supports SSE4.2 calculation is done in hardware and is faster.
16
X X  
keep connected <time in seconds> The number of seconds that the connection should be kept open after retrieving all files. Note this can also be set via Keep connected in edit_hc dialog.   X 0 (disabled)
no delimiter In AWS4 HTTP retrieve it is possible to set a delimiter for listing directories which helps to reduce the volume of the listing to just the specified directory. If no delimiter and no path is set one will get the complete content of the bucket because AWS4 has a flat structure and does not know about directories. If a path is set everything in this path an all subdirectories will be listed. If this option is not set, AFD will set as default the / as delimiter.   X /
keep path If the above option no delimiter is set, it can be useful to set 'keep path' so that files in different directories but with the same name will not be overwritten. The directory delimiter / will be changed to \. With srename you can change the filename to a more meaningful one, if required.   X not set
local remote dir When AFD retrieves files from a remote host it stores them in $AFD_WORK_DIR/files/incoming. If the target is a local directory not in the same filesystem as $AFD_WORK_DIR, AFD must do a copy. To avoid this copy you can set this option to replace the $AFD_WORK_DIR part, to a directory in the same filesystem as the target. Then AFD will just do a move and will not copy the data of the file.   X $AFD_WORK_DIR
Option Description Local Dir Remote Dir Default

The files entry [files]

In this section it possible which files are to be distributed from the directory specified in the directory section. Each file name may contain the wild cards '*' and/or '?' in any order. One may also list characters enclosed in []. A pair of characters separated by a hyphen denotes a range expression; any character that sorts between those two characters is matched. If the first character following the [ is a ! then any character not enclosed is matched.

If the file entry starts with a '!' it means NOT this file. It is important to put these entries first otherwise the '!' option is meaningless. Here follow some more additional options that one can insert into the file filter:

Feature Description Example [files] entry Resulting filter
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
cloud_%tH-*-%tY.png
cloud_19-*-2006.png
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
*.%td_%T+1d%td_%T0%td
*.28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
*.%h
*.ducktown

The number file mask entries is unlimited, however the maximum length of a single entry should not exceed 256 (FILE_MASK_STEP_SIZE) characters. There is also no limit to the number of file groups.

To make the handling of large number of file mask entries easier, one can use file mask groups. The group is defined in the same syntax as those for directory entries, it begins with a & and a group name in brackets, either {} or []. In case the square brackets [] are used the group name will be looked up in $AFD_WORK_DIR/etc/group.list and the list of file mask entries will be inserted. Curly brackets make it look into the file with the name $AFD_WORK_DIR/etc/groups/files/group_name.

The destination entry [destination]

This marks the beginning of the destination section which contains one recipient and one options section. There is no limit in the number of destination entries per files section.

The recipient entry [recipient]

In the recipient section you specify the user name, password, remote host and remote directory in the following URL format:

<scheme>://[[<user>][;fingerprint=<SSH fingerprint>][;auth=login|plain;user=<user name>][:<password>]@]<host>[:<port>][/<url-path>][;type=<i|a|d|f|n>][;server=<server name>][;protocol=<protocol number>][;auth=<basic|digest|aws4-hmac-sha256>][;region=<region name>][;service=s3]

The items are separated by special characters :, /, @ and ;. If these characters appear within one of the items, they must be masked with a \.

If you do not want to write the password in clear text into the DIR_CONFIG you can set it via the program set_pw. If you have an external tool that generates the DIR_CONFIG automatically and you want to write the encoded passwords, ie. not in clear text, it can be done specifying the encoding with $<encode type>$ and then the encoded password (eg. $2$XRZ5WwxA). For the exact details please read the code or contact the author. Currently only 3 types are known: 0, 1 and 2. 0 is in clear plain text. 1 is the AFD internal version. And 2 is a simple XOR encryption with a key that you need to specify in $AFD_WORK_DIR/etc/.xor.key or by compiling it in with the configure option --with-xor-key=KEY.

The fingerprint= is only valid for scheme scp and sftp. To get the fingerprint from a host with OpenSSH enter the following command: 

                ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub

This will generate the fingerprint as shown in the following example: 

     1024 11:ff:22:ee:33:dd:44:cc:55:bb:66:aa:77:99:88:88 /etc/ssh/ssh_host_rsa_key.pub

This fingerprint is then added as follows in the recipient line: 

     sftp://donald;fingerprint=11-ff-22-ee-33-dd-44-cc-55-bb-66-aa-77-99-88-88:secret@hollywood//home/user

Another valid way of writing this is as follows: 

     sftp://donald;fingerprint=ssh-rsa-11-ff-22-ee-33-dd-44-cc-55-bb-66-aa-77-99-88-88:secret@hollywood//home/user

Note that the : are replaced by a minus -.

It is not necessary to specify the real hostname or IP number for host. In fact it might be much more user friendly to use an alias name here, since this will be the name displayed in the afd_ctrl dialog. This alias name is then resolved via the HOST_CONFIG file. However, the host must always be supplied.

When scheme is sftp or scp one can use the option protocol= to specify the SSH protocol version. For scheme https with AWS4 (auth=aws4-hmac-sha256) this can be used to set the list object version to either 1 or 2. 2 is the default.

The server= option is only useful for the mailto scheme. With this option it is possible to set the mail server address. If it is not set the default is localhost (SMTP_HOST_NAME in protocols/smtpdefs.h). If it is set this will be the hostname that AFD will take as alias/real hostname.

The type= option has the following meaning:

character scheme Description
i ftp[s|S] Does no modification to the data and sends as data representation TYPE IMAGE.
  wmo In WMO header a BI is inserted to indicate binary message.
a ftp[s|S] Inserts a linefeed (decimal 13) to all newline characters (decimal 10), if the linefeed does not exist. As data representation TYPE ASCII is send.
  wmo In WMO header a AN is inserted to indicate ASCII message. Note AFD does not check if the following data does not have any binary characters.
d ftp[s|S] Inserts a linefeed (decimal 13) to all newline characters (decimal 10), if the linefeed does not exist. As data representation TYPE IMAGE is send.
f wmo In WMO header a FX is inserted to indicate FAX message. Note AFD does not check if the following data is a fax.
n ftp[s|S] Does no modification to the data and sends no data representation TYPE.

The auth= at the end can be used to specify what authentication method to use. It is only valid for scheme http and https and can be set to basic, digest or aws4-hmac-sha256. basic is the most common used by HTTPS servers and is the default when no value is set. This is only useful if you need to authenticate to the server. aws4-hmac-sha256 can be used to connect to a S3 storage server. Via service= you can specify the service. The only service currently implemented is s3 which will be the default. A specific region for the storage can be set via region=. If none is set the default is eu-central-1.

Another useful feature for the mailto scheme is the possibility to group several mail addresses into one group. Under recipient it is then only necessary to specify one group name and not each recipient each on a separate line. This can be done as follows: 

         [recipient]
         mailto://$hotels@HOTELS

You then need to create a file in $AFD_WORK_DIR/etc/group.list with a list of mail addresses as shown in the following example: 

         [hotels]
         manager1@hotel1.com
         manager2@hotel2.com
         manager3@hotel3.com
         manager4@hotel4.com

In this case it is also possible to just write mailto://$hotels. In the afd_ctrl dialog it will then appear as hotels. The group name may not be longer then 40 characters.

The directory name may contain modifiers for which AFD will insert the current or a modified time, the current host name where AFD is running and/or the output of some program executed. The modifiers are as follows:

Feature Description Example [recipient] entry Resulting directory name
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
ftp://daisy@ducktown/diary/%tY/%tm/%tH
 ftp://daisy@ducktown/diary/2006/04/01
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
ftp://goofy@hollywood/%td_%T+1d%td_%T0%td
 ftp://goofy@hollywood/28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
ftp://donald@daisy/archive/%h/todo
 ftp://donald@daisy/archive/ducktown/todo
%e`<command to execute>` Insert the output of the executed command. 
ftp://donald@daisy/archive/%e`date "+%Y-%m-%d"`/todo
 ftp://donald@daisy/archive/2013-11-24/todo

There are currently seven schemes supported:

Scheme Description Example
ftp Normal FTP transfers. ftp://donald:secret@ducktown/directory
ftp[sS] FTP transfers over secure sockets (TLS/SSL). When ftps is used only the control connection is encrypted. With ftpS both control and data connection are encrypted. ftps://donald:secret@ducktown/directory
mailto Sending files via SMTP. mailto://daisy@ducktown
file Copies files to another directory. file://local/directory
exec Calls an external program to transmit the file to another destination. exec://printer/lpr -Pmy_printer %s
wmo Sends files via TCP according to WMO regulations at port 1111. wmo://@wmo-host:1111
sftp Sends files via SSH using the SFTP protocol. sftp://donald:secret@ducktown/directory
scp Sends files via SSH using the SCP protocol. scp://donald:secret@ducktown/directory

Each recipient section may have an unlimited number of recipients and each may not be longer then 256 (MAX_RECIPIENT_LENGTH) characters.

The options entry [options]

For each destination section you may specify one options section and may contain up to 15 (MAX_NO_OPTIONS) options each 256 (MAX_OPTION_LENGTH) characters long. All options specified here count for all recipients that are specified in the recipient section. Valid options can be taken from the following table:

Option name Description AMG FD ftp mailto file wmo sftp scp exec
priority x The priority of this job. Valid entries for x are 0-9, 0 is the highest priority and 9 is the lowest. (default 9) X X X X X X X X X
delete Deletes all files. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
time <min hour dom month dow>

This option allows to enter cron like time jobs in the AFD. The five time and date fields in the time option specify when files are to be distributed, that is when the minute, hour, and month of year fields match the current time, and when at least one of the two day fields (day of month, or day of week) match the current time. The time and date fields are as follows:

time and date fields
minute
hour
day of month
month
day of week
allowed values
0-59
0-23
1-31
1-12
1-7

In the <day of week> field 1 is Monday and 7 is Sunday.

A field may be an asterisk, which always stands for all values in this field.

Ranges of numbers are allowed. The ranges consist of two numbers and a hyphen. There may not be any blanks! The specified range is inclusive. Example: "19-22"

Its also possible to list a set of numbers or even ranges. Examples: "1,2,4,15", "4-8,18-20"

Step sizes can be used after an asterisk. So to specify every five minutes, just use "*/5". They can also be used in combination with a range. For example, "0-30/5" would equal "0,5,10,15,20,25,30".

There may be more then one time option. The number of time option is not limited, its however limited to the total number of options 15 (MAX_NO_OPTIONS) that may be set per job.

X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
time no collect <min hour dom month dow> This option is the same as the above time option only that no files will be collected (these will be deleted) outside the given time frames. The number of time no collect option is not limited. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
timezone <zone> Here one can set the timezone for the above two time options. When not set it will take the default timezone from the system. Valid timezone names are system dependent and can be found in /usr/share/zoneinfo, /usr/lib/zoneinfo, /usr/share/lib/zoneinfo, ... X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
basename[ overwrite] Sends only the basename of the file. For example if you have the following file name xxxx.yyyy.zzzz, it will rename the file on the remote site to xxxx. If the optional parameter overwrite is not set and the rename would overwrite another file, it will add a minus (-) and a number to the end of the filename so that no files will be overwritten as follows: xxxx-0. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
extension[ overwrite] Removes only the extension from the file name. Taking the example file name from above, the file name on the remote site would be xxxx.yyyy. If the optional parameter overwrite is not set and the rename would overwrite another file, it will add a minus (-) and a number to the end of the filename so that no files will be overwritten as follows: xxxx.yyyy-0. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
prefix add xx Adds a prefix to the file name. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
prefix del xx Remove a prefix from the file name. If the AMG does not find this prefix, this option is ignored. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
toupper Convert all letters of the file name to upper case. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tolower Convert all letters of the file name to lower case. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
lchmod mode Changes the permission of each file before being distributed. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
mirror source Deletes all files from destination not found in source. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
srename <filter> <rename to>[overwrite] [with_path] Rename files as defined by the filter and 'rename to' part. The pattern possible here are as those specified in the etc/rename.rule file. If the overwrite option is NOT set, multiple files with the same name will not be overwritten. This is done by appending a minus (-) and a number to the file name that is renamed. When with_path is set it will add the source path to the original filename. The filter and 'rename to' part may not be longer then 256 (MAX_FILENAME_LENGTH) characters. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
rename <rule> [overwrite] [with_path] Rename files as defined in the 'rule'-section of the etc/rename.rule file. If the overwrite option is NOT set, multiple files with the same name will not be overwritten. This is done by appending a minus (-) and a number to the file name that is renamed. When with_path is set it will add the source path to the original filename. The rule name itself may not be longer then 80 (MAX_RULE_HEADER_LENGTH) characters. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
trans_srename <filter> <rename to> [primary_only|secondary_only|dupcheck[see dupcheck options]] Rename files on the remote site after the files have been transmitted, as defined by the filter and 'rename to' part. The pattern possible here are as those specified in the etc/rename.rule file. This renaming may include directory names, however the directories must exist on the remote site. If the additional option primary_only is specified and the system has a secondary host specified, the renaming will only occur on the primary system. When secondary_only is set, the renaming will only happen on the secondary node. The filter name an 'rename to' part itself may not be longer then 256 (MAX_FILENAME_LENGTH) characters. If dupcheck is specified it will perform a duplicate check on the renamed file. 
 
 
 
 X 
 
 X 
 
 X 
 
 
 
trans_rename <rule> [primary_only|secondary_only|dupcheck[see dupcheck options]] Rename files on the remote site after the files have been transmitted, as defined in the 'rule'-section of the etc/rename.rule file. This renaming may include directory names, however the directories must exist on the remote site. If the additional option primary_only is specified and the system has a secondary host specified, the renaming will only occur on the primary system. When secondary_only is set, the renaming will only happen on the secondary node. The rule name itself may not be longer then 80 (MAX_RULE_HEADER_LENGTH) characters. If dupcheck is specified it will perform a duplicate check on the renamed file. 
 
 
 
 X X X 
 
 X 
 
 
 
force copy Force files to be copied and not linked nor memory mapped (Good for copying files to /dev/null). 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
mirror source Will delete files from target directory if they are not found in the source directory. This will only work if the [dir options] store retrieve list is set! Also note currently the implementation is incomplete in that it only deletes from destination when it finds files to fetch from source. 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
chmod mode Changes the permission of each file distributed. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). 
 
 
 
 X 
 
 X 
 
 X X X
hardlink <newname_1>[ <newname_n>] Creates a hardlink with name <newname> of each file distributed. 
 
 
 
 
 
 
 
 
 
 
 
 X 
 
 
 
symlink <newname_1>[ <newname_n>] Creates a symlink with name <newname> of each file distributed. 
 
 
 
 
 
 
 
 
 
 
 
 X 
 
 
 
chown [user][:.][group] Changes the user and/or group ownership of files. If only a user name (or numeric user ID) is given, that user is made the owner of each given file, and the files group is not changed. If the user name is followed by a colon or dot and a group name (or numeric group ID), with no spaces between them, the group ownership of the files is changed as well. 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
create target dir[ mode] Creates the target directory recursively if it does not exist. This will not work when the option Fast cd is set, since the directory will be created during the cd command. The optional mode can be used to set the permission of created directory. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). If the option CREATE_TARGET_DIR is set to YES in AFD_CONFIG, then this option does not need to be set since then it is the default. 
 
 
 
 X 
 
 X 
 
 X 
 
 
 
do not create target dir If the option CREATE_TARGET_DIR is set to YES in AFD_CONFIG, then it can be overridden with this option. 
 
 
 
 X 
 
 X 
 
 X 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
tiff2gts Converts T4-coded TIFF files to T4-files having a GTS header and end. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
gts2tiff Converts T4-GTS files to TIFF-files. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
fax2gts[ <fax ID>]

Converts FAX to a GTS T4 file. The GTS header is taken from the filename, so this must be a typical "TTAAii CCCC YYGGgg" WMO header. Optionally one can specify the following fax ID's:

1 DFAX1062 1728 Pixels 100 - 2352 lines
2 DFAX1064 1728 Pixels 100 - 2352 lines
3 DFAX1074 3456 Pixels 100 - 2352 lines
4 DFAX1084 2432 Pixels 100 - 3120 lines
5 DFAX1099 4752 Pixels 100 - 3040 lines

If no fax ID is specified it will take by default 2.

 X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
grib2wmo[ CCCC] Converts GRIB to WMO files with "TTAAii CCCC YYGGgg" headers, SOH and ETX at start and end and 10 Byte ASCII length and type indicator. You can optionally supply the CCCC if you do wish to overrule the one from the GRIB. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
wmo2ascii wmo2ascii will remove SOH, ETX and the two carriage returns from the given WMO file. It will convert the following text from: 
          <SOH><CR><CR><LF>nnn<CR><CR><LF>
	  WMO header<CR><CR><LF>WMO message<CR><CR><LF><ETX>
to 
          WMO header<LF>WMO message<LF>
in addition <CR><CR><LF> will be changed to <LF>. The file name will not be changed.		 X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
afw2wmo afw2wmo will convert the old AFW format used at DWD to WMO bulletin format. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
assemble <type>[ <file name>] This options assembles all files to a single file. The original files will be deleted. To help extraction of the files a length indicator may be inserted. The following types can be assembled:
VAX Two byte length indicator.
LBF Four byte length indicator with low byte first.
HBF Four byte length indicator with high byte first.
DWD Four byte length indicator DWD standard, that is like HBF but with four NULL bytes at end.
MSS Four byte length indicator MSS standard.
WMO Eight byte ASCII length indicator plus two byte type indicator (00 - with SOH and ETX, 01 - no SOH and ETX) . This is the current WMO standard.
WMO+DUMMY Same as WMO above but with a dummy length indicator at end of file.
ASCII No length indicator. The data of the next file is always appended, no matter if it contains binary or ascii data.
The <file name> may contain a %n to insert a unique number four characters long (hexa decimal) and/or with the help of %tx insert the actual time, where character x may be one of the following: 
         a - short day "Tue",           A - long day "Tuesday",
         b - short month "Jan",         B - long month "January",
         d - day of month [01,31],      m - month [01,12],
         j - day of the year [001,366], y - year [01,99],
         Y - year 1997,                 R - Sunday week number [00,53],
         w - weekday [0=Sunday,6],      W - Monday week number [00,53],
         H - hour [00,23],              M - minute [00,59],
         S - second [00,60]
         U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC

With the help of the %T[+|-|*|/|%]xS|M|H|d time modifier it is possible to modify the current time. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier.

A counter with a fixed length can also be specified. If you want a fixed 5 digital counter, just specify it as: 

         %5d.xyz
The resulting name will then be: 
         00001.xyz
A hexadecimal number can be produced by setting %5x. If the length is not set, ie. one just uses %d or %x, the default field length will be 8.

If no file name is specified the default will be no_file_name. 		X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
convert <type> This options converts all files from one format into another. The original file will be overwritten with the new one. The following types can be converted:
sohetx[+nnn] Adds <SOH><CR><CR><LF> to the beginning of the file and <CR><CR><LF><ETX> to the end of the file. Optionally one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
wmo[+nnn] Just add WMO 8 byte ascii length and 2 bytes type indicator. If the message contains <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> at the end, these will be removed. Optionally one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetxwmo[+nnn] Adds WMO 8 byte ascii length and 2 bytes type indicator plus <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> to end if they are not there. Optionally one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetx2wmo0[+nnn] converts a file that contains many ascii bulletins starting with SOH and ending with ETX to the WMO standard (8 byte ascii length and 2 bytes type indicators). The SOH and ETX will be copied to the new file. Optionally one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetx2wmo1[+nnn] Same as sohetx2wmo0 only that the SOH and ETX will NOT be copied to the new file.
mrz2wmo Converts GRIB, BUFR and BLOK files to files with 8 byte ascii length and 2 bytes type indicator plus <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> to the end, for the individual fields.
iso8859_2ascii Converts ISO8859 to ASCII. Currently only the following conversions have been implemented: 
                ä (e4) -> ae    Ä (c4) -> Ae | AE | A E
                ö (f6) -> oe    Ö (d6) -> Oe | OE | O E
                ü (fc) -> ue    Ü (dc) -> Ue | UE | U E
                ß (df) -> ss
Please contact the author if more conversions are required.
unix2dos Converts text files UNIX format to DOS format.
dos2unix Converts text files in DOS format to UNIX format.
lf2crcrlf Converts a single <LF> or a <CR><LF> to a <CR><CR><LF>.
crcrlf2lf Converts <CR><CR><LF> to a single <LF>.
X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
extract [-b|-B|-c|-C|-d|-D|-f|-F|-H|-n|-N|-r|-R|-s|-S|-t|-T] <type> [<pattern>] This options extracts files with WMO bulletins into one file per bulletin and will delete the original file. Since it is not possible to automatically detect the file type and its length encoding, you to specify what type of file it is. To indicate the length of the bulletin there is either a two or four byte length indicator at the beginning of the following format:
VAX Two byte length indicator.
LBF Four byte length indicator with low byte first.
HBF Four byte length indicator with high byte first.
MSS Four byte length indicator MSS standard.
WMO Eight byte ASCII length indicator plus two byte type indicator. This is the current WMO standard.
WMO+CHK Eight byte ASCII length indicator plus two byte type indicator plus a check if the length is correct.
ASCII No length indicator. The start of a bulletin is marked by SOH (binary 1) and end is ETX (binary 3).
BINARY This option is only useful together with the -H option to cut away SOH (binary 1), WMO header and the end ETX (binary 3). This only works if there is only one data set.
ZCZC No length indicator. The start of a bulletin is marked by ZCZC and end is NNNN.
MRZ Special four byte length indicator used in the DWD. The resulting file name will have the following name: <data type>_<originator>_YYYYMMDDhhmmss_<hexadecimal counter>
GRIB Standard GRIB, however since there are so many different GRIB types and limited name space for WMO header, many GRIB's will overwrite each other.

The file name of the new file will always be the WMO-header, that is: TTAAii_CCCC_[<year><month>]YYGGgg[_BBB][_<station indicator>]-<CRC checksum> The year will be shown with century and month will always be shown as a two digit number. So March will be shown as 03. The year and month will only be added when the -d parameter is supplied.

The optional options have the following meaning:
      -b     extract individual reports
      -B     do not extract individual reports [default]
      -c     adds a CRC checksum to end of filename [default]
      -C     do not add a CRC checksum to end of filename
      -d     add full date to bulletin header in filename
      -D     do not add full date to bulletin header in filename
      -f     add the full bulletin name and the original filename
             where this bulletin was extracted from as the first
             line of the bulletin/report
      -F     the opposite of the above option
      -H     remove WMO header line including SOH and ETX
      -n     adds a four digit unique number to end of filename
      -N     do not add a four digit unique number to end of filename [default]
      -r     generate file name with report type
      -R     generate file name without report type [default]
      -s     add SOH and ETX if not there [default]
      -S     remove SOH and ETX
      -t     in report add additional header showing report type
      -T     in report do not show the additional header with report type [default]

If a pattern is supplied, only those files will be extracted that match the given pattern.

X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
exec [-d|-D|-l|-L|-s|-t] <command> Executes a command specified in <command> for each file to be distributed. The file name can be supplied with %s. There may be up to 10 places in the command where you can insert the file name. The optional options have the following meaning:
          -d                  Deletes the original file after executing
                              the command regardless if the command
                              was successful or not.
          -D                  When an error occurs during execution all
                              files will be deleted. This only works if
                              no %s is found in the command string
                              otherwise it will be ignored.
          -l                  This sets a lock so that only one process
                              can be executed at one time for each file.
                              If %s is being used the lock is only set
                              for the execution duration of that file,
                              this gives other jobs a chance but only
                              one can do the execution at one time.
          -L                  Some as above only that the lock is held
                              for the duration of all files when %s is
                              being used.
          -s                  On error save the original to the directory
                              $AFD_WORK_DIR/files/store/<creation time>_
                              <unique number>_<split job counter>
          -t <time in second> If after the given timeout the execution
                              has not finished the process will be killed.
X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
execd <command> Same as the exec option above, but deletes the original file after executing the command. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
execD <command> Again this is similar to the exec option and differs in that when an error occurs during execution that all files will be deleted. This only works if no %s is found in the command string, otherwise it will behave just like the exec option. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
pexec [-l|-L|-r|-t] <command> Executes a command specified in <command> for each file after it was distributed. The file name can be supplied with %s. There may be up to 10 places in the command where you can insert the file name. You may also specify the full new file name with the %n parameter. The files stored in the archive will NOT be effected by this option. The optional parameters have the same meaning as those above in the exec option. The following additional parameters only used for pexec are as follows:
          -r                  Execute the command in the target directory.
                              When this parameter is not set the command
                              is executed in an internal AFD directory.
                              Note that this only works with the scheme
                              file, ie. when the directory is local. Also
                              note that this command is executed after the
                              file has been renamed, so if another process
                              is using that file, both AFD and that process
                              work on that file at the same time.
       
 
 
 
 X X X X X X X
[u]lock <type>
DOT When transmitting file "xxxx.yyyy" it creates the file on the remote site as ".xxxx.yyyy". When the file has been completely transmitted it is renamed to its original name. This ensures that the remote user can read the file when it is complete.
DOT_VMS When transmitting file "xxxx.yyyy" it creates the file on the remote site as ".xxxx.yyyy". After transmission this file is renamed to "xxxx.yyyy.". This ensures that the remote user can read the file when the transfer is complete.
OFF No locking on the remote site.
LOCKFILE [file name] Before starting transmission of the actual data files, the lock file "CONNECTED______.LCK" is send or if given the file name after LOCKFILE is used. When all files have been send successful this file is removed again.
xyz. When transmitting file "xxxx.yyyy" it creates the file on the remote site as "xyz.xxxx.yyyy". This ensures that the remote user can read the file when the transfer is complete.
If this option is not set the default is lock DOT. If ulock is used instead of lock, a unique number is added to the end of the filename. If you for example set 'ulock DOT' and taking the above example name "xxxx.yyyy", it is first transmitted with the name ".xxxx.yyyy.zzz", where zzz is some unique number.		 
 
 
 
 X 
 
 X 
 
 X 
 
 
 
lockp .zz When transmitting file "xxxx.yyyy" it creates the file on the remote site as "xxxx.yyyy.zz". This ensures that the remote user can read the file when the transfer is complete. 
 
 
 
 X 
 
 X 
 
 X 
 
 
 
sequence locking When transmitting file "xxxx.yyyy" it adds the number of retries to the end of the file name "xxxx.yyyy-<retries>" during transmission. After the file was transmitted successful it is renamed to the original name "xxxx.yyyy". Note, when set appending will NOT be possible. This option is useful when the remote side has problems with still having this file open by some other process, since with every try it will open a new file with a different name. It will try to delete the previous before it opens a new one. This option can be combined with the other locking options. 
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
archive x[d|h|m|s] Tells the AFD to archive files to this user for x days if no unit [d|h|m|s] is specified. The meaning of the units is as follows:
d Days
h Hours
m Minutes
s Seconds
When no archive option is set, the default (0 = no archiving (DEFAULT_ARCHIVE_TIME)) is set. With afdcfg -A the system administrator can disable archiving completely for all jobs.		 
 
 
 
 X X X X X X X
age-limit x Files older then x seconds will not get send. They will be removed. X X X X X X X X X
ageing x Jobs land in a queue of process FD where they start ageing if the job cannot be distributed. Ageing means that it falls back in the queue, so that new files of the same job ID will be inserted before the older job. When the problem is solved and data flows again, new files will be distributed before old files. With this option one can influence how quickly a job ages. With a value of 0 one can disable ageing. 1 is very slow ageing of a job while the largest value 9 is the fastest ageing. If this is not set the default is 6. 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
 
 
match size After file is transmitted check the size of the destination file with that of the local file just send. If they do not match, show an error and then resend the file. 
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
silent not locked This silence the warning in receive log that a file has been send without locking. The warn sign is changed to a debug sign. 
 
 
 
 X X X X X X X
no log output When set each file name distributed will NOT be logged to OUTPUT_LOG. 
 
 
 
 X X X X X X X
file name is subject The subject of the mail is the file name. This can only be used when sending mails. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
file name is user[ <rename rule name>] The user name of the mail address (the first part up to the @ sign) is the file name. This can only be used when sending mails. The optional rename rule allows to filter or create the user name from the file name. For a detailed description of the renaming procedure see rename rule. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
file name is target[ <rename rule name>] The full recipient of the mail address is the file name. So the file name must be a fully qualified mail address. This can only be used when sending mails. The optional rename rule allows to filter or create the user name from the file name. For a detailed description of the renaming procedure see rename rule. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
mail header[ <full path to mail header file>|<rename rule name>] Is used to insert some default information at the top of the main body of the mail. If the option is used without the optional mail header file, it will take the file MAIL-host_alias in the $AFD_WORK_DIR/etc directory as the header file. If the optional mail header file does not start with a / it take this as a rename rule identifier. For a detailed description of the renaming procedure see rename rule. Note that in this case if the resulting file name does not start with a / it will search for this file in $AFD_WORK_DIR/etc/mail_header. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
attach file[ <rename rule name>] With this option it is possible to attach a file to a mail. Each file will be attached to a separate mail (one file per mail). The encoding will be done in BASE64 according to RFC 1521. If a rename rule is specified the filename will be renamed in the attachment. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
attach all files[ <rename rule name>] With this option it is possible to attach files to a mail. If more than one file is available they will all be attached to ONE mail. The encoding will be done in BASE64 according to RFC 1521. If a rename rule is specified the filename will be renamed in the attachment. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
show all group members When using groups the name of each member is not shown in the To: line. It will just show the group name. When this option is set it will show all group names. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
show no to line When using groups this writes no to line in the header. Most mail systems will then set 'undisclosed recipient'. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
subject <full path to subject file>[ <rename rule name>] Adds the subject in the <subject file> to the mail. It is possible to insert the current date or filename, for more details see the next option. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
subject "xxx"[ <rename rule name>] Adds the line xxx after the subject identifier to the mails subject. Note that only 7-bit ASCII characters are allowed in the subject. It is possible to insert the current date in any form with %tx where character x determines the time format which can be one of the following:
            a - short day "Tue",           A - long day "Tuesday",
            b - short month "Jan",         B - long month "January",
            d - day of month [01,31],      m - month [01,12],
            j - day of the year [001,366], y - year [01,99],
            Y - year 1997,                 R - Sunday week number [00,53],
            w - weekday [0=Sunday,6],      W - Monday week number [00,53],
            H - hour [00,23],              M - minute [00,59],
            S - second [00,60]
            U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC

With the help of the %T[+|-|*|/|%]xS|M|H|d time modifier it is possible to modify the current time. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier.

Additionally it is possible to add the filename with %s. If more then one file is send it will send a comma separated list with all the filenames. The maximum length of the subject line depends on the Transfer Blocksize and/or Transfer rate limit, whichever is larger. Should the subject exceed this length, the subject is just truncated. 

 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
from <mail address> This allows to change the From: line of a mail. This can also be set in AFD_CONFIG with DEFAULT_SMTP_FROM globally. One can also insert the current hostname, for example user.%H@ducktown.de would resolve to user.node1@ducktown.de if the hostname is node1. To insert the full hostname one can use %h. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
reply-to <mail address> This adds Reply-To: header to a mail with another mail address to which the recipient should reply. This can also be set in AFD_CONFIG with DEFAULT_SMTP_REPLY_TO globally. As explained in from above one can also use the %H or %h option to insert the current hostname. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
group-to <mail address> When using groups this writes the given mail address as to line in the header. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
hide all group members When using mail groups, this option can be used to hide all group members of the group (BCC). Only the first member of the group will be shown in the To: line. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
charset <character set> This adds a MIME header, sets the transfer encoding to 8bit and specifies the character set. It does NOT check if the character set specified is a valid one. Possible values could be: 
          utf-8          --> UTF-8
          iso-8859-1     --> Latin 1; Western European Languages
	  iso-8859-2     --> Latin 2; Slavic/Central European Languages
	  iso-8859-3     --> Latin 3; Esperanto, Galician, Maltese, Turkish
	  iso-8859-4     --> Latin 4; Estonian, Latvian, Lithuanian
	  iso-8859-5     --> Cyrillic
	  iso-8859-6     --> Arabic
	  iso-8859-7     --> Modern Greek
	  iso-8859-8     --> Hebrew
	  iso-8859-9     --> Latin 5; Turkish
	  iso-8859-14    --> Latin 8; Celtic
	  iso-8859-15    --> Latin 9; Western European Languages with Euro
       
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
encode ansi This converts ASCII to ISO8859 when mailing the content. Note not all characters are converted, mainly only the German characters are working. Please contact developer if you need more. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
file name is header The file name is send as WMO header. This can only be used when sending files via TCP according to WMO regulations. The file name must then be of the following format: TTAAii_CCCC_YYGGgg[_BBB]. 
 
 
 
 X 
 
 X X X X 
 
sequence numbering Puts in a three digit sequence number into the WMO header. 
 
 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
check reply According to the WMO protocol messages are send as a continuous stream without any form of handshaking. When this option is set AFD waits for an acknowledge for each message its sends. The acknowledge consists of the following 10 characters: 00000000AK for a positive acknowledge or 00000000NA for a negative acknowledge. 
 
 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
mode <active | passive> There two ways to communicate with another FTP server, either active or passive. Active mode is the more common one while passive is more secure. If no mode is specified the default will be active. NOTE: It's much better to use the HOST_CONFIG to set active or passive mode. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
login site <command> This option is used to execute the command once after login. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
site <command> This option is used to execute the command after each file is transmitted. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
file busy rename <character> Appends the given character when we fail to open the remote file due to some broken FTP server that will not allow to open it because a hanging child still has this file open. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
dupcheck[ <timeout in secs>[ <check type>[ <action>[ <CRC type>]]]] Performs a duplicate on all files for this job. The options have the following meaning:
Option Description Default
timeout Time in seconds when this CRC value is to be discarded from the database. 3600
check type What type of check is to be performed, the following values are possible:
1 On filename only.
2 File content.
3 Filename and file content.
4 On filename without last suffix.
5 On filename and size.
1
action What action is to be taken when we find a duplicate. The following values are possible:
24 Delete the file.
25 Store the duplicate file in the following directory: $AFD_WORK_DIR/files/store/<id>. Where <id> is the job id.
26 Only warn in SYSTEM_LOG.
33 Delete and warn.
34 Store and warn.
24
CRC type What type of CRC check is to be performed. The following values are possible:
16 CRC32
17 CRC32C. This CRC algorithm computes better checksums and if your CPU supports SSE4.2 calculation is done in hardware and is faster.
16 
 
 
 
 X X X X X X X
socket send buffer <buffer size> Specifies the socket send buffer size in bytes. 
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
socket receive buffer <buffer size> Specifies the socket receive buffer size in bytes. 
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
eumetsat <DestEnvId> This option creates the header and sub header for EUMETSAT according to "MSG Ground Segment Design Specification Volume F". You might also need to execute create_eumetsat_name to create the correct file name. Syntax of create_eumetsat_name is as follows: 
       create_eumetsat_name <file name>[ <rename rule name>]
       
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
restart <file> Append file on remote site. This option is for internal use only! 
 
 
 
 X 
 
 
 
 
 
 X 
 
 
 
Option name Description AMG FD ftp mailto file wmo sftp scp exec
Copyright © 1997 - 2024 by H.Kiehl
Holger.Kiehl@dwd.de
Last updated: 02.04.2024
[red dot]Index [red dot]Home