%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%deffont "normal" xfont "times"
%deffont "typewriter" xfont "courier new-r"
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%default 1 area 90 90, leftfill, size 2, fore "black", back "white"
%default 2 size 7, vgap 10, prefix " "
%default 3 size 2, bar "gray70", vgap 10
%default 4 size 5, fore "black", vgap 30, prefix " "
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%tab 1 size 5, vgap 40, prefix "  ", icon box "black" 40
%tab 2 size 4, vgap 40, prefix "      ", icon arc "black" 50
%tab 3 size 3, vgap 40, prefix "            ", icon delta3 "black" 40
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Agenda


%size 4
          Introduction
          Architecture of AFD
          Compiling
          Setting up AFD
          Graphical User Interface
%size 6

          Configuration by examples
       
%size 4
          Evaluating Log Data
          Performance and other considerations
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Configuration by examples


	Simple configuration
	Send one file to multiple destinations
	Renaming files locally and/or remotely
	Sending files via mail
	The exec option
	Working with priorities
	Configuring checks for duplicates

	Exercise
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Simple configuration

	copying/moving files from one directory to another locally
%font "typewriter", size 2.5
        [directory]
        /home/afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://local//home/afd/data/out

%font "normal"
	shorter form:
%font "typewriter", size 2.5
        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Send one file to multiple destinations


%font "typewriter", size 2.5
        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out1
                 file://afd@local/data/out2
                 sftp://afd@local/data/out3
                 ftp://ftponly:secret@local//home/afd/data/out4

                 [options]
                 create target dir
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Send different data to different destinations

%font "typewriter", size 2
        [directory]
        ~afd/data/in

           [files]
           out1*

              [destination]

                 [recipient]
                 file://afd@local/data/out1

           [files]
           out2*

              [destination]

                 [recipient]
                 file://afd@local/data/out2

           [files]
           out3*

              [destination]

                 [recipient]
                 sftp://afd@local/data/out3

           [files]
           out4*

              [destination]

                 [recipient]
                 ftp://ftponly:secret@local//home/afd/data/out4
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Selecting files from a directory


%font "typewriter", size 2.5
        [directory]
        ~afd/data/in

           [files]
           out1*

              [destination]

                 [recipient]
                 file://afd@local/data/out1

           [files]
           !out1*
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out2
                 sftp://afd@local/data/out3
                 ftp://ftponly:secret@local/data/out4
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Source directory on a remote system


%font "typewriter", size 2.5
        [directory]
        ftp://ftp.dwd.de/pub/afd

           [dir options]
           do not remove
           store retrieve list
           time */10 * * * *

           [files]
           Changelog
           afd-*.bz2
           patch-*.bz2

              [destination]

                 [recipient]
                 file://afd@local/data/afd
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Renaming files locally


%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out

                 [options]
                 rename local_simple


        $AFD_WORK_DIR/etc/rename.rule

        [local_simple]
        * XYZ
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Rename locally, complicated


%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out

                 [options]
                 rename local_complicated

        $AFD_WORK_DIR/etc/rename.rule

        [local_complicated]
        *_*_*_???  %?3%?2%?1_%*3_%*2_%*1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Rename locally, complicated (cont.)



%font "typewriter", size 5
         111_222_333_123
%pause

     *_*_*_???  %?3%?2%?1_%*3_%*2_%*1
%pause

         321_333_222_111
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Rename at target


%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/out

                 [options]
                 trans_rename tree
                 create target dir

        $AFD_WORK_DIR/etc/rename.rule

        [tree]
        *_*_*_*    */*/*/*
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Sending files via mail


	Always remember mail is not reliable and not save! Also note that by default AFD will try to send its mail to localhost. If you do not want this you can use the server=<mail server> parameter in the recipient string. Even better is to use the DEFAULT_SMTP_SERVER parameter in $AFD_WORK_DIR/etc/AFD_CONFIG file.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Simple example sending a picture via mail


%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in/pictures

           [files]
           *.jpg

              [destination]

                 [recipient]
                 mailto://Holger.Kiehl@dwd.de

                 [options]
                 attach file
                 subject "Picture %s"
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Example sending mail to any destination


%font "typewriter", size 3
  $AFD_WORK_DIR/etc/DIR_CONFIG             |  $AFD_WORK_DIR/etc/rename.rule
                                           |
  [directory]                              |  [mail_subject]
  ~afd/data/in/mail                        |  *\\ *\\ *       %*3
                                           |
     [files]                               |  [mail_target]
     *.txt                                 |  *\\ *\\ *       %*2
                                           |
        [destination]                      |  [mail_filename]
                                           |  *\\ *          %*1
           [recipient]                     |
           mailto://mail_out               |
                                           |
           [options]                       |
           subject "%s" mail_subject       |
           file name is target mail_target |
           attach file mail_filename       |
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Mail to any destination (continued)


So, if a file with the following name is put into the directory ~afd/data/in/mail:

%font "typewriter", size 3.5
       "afd.c Holger.Kiehl@dwd.de This is the AFD code."
%font "normal", size 5

Will then have to following effect:

%font "typewriter", size 3.5
        To         : Holger.Kiehl@dwd.de
        Subject    : This is the AFD code.
        Attachment : afd.c
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

The exec option

	This allows you to execute any command before AFD transmits the data. If one wants to resize the size of an image to a certain size one can use the following:

%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 file://afd@local/data/png

                 [options]
                 exec convert -geometry 848x480! %s resized_%s
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

The exec option (continued)


	The %s is the place holder for file name. If AFD picked up more then one file it executes the command for each file in a loop. The %s place holder can only be used 10 times for each exec line.

	If the user wants to put all file names at once on the exec command line, just use the * as shown below:

%font "typewriter", size 3.5
         [options]
         exec tar --remove-files -cf all_files.tar *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

The exec option (continued)


	It is possible to path arguments to the exec command:

%font "typewriter", size 3.5
       -d                    Deletes the original file after
                             executing the command regardless if
                             the command was successful or not.
       -l                    This sets a lock so that only one
                             process can be executed at one time
                             for each file.
       -t <time in second>   If after the given timeout the
                             execution has not finished the
                             process will be killed.
%font "normal"

	For more information see:

%font "typewriter", size 4
    http://www.dwd.de/AFD/html-en/dir_config.html#exec
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

The pexec option


	This allows you to execute any command after AFD transmits a file.

%font "typewriter", size 3
        $AFD_WORK_DIR/etc/DIR_CONFIG

        [directory]
        ~afd/data/in

           [files]
           *

              [destination]

                 [recipient]
                 ftp://ftp_only@radolan/data

                 [options]
                 pexec ssh user@somehost bin/register_radolan_rw.sh
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

File Locking

	When one sends data to another host it is important to understand that one must use some form of locking. If you do not consider this you will loose data! See the following diagram:
%center, newimage "../images/locking.jpg"
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

File Locking (continued)


	Each level must communicate with each other. That also goes for the last level, your application. Most common way to do this is via renaming files.

	AFD supports all forms of locking, here follow just a view:

%font "typewriter", size 4
          locked | unlocked |  AFD [options]
          -------+----------+---------------
          .*     | *        |  lock DOT
          tmp.*  | *        |  lock tmp.
          *_tmp  | *        |  plock _tmp
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Age limit


	If one does not reach a partner, for what ever reason (network not working, system down), it does not make sense to send data that is  old. For this reason it is possible to set the following:
%font "typewriter", size 3.5
                  [options]
                  age-limit 3600
%font "normal"

	The value here is in seconds. So in the above example if the age of the file is older then 1 hour it will be deleted.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Working with priorities

	To give some files preference over others because they are more important one can do this via the option priority and the priority number. 0 is the highest priority and 9 is the lowest. If no priority is specified the default is 9.
%font "typewriter", size 3.5

                  [options]
                  priority 3
%font "normal"

	Note that priorities within AFD have an aging effect. Each time AFD fails to transit a file for whatever reason it's priority decreases. The reason for this is that a single file does not block the transfer to a host because for example the file already exists and one may now overwrite files on this host. If one does not want this behavior one can disable this via the edit_hc dialog by setting "No aging jobs".
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Configuring checks for duplicates

	There are two ways to check for duplicate files, one is on input and is set via the option dup_check under [dir options]. And the other is on output either for a single host (via HOST_CONFIG) or for a single job by setting dup_check under [options].
	There are several ways one can do the check itself. The most common is on the file name. Doing it on the content of the file is also possible but takes longer and consumes more computer resources. The check itself is just a simple CRC32 checksum.
	If a duplicate is detected there are several things one can do, the most common is to delete the file.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Exercise


In this exercise we all need to work together. We have four virtual systems (ubuntu, redhat, fedora, suse). Each user must configure his AFD to send all data coming into the directory ~/data/wmo to the next 'higher' user.

That is usr1@ubuntu sends to usr2@ubuntu. And usr2 sends to usr3@ubuntu. The last ubuntu user then sends to usr1@redhat. Aim is to create a big loop so the last user on suse must send the file to usr1@ubuntu.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Exercise (continued)

The DIR_CONFIG entry could look as follows:

%font "typewriter", size 2.5
        [directory]
        ~/data/wmo

           [files]
           *

              [destination]

                 [recipient]
                 ftp://usr2:afdusr2@ubuntu/data/wmo

                 [options]
                 create target dir
%font "normal", size 3.5

Once configured, each user issues the following command:

%font "typewriter", size 2.5
         touch ~/data/wmo/<your user name>@<your hostname>
%font "normal", size 3.5

So usr1 on ubuntu issues the following command:

%font "typewriter", size 2.5
         touch ~/data/wmo/usr1@ubuntu