FD (File Distributor)

The FD is responsible to send and retrieve the files via the corresponding protocols. Currently only the FTP, FTPS, SMTP, HTTP, HTTPS and SCP protocol is implemented.

For sending files (push) FD waits for messages to arrive in the message FIFO ($AFD_WORK_DIR/fifodir/msg.fifo) from the AMG. It will then check if it may start the corresponding process sf_loc, sf_smtp or sf_ftp, so that the files can be distributed. Only 30 (MAX_DEFAULT_CONNECTIONS) processes may distribute files at one time. If not, it will store the message in a internal queue where the message will stay until a process slot is free. This queue will be sorted by priority and in what order the messages arrive from the AMG. It can be viewed with the tool queue_spy in the sbin directory. The output looks as follows:

   Message number  Pid    time        Pos  FC  FS         CP R    Message name
   100874222586459 30320  1120824695  576  1   114        0  29   77a6014/0/42ce6d77_fd3_0
   112071721003810 -2     1120717210  333  5   6560040    60 0    fff2e797/0/42ccc99a_17d_0
   112071925058200 -2     1120719250  333  5   6560040    0  -1   fff2e797/0/42ccd192_16bc_0
   112082387417220 -2     1120823874  333  20  26240160   0  -1   fff2e797/0/42ce6a42_6ba_0
   112082446033890 7928   1120824460  169  13  601756     0  6    21ed24b6/0/42ce6c8c_d3d_0
   112082457036540 1315   1120824570  169  11  288224     0  15   21ed24b6/0/42ce6cfa_e46_0
   112082461037750 1312   1120824610  169  11  614673     0  8    21ed24b6/0/42ce6d22_ebf_0
   112082469040390 29983  1120824690  1592 1   8192       0  65   a12e79ee/0/42ce6d72_fc7_0
   112082470040690 30695  1120824700  785  6   45792      0  49   4b1d8aac/0/42ce6d7c_fe5_0
   112082470040700 30705  1120824700  966  1   316        0  57   11a21170/0/42ce6d7c_fe6_0

FD is also responsible to collect files (only FTP, FTPS, HTTP and HTTPS) from a remote host. For this it makes a connection to the remote host at a regular interval or at certain configurable times and check if there are any files to fetch. Such files will be stored into a internal directory of the AFD which is monitored by AMG. Thus giving the full flexibility of the DIR_CONFIG file.

Get and Send File FTP

'sf_ftp' is the process responsible to transmit (push) files via FTP. It does so by using its own FTP-client that has build in a subset of the FTP protocol ( RFC 959 and RFC 2228). This has the advantage that the transfer of a file can be monitored byte for byte and not file by file, when using the client of the implementation. Another advantage is that, since the clients of different implementation differ, it makes the AFD more portable.

The process 'gf_ftp' is very similar to 'sf_ftp', the only difference being that it fetches files from a remote FTP-server (pull).

The FTP-clients of the AFD currently has the following features:

Do secure communication for both control and data connection using secure sockets (TLS/SSL).
Change the transfer type.
Change the remote directory.
Create the remote directory, recursively if necessary.
Rename a file.
Delete files on the remote site. This is useful when sending lock files.
Both active and passive mode are supported and the extended version.
Change the mode of a file that was transmitted.
Execute a command on the remote site for each file transmitted.
It appends a file on the remote site when the transmission was interrupted. This is very useful with very large files and buggy connections.
Get a 'dir' listing from the remote site.
To reduce latency it can combine the rename procedure RNFR and RNTO and/or not send the TYPE command by default.
Show all commands it sends and receives from the remote server in a log file (see Transfer Debug Log). This helps to locate errors very quickly when the transfer has failed.

Get and Send File SFTP

'sf_sftp' is the process responsible to transmit (push) files via SFTP. To connect to a remote system it uses the local ssh program, once the connection is setup it uses it's own client to do the communication according to the draft SSH File Transfer Protocol . This has the advantage that the transfer of a file can be monitored byte for byte and not file by file, when using the sftp client of the SSH implementation.

The process 'gf_sftp' is very similar to 'sf_sftp', the only difference being that it fetches files from a remote SSH-server (pull).

The SFTP-clients of the AFD currently has the following features:

Change the remote directory.
Create the remote directory, recursively if necessary.
Rename a file.
Delete files on the remote site. This is useful when sending lock files.
Change the mode of a file that was transmitted.
It appends a file on the remote site when the transmission was interrupted. This is very useful with very large files and buggy connections.
Get a 'dir' listing from the remote site.
To increase speed for writing it can send several write request at once.
Show all commands it sends and receives from the remote server in a log file (see Transfer Debug Log). This helps to locate errors very quickly when the transfer has failed.

Send File SMTP

This program is used to send mail via SMTP (RFC 821) and according to MIME (RFC 1521 and RFC 1522). With the attach file option it is possible to send binary files as mail. The encoding will be done in BASE64.

Send File Local

This process moves (when in the same file system) or copies the files from one directory to another. It is able to change the mode or owner of each file that it has copied.

Send File WMO

This program is only useful if files should be send with the WMO-socket procedure. Since WMO-sockets are insecure (data can be lost) it can be optionally configured to acknowledge each message.

Get and Send File HTTP

'sf_http' is the process responsible to transmit (push) files via HTTP. It does so by using its own HTTP-client that has build in a subset of the HTTP protocol ( RFC 2616 and RFC 2617). This also supports AWS4 S3 storage via HTTPS. Note, only sending via PUT is implemented.

'gf_http' is the process responsible to receive (pull) files via HTTP.

The HTTP-client of the AFD currently has the following features:

Do secure communication using secure sockets (TLS/SSL).
Is able to do chunk reading.
Change the remote directory.
Handle Apache and Nginx directory listing.
Delete files on the remote site.
Supports AWS4 S3 storage (GET + PUT).
Show all commands it sends and receives from the remote server in a log file (see Transfer Debug Log). This helps to locate errors very quickly when the transfer has failed.

Index

Home