Description : AFD FAQ
Author : Holger Kiehl
email : Holger.Kiehl@dwd.de
Version : 0.2
Last Updated: 12 December 2018
Copyright : GPL
======================================================================
Table of Contents
1. General Information
1.1 What is AFD?
1.2 Where can I get AFD?
1.3 Will AFD run with Windows XXX?
1.4 What is the history of AFD.
2. Compiling
2.1 How do I compile AFD under XXX?
2.2 Security considerations
2.2.1 Single user
2.2.2 Group access
2.2.3 Any users
2.3 I get the following error when compiling:
2.3.1 Can't open include file 'X11/Xaw/StripChart.h'
2.3.2 Can't find library for -lXpm (only versions < 1.2.2)
2.3.3 /usr/bin/ld: cannot find -lXm
3. Installing
3.1 How do I install AFD?
3.1.1 Installing from source
3.1.2 RPM
4. Configuration
4.1 When starting AFD or one of its process I get the following
error:
4.1.1 ERROR : Failed to determine AFD working directory!
4.1.2 You are not permitted to use this program. [<username>]
4.1.3 ERROR : Cannot read DIR_CONFIG file : No such file or
directory
5. General usage
5.1 DIR_CONFIG/HOST_CONFIG
5.1.1 How do I activate the changes in the DIR_CONFIG or
HOST_CONFIG file?
5.1.2 The password has changed but it is still seems to use
the old one. What's wrong?
5.1.3 Do I always have to press the 'Reread HOST_CONFIG' and
'Reread DIR_CONFIG' button?
5.1.4 I have lost some or all the content of the DIR_CONFIG,
what can I do?
5.2 Operational questions
5.2.1 The network connection to ALL hosts is down. What should
be done?
5.2.2 The connection to one host is down and the job queue counter
is constantly increasing. What action is necessary?
5.2.3 When a connection to one host is down should I stop transfer
or stop the queue?
5.2.4 An error has occurred for connection XYZ. How do you know
what type of error has occurred?
5.2.5 The problem to a host has been solved, but it always takes
such a long time before it tries to send the files again.
How can one test immediately whether the problem is solved
or not?
5.2.6 There is no response or action at all. Even starting or
stopping the AMG or FD does nothing. What's wrong?
5.2.7 There are files in the directory, but AFD does not send
them.
6. Error Messages
6.1 FTP Error Messages
6.1.1 Disc quota exceeded.
6.1.2 Disc full.
6.1.3 The process cannot access the file because it is being used
by another process.
6.1.4 Failed to send password for user XXXXX.
6.1.5 Failed to send password for user XXXXX due to timeout.
6.1.6 Failed to send password for user XXXXX.
530-User XXXXX: can't change directory to YYYY
6.1.7 Failed to connect() to XXXXX : Connection timed out.
6.1.8 Failed to connect() to XXXXX : Connection refused.
6.1.9 421-Service not available, closing control connection.
6.1.10 Remote hang up.
======================================================================
1. General Information
1.1 What is AFD?
The Automatic File Distributor provides a framework for very flexible,
non-stop, log and debug-able delivery of an arbitrary amount of files to
multiple recipients as expressed in URLs (currently mailing and ftp
supported with the mailto://user@domain and ftp://user:password@host URL
conventions). AFD stands for Automatic File Distributor.
1.2 Where can I get AFD?
You can get information from the following locations:
https://download.dwd.de/pub/afd/stable/doc - Homepage
https://download.dwd.de/pub/afd - directory containing
the stable release of
AFD
https://download.dwd.de/pub/afd/development - directory with unstable
releases of AFD
https://download.dwd.de/pub/afd/RPMS - pre compiled RPMS for
some Linux Distributions
https://github.com/holger24/AFD - get or follow the lattest
development of AFD
1.3 Will AFD run with Windows XXX?
No. Currently there is some work to get AFD running under Cygwin.
1.4 What is the history of AFD?
In 1995 the German Weather Service (Deutscher Wetterdienst (DWD)) was
doing its international file exchange via scripts. It was soon realized
that this did cause a very high cpu load and was not very flexible.
For this reason it was decided to develop a file distribution system
that was to be very portable and easy to handle by a single person.
Here follow some important milestones of the AFD:
- 04.02.1996 First release and first operational use at DWD
(only for distributing files to international partners).
- 22.06.1998 Release of version 1.0.0 with a complete rewrite
of the process FD to handle the internal
communication between FD and AMG via FIFO's.
- 08.08.1999 Released 1.1.0. Included time job and process to
monitor other AFD's.
- 03.02.2000 AFD released under the GENERAL PUBLIC LICENSE (GNU).
- 24.09.2000 Released 1.2.0. AFD is now able to retrieve files
via FTP from a remote host.
----------------------------------------------------------------------
2. Compiling
2.1 How do I compile AFD under XXX?
Run ./configure to generate config.h and the various Makefiles.
./configure --help gives a list of possible options with slightly
longer descriptions in README.configure. See also 2.2 Security
considerations.
Some systems require unusual options for compilation or linking that
the `configure' script does not know about. You can give `configure'
initial values for variables by setting them in the environment.
You can do that on the command line like this:
./configure CC=c89 CFLAGS=-O2 LIBS=-lposix
Before compiling you might want set some other preferences in
src/afdsetup.h and/or src/init_afd/afddefs.h. But note, changing some
values here might make this incompatible with a previous AFD version
and AFD will no longer work when it is started. So best to leave these
values as they are. If you do change them it's best to run the following
command before starting AFD again:
afd -i
This will initialize the internal database of AFD. After this it will
be possible to run AFD with the new values.
2.2 Security considerations
Before compiling AFD you need to decide where you want to install
AFD and if you want other users to have access to your AFD. There
are currently three possibilities:
2.2.1 Single user
This is the default and you need to set no additional flags
with configure. Here only the user that has started the AFD
may have access and may control it.
2.2.2 Group access
Via this method all users of the same group have access and may
control the AFD. This access can be further restricted in the
$AFD_WORK_DIR/etc/afd.users file. To use this you need to start
configure with the option --enable-group_can_write.
2.2.3 Any users
To allow any user to have access you must set the flag
--enable-setuid_progs in configure. This will set the setuid flag
for some binaries. The access can be further restricted in the
$AFD_WORK_DIR/etc/afd.users file. It is not recommended to set
this if you install the binaries as root.
2.3 I get the following error when compiling:
2.3.1 Can't open include file 'X11/Xaw/StripChart.h'
HPUX for example does not include the Xaw libraries and headers.
You can either install these on your system or in
src/UI/Motif/Makefile.am comment out all occurrence of afd_load. Then
AFD will compile without the dialog afd_load, which is not required
for running AFD.
2.3.2 Can't find library for -lXpm (only versions < 1.2.2)
Most commercial unix vendors do not include the Xpm libraries.
so either install these on your system or comment out the line:
#define _WITH_XPM
You can savely do this and you will hardly know any difference.
2.3.3 /usr/bin/ld: cannot find -lXm (only versions < 1.3.0)
Either you do not have the Motif library or you need to supply
the correct path to it in Include.mk with the variable XLIBPATH.
On a Linux system this is for example set as follows:
XLIBPATH = -L/usr/X11R6/lib
If you do not have Motif install either OpenMotif or LessTiff.
----------------------------------------------------------------------
3. Installing
3.1 How do I install AFD?
3.1.1 Installing from source
In the source directory just issue the command 'make install' to
install the binaries in the ../bin and ../sbin directories.
3.1.2 RPM
This is the easies method, just issue the following command:
rpm -Uvh <RPM filename>
To create an RPM, the simples way is to issue the following
commands as root:
rpm -Uvh <SRPM filename>
cd /usr/src/redhat/SPECS # This is distribution dependent
rpmbuild -ba afd.spec
----------------------------------------------------------------------
4. Configuration
4.1 When starting AFD or one of its process I get the following error:
4.1.1 ERROR : Failed to determine AFD working directory!
Either the environment variable AFD_WORK_DIR is not set or you did
not supply the working directory of the AFD with the '-w' option.
Without it AFD will not run.
4.1.2 You are not permitted to use this program. [<username>]
The user <username> is not permitted to start the program. If you
want to give the user the permission to start this program add
the users name to $AFD_WORK_DIR/etc/afd.users and the programs he
may use.
4.1.3 ERROR : Cannot read DIR_CONFIG file : No such file or
directory
There is no DIR_CONFIG file in the $AFD_WORK_DIR/etc directory.
To run AFD you must have a valid DIR_CONFIG file!
----------------------------------------------------------------------
5. General usage
5.1 DIR_CONFIG/HOST_CONFIG
5.1.1 How do I activate the changes in the DIR_CONFIG or
HOST_CONFIG file?
You need to press 'Reread DIR_CONFIG' or 'Reread HOST_CONFIG' in
the afd_ctrl dialog under 'Control'. Or if you do not want to use
the graphical interface use udc to update the DIR_CONFIG or uhc
to update the HOST_CONFIG from the sbin directory.
5.1.2 The password has changed but it is still seems to use
the old one. What's wrong?
The message that is causing this error still has the old password.
You must manually change this yourself (this will hopefully be
fixed in Version 2.x.x). To get the message number look into the
TRANSFER_LOG, you will see the number right after the '#' sign
at the end of the error message. Now edit ths message with this
number in the message directory and then press the 'Retry'
button.
5.1.3 Do I always have to press the 'Reread HOST_CONFIG' and
'Reread DIR_CONFIG' button?
No, if you have made changes to both the DIR_CONFIG and HOST_CONFIG
file you just need to press the 'Reread DIR_CONFIG' button (or call
udc from the sbin directory).
5.1.4 I have lost some or all the content of the DIR_CONFIG,
what can I do?
There is a tool get_dc_data which when called without arguments
will print the whole DIR_CONFIG, as stored within AFD, to standard
out.
5.2 Operational questions
5.2.1 The network connection to ALL hosts is down. What should
be done?
No action is necessary. AFD will stop the queue those hosts where
a lot of jobs queue up and it will start them again as soon as the
network comes back. If unsure you can stop AMG so no more jobs will
be generated for FD. FD will still continue to try and distribute
files at regular interval's.
5.2.2 The connection to one host is down and the job queue counter
is constantly increasing. What action is necessary?
Actually no action is necessary. Since the AFD will itself stop
the queue for this host if it reaches any critical value (and it
will automatically reactivate the queue when everything is okay
again). But it also does not hurt when you stop the queue for
this host. But don't forget to turn it on again!
5.2.3 When a connection to one host is down should I stop transfer
or stop the queue?
It depends on how long the connection to host is down and on the
number of files that are to be generated. Stop the transfer if you
wish to interrupt the current transfers and/or you know this is
only a short interrupt. But be careful that not to many jobs queue
up (AFD will automatically stop the queue if this is the case and
start it again). Best thing is to just stop the queue. Or leave it
to AFD, it will automatically stop the queue and start it again.
5.2.4 An error has occurred for connection XYZ. How do you know
what type of error has occurred?
Look in the Transfer Log file and see what type of error it is. If
it says "Failed to connect." then it is a network error or the
remote host is down. Check the network to the remote host. With
other errors it is best to turn on the debug mode for this host
and evaluate the output in the Transfer Debug Log. (see also 6.1)
5.2.5 The problem to a host has been solved, but it always takes
such a long time before it tries to send the files again.
How can one test immediately whether the problem is solved
or not?
Press the retry button.
5.2.6 There is no response or action at all. Even starting or
stopping the AMG or FD does nothing. What's wrong?
Most properly the AFD was not shutdown correctly. Log out of the
afd_ctrl and log in again. Be patience this can take a while
(up to 30s) because every time you call the program afd it will
check whether there is an AFD already running. If not it will
restart the AFD. All other programs and shared memory areas
will be removed.
5.2.7 There are files in the directory, but AFD does not send
them.
There could be many reasons why the files are not being send:
- Make sure that there is a rule in the DIR_CONFIG to
transmit those files.
- The files in the source directory must have the correct
permission, ie. AFD must have read permission for those
files.
- Ensure that the input queue or transfer is NOT stopped
in the afd_ctrl dialog. The two LED's next to the hostname
and debug led must be green.
----------------------------------------------------------------------
6. Error Messages
6.1 FTP Error Messages
6.1.1 Disc quota exceeded.
The user may not write any more data into this directory since since
the quota is exceeded.
6.1.2 Disc full.
The disk on the remote host is full.
6.1.3 The process cannot access the file because it is being used by
another process.
Another process on the remote machine has the file we wish to send
open and the operating system does not allow the ftp daemon to open
the same file again.
6.1.4 Failed to send password for user XXXXX.
The password for user XXXXX is wrong. To correct this correct the
password in the DIR_CONFIG and read 5.1.2.
6.1.5 Failed to send password for user XXXXX due to timeout.
If this error message appears frequently then its very likely that
the remote machine is overloaded and not able to handle your
request. This is a common error on computers with NT.
If this message appears only once then the network connection is
broken.
6.1.6 Failed to send password for user XXXXX.
530-User XXXXX: can't change directory to YYYY
The directory YYYY for user XXXXX on the remote host, does not
exist or does not have the correct permissions.
6.1.7 Failed to connect() to XXXXX : Connection timed out.
The remote host is not reachable. Either the network to this host
is not functioning properly or the host is down. This seems to be
a very common error.
6.1.8 Failed to connect() to XXXXX : Connection refused.
Here we do reach the remote host via the network but there is no
FTP daemon running or listening at the expected port.
6.1.9 421-Service not available, closing control connection.
This error message can appear after we connected to the remote
host. It means that it is currently being shutdown and not able
to handle any further requests.
6.1.10 Remote hang up.
If you only receive this error it means that the remote host
refuses any connections to this port. It could be an indication
that the remote host is overloaded or is in big trouble.
If this error appears in conjunction with another error message,
for example: Failed to change directory to YYYY. Then the remote
host has just cut the connection, either because it is being
shutdown or has a very high load.