Re: "Push" mirroring of servers

1998-03-03 08:12:44
Sorry, coming back to this thread a bit late ...

On 27 Feb 98 at 19:15, Steve Pacenka wrote:

Regarding writing a FTP push-mirror, the most complexity seems to be in the
heterogeneity of the FTP servers.  Ones on different platforms send back
different formats of directory listings that have to be parsed to obtain
file timestamps (and sizes if those are used in comparison; not useful in
cross-platform text files).

I hit this issue last autumn when I started using mHonarc, and found 
no satisfactory tools available for my platform: I use KA9Q under DOS 
for most of TCP/IP stuff, because its so reliable and so flexibly 
configurable, though I use Winsock undr Win 3.1 for WWW stuff.

So I sat down and wrote something myself, in Turbo C (having since 
learnnt Perl, I'd use that if I was doing it again, but I haven't 
bothered with a rewrite).

Basically, my proggie (FTPmir) generates scripts for KA9Q's 
scriptable FTP client, which means that I can do all my mirroring 
unattended: in the early hours of the morning, a batch file updates 
the archives, FTPmir updates the scripts, and then KA9Q does all the 
transfers at off-peak phone rates.

FTPmir only processes sigle directories at a time, but by using 
multiple configs, it can generate scripts for each dir that needs to 
be mirrored.  Since my mHonarc directories only change monthly, 
that's fine by me.

I wrote it for a UNIX server with my homepages account, mnut it 
needed no modification when I pointed it at a commercial server, 
also UNIX.  Basically, its methodology is:

First invocation: just get directory listing

Then build script.  Files are compared by time and size: newer local 
files are always uploaded.  Other files are compared by size: 
different sizes cause a file to go into the update queue, and ASCII 
conversion issues are handled by speccing which extensions are to be 
treated as ASCII -- local files in that category are scanned to 
calculate an ASCII size before comparison with remote size.

Finally, it generates an upload script, with a cmd line flag to
enable deletion of redundant remote files.  The first and last tasks
in each script are retrieving a remote dir listing, so it is
self-correcting, though with some potential for lag (discussed in the 

I haven't found a scriptable FTP client for Windows, but if there is 
one, I'm sure that FTPmir could be easily adjusted to generate the 
appropriate scripts.

Anyways, binary and source code and some docs available to anyone who 
asks in private mail: as I say, its in Turbo C 2, but with ANSO 
compliance checking on, so it should (<g>) compile under other 


Claire McNab -- Claire(_at_)siberia(_dot_)demon(_dot_)co(_dot_)uk

<Prev in Thread] Current Thread [Next in Thread>