This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new!

Feature: Networking

Automate FTP with macros

By Mark Alexander Bain on October 05, 2007 (4:00:00 PM)

Share    Print    Comments   

Has it been a while since you used FTP from the command line? While there are decent GUI-based FTP clients (such as gFTP), you can automate operations with the command-line version and handle file transfers with no user interaction at all.

When you try connecting to an FTP server and specify a username, FTP won't do anything unless you type in your password, or specify it in the connection command, as in ftp bainm:myftppassword@ You can take advantage of this syntax to automate your FTP transfers using a pipe:

echo get testfile.txt | ftp://bainm:myftppassword@

Unfortunately, this doesn't work for every distro. For example, it's fine on SUSE but not on Red Hat or Debian. Therefore, you need a solution that works for all distros.

Create a file that contains the IP address or name of the machine that you want to access, your username, and your password, and save it as ~/.netrc:

login bainm
password myftppassword

With ~/.netrc in place, you can send a command to FTP without having to input a password manually:

echo get testfile.txt | ftp
Connected to
220 acamas.ilium FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.17) ready.
331 Password required for bainm.
230 User bainm logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
local: testfile.txt remote: testfile.txt
227 Entering Passive Mode (192,168,1,2,9,203)
150 Opening BINARY mode data connection for 'testfile.txt' (0 bytes).
226 Transfer complete.
221 Goodbye.

At this point, you may want to think about creating a shell function that can carry out data transfers for you -- for example:

function compileFTP {
cat <<STOP
prompt off
lcd $1
#Don't worry if the target directory already exists - FTP won't complain:
mkdir $2 
cd $2
mput *

You can then call the function with a command like:

compileFTP ~/webfiles webdirectory | ftp

FTP runs the commands in the function without you having to input a password. This means that you can automate any FTP operations that you need to carry out. For example, you might want to save the code from above into a file, make it executable with a command like chmod +x ~/FTP/updateWeb, then run that file daily at midnight using cron:

0 0 * * * ~/FTP/updateWeb

You're not limited to writing functions in shell scripts -- you can also write them for FTP. However, in that case, they're called macros.

If you want to write an FTP macro, you'll need to go back to the .netrc file, because that's where the macros are stored. The rules for writing a macro are simple:

  1. Start a new macro definition by using the macdef keyword.
  2. Write your FTP commands.
  3. End the macro definition by adding a blank line.

For example, you could add the following code to .netrc:

macdef updateWeb
prompt off
lcd /home/bainm/webfiles
mkdir webdirectory
cd webdirectory
mput *

Don't forget that blank line!

You can call a macro from the Linux command line, but in a slightly different manner than the way you call a shell function:

echo "$ updateWeb" | ftp

You have to include a $ in the command to be sent to FTP to tell FTP that the command is a macro. Now you can change the crontab entry accordingly:

0 0 * * * echo "$ updateWeb" | ftp

At this point, you're probably wondering about variables -- you don't want to have to create a new macro for every upload that you want to carry out. Fortunately, macros use variables the same way shell functions do. The macro, therefore, becomes:

macdef updateWeb
prompt off
lcd $1
mkdir $2
cd $2
mput *

You can now send the names of the directories to your macro rather than hard-coding them:

echo "$ updateWeb /home/bainm/webfiles webdirectory" | ftp

You can also create another macro that supplies the inputs to your generic one (remembering to use the $ to tell FTP that you're calling a macro):

macdef myWebUpdate
$ updateWeb /home/bainm/webfiles webdirectory

What we're doing here is defining the new macro by using the macdef keyword, then calling our original macro (updateWeb) and supplying it with the directories that we want to copy from and to (i.e. from /home/bainm/webfiles on the local machine to webdirectory on the server. You can now run the macro from the command line:

echo "$ myWebUpdate" | ftp

Of course, you're not going to throw away your favorite FTP GUI. However, with some command line magic, you can automate FTP and run it with no human intervention at all. You can go home each evening knowing that all your Web servers will be updated or all your files backed up -- all through the power of FTP macros and the .netrc file.

Share    Print    Comments   


on Automate FTP with macros

Note: Comments are owned by the poster. We are not responsible for their content.

Automate FTP with macros

Posted by: Anonymous [ip:] on October 05, 2007 04:17 PM
Another client that has some nice features (reverse mirroring) is lftp. I use this to automate website updates on a schedule:

lftp << EOF

My work would be much harder without it.


Re: Automate FTP with macros

Posted by: Anonymous [ip:] on October 07, 2007 04:16 PM
on lftp you can make a script
#using automated solution non interactive mode
#create file with rules
vi configfile
open -u username.password IP
lcd directory
cd directory
mirror -Rv
#Run the lftp with configuration file
lftp -f configfile


Automate FTP with macros

Posted by: Anonymous [ip:] on October 05, 2007 05:06 PM
You need to checkout lftp.


Why use *ftp when sftp exists?

Posted by: Anonymous [ip:] on October 05, 2007 05:08 PM
What does ftp offer that sftp does not?


incryption and tuning

Posted by: Anonymous [ip:] on October 05, 2007 08:01 PM
Maybe its been said allready yet may bare repeating as techknowledgy increases the mesage becomes the medium. This was demonstrated particularly by rap music making songs with lyrics that not only advertized the song but the proformer as well. There was a Mad Magazine spoof years ago about the market cycle of a performer beeing determined by how big of close up they got on a magazine cover. With broadcasting going digital, would this allow incryption methods analogous to a tunner allowing standard content as well different decoders for uncensored material perhaps rated like movies. These could solve drm conflicts and allow users to police and define content.


Automate FTP with macros

Posted by: Anonymous [ip:] on October 06, 2007 12:06 AM
may I direct your attention to rsync or unison


Automate FTP with macros

Posted by: Anonymous [ip:] on October 06, 2007 06:02 PM
I had to make an additional correction before downloading files:

[root@myhost ~]# echo get testfile.txt | ftp
Error - .netrc file not correct mode.
[root@myhost ~]# ls -l .netrc
-rw-r--r-- 1 root root 47 Oct 7 02:19 .netrc
[root@myhost ~]# chmod 600 .netrc


Automate FTP with macros

Posted by: Anonymous [ip:] on October 07, 2007 12:06 AM
The above solution gives the user access to the ftp regardless which script that is executed, and that might become a security flaw.

Another (better, imho) guide about this:


Why shell scripting? How about perl and NET::FTP?

Posted by: Anonymous [ip:] on October 09, 2007 12:47 AM
Perl with NET::FTP (standard included module) can be used for much more robust automation of ftp sessions.
If you prefer python there's ftplib (I'm not a python guy so I'm not sure how good that is.)


This story has been archived. Comments can no longer be posted.

Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya