This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: System Administration

Automatically process new files with fsniper

By Ben Martin on October 17, 2008 (9:00:00 AM)

Share    Print    Comments   

fsniper lets you monitor specified directories and execute scripts on any new files that are created in them. Because fsniper uses inotify to monitor its directories, the actions you define are executed as soon as filesystem changes happen. This makes fsniper both more immediate than an hourly cron job and more efficient.

One possibility that suggests itself when you think about automatically processing files as they are placed in a directory is to have some sort of classification of files that you download from the Web. In fact, this is the first example that the fsniper Web site gives.

Another use case presents itself in the context of security. Suppose you have a collection of media files that you share read-only on a file server. This security makes it impossible for a user to add new files to the server. With fsniper, you can share an "upload" directory and have scripts move files that are uploaded to it across to the read-only fileserver automatically. If the client machine has a problem it could write random files to the upload directory, but then you have control of new files in the upload directory with the scripts that fsniper calls. For example, your move scripts might check that the file names are sane and that the file types are valid, not allowing random files, only valid JPEG files. At the very least your move scripts might block overwriting files or check the old version of a file into a revision control system before overwriting it.

As an added bonus, because fsniper uses a script that you create to perform the file move, you can change the file protection, log the new file to a database, send an email, or update an RSS newsfeed so that interested parties know about the new file right away. The scripts used by fsniper can also use sudo and other features to execute actions on the files that the users themselves cannot perform, such as changing the user and owner of a file before moving it.

fsniper is not packaged for openSUSE 11, Ubuntu Hardy, or Fedora 9, but you can build it from source using the normal ./configure && make && sudo make install process. fsniper uses libmagic to determine what sort of data is in a file, so you'll need to first install the file-devel package if you're running Fedora 9 or openSUSE 11, or libmagic-dev for Hardy.

If you execute fsniper at this stage you will get an error message telling you that the configuration file ~/.config/fsniper/config was not found, and fsniper will exit. The fsniper tarball includes an example configuration file, example.conf, to help you get up and running. The format of the configuration file is nested scopes delimited by {} characters. The outermost scopes define watches. The first scope inside a watch defines the directory to watch, and the innermost scope defines match patterns that are executed whenever something changes in the directory that the watch is observing.

An example like the one below, from the example.conf file, may help visualize the pattern. First a watch scope is defined and the ~/drop scope gives the path that this watch will observe. Inside the ~/drop scope the image/* scope will be executed on files that have been written to and closed in the ~/drop directory and that have a MIME type starting with image/. To make things clearer I'll refer to the outermost scope as the watch scope, the next one in that defines the directory path the directory scope, and the innermost as the match scope.

watch { # watch the ~/drop directory for new files ~/drop { # matches any mimetype beginning with image/ image/* { # %% is replaced with the filename of the new file handler = echo found an image: %% } ... } }

The directory scope is not recursive by default, so if you created a new image file inside ~/drop/foo, the above match scope would not run on it. You can change this by setting recurse = true as the first setting for a directory scope.

The handler setting in a match scope defines what command to execute if the match scope succeeds. In the above example it displays the path on the terminal that you started fsniper in if any new images are found. fsniper replaces the %% variable with the path to the file that the watch found. You can also use %f if you just want the file name, and %d if you only want the path of the directory that contains the file. If your handler line does not include any of these special path characters, fsniper will append the path to the file as the final argument to whatever command you specify.

While the handler defined in the above match scope is a command to execute, you might like to use multiple commands in scripts and call them from your match scopes. This separates the functionality from the configuration and allows you to define more complex commands or use Perl scripts. fsniper prepends ~/.config/fsniper/scripts to your $PATH before executing the match scope, so you can use simple script names and not bother specifying the full path for each custom script you use, as long as you add new scripts to ~/.config/fsniper/scripts.

You might wonder why you define the command inside a handler setting instead of directly inside the match scope itself. It's because fsniper allows you to chain multiple handlers together for a match scope. If the return code of a handler is zero, then fsniper assumes that the handler has fully taken care of the file and does not execute any subsequent handlers inside the same match scope. If the return code is 2, fsniper assumes that the handler wants to handle the event but cannot do so right now. This is handy in cases dealing with networked computers, because the handler might need to contact another machine that is temporarily down. If the handler returns a value other than 0 or 2, fsniper executes the next handler in the match block. This way you can have a script that is very selective about the file it wants to handle, and if the file is not right the script can just return 1 to have fsniper use the next handler for this file.

The order you specify your handler lines in a match scope makes a difference, because the return code of any handler affects whether the next one is executed.The match scopes are also executed top to bottom in the order you specify them. Only one match scope is executed for an event, so it is best to place the most specific match scopes at the top of your directory scope and the most generic ones at the bottom.

The doc/doc.txt file includes information about using fsniper with a Firefox download directory. The complication is that Firefox creates both a zero-byte file and a file.part file which is used to contain the data as it is downloaded. You have to set up fsniper so that your script will be executed only when the file.part is renamed to file after the download is complete.

As a word of warning, I found that comments inside match scopes caused fsniper to crash on startup, and comments above match scopes caused fsniper not to execute the directory scope correctly. I tried this with tab-indented and non-indented comments; the only way to get the correct behavior was to strip out the comments completely.

With fsniper you can set up automatic file processing without any delays. As an added bonus, these processes can be executed with greater privileges than those the user who writes the files has. For example, you could create a special user on a file server that has the ability to change file ownership. In any event, file processing becomes a matter of following the rules once you have set up your fsniper directories and scripts.

Ben Martin has been working on filesystems for more than 10 years. He completed his Ph.D. and now offers consulting services focused on libferris, filesystems, and search solutions.

Share    Print    Comments   

Comments

on Automatically process new files with fsniper

Note: Comments are owned by the poster. We are not responsible for their content.

Automatically process new (OR CHANGED) files with fsniper

Posted by: Anonymous [ip: 66.65.48.101] on October 17, 2008 08:14 PM
From the documentation:
...
fsniper is a utility that waits for a file to be changed or created,
and then executes a command on that file.
...
fsniper uses inotify to watch for when a file is closed after being written
to. This means utilities such as touch will cause the event to trigger.
...


The utility is more useful than you mentioned in the article. This utility can also allow users to
automatically backup files that have changed, such as source code or /etc configuration files.
Please be more thorough in your description of the features.

#

Automatically process new files with fsniper

Posted by: Anonymous [ip: 168.105.119.64] on October 18, 2008 01:52 AM
what happens if a file is being used while processed by this application? I mean for example a backup that takes some seconds/minutes to finish.

#

Re: Automatically process new files with fsniper

Posted by: Anonymous [ip: 66.65.48.101] on October 18, 2008 05:51 PM

Re: Automatically process new files with fsniper

Posted by: Andrew Yates on October 18, 2008 07:29 PM
The handlers are only called after a file has been opened for writing and then closed, so fsniper will not call a handler on your backup file until the file writing the backup has finished and closed the file.

If you somehow create a situation in which the backup file is closed and then opened again, you would want to check for that condition in your handler so that the file is only handled when appropriate.


#

fsniper with torrents

Posted by: Anonymous [ip: 75.71.17.31] on October 18, 2008 05:04 PM
I use fsniper for fully automated torrent management and a few other odds and ends, it's great. On my desktop, .torrent files that I download from Firefox will automatically mv to ~/.torrents where they are picked up by rtorrent. On my laptop, (with ssh keys authorized for no password) .torrent files scp to my desktop:~/.torrents where then, rtorrent picks them up. The result is queueing a torrent to start downloading is a one-click process from wherever I am.

#

Comment Bug

Posted by: Andrew Yates on October 18, 2008 07:49 PM
As a word of warning, I found that comments inside match scopes caused fsniper to crash on startup, and comments above match scopes caused fsniper not to execute the directory scope correctly. I tried this with tab-indented and non-indented comments; the only way to get the correct behavior was to strip out the comments completely.

This was a regression in v1.3 caused by some new config validation code. It should be fixed as of v1.3.1, which was released earlier today.

#

Automatically process new files with fsniper

Posted by: Anonymous [ip: 64.126.190.119] on October 19, 2008 12:41 AM
Also see incron
www.debian-administration.org/articles/581

#

Useful for little tasks, too.

Posted by: Anonymous [ip: 68.227.113.156] on October 19, 2008 06:40 AM
The first thing that I used this for, was automatically chmod-ing all files in ~/bin to be executable. May be a bit simple, but it's a time saver, as I only keep executable scripts in there anyway. Before, I had been doing this via .bashrc which would autmatically chmod those files on startup. This way seems much more clean.

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya