As Jeremy and I get more down the road of finishing our book on Python for *NIX systems administration, working title, we are both going to start dishing out some meatier nuggets of Python that we expose in our book. I have a background in Feature Film Animation Pipelines, and I thought I would share an interesting Python module that Python programmers in the Animation world might enjoy. The module is called pyinotify, and it “monitor’s filesystem events with Python under Linux”. Sound cools right, well, it is, so lets get a little background.

Inotify went into the linux kernel in release 2.6.13, and, according to Wikipedia, “Inotify uses an API that uses minimal file descriptors, allowing programmers to use the established select and poll interface”, in plain english it notices changes to the filesystem and reports those changes to applications. In animation pipelines, generally, files need to make a round trip between CGI, and Editorial. The Animators are given a scene to create, and very large, raw image files, like 16 bit TIFFs, are created. The files generally need to be processed in many different ways depending on where those images need to go. They may need to be converted into a HD Quicktime Movie for viewing on a 2K or 4K, stands for thousands of pixels, digital projector, or they may need to be converted into a compressed format like MXF, or DNxHD, a variant on the MXF file format that Avid has developed. If the files are to get sent to the editorial department for editing, then they will need to be processed with a timecode, or keykode, value embedded into the metadata of the file.

Embedding either a timecode, or keykode, is mandatory, as it allows image files to be assembled automatically according to a shot list, or else they would need to be visually arranged by hand, which just wouldn’t work in an animation pipeline. One of the reasons why it wouldn’t work, is that the image files could be cut out of a scene, and it would be an incredible waste of money to animate a frame or many frames, that were cut out in editorial. This is just one of the reasons why embedded metadata in files is so important in animation. Of course things need to be kept track of, and that is why using something like SQLAlchemy could make a lot of sense. I am actually building a metadata management system into open source tool I am creating called Liten, and, when I get more time, it will use SQLAlchemy.

All of this rambling background material, leads me back to Python, and Pyinotify. Pyinotify, by talking the Linux Kernel API, can watch a directory or a whole filesystem, for the moment that say, a Maya Artist, has exported a sequence of frames to the “shot tree”, or file server, in plain english. At that point, when Pyinotify notices these changes, it could begun to process these files and perhaps move them to a High Speed Fibre SAN like Avid Unity, or XSAN, that is built for playing back HD media files.

Lets take a look at how that might work:

I threw this “toy code”, together in about an hour or so, and all it does is pretend to do things when a file is added to the /tmp directory and it closes. I might get around to actually making a useful tool this weekend and adding threading, etc.

If you would like to check this code out, I put up a Google Code Project here. These are some pictures of my “toy code” pretending to do things, when I create a file in a directory, I am watching. Thanks to author of pyinotify for making this so easy to work with!

import os
import sys
import optparse
from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes

class PClose(ProcessEvent):
    """
    Processes on close event
    """

    def __init__(self, path):
        self.path = path
        self.file = file

    def process_IN_CLOSE(self, event):
        """
        process 'IN_CLOSE_*' events
        can be passed an action function
        """
        path = self.path
        if event.name:
            self.file = "%s" % os.path.join(event.path, event.name)
        else:
           self.file = "%s" % event.path
        print "%s Closed" % self.file
        print "Performing pretend action on %s...." % self.file
        import time
        time.sleep(2)
        print "%s has been processed" % self.file

class Controller(object):

    def __init__(self, path='/tmp'):
        self.path = path

    def run(self):
        self.pclose = PClose(self.path)
        PC = self.pclose
        # only watch these events
        mask = EventsCodes.IN_CLOSE_WRITE | EventsCodes.IN_CLOSE_NOWRITE

        # watch manager instance
        wm = WatchManager()
        notifier = Notifier(wm, PC)

        print 'monitoring of %s started' % self.path

        added_flag = False
        # read and process events
        while True:
            try:
                if not added_flag:
                    # on first iteration, add a watch on path:
                    # watch path for events handled by mask.
                    wm.add_watch(self.path, mask)
                    added_flag = True
                notifier.process_events()
                if notifier.check_events():
                    notifier.read_events()
            except KeyboardInterrupt:
                # ...until c^c signal
                print 'stop monitoring...'
                # stop monitoring
                notifier.stop()
                break
            except Exception, err:
                # otherwise keep on watching
                print err

def main():
    monitor = Controller()
    monitor.run()

if __name__ == '__main__':
    main()

Sound fun? if so, let me know.