This is the third in an N part series on rewriting my podcast grabbing application. Here are the links to parts one and two. In part two, I promised to get into a common way of synchronizing media files between media stores.
Delivering on that promise, here is my SyncManager:
from sets import Set class SyncManager(object): """This is a concrete implementation of a syncronization manager which is intended to be subclassed if necessary. A SyncManager connects two mediaStores with filters and processing steps. It should be able to copy files from the fromStore to the toStore, exclude any files which were filtered out, and execute any processingSteps along the way. """ def __init__(self, fromStore, toStore, copyFilters, deleteFilters, preProcessingSteps, postProcessingSteps): self.fromStore = fromStore self.toStore = toStore self.copyFilters = copyFilters self.deleteFilters = deleteFilters self.preProcessingSteps = preProcessingSteps self.postProcessingSteps = postProcessingSteps self._init() def _init(self): pass def getCopyList(self): """return a list of files we need to copy from the fromStore and to the toStore""" copySetAll = Set(self.fromStore.list()) copySet = Set() for filter in self.copyFilters: copySet = filter.filter(self.fromStore, self.toStore).union(copySet) ##a poorly written filter could feasibly add things to the list that weren't ##initially there. Use set arithmetic to return all things which the filters ##returned which were also in the original set. copySet = copySet.intersection(copySetAll) return copySet def getDeleteList(self): """return a list of files that we need to delete from the toStore""" deleteSetOriginal = Set(self.toStore.list()) ## set deleteSet to a null set - we'll build up a set of what to delete. deleteSet = Set() for filter in self.deleteFilters: deleteSet = filter.filter(self.fromStore, self.toStore).union(deleteSet) ##a poorly written filter could feasibly add things to the list that weren't ##initially there. Use set arithmetic to return all things which the filters ##returned which were also in the original set. deleteSet = deleteSet.intersection(deleteSetOriginal) return deleteSet def setOverrideCopyList(self, copyList): pass def syncCopy(self): for mediaFile in self.getCopyList(): for preProcessingStep in self.preProcessingSteps: mediaFile = preProcessingStep.process(mediaFile) self.toStore.addFile(mediaFile) for postProcessingStep in self.postProcessingSteps: mediaFile = postProcessingStep.process(mediaFile) def syncDelete(self): pass def syncAll(self): ##do delete first because that is often nicer to lower-memory ##media devices self.syncDelete() self.syncCopy()
This was nearly the final piece to the puzzle for creating a simple automated RSS download manager. I introduced several new concepts to the mix, including copy and delete filters, and pre and post processing steps. The copy filter(s) help determine which files to copy, or more to the point, which files not to copy, from the source media store. I’m not using delete filters yet, but they would determine which files to remove from the source media store when I get the “sync to MP3 player” functionality working.
Processing steps were how I decided to handle the problem of keeping track of which files have been downloaded. This is also how I plan on accommodating changing of certain ID3 tags upon download or sync-to-MP3-player. After copying a file from one media store to another, the post processing steps are called. If you notice in the syncCopy() method, the processing steps, both pre and post, return the mediaFile and re-bind the returned media file to the same name we were using in processing it.
def syncCopy(self): for mediaFile in self.getCopyList(): for preProcessingStep in self.preProcessingSteps: mediaFile = preProcessingStep.process(mediaFile) self.toStore.addFile(mediaFile) for postProcessingStep in self.postProcessingSteps: mediaFile = postProcessingStep.process(mediaFile)
This will allow the ability to chain processing steps and manipulate the file with each successive step. I actually had more things in mind that just podgrabber for this. I’m seriously considering removing the virtual filesystem functionality from podgrabber and spinning off another open source project. But one thing at a time, right?
So, we finally have a working podgrabber. Here is a script which will run podgrabber in an unattended way and keep track of what it has downloaded:
from podgrabber import syncManager from podgrabber import mediaStore from podgrabber import filter from podgrabber import processingSteps import os dl_base = 'download' db_filename = 'podgrabber.db' rss_list = ( ##Name, rss url, dl directory ('Buzz Out Loud', 'http://www.cnet.com/i/pod/cnet_buzz.xml', 'buzz'), ('News.com Daily', 'http://news.com.com/2325-11424_3-0.xml', 'news.com'), ) db_copy_filter = filter.DbCopyFileFilter(db_filename) update_processing_step = processingSteps.UpdateDBStep(db_filename) sm_list =  copy_filters = [db_copy_filter] delete_filters =  pre_proc =  post_proc = [update_processing_step] for name, url, directory in rss_list: fromStore = mediaStore.RSSMediaStore(url) toStore = mediaStore.FileSystemMediaStore(os.path.join(dl_base, directory)) sm = syncManager.SyncManager(fromStore, toStore, copy_filters, delete_filters, pre_proc, post_proc) copy_list = list(sm.getCopyList()) print copy_list sm.syncAll() sm_list.append(sm) #for f in copy_list[:-3]: # print 'Updating', f # update_processing_step.process(f)
This is a working, but limited script. There is no GUI. There are no status updates on how far along the download has progressed. This is a totally single threaded process. The single threadedness is probably the biggest stinker among the limitations. I can either handle threading at the very top level and spin off each syncManager into its own thread, which would be the easy way, or introduce something like a syncTaskManager which the syncManagers would pass off a task to and it would manage a thread pool and execution of those tasks. Hmmmm….a task manager isn’t a half bad idea, come to think of it.
OK - so for next time, I’ll either start re-implementing the GUI for podgrabber, or I’ll introduce a task manager for nicer, cleaner threading. What am I talking about? I don’t even have threading yet!