I’ve been running local special need gem servers as well as a gem server for beta versions of Ruport and support code (namely plugins). I got tired of manually wget’ing the gems I needed and their dependencies, so with a little help from mechanize and archive_tar_external, my friend Dinko and I came up with this quick (pretty messy) hack.
require 'rubygems'
require 'archive/tar_external'
require 'mechanize'
require 'fileutils'
include FileUtils
include Archive
module RFDownloader
def populate(archive)
mkdir ".rf_downloader"
cd ".rf_downloader"
yield
Tar::External.new( "../#{archive}.tar",
"*.gem", "gzip")
cd ".."
end
def get(pattern,options={})
link = "http://rubyforge.org/projects/#{options[:project]}"
agent = WWW::Mechanize.new
page = agent.get(link)
page.links.find { |l| l.text[/File/] }.click
link = agent.page.links.find { |l| l.text[pattern] }
link.click.save_as(link.text)
end
def fetch(opts,&block)
populate(opts[:archive]) do
module_eval &block
end
end
module_function :fetch, :populate, :get
end
With this, we were able to grab automatically a tarball filled with all the gems needed to run Ruport 0.9.0 with a tiny little script:
require "lib/rf_downloader" RFDownloader.fetch(:archive => "ruport_deps") do get /fastercsv.*gem/, :project => "fastercsv" get /pdf-writer.*gem/, :project => "ruby-pdf" get /color-tools.*gem/, :project => "ruby-pdf" get /transaction-simple.*gem/, :project => "trans-simple" get /RedCloth.*gem/, :project => "redcloth" get /hoe.*gem/, :project => "seattlerb" get /rubyforge.*gem/, :project => "codeforpeople" get /mailfactory.*gem/, :project => "mailfactory" get /mime-types.*gem/, :project => "mime-types" get /scruffy.*gem/, :project => "scruffy" get /build.*gem/, :project => "builder" get /gem_plugin.*gem/, :project => "mongrel" end
Elegant? Nope! A way to be lazy and quickly snapshot the newest versions of a number of gems, you bet ya. Maybe someone will see this and build a nice little library that does this ‘the right way’ :)
UPDATE: Thanks to Aaron Patterson for helping me get rid of the wget call
What are you using this for exactly? I'm curious b/c this comes very close to my own needs -- I have project made up of many subprojects. setup.rb handles this layout gracefully via the packages/ directory, but gems has no way to handled it. And given how gem works I have no desire to add tens of depependencies to a "stub pacakge".
Among other possibiles, I've been considering creating a tool that can merge multiple gems into a single gem. Would that be of use to you?
Personally I wish -y was the deafult mode of gem. This is mostly a perceptual issue.
Trans,
As far as I can tell, if you want automatic dependency resolution, the dependencies must be available on your gem_server.
(Or you must have your users fire up a laundry list against RubyForge, or post a stub package there, which I consider to be pollution)
So i'm using this tool to identify a series of gems that I need for any given gem server, run it, get a tarball back, sftp it to wherever it needs to be, and then reindex.
It's actually a fairly common operation for me, because I build lan appliances using Ruport+Camping for clients, and some want a very smooth install process, which I solve through setting up a gem server for them.
About a gem merger: Yes, that'd be nice. Then I could sling around one or two gems instead of 20.
about -y, it's unrelated to the issue. If you use -y against a non-rubyforge gem server, it will cry that it cannot find the gem in the repository if you don't have it there. (But I agree it could be a nice default, since saying no cancels the install). If it were the default, we'd need a Linuxy style pretend flag, for auditing.
Someone's using archive-tar-external? Yay! :)
Five minutes of zlib didn't find me multiple file love so I searched RAA for tar. archive-tar-external works great. Wish it'd glean format from the filename, though.
Wish it'd glean format from the filename, though.
You mean for the uncompress method? That makes sense. If you're referring to something else you'll have to clarify. Or, put in a feature request. :)
Tar::External.new( "../#{archive}.tar","*.gem", "gzip")
becomes
Tar::External.new( "../#{archive}.tar.gz", "*.gem" )