I’ve been thumbing through the Perl Hacks book lately. Some of the “Productivity Hacks” are (nearly) applicable to Python as well as Perl hackers. For example, most of “Hack #5: Autocomplet Perl Identifiers in Vim” applies directly to Python vim users. I’ve been using the CONTROL-N thing for years now. Basically, if you have a “word” in a file that you are vimming, you type the first couple of letters of it, hit CONTROL-N, vim will match and complete that already-entered word. But, as chromatic pointed out, it won’t match (in Perl) “words” that look like “Foo::Bar”. Nor will it match (in Python) “words” that look like “foo.bar”. The solution to this for Perl is to add “set iskeyword+=:” to your .vimrc. The solution in Python is to add “set iskeyword+=.” to your .vimrc. Actually, I added this to my “.vim/ftplugin/python.vim” so that this would only get loaded when I’m editing a Python file. Now, I have greater joy in my vimming experience.

Anyway, this was a great hack that was easily modified to work for Python. There are other useful hacks that are very applicable to Python folk. I don’t know how many of these I want to give away for free, though. I might make you go buy the book :-)

This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainability.

Here’s what I learned about programming, refactoring, and 30-day projects during this series.

Anthony Baxter posted that Python 2.5 Beta 1 has been officially released.

You can find downloads here and “what’s new” here. It looks like they have included a quasi-currying function called “partial”, which is interesting. Also included is a ternary styled conditional expression in the form of “x = true_value if condition else false_value”. Also, you can have try, except, finally, and else all at the same time. And, ctypes, ElementTree, hashlib, sqlite, and wsgiref are all part of the standard library.

These are some pretty big changes and there are more changes on the “what’s new” page referenced above.

I have looked at several podcast grabber applications and have been unhappy with each one of them in one way or another. So I decided to roll my own. I’ve been storing the configuration in XML, but decided to use a different format that 1) looks more human readable, and 2) provides a good method of serialization and deserialization. I decided to look at Yaml. Yes, I know that I can serialize and deserialize XML and even do pretty XML, but that kind of violates intention number 1 above.

According to the Yaml website, Yaml is a “straightforward machine parsable data serialization format designed for human readability and interaction with scripting languages such as Perl and Python.” I found a Python library for dealing with Yaml called PyYaml. It installed easily with “easy_install” and appears pretty easy to work with.

If I have a Yaml file that looks like this:

-
db_file: /home/jmjones/podcasts/oreilly.db
description: OReilly Future
download_dir: /home/jmjones/podcasts/oreillyfuture/
mode: OK
-
db_file: /home/jmjones/podcasts/change.db
description: Accelerating Change
download_dir: /home/jmjones/podcasts/change/
mode: OK


Here’s all it takes to deserialize it:

In [1]: import yaml

In [2]: yaml.load(open("podcasts.yaml", "r"))
Out[2]:
[{'db_file': '/home/jmjones/podcasts/oreilly.db',
'description': 'OReilly Future',
'download_dir': '/home/jmjones/podcasts/oreillyfuture/',
'mode': 'OK'},
{'db_file': '/home/jmjones/podcasts/change.db',
'description': 'Accelerating Change',
'download_dir': '/home/jmjones/podcasts/change/',
'mode': 'OK'}]


And given the exact data structure we took in, here’s all that’s required to get it back out to Yaml:

In [5]: y = yaml.load(open("podcasts.yaml", "r"))

In [6]: print yaml.dump(y, default_flow_style=False)
- db_file: /home/jmjones/podcasts/oreilly.db
download_dir: /home/jmjones/podcasts/oreillyfuture/
mode: OK
description: OReilly Future
- db_file: /home/jmjones/podcasts/change.db
mode: OK
download_dir: /home/jmjones/podcasts/change/
description: Accelerating Change


If you don’t give it the “default_flow_style=False” keyword argument, the output is not as pretty as otherwise:

In [4]: print yaml.dump(y)
- {db_file: /home/jmjones/podcasts/oreilly.db, download_dir: /home/jmjones/podcasts/oreillyfuture/,
description: OReilly Future, mode: OK}
- {description: Accelerating Change, db_file: /home/jmjones/podcasts/change.db, mode: OK,
download_dir: /home/jmjones/podcasts/change/}


It looks more like a plain Python dictionary. When I get a chance, I’m going to rework my podgrabber to incorporate Yaml.

A lengthy thread on the Perl 5 development mailing list has asked if there’s self-generated FUD about the future of Perl worth debunking.

I’m not the person to ask about the viability of using Perl 5 for new development projects in your own environment (and I can’t wait for Perl 6), but I had an interesting thought about how to gauge the suitability and evolution of a language today….

This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.

Today’s task is a race against the clock to get just one single test running against the live, not a mocked, database. Can chromatic do it? His reputation is on the line!

As the sometimes baffling yet ultimately sage Yogi Berra once put it: “You got to be very careful if you don’t know where you’re going, because you might not get there.” This slightly corkscrew bit of advice holds a nugget of truth for managing successful projects–you gotta know what can cause a project to fail, to know how to make it succeed.

Andrew Stellman and Jennifer Greene offer more straightforward advice on what programmers need to know to save floundering software projects in Why Do Projects Fail?, an excerpt from their PDF with the very apropos title: How to Keep Your Boss from Sinking Your Project.

Perhaps my keenest frustration with Perl 5 is its lack of a clean metaprogramming facility. Stevan Little’s Class::MOP has filled in much of the gap. Here’s what I learned after playing with it for an afternoon.

In a recent post to the Perl 5 Porters mailing list, Yves Orton argued that it’s time to debunk the myth that Perl 5 is dying.

Ned Batchelder announced about a month ago (announcement here) the release of Tabblo, his company’s new web app. Interestingly, Django was used to create Tabblo.

Tabblo is a photosharing site with a twist. Your standard, run-of-the-mill photosharing service allows people to upload and categorize their photos for other people to view. Tabblo’s twist is that users can organize the photos on a page with their own words in order to tell a story about the photos. Each photo story page is itself referred to as a “tabblo”.

I’ve created a couple of tabblos, just to play around and see how functional it is. I have to say that it is really easy to create a great looking tabblo. Organizing the images on your tabblo is literally as easy as dragging and dropping. Tabblo feels more like a photo scrapbook than groups of images. It has actually been fun to just browse through tabblos that other people have created. This is a great app built in a great framework.

I’ve been trying for the last couple days to use MySQL’s full-text indexing with accent-insensitivity with no success.

From what I know, when you’re using accent-insensitive strings it doesn’t matter what is the case of the accented characters. For example, café and CAFÉ (coffee, in portuguese) are considered the same.

Well, it just doesn’t work that way. I’m using the utf8 charset and the utf8_unicode_ci collation. I created a fulltext index on two columns and it will return different results when using different cases of the same accented character.

Does anybody have any clues? I’m about to create two lower-case columns specifically for the purpose of indexing. Although I really don’t like this solution, I think it’s the only way to make it work as supposed.

Related link: http://www.futuresalon.org/2006/06/space_elevator_.html

Far out as in going life in 11 years and 306 days from today. Really far out as in

100,000 km (62,000 mile)

out into space. I remember late night conversations after Future Salons where some of the more-out-there Saloners would marvel over Dyson Spheres and Space Elevators. Accelerating Change and all (or is it just some Internet riches) makes it viable to built one now, or well within the next 11 years. The Dyson Sphere will take a bit longer, but the

LiftPort Group is a funded company that is working on nothing else but building a Space Elevator. (It may be only a front to sell books about it and get on the future conference circus or was that circuit :-)

Personally I think there are some problems down here on earth that we should tackle first. O.K. there is the famous meteor that may wipe us out any minute. To spread ourselves a bit in our solar system would do us good in that regards. Although shouldn’t our focus be on creating an early warning system and destroying the meteor, changing its course, or even better harvest the iron or whatever it is made of for use on earth? Well, to do that efficiently we could really use the help of a Space Elevator. Makes sense to me.

Please join us at the Space Elevator Future Salon next Thursday 22nd of June where

Michael Laine , president and founder of LiftPort will present his vision and answer all your skeptics questions.

In Silicon Valley one gets a bit jaded by all the cool stuff that gets presented here every other day it seems, but a company seriously working on a Space Elevator is something special.

Details: Thursday, June 22. 6-7 Reception, 7-9 Presentation and Discussion
SAP Labs Building D, 3410 Hillview Ave, Palo Alto, CA 94304

As always the Future Salon is free and open to the public, please RSVP so we can order enough food and drinks for the reception proudly sponsored by SAP.

See you on Thursday, Mark.

This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.

Today’s task is to get just one single test running against the live, not a mocked, database.

Greg Law of Undo Software bent my ear the other day about his company’s new release of UndoDB, a piece of software that adds reversible debugging to GDB.

That is, when running your program under UndoDB, you can set a breakpoint at the point of failure and rewind and replay the program’s state repeatedly. For trivial bugs this is perhaps nice, but for difficult bugs, whether arduous to reproduce or impossible to find, being able to move back and forth between a known good state and a known bad state can be invaluable to fixing the bug.

Like the also impressive Coverity static analyzer, UndoDB is not open source, but it is available freely for non-commercial use. Hopefully we will see improvements to the underlying tools as well as improvements in existing software from better development tools.

I just downloaded and installed Google Earth R4 Beta for my Ubuntu laptop. It installs from a .bin file rather than a package and it feels a little jerky at times, but it’s great to have this app out for Linux. I can see myself using this more than Google’s other recent release. This is great news.

I seem to have spent a large part of the last year ranting about people who don’t use database metadata properly. Usually I’m talking about things like primary keys, foreign keys and unique indexes, but recently I’ve come across a far more basic level of idiocy - using the wrong data types.

Currently part of my work involves taking data dumps from one database and loading it into another database (and then generating some reports from the data). Some of the fields involved contain dates. So being a sensible programmer I’m storing them in DATETIME columns in my database. It’s simply a case of working out what format the other database uses as its default export format for datetime values and converting that to my database’s default import format.

For a company so instrumental in enabling the next-generation web, Google’s behavior is unconscionably Web 1.0.

Not everyone has the time to write a book or a blog, or speak at conferences, but you can still help spread what you know. Just start talking.

Last night’s Chicago Perl Mongers meeting was a good one. We had a bunch of new folks show up, and we had pseudo-lightning talks, so there was a lot of good interaction. In the strict sense, lightning talks are 5 minute talks, where you’re given exactly 5 minutes and at the end of that time, you’re done. When you only have 5 people doing lightning talks, you get lax with the timer.

My lightning talk was supposed to be talking about ack, my grep replacement for large heterogeneous source trees, but as other people did their talks, I kept adding little notes of things I wanted to mention.

I try to remember that not everyone who does Perl sits immersed in it 24/7, or maybe even do it as their primary programming activity. I try to do as much as I can to let people know cool things they can do that they might not have known about, just by mentioning them.

I discussed all of the following to some degree, and in every case, it was new information for at least one of the dozen people around the table.

• ack
• Why the CPAN is better than PEAR
  • Because the PEAR is regulated and you can only put stuff on it if it’s approved and works well.
  • The CPAN thrives because of the unfettered uploading of shit, not in spite of it.
• How to use Carp and $SIG{__WARN__} to dump stacktraces in modules you don’t own.
• Damian Conway’s Perl Best Practices
• Who’s Damian Conway anyway?
• Module::Starter
• del.icio.us
• Writing Perl Modules For The CPAN, Sam Tregar’s excellent book now available for free at http://apress.com/free
• The CPAN shell

Next time you’re at a group, speak up. If you talk about something that might not be universally known, ask. “Anyone here not heard of del.icio.us?” The hands go up, and you can spend ten seconds saying “It’s a public bookmarking site that lets you post your bookmarks where others can see them, and see what other people are bookmarking.”

It’s easy to do, and there’s no better way to help spread knowledge.

I’m creating yet another website in Django, mostly for fun. One of the things I wanted this website to have was an RSS feed. I glanced at the documentation for generating a feed from Django and was initally put off because it looked a little cumbersome. I then began investigating my other options, like generating the feed manually, and decided to give the Django syndication doc a fuller read. I’m very glad that I did. The reason that Django’s RSS generation looked cumbersome at first glance is because it is featureful, flexible, and has every option you will probably need.

In order to create an RSS feed, all I needed to do was create a “feeds” dictionary in my url.conf:

feeds = {
'articles': ArticlesFeed,
'blog': BlogFeed,
'podcasts': PodcastFeed,
}


a URL pattern:

urlpatterns = patterns('',
##lot of URL configuration omitted here
(r'^feeds/(?P.*)/$', 'django.contrib.syndication.views.feed', {'feed_dict': feeds}),
)


and a few Feed objects. Here is an example of one:

class BlogFeed(Feed):
title = "Blog Title"
link = "/blog/"
description = "My random thoughts and musings on a variety of subjects."
def items(self):
return get_items("blog")


And everything just works. I tested my feeds with Blam and Sage and they work just fine. I also created a feed with enclosures and both iPodder and a custom podcast grabber I created had no problems with the feed. I can’t imagine this being easier.

I just noticed that IPython 0.7.2 was released, so I updated my dev environment to this version by doing easy_install --upgrade ipython. Anyone who uses Python even casually and doesn’t know about IPython needs to install it. This is one of a few of what I consider to be “must have” tools. It is an enhanced, customizable shell for Python.

I started IPython just to make sure it upgraded properly and to thumb around a bit. It said that it was on the correct version, so I gave the %magic command and started browsing through. I don’t know if I have just been asleep for a few months or if there are new features, but there’s some really cool stuff in IPython that I hadn’t noticed before. For example, dhist prints your history of visited directories. This has been in since before 0.7.1, so I’ve apparently been asleep. Another feature I’ve missed is pfile. I’m sure I’d be embarrased to know how long these features have been in and how much time they would have saved. Looks like my article needs updating. I guess I’ll be reading the docs on IPython today. Great work, folks.

I was fortunate to receive an invitation to test Google Spreadsheets. So how does it stack up against a desktop spreadsheet application –can I do everything that I can do in Microsoft Excel or OpenOffice? I am particularly interested in:

1. Formatting cells
2. Creating and editing formulas
3. Loading and saving
4. Importing an Excel spreadsheets
5. Creating multiple sheets
6. Reference cells from multiple sheets
7. Cutting, dragging, and pasting
8. Advanced stuff (e.g. pivot tables, macros)

So here are some screenshots in action (click on thumbs to enlarge):

Above: Renaming a spreadsheet

Above: Selecting a column

Above: Formatting cells

Above: Reference cells from another sheet

Above: Simple formula editing

Above: Saving the spreadsheet

Above: Opening an existing spreadsheet (saved from Google Spreadsheets)

Above: Exporting the spreadsheet to Microsoft Excel

Above: Importing an Excel spreadsheet from my hard disk

Above: The imported Excel spreadsheet from my hard disk

Above: Uh oh… Trying to open a complicated spreadsheet with macros and pivot tables

Note: tested on Windows XP w./ Firefox.

Overall, it is great. It should suit most computational needs. Most of the common actions, such as keystroke commands in Excel/OpenOffice, are applicable in Google Spreadsheets. The only things that I see are missing are support for macros (ugh!), pivot tables, and of course, charts.

This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.

Today’s task is adding a SQLite backend.

This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.

Today’s task is to test the new Workspace code.

I’ve been all but gushing about Ubuntu since my first experience with it over a year ago and I believe my praise has been well deserved. I just installed Dapper final yesterday and I have a less gushing report. I’ll just give a quick summary here.

• I installed from the “Desktop CD”. This started a “live CD” session and gave me an “install” icon on the desktop. That was a little disappointing. I’m sure there is a more efficient way to get to the GUI installer than starting up Gnome.
• I wanted to look on my hard drive before I began the install and the live Gnome instance refused to mount it. It also refused to mount a USB drive I plugged in before I began the install.
• The “install interview” process was quick and painless. There was a total of six screens to go through and not one of them required much detailed information. I count this a plus for ease of use and a minus for functionality.
• grub just automatically installed itself to the master boot record, found my existing Windows XP install and put it in the grub configuration. That’s what I wanted to happen, but I’m not happy it just did it.
• Pretty much everything was working after rebooting: sound, networking, synaptic touchpad including vertical scrolling.
• NetworkManager (after I installed it) was not working appropriately after resuming from suspend. I had to do a little bit of work in the /etc/acpi scripts to get it working right. At least, I hope it is working right.

In the book Surely You’re Joking, My Feynmn!, Richard Feynman tells an amusing story of engineers showing him complicated blueprints and him pointing to them and asking “what happens if this valve sticks?”

The engineers were astonished. They pored over their blueprints and realized that they had a design flaw. What’s amusing about the story is that Mr. Feynman was bluffing. He didn’t know if he was pointing to a valve or not and assumed the engineers would gently correct his misunderstanding. By mere chance, a serious problem was averted.

In yet another anecdote I read, a contractor finished building a hospital but its opening was delayed because somehow nobody realized that they had forgotten to have phones in the building. Oops.

Today, these sorts of problems are less likely to occur because engineers, architects and others have sophisticated software tools which can alert them to potential problems. But what about “software engineers”?

Ubuntu announced today the release of Ubuntu 6.06 LTS (Long Term Support). This release has been referred to as “Dapper Drake” during its development cycle. The name “LTS” sums up some of the differences between this release of Ubuntu and previous releases. One of the driving goals for this release has been to get Ubuntu ready for deployment into the enterprise. That includes ease of install and use, automatic hardware recognition and configuration, and support.

The download options seem a little different this time. Here is an example mirror to a download page. The options are “Desktop CD”, “Server install CD”, and “Alternate install CD”. The “Desktop CD” looks like what was formerly referred to as the “Live CD”. The description for the “Desktop CD” says

The desktop CD allows you to try Ubuntu without changing your computer at all, and at your option to install it permanently later. This type of CD is what most people will want to use. You will need at least 192MB of RAM to install from this CD.

The description for the “Server install CD” says

The server install CD allows you to install Ubuntu permanently on a computer.

And the description for the “Alternate install CD” says

The alternate install CD allows you to perform certain specialist installations of Ubuntu. It provides for the following situations:

* creating pre-configured OEM systems;
* setting up automated deployments;
* upgrading from older installations without network access;
* LVM and/or RAID partitioning;
* installs on systems with less than about 192MB of RAM.

I just updated my already-installed-from-Dapper-Flight-7-and-daily-updated-system this morning and only had two updates, so I don’t think a reinstall of Dapper would buy me much. I may install this final release just for fun and to see how smoothly it runs. But so far, this has been the best OS install of my life. I would highly recommend Ubuntu to newbies and veterans alike.