Python 3.0a1 has been released. See Guido’s announcement here. This is awesome news. I’ll have to download it tonight and start poking around.

Following my recent post about a color-coded reputation management system* for politicians (indicating their truthfulness by the color used to display their names on web pages), I came across Politifact and its Truth-o-Meter.

The term Identity Theft is usually assumed to be related to a malicious entity abusing someone’s credit information to commit financial fraud. This continues to be a big problem, but I’d like to extend the problem of identity theft in the social-networking aspects of so-called Web 2.0 applications. I feel this is an important topic of discussion because, unlike technical vulnerabilities that can be remediated with a software patch, the problem at hand is a design issue that poses significant risks to society’s ability to securely leverage the usefulness of social networking.

The optparse module is a modern alternative for command line option parsing that offers several features not available in getopt, including type conversion, option callbacks, and automatic help generation. There are many more features for to optparse than can be covered here, but hopefully this introduction will get you started if you are writing a command line program soon.

After reading one of my favorite blogs this morning, it got me thinking about the Zen of Mac. In the article that I linked to, Shannon mentioned that one of his hangups is not using OS X because it isn’t Open Source. The topic of smart people with hangups is very interesting actually and I would love to talk even further about it, but I am going to talk specifically about the Zen of Mac in this post.

Back in 2002, I was a systems administrator for the administration building at a Caltech, and I was solely responsible for providing support for Dr. Baltimore. Dr. Baltimore is smart. He received a Nobel Prize in his 30’s, he went to MIT, and he was the President of Caltech at the time. He is a mac guy, and that is partially why I was hired to do that job, as I have been into Mac computers for quite some time.

For those of you that remember, it was a huge deal to switch from OS 9 to OS X. It was my responsibility to design and build his OS X laptop and make sure that it was an easy transition. When the time came for me to actually deliver the laptop, things worked reasonably well and were intuitive as he would expect them to be. The punchline of this story is when I finally decided to show Dr. Baltimore the terminal. I think I mentioned something like, “One of the nice things about OS X is that it has a terminal and you can do nice things like……”.

The look on his face was priceless. I don’t remember the exact specifics of the conversation, but it was roughly, “Why would I use a terminal, it is a mac, that is the whole point!”. That taught me quite a bit, as here was one of the most successful and intelligent people on planet earth and he “got” Mac. He didn’t want to think about his computer as his thinking time was spent in other areas like making sure Caltech continued to be one of the top Science Universities and doing research on curing AIDS.

That is the Zen of Mac. You don’t think, it just works. As software engineers, systems administrators, or people that are very technical, it can be difficult to just not think about your desktop computer. The reality is that you are more productive on solving your other problems when you don’t think about your computer, it just works.

So as far as I am concerned, even though I work literally all day on linux machines from a shell, my desktop experience is OS X because it just works and I don’t have to think, my thinking can be devoted to solving my problems. Finally, for those people that are switching from a Linux desktop to an OS X computer, I would give this advice. Don’t think about how you used to do it on Linux, just forget what you know and try things out. OS X is designed to be intuitive and effortless. Fighting it to make it do what Linux does is not the proper approach. You must submit your will, relax your mind and float downstream on the white glow of the macbook pro monitor. Once you surrender your desires, and realize that desire leads to suffering you will truly appreciate the mac.

Update: excellent tip from Eric Larson.

I’m a long-time CVS head, and I recognize CVS is quite showing is age, but I’ve never had much of a taste for the pundit’s heir apparent Subversion. I could go on about why, but I think it all boils down to SVN’s not fitting my head. I’ve been thinking of VCS and hosting options for my FOSS projects, and have discussed some of this in the community. I’m leaning heavily towards Mercurial (with Bazaar a close second). I decided to set up Hg on my MacBookPro so that I can play with it, and so that I can see if I can start doing local work on my machine in Hg and then push to a centralized SVN repo as a bit of insulation from SVN. Herein a few notes on the matter.

On behalf of the Parrot team, Patrick Michaud released Parrot “Augean Stable” 0.4.15. The most important feature in this release is the addition of the NQP compiler. NQP is a stripped-down version of Perl 6 suitable for writing compiler transformations.

Within the next day or so (if it hasn’t happened already), NQP should displace PIR (Parrot’s native assembly language with a touch of syntactic sugar) as the preferred language for writing compilers. NQP’s also a useful language in its own right; by the time you read this, it might already be capable of using libraries and objects written in other languages running atop Parrot.

Also of note is that Gabor Szabo has written guidelines on smoke-testing Parrot. If you have a machine with spare cycles (and especially a machine somewhat more exotic than x86 Linux), you would do the Parrot porters a great service by reporting tests results.

The discussion turned, as it occasionally does, to licensing and philosophy. (This is what happens when you hang out with smart people who also care about the subject.) I may have surprised a couple of them by saying, fire-breathing zealot for freedom that I am, that I don’t particularly care about the mythical software as a service loophole.)

Why not?

My current camcorder is over 5 years old now and I’m looking to replace it. I’m hoping one of you can point me in the right direction. Specifically, I’m looking for a camcorder which uses flash memory, writes files in a format which iMovie can directly start manipulating, has a crisp picture (which implies 3CCD, but not necessarily HD), and has a firewire connection.

I was looking at the Panasonic HDC-SD1, but iMovie 08 looks like it transcodes the files as it imports them. The reason that I wanted to go with a flash based camcorder is the ability to bring movies onto my computer more quickly than I have been able to with my current digital 8/firewire camcorder. But if iMovie has to transcode files as it imports, them, that doesn’t really buy me much. I suppose I could get the SD1, create an image of the files on the camcorder and mount it later to allow me to free up the flash drive in the camcorder, but that feels like a pain. I’d like flash as opposed to hard drive because of the ease of replacement if it goes bad and fewer moving parts. I want firewire because sometimes it’s nice to be able to record directly in iMovie.

So, 2 questions: 1) Am I thinking correctly about AVCHD and the SD1? and 2) Do you have any specific camcorder suggestions which roughly meet my needs?

The subject of “Non-Toxic Stimulation” has been on my brain for a bit now. I remember the subject being brought up a few years ago by my mentor, who passed away a couple of years ago. The best way to describe our relationship was to say that is was very similar to tuesday’s with morrie. One night Dr. Bogen described a friend of his that was in constant search for Non-Toxic Stimulation.

The definition for it is quite simple. Non-Toxic Stimulation is something that excites you but doesn’t kill or harm you. Toxic stimulation is quite easy to define as well. It is something that excites you but could kill you literally or figuratively. Examples of toxic stimulation could be smoking crack, driving a motorcycle down the freeway at 160 mph on your back tire, running with the Bulls in Spain, rock climbing without a rope, or fighting dogs. These are all optional activities that might leave you or others dead, maimed or in jail.

There are equally exciting Non-Toxic forms of stimulation, but they for the most part involve your brain. One form could be taking a class at Big Nerd Ranch and starting a Web 2.0 company. It could be finally teaching yourself higher math because it scares you and make you feel inadequate, but you know you can do it anyway.

The Non-Toxic form of stimulation I am engaging in currently is to teach myself Ruby as a way to practice test driven development. It is too hard to do test driven development in Python as I don’t want to slow down, but in Ruby, which is very similar to Python, I am not tempted to crank out some code as I can’t. It makes it much easier to practice doing TDD with Ruby and then apply that knowledge to Python.

Most of the code that I’ve been writing for the book has been getting its own unit tests. I’ve been working on a chapter on networking for the past week and a half and have written a little code for the chapter. One of the challenges of writing tests for something like networking code is that there are so many variables which may influence a suite of unit tests. For example, if my unit tests rely on hitting some Google webserver and I encounter problems, trouble shooting questions may include: is my router acting up, is my ISP acting up, am I failing to get DNS resolution, is that particular server down, have they changed the URL for this resource, etc.

So, for the purpose of testing, I decided to bypass the socket module in this case and handle everything locally. I created a faux socket class fleshed out with the methods that I needed. I then monkey patched my module under test with the new faux socket class. All attempts to connect to a real socket actually “connected to” a fake socket from which I could totally control the behavior.

I thought it would be about time to announce that Jeremy and I will be doing a monthly Video Podcast hosted by O’Reilly. We are going to keep things short, just a few minutes, tight, professional and hopefully put on a good show once a month. It happens to be a brave new world and video is dirt easy to make, produce and distribute. I pitched the idea to O’Reilly and they liked it, so here we are.

We will be shooting on a combination of devices ranging from HD camcorders to our built in macbook cameras. We will be doing post production using Soundtrack Pro and Final Cut Pro. In another life, when I was a teenager I was working for ABC Network Television as a freelance editor. Oh, and I also worked on the first digital feature animated film for Disney and for Sony Imageworks. So I have a little experience :)

If you have any ideas for our show let us know. We will be keeping it broad and covering all of open source, but we are also open to suggestions for future topics and show ideas.

I’ve been reading Beautiful Code, picking out chapters here and there as I have time. While reading Brian Kernighan’s explanation of Rob Pike’s regular expression program from The Practice of Programming, I had an idle thought. “Hey, that’s a highly recursive program with complex behavior suitable for didactic purposes.”

Of course, Kernighan says that almost verbatim in the text. He also says “It’s a nicer example than Yet Another Fibonacci Sequence Generator.”

So I ported it to Haskell. I don’t promise it’s necessarily great Haskell, and I wouldn’t consider it entirely beautiful, but it appears to function.

The csv module is very useful for working with data exported from spreadsheets and databases into text files.

Jesse Noller is leading a campaign to have Python developers form a network via LinkedIn.com.

News of University of California Santa Cruz computer scientist Luca de Alfaro’s Wikipedia trust-coloring system revived - and improved - an idea I’ve been playing with: automated reputation-management for politicians. The idea is to make the concept of honor meaningful again, by creating new social rewards and penalties for behavior that affects the rest of us. (It could, of course, also be applied to journalists, corporate leaders or other public figures.)

I rarely use a word processor: if I can’t write plain text in a text editor (strangely optimized for editing text), something’s gone weird in my world.

I can’t always avoid word processors, however. Occasionally I really need to read (or even edit) text saved from a word processor in something other than plain text. Usually my first task is to extract the relevant text from that format into something I might actually want to use.

AbiWord is my underappreciated aid. Not only is it quick and fast (especially in startup time), but it has one stunningly useful command line option: --to=format. That’s right. From the command line, you can convert a document in whatever evil, proprietary format AbiWord supports into something useful, without splash screens or menu popups or even taking your hand off of the keyboard to move a mouse.

As a word processor, it’s fairly nice too–doing exactly what I need to do on those rare occasions when I really really need a word processor.

Thanks to the AbiWord developers and everyone who’s contributed to the project!

I have been meaning for the longest time to scratch the personal itch of creating a robust command line tool to analyze duplicates on a file system. There are a few scripts floating around in various languages and the problem is not all that difficult to solve, but I went the whole nine yards and wrote a reasonably cool command line tool that uses md5 checksums to detect duplicates. A report is generated in addition to stdout dup messages in CSV format, so you can manually look through dupes and decide what you want to do with them.

Liten can be downloaded from the cheeseshop: http://cheeseshop.python.org/pypi/Liten/0.1a

GoogleCode: http://code.google.com/p/liten/

I have a rather long list of things to finish, like threading, daemonizing, caching ORM backend, way more unittesting etc. Give it a whirl and let me know what you think…

Oh, and thanks to the following people that I bugged with dumb questions like I usually do :)
Titus Brown, Shannon Behrens, Rick Copeland, Jeremy Jones, Scott Leerssen.

Why the Lucky Stiff’s fast, enjoyable Hpricot library makes hard Rails View tests effective and fun. Hpricot is a deep and useful HTML parser with a wide, flexible interface. It supports many clever systems to read and edit HTML. When we put it to work in Rails functional tests, it offers lots of different ways to solve hard problems.

Hearing that Uncle Bob feels afraid to refactor Ruby code without an IDE sends me back to the stern-but-loving embrace of Vim, my comprehensive test coverage, and a decent version control system.

Yikes. That’s like forgetting how to type. (No, that’s not a Hindley-Milner joke.)

In a now famous article by Joel Spolsky, he argues that you should never rewrite projects from scratch. To be fair, I’ve done this, but generally on open-source projects where I’m donating my time. I’m less worried about financial constraints or competitive advantage.

Aside from that edge case, I generally agree with Joel Spolsky. If something is a tiny project, refactoring is often trivial and if you want to do a rewrite, so be it. However, large projects are often dangerous to rewrite. But how did they become large projects? The vast majority of “large” projects I have worked on started out as small projects which gradually had features and cruft added. Interestingly, this is very similar to how agile methodologies work: build something small, but useful, in the first iteration or two. Always have working code and keep building on it. So the rewrite seems like a good idea because we’ve already shown we can create the project, right?

The subprocess module provides a consistent interface to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as os.system, os.spawn*, os.popen*, popen2.* and commands.*.

I’m pleased to bring the Python Module of the Week series to ONLamp. The goal of the series is to cover one of the Python standard library modules each week with a brief description and examples.

I will post one or two of the popular past episodes here, but for the complete archive check out the main feed. Future posts will be posted to both sites.

In comments on Sci Foo 2007, Tim O’Reilly wondered if giving Larry Wall a research job may have reduced his immediate practical concerns influencing the development of Perl 6.

I interpret the line of thought as “To what degree does the lack of a single well-defined problem to solve influence the concreteness or the abstractness of the solution and its delivery date?” (I won’t claim that that’s Tim’s line of thought, but it’s the question I’ve pondered this past week.)

Parrot’s monthly bug day will take place this Saturday, 18 August. Parrot committers and developers will be in #parrot on irc.perl.org all day to prepare for the upcoming release.

This month, our focus is on cleaning up compiler warnings from all of the additional static checking we’ve added in the past two releases. There are also several small coding tasks for Parrot novices. We’re all happy to help you download, configure, and build Parrot and will gladly help you write your first patch for the project.

Maybe you know how to drive a car. Maybe you can fix the carburetor. You probably don’t assume that you know how to design the car. However, programmers who can write huge SQL statements and understand the output of an EXPLAIN statement often assume that they can design a database. Admittedly, unless we’re comparing Oracle 9i with the 1967 Volkswagen Beetle, databases usually aren’t more complex than cars, but a properly designed database can be very hard to come by yet it’s the key to a solid application.

I rarely see my colleagues, both in my work and in my hobbies. I spend most of my time collaborating with them, and one primary communication medium is email.

My personal mail server runs Postfix. Besides one small problem with a spam filtering message loop (mail forwarding to an account which rejects spam messages and includes the spam in the response is at least as evil as challenge-response), I’ve never had a lick of trouble.

Throw in server-side filtering with procmail or Email::Filter, as well as extended addresses, and I’m thrilled.

Then I discovered how to add just a couple of lines of configuration to make temporary, expirable addresses stunningly easy to create and manage and expirable, lightweight mailing lists, both of which require MTA administration beyond one initial configuration.

I never have to think about Postfix. It silently hums along, delivering buckets of mail. Thank you to all of its developers and contributors!

I ran into a beehive of standardization activity at LinuxWorld Expo this year. The outcomes of these efforts aren’t the traditional standardization documents, numbingly complex yet short on critical detail, that companies have to rush to implement. Instead, the outcomes are working code, the open source way to achieve interoperability.

This blog will talk about two organizations I met with and their members–the Linux Mobile Foundation and the Open Solutions Alliance–plus other interesting people I met at LinuxWorld Expo.

A few years ago I was at a meeting where a salesman was trying to pitch his terribly expensive closed-source software to our company. Since security of our data was very important, this topic was raised a few times. I asked about how they encrypted their data. The salesman replied that since security was so important, the company created a proprietary encryption algorithm which was secure because no one knew how it was implemented. He seemed a bit flustered when I burst out laughing.¹

If you’re reading this blog, you probably have a technical bent and know that the vast majority of software out there has bugs. The larger the project, the more bugs. In fact, I’ve never worked on any significant (you know, large) piece of corporate software without known bugs. I’m constantly talking to friends who complain bitterly about long-standing problems with their systems. So why is closed-source software allowed to take a witness stand and accuse you of crimes when you’re not allowed to cross-examine it?

I’m running Firefox 2.0.0.6 on my (relatively) new (intel) Mac. Problem is that Firefox locks up several times a day. I found a forum which suggested disabling the anti-phishing functionality. Several people seemed to have benefited from said disabling.

I also found a Bugzilla report about the same behavior. It sounds like the anti-phishing thing can cause some problems during startup, but is probably not responsible for hanging during regular browsing. My problem is not on startup, but on browsing, so disabling anti-phishing probably won’t help me. But I’ve disabled it just on the off-chance that it will. If this hanging persists, I’ll either switch to Opera or Safari.

What are your experiences with FF on Mac? Crashes, hangs? Or is your world just peachy?

Are you one of the multitudes of people that want to know once and for all who is the king of web application frameworks in the Python/Ruby arena? You might want to attend a yet to be announced Ruby vs. Python Webframework Charity Deathmatch here in muggy Atlanta, GA. These are some ideas of how we might organize it, if you have some advice or ideas, please let me know as we are just now brainstorming how the event would shake down.

I don’t think we would limit attendance to just Atlanta, GA programmers either. I believe anyone that wanted to attend and enter the competition would be allowed.

Mission: Sort out once and for all who is the king of all web frameworks…for Charity.

Rules:

1. We find a needy charity, and they give a judge the design spec.
2. Teams of Ruby and Python programmers are given the spec on Friday and have until Saturday morning to come up with a prototype. Saturday morning each Ruby and Python teams internally vote on the best design.
3. One group from Ruby and one group from Python code then code until Sunday morning when judging starts.
4. Winner gets to be the design for the Charity, and the loser..well maybe they get VC startup money?

The initial interest has been strong so far, and there will be both technical and business people who will attend

Lets face it Ruby and Python are both great languages and fun to code in. I think the more of these type of events we have the better. It might be cool to have a future event where groups have to design an application in both Ruby and Python and you get judged on both efforts collectively.

Comments or Suggestions?

This is part 4 of an N part series on rewriting my podgrabber application. Here are links to part one, part two, and part three. In part 3, I outlined my strategy for synchronizing between mediaStores. This post will update that strategy slightly to show how I’m now handling threading.

For the curious, the code lives in a Bazaar repository at http://bzr.podgrabber.org/trunk/

The SyncManager now takes a taskManager in its constructor.

class SyncManager(object):
    """This is a concrete implementation of a syncronization manager which is
    intended to be subclassed if necessary.

    A SyncManager connects two mediaStores with filters and processing steps.
    It should be able to copy files from the fromStore to the toStore, exclude
    any files which were filtered out, and execute any processingSteps along
    the way.
    """
    def __init__(self, fromStore, toStore, copyFilters, deleteFilters, preProcessingSteps, postProcessingSteps, taskManager):
        self.fromStore = fromStore
        self.toStore = toStore
        self.copyFilters = copyFilters
        self.deleteFilters = deleteFilters
        self.preProcessingSteps = preProcessingSteps
        self.postProcessingSteps = postProcessingSteps
        self.taskManager = taskManager
        self._init()

And on copying a file, the SyncManager pushes the request to the task manager:

    def syncCopy(self):
        for mediaFile in self.getCopyList():
            print "ADDING MEDIA FILE", mediaFile
            logger.info("Copying file %s" % mediaFile)
            self.taskManager.addCopyFile(mediaFile, self.toStore, self.preProcessingSteps, self.postProcessingSteps)

Here is the task manager code in its entirity:

from Queue import Queue
import thread
import threading
import time

import logging
logger = logging.getLogger("podgrabber.syncTaskManager")

class Shutdown(object):
    pass

class CopyWorker(threading.Thread):
    def __init__(self, q, fileDict):
        self.q = q
        self.fileDict = fileDict
        threading.Thread.__init__(self)
    def run(self):
        #print "Running copy thread", self.getName()
        logger.info("Running")
        while 1:
            logger.debug("Blocking while pulling items from queue")
            mediaFile, mediaStore, preProc, postProc = self.q.get()
            if type(mediaFile) == Shutdown:
                #print "Break"
                logger.info("Shutting down")
                break
            logger.debug("Retrieved items from queue")
            for preProcessingStep in preProc:
                mediaFile = preProcessingStep.process(mediaFile)
            logger.debug("Retrieving file %s" % mediaFile)
            mediaStore.addFile(mediaFile)
            logger.debug("Done etrieving file %s" % mediaFile)
            for postProcessingStep in postProc:
                mediaFile = postProcessingStep.process(mediaFile)
            self.fileDict[mediaFile][1] = time.time()

class TaskManager(object):
    def __init__(self, numCopyThreads=5, numDeleteThreads=2):
        self.copyQueue = Queue()
        self.deleteQueue = Queue()
        self.numCopyThreads = numCopyThreads
        self.numDeleteThreads = numDeleteThreads
        self.threadList = []
        self.fileDict = {}
        for i in range(numCopyThreads):
            #thread.start_new_thread(self._copyFile, ())
            copyWorker = CopyWorker(self.copyQueue, self.fileDict)
            copyWorker.setDaemon(True)
            copyWorker.start()
            self.threadList.append(copyWorker)
        for i in range(numDeleteThreads):
            pass
    def addCopyFile(self, mediaFile, mediaStore, preProc, postProc):
        self.fileDict[mediaFile] = [time.time(), None]
        self.copyQueue.put((mediaFile, mediaStore, preProc, postProc))
    def addDeleteFile(self, mediaFile, mediaStore, preProc, postProc):
        self.deleteQueue.put((mediaFile, mediaStore, preProc, postProc))
    def _copyFile(self):
        while 1:
            mediaFile, mediaStore, preProc, postProc = self.copyQueue.get()
            if type(mediaFile) == Shutdown:
                break
            for preProcessingStep in preProc:
                mediaFile = preProcessingStep.process(mediaFile)
            mediaStore.addFile(mediaFile)
            for postProcessingStep in postProc:
                mediaFile = postProcessingStep.process(mediaFile)
    def _deleteFile(self):
        pass
    def shutdown(self):
        for i in range(self.numCopyThreads):
            self.copyQueue.put((Shutdown(), None, None, None))
        for i in range(self.numDeleteThreads):
            self.deleteQueue.put((Shutdown(), None, None, None))
        for t in self.threadList:
            t.join()

Basically, the task manager creates a queue for copies and one for deletes and a number of threads for each operation. When the sync manager passes the request to copy files to the task manager, it is a non-blocking call. The downloading/processing of each file happens N files at a time, depending on how many threads you’ve allowed to be active. The defaults are 5 for copying and 2 for deleting.

So far, this seems to be working pretty well. The only thing that I see that could use some immediate improvement is to either thread the downloading of the RSS feed(s), or to use Doug’s feedcache, or both. I’m going to try to work on trying to get that supported over the weekend.

The next area of functionality addition is that of creating a GUI. From the feedback I received in a post today, I’m going to have to check out wxPython.

As reported first on Groklaw, it appears that Judge Kimball has ruled in summary judgment on the major point of the SCO vs Novell case, that of who actually owns Unix. Not surprisingly, the answer to that question is not SCO.

While IANAL, a brief reading of the juicy parts seems to essentially say: A) Novell owns Unix; B) Since SCO doesn’t own Unix, they can’t sue for slander of title; C) Since Novell DOES own Unix and SCO’s been claiming they have (and collecting license fees for it…), Novell should feel free to continue their own slander of title claims.

Anyone like to take bets on the current half-life of SCOX? Now on to FUD round 2: Steve Ballmer and the Goblet of Patents.

Does anyone have any tips on GUI building on Mac using Python? It looks like PyGTK for Mac isn’t totally there yet. PyQT for Mac looks really out of date. Is building a Cocoa app using PyObjC my best bet? The more native, the better. The fewer dependencies, the better.

A couple of disparate threads have bounced around in my head lately, which makes me think that there’s some fundamental notion at work in the world.

Mitchell Baker and Matthew Gertner have had a brief back and forth over the nature of a public good (such as the Firefox web browser in specific, or free software in general). Mitchell’s position is:

A people-centered Internet needs some way for people to interact with the Internet that isn’t all about making money for some company and its shareholders.

— Mitchell Baker, Firefox is a Public Asset

Matthew Gertner wonders if corporate backing is necessarily, in itself, inimical to the creation and community-based maintenance of such a public good:

This isn’t about a small group of people trying to get rich. It’s about putting into place the most efficient overarching structure to achieve our common goals of choice and innovation on the internet.

— Matthew Gertner, More on Mozilla and Capitalism

The other thread synchronous in time comes from a comment Tim O’Reilly made a week ago:

I will predict that virtually every open source company (including Red Hat) will eventually be acquired by a big proprietary software company.

— Tim O’Reilly at [08.02.07 11:47 AM] in Microsoft to Submit Shared Source Licenses to OSI

Sometimes I wonder if the fateful 1998 meeting which gave birth to the term “Open Source” led the world of software freedom down a dark path. If the only way to get business to adopt the idea of embracing the power of communities to build software and ecosystems larger, more powerful, and more efficient than individuals could build on their own was to focus on economic principles, rather than the notion of the public good, is it any wonder that so many businesses seem to be indifferent at best to the health of those public goods?

To switch rhetorical metaphors, do you find it more likely that any given business would invest N% of its budget in energy-saving measures because it considers the investment ethically right on its own merits, or because it saves money and provides the basis for a nice, friendly press release?

Perhaps it’s inevitable that community-driven development, maintenance, and support will reduce markets for proprietary software up and down all of the stacks. Perhaps the most successful projects will have the strong support of businesses.

Do you want to rely on their goodwill to allow you to use, study, and redistribute software as you see fit? Are you willing to take the risk that they will encourage a healthy commons which allows you to use your data as you see fit?

I’m not sure.

Updated on 2007-08-10; corrected misattribution of Matt Asay to Matthew Gertner

Piers Cawley decided that his little language embedded in Ruby is a pidgin:

Casting the problem domain as the colonial power and ruby as the native language, it’s obvious that I’ve invented a pidgin language.

The nice thing about a pidgin (or a dialect, as Ben Scofield suggests) is that all of the power and syntax and semantics of the host language are available if you need them to express concepts that your little language cannot–without modifying the little language at all.

There you go; there are two good terms for describing the use of domain-specific language within a full-fledged programming language without co-opting a term with a perfectly good, existing definition.

Yesterday I heard of a government agency where the manager required employees to post their account names and passwords on a bulletin board, so they could get into each other’s systems in case one of them was out for a day. This was told to me, along with examples of other security policy lapses, by an IT consultant who works for government agencies.

Many workers in human services, she told me, are reluctant to provide data that would be useful to improve the services. She’d like to track homeless people as they move from one jurisdiction to another for instance, to provide better continuity of service and find out what works and what doesn’t. The agency staff are afraid that sinister forces within government will misuse data. While we have no lack of sinister forces in government, it appears that the people needing human services are more at risk of snooping by random staff people, facilitated by the awful security practices just mentioned.

I’m not surprised that employees would treat passwords as just one of the many random impediments they have to bypass each day to do their jobs. Given how many regulations reflect political grandstanding rather than life on the street, and how many well-meaning regulations outlive their usefulness, workers have to interpret the rules in a (shall we say) creative manner. I’m sure many employees in private industry get through the day the same way; it’s not limited to government. But an even deeper issue is at work.

For everyone who read my A Taste of Haskell, A Taste of C, and whose eyes glazed over when I told Nat that Haskell functions only take one argument, John Goerzen explained what I meant in Haskell manipulates functions with the same ease that Perl manipulates strings..

If you watched A Taste of Haskell - Part I, Simon Peyton-Jones answers my question about parenthesizing expressions about halfway through; that’s when I was trying to explain this feature to Nat and made his brain hurt. (I realized halfway through SPJ’s answer that I had forgotten about associativity.)

If none of this makes sense to you and it still all sounds somehow cool (and once you get past the syntax and avoid some of the math, it’s seriously cool), check out Adam Turoff’s Why Haskell?, Pure Functions in Haskell, and Monads, for a great introduction to the language for LAMP programmers.

You could do a lot worse than to grab a lazy Saturday afternoon and work through the code. It’ll expand your mind in very good ways.

XPath is a complex query language that provides substantial benefits. It treats XML as a database, permitting queries as powerful as SQL SELECT. This post shows how to use a nifty visual explorer, XPath Checker, to write aggressive and accurate queries. Then we install these queries into test cases using assert_xpath.

But I don’t want to live in a world where the only thing the Internet is useful for, or effective at, or pleasant or fun, are activities where someone is making money from me.

— Mitchell Baker, The Internet and the Public Good

I feel the same way about free software.

I spent some time the past couple of days wrestling with HTTP authentication for a WSGI atom store implementation. It’s well known to be a tricky topic, and I went through a lot of bother, especially trying to figure out a Python WSGI library for HTTP auth that was clean, well-documented, and sported a flexible API that worked well across framework choices. I ended up using httplib2 for the client and decided on AuthKit on the WSGI server. I generally like to test my WSGI Web components with CherryPy, Pylons and raw Paste. In this case I didn’t get around to Pylons, I was able to get raw paste working well with code along the lines of the bundled example http://authkit.org/trac/browser/AuthKit/trunk/examples/authorize.py. I was never able to get things working with CherryPy, and I’m not sure why. I ran out of time to debug further. It seems CherryPy doesn’t like the HTTP status line sent by httplib2 with a www-authenticate response to a 401 using Digest authentication. I think this might have nothing to do with Authkit.

One problem I found with AuthKit is that I had to manually place ez_setup.py in the PYTHONPATH before the install would work. I might be doing something wrong, but this is not a problem I’ve had with other packages.

What Python tools do you use for your HTTP auth needs?

Kirrily Robert dissects a meaningless job posting for a Perl programmer.

This reminds me of a story Jim Shore told me about Fit. Developers in a company wanted to use the software, but their lawyers had grave concerns about the license. Eventually, the developers appealed to Ward Cunningham, who said that they were using it in the way he intended and he had absolutely no intention of bringing suit or other legal action against anyone who used his software appropriately. Even so, the lawyers saw that as an unacceptable risk.

The punchline is Ward’s final question. “You have to ask, do you work for your lawyers or do they work for you?”

Perhaps it’s time to ask that of HR departments.

This is the third in an N part series on rewriting my podcast grabbing application. Here are the links to parts one and two. In part two, I promised to get into a common way of synchronizing media files between media stores.

As I read reactions from people to JT Smith’s Perl is Dead. Long live Perl. I see the usual knee-jerk claims that Perl inherently leads to unmaintainable code.

In my vast experiences in dealing with difficult-to-maintain code, I’ve noticed that nearly all of the messes I’ve seen had comments, documentation, and identifiers written in English. English is not an easy language to learn. It has inconsistencies (irregular verbs! homophones! homonyms!) and quirks (idioms! punctuation styles! possessive marks!) which make writing perfectly correct English–or even succinct and direct English–difficult.

There’s one good reason it’s difficult to produce correct and successful software from a full specification written at the start of a project.

Yet somehow it’s acceptable to allow programmers, presumably smart people with the ability to juggle small details, to write terrible, horrible, incomprehensible English but not comprehensible, concise, and correct code in languages which are orders of magnitude simpler than English. Hey, if you can use the pronoun “it” appropriately in English, Perl’s topic variable $_ should give you absolutely no trouble!

If (perceived) simplicity and regularity of the language were truly an important factor toward the maintainability of software, we should all be using Esperanto or Lojban to talk about our software. It’s not as if we don’t expect programmers to be able to learn new programming languages where appropriate.

Perhaps the true source of maintenance problems lies elsewhere.

Saturday, November 9, 2002 was my first Weblog entry, right here on O’Reilly Network. In 2005 I started using my personal Weblog Copia and soon I was posting exclusively there. On Copia I post on everything from music to literature to politics to technology, and I’ve long wanted to find a way to primarily post my tech writings here on O’Reilly Network, while having this aggregated with my other personae on Copia. This, “personal aggregation” as I call it, is one of the motivations for me to move from the PyBlosxom platform of Copia to a new one I’m building from scratch, with a few other co-developers. Bright Content (I don’t yet want to link to it because i think it’s not entirely ready for any attention) is a Weblogging system that embodies much of my long-standing interests in technology architecture relating to content.

But I’ll talk more about that project later. Meanwhile I’ll start the process of moving my technological Weblogging persona back to this Weblog, while ‘ll continue with other topics on Copia. It’s actually good timing for a different reason: after a separate one-year hiatus I recently renewed my agile Web column with the new article Introducing OpenSearch, a topic I chose in large part because of the resonance of OpenSearch with my philosophy of content technology. I look forward to continuing my conversations with you, through my articles and Weblog entries on this site.

Summer is flying by and as we usher in fall, we wanted to give all New Englanders a heads-up that we are having a second Ignite Boston. The second Ignite Boston will take place on Thursday, September 6, from 6 to 10pm at Hurricane O’Reillys. Yes that is right, Hurricane O’Reillys. No, it’s not Tim’s office after FOO Camp. We’ve picked a venue that is more acoustically-oriented and should allow everyone to hear what is going on. And we are planning to mix-up the format a little bit. There will be some short “launches,” followed by lightening talks, and a couple of other ideas that we will inform you of in the coming weeks. Let’s show our tech colleagues around the country that Boston/Cambridge have a vibrant tech community that gets involved in talking about cool new technologies and ideas. Not to mention that it is a social event to get to know other developers in the area.

If you plan to attend, email IgniteBoston at oreilly dot com for the chance to win $300 worth of O’Reilly books of your choosing. You must be present to win.

If you are interested in connecting with some of the folks who attended the first Ignite Boston, we have a social network set up for this purpose. You can reach our Crowdvine network here.

Another reason we wanted to announce this event this early, is so those of you who would like speak for five minutes on something cool, new, or exciting you can get into the queue sooner rather than later. Please submit your idea/s here:

Presentation Guidelines

• Be no longer than 5 minutes.
• Be on an innovative topic (no sales pitches, please!).
• Be viewable on a PC [a MacBook Pro with Powerpoint, Keynote/has remote control, and PDF] with standard AV equipment.

To submit a proposal.

For anyone that’s never been to Ignite, you may find it useful to see a talk or two. Here’s a link to a good example [but poor audio quality] from the first Ignite Boston talks.

Technorati Tags: Boston, Cambridge, Event, Ignite, O’Reilly Media

JT Smith, president of Plain Black, the creator of WebGUI, and one of the unsung successes of using Perl in business, recently sent me this essay. He gave me permission to publish it in its entirety here.

(In the interest of full disclosure, the plush WebGUI octopus I have from YAPC::NA last year is one of the coolest pieces of swag outside of Hollywood, ever.)

c|net had a story up a couple of weeks ago entitled “Kids say e-mail is, like, soooo dead”. The story was about how kids were moving away from email and using instant messaging and social networking sites for peer communication. Maybe I know why. Or maybe I know why that could be such a temptation. The reason is simple. Email sucks. All email clients suck. No one of them has the feature set that I would like. Their handling of the email protocols can be atrocious, particularly IMAP. And don’t even get me started on having to interface with Exchange with anything other than Outlook. Also, you don’t know when someone has read an email (unless you’re using X.400, but that’s another story) or when they’re online. I can see why IM, social networks, and text messaging are gaining usage. Pownce, anyone? (BTW - I have a few invites left. If you’re interested, email me.)

UPDATE: The Pownce raffle is now officially closed. Thanks for everyone who emailed me.

My wife’s business has the need to make some sales online and so naturally I want to help. I know next to nothing about E-Commerce, so I started to do a little bit of research. It appears there are three options available to people currently in the market for a shopping cart that will process credit card transactions. There may be more options as well, but this is what I have found on short notice.

Option 1:

It appears to be relatively cheap to just host a whole website with a shopping card and credit card transactions built in. Google Checkout has links to a couple of complete solution providers that offer WYSIWYG admin panels and editors for somewhere between 30-50 dollars a month.

Option 2:

Integrated Solutions that you build into your existing framework/website. They appear to offer differing levels of flexibility, ranging from an API, to a link to another website that has a cart.

Option 3:

Something like Satchmo which is a “webshop” for perfectionists with deadlines. I found out about Satchmo through a fellow member of PyAtl who had some success with it.

Why does E-Commerce have to be so hard anymore? Maybe it isn’t that hard, and I am very ignorant. It seems like 2007 might be a good time for mere mortals and Mom and Pop businesses to start taking orders online. So, what is the best solution for minimal effort? Is there a solution that works well with Python, and/or Python Web Application Frameworks?

On one level it would be great to know there is a completely Open Source solution to a relatively simple problem. After all, why pay 50 bucks a month if you can easily do it yourself. I think VOIP is a good example of small business doing Phone Service themselves, and the same model might apply for E-Commerce. On the other hand, processing credit cards is somewhat risk prone, so perhaps there is a middle ground that still serves as a good price point for small business owners.