At OSCON last week, my favorite tutorial was Simon Peyton-Jones’s A Taste of Haskell (A Taste of Haskell presentation materials (PDF)). I’ve dabbled with the language before (and pair programmed with Audrey Tang once to find and fix a bug in Pugs), but I’ve never really felt comfortable with the language.

It’s nice to report that function composition actually makes sense to me now. (At Foo Camp, I spent ten minutes explaining to Brian O’Sullivan why and how it tripped me up every other time I tried to figure it out, so hopefully Real World Haskell will smooth out that path for other people.

Most of my work these days is in C, though. As a junior language designer, I spend a lot of time considering the differences between languages (and, yes, rarely at the syntactic level).

CIO Magazine just ran an article looking at how one CIO coped with going cold-turkey from Windows in a corporate environment. Although there were the usual glitches with proprietary corporate software, it is on the whole a very positive look at the practicality of bringing da Penguin into enemy territory.

A Perl demographic survey is now open. From perlsurvey.org:

Take part in the 2007 Perl Survey!

The Perl Survey is an attempt to capture a picture of the Perl community in all its diversity. No matter what sort of Perl programmer you are, we’d love to hear from you.

The survey can be found at: http://perlsurvey.org

It only takes about 5 minutes to complete.

The survey will be open until September 30th, 2007. After that, we’ll be reporting on the results and making the data freely available.

Please feel free to forward this email to any other Perl programmers you know.

Thanks for your help!

Yours,

Kirrily “Skud” Robert
The Perl Survey
info@perlsurvey.org

Scott Walters recently ranted on how Perl programmers tend not to perform OO analysis and design, at least when compared to Java programmers.

Setting aside issues of language design–though they’re important, I think there are other insights available–it does seem as if there’s much more literature available for developers who want to design large systems in Java than there is for developers who want to design large systems in Perl. (I operate from the assumption that the number of people who could write a program in Perl is probably the same as the number of people who could write a program in Java, if not greater, so I set aside questions of market size as well.)

Aristotle Pagaltzis suggested I repost my analysis here. In short, it’s all about the teachers.

I’ll be at the Black Hat briefings in Las Vegas this week. In addition to the briefings, I enjoy going to Black Hat to reconnect with old friends, and to make new friends in the security industry. It is also a delight to meet people who read my blog and to have the chance to hear their thoughts and philosophies.

If you will be there as well, and would like to catch up, please send me an email.

At OSCON I attended a handful of the sessions that were sessions were related to web application development. I’ve compiled a list of features I’d like to see in the next web application development framework:

• Give me continuations. Let my request handlers check whether the user has authenticated and, if they haven’t, prompt the user and resume execution in the handler without losing state. If the user has requested to delete something, confirm the deletion first. I want this type of logic in the controller, not as a confirm() method in the JavaScript.
• Let me attach listeners to server-side models and update the interface when the data structures have changed. For example, given a list of favorite movies that is stored in memory or in the database, I want a <div> to be modified automatically when a favorite is added or removed.
• Keep the API simple. Give me little languages, DSLs, data structures or whatever the fancy name for them is. Let me focus on the logic of what I’m doing. Don’t limit these little languages to only configuration files — Let me write my HTML templates in whatever language I’m using throughout the rest of the framework.
• Make sure testing is a breeze. Let my tests “click” on <div>s and make assertions on the contents of elements.

Fortunately, there are a few frameworks that have some of these features already.

• Google Web Toolkit lets you write applications entirely in Java. The Java that needs to be run client-side is compiled into JavaScript. Designing interfaces in GWT is similar to the popular toolkits for desktop applications, and the testing framework seems solid.
• Jifty, a Perl framework created by Best Practical, includes a lot of the aforementioned. Templates and tests are all written in Perl in a mini-syntax, and they’re working on compiling Perl to JavaScript. It even has continuations.

Above all, I never want to have to write an <a href="..."> tag again. Giles Bowkett, in his HREF Considered Harmful talk, explained that it’s the modern equivalent of GOTO. I agree.

When a test case calls methods that write new records to a database, sometimes the test needs to fetch those records back and inspect them. This post develops assert_latest, an assertion that detects newly created records.

The news over the past few years in open source establish it as the natural way to release software. If there’s anything else you can do to earn money–whether setting up a social environment like Second Life, putting up ads like Google, or selling hardware like Intel–you really need to search hard for a reason to keep software proprietary. The benefits that free software reaps from contributions and community are demonstrated beyond a doubt, and the mechanism for releasing software as open source is now familiar.

Here’s my wrap-up of the Open Source convention. I published an earlier blog on it as well.

Andy Lester is renown for his evangelism of technical debt awareness, and his talk at OSCON was full of rich lessons on improving your code and yourself.

Managing upward is a theme in his talk, but it isn’t his primary focus. Andy wields upward management as a tool which can be used to improve code quality, but it’s a topic that deserves significant attention as well.

“Talk in dollars. This is the language that management understands.” — Andy Lester

Nobody develops for Perl anymore, CPAN is too crowded.

    Jordan Henderson

This post introduces developer tests that constrain exceptions. Our platform, as usual, is Ruby, yet these topics apply to any system. We will extend assert_raise() for more control over program faults.

The best developers write tests to keep their projects on track. Tests should cover every aspect of a program, and should take special care with program details that are sticky, hard, and mysterious. Exception handling is a murky topic, because when a program fails its control flow might not be obvious. Many a program has failed in the field because nobody in the lab tested all its error paths. Our test cases must ensure that faults make our programs degrade gracefully, not derail.

Twin conferences have been taking place at the convention center in Portland, Oregon this week: the O’Reilly Open Source convention and Ubuntu Live (also partly sponsored by O’Reilly). As we ramp up at OSCon, evidence of the drive toward openness in society continues to roll in.

Senator Dick Durbin has announced an online forum about broadband policy that started last night. Today’s news that iPhone sales are disappointing provides one illustration of the importance of this issue. While the device is overpriced, I have no doubt that a key drag on its uptake is the slow network AT&T has devoted to it. This would have been considered ludicrous in East Asia.

US Government, as we all know, is fleeing from openness as fast as its so-called leaders can run. At Tim O’Reilly’s open source briefing yesterday, open source advocate and version control expert Karl Fogel presented a case for recording and releasing all communications that go into making laws. Apparently the New York Times picked up Tim’s blog on the subject.

The briefing also presented open web APIs, open source hardware, and other examples of how the open source movement has spilled over its origins in free software. The most popular free software packages are still infrastructure: operating systems, languages and language tools, system administration packages, and so forth. But there’s no doubt that everybody is evolving in response to this powerful model for encouraging creativity.

Billy Rios has let me know about another vulnerability he has found along with Nate McFeters. Here are the URLs, which when clicked from Firefox running on Windows should spawn cmd.exe and calc.exe in order to demonstrate remote execution flaws in Firefox:

At OSCON, David Adler of The Perl Foundation presented the 2007 White Camel Awards to recognize significant non-technical acheivement in the Perl community. Perl Mongers started the White Camels in 1999 and has presented three of them every year. This year’s recipients are:

Allison Randal

Allison is at the center of the Perl community. She’s been president of The Perl Foundation, a leader and manager of various parts of the Perl 6 and Parrot efforts, as well a Perl author and editor. Her latest contribution to Perl is version 2 of the Artistic License, under which most open source Perl code, and Perl itself, is licensesd “under the
terms of Perl itself”.

Tim O’Reilly

You may think of Tim as the guy who published
Programming perl and Learning Perl, but he also kick-started the current form of the Perl community by giving it a place to come to together once a year. O’Reilly & Associates started The Perl Conference in 1997. Perl Mongers, the organization that helped start Perl users groups all over the world, started at that first Perl Conference. O’Reilly Media has been incredibly gracious and helpful to the Perl community.

Norbert E. Grüner

Norbert help start the German Perl Workshop in 1999 and now is involved in several of the Perl conferences and workshops that take place in Europe. He’s the chair of the YAPC::Europe committee.

The best hack I saw in OSCON yesterday was in Jonathan Oxer’s Hardware / Software Hacking: Joining the Real and the Virtual.

Jonathan brought a table lamp in a whole box of breadboards and other electronic components from Australia. He connected the table lamp to a remote appliance control. Then he opened the remote control and connected a relay across one fof the buttons. The relay hooked up to an Arduino also connected to his laptop.

He then wrote a small program to run on the Arduino. By sending a single command through the USB port, the Arduino flipped the relay and turned off the light.

That’s not the end of it. He ran ser2net to redirect requests from a network socket to the USB port. Then he wrote a very small PHP program which translated CGI parameters and sent the appropriate results to the local network socket.

Within Second Life, you can create an object which invokes a script when touched. This script can make HTTP requests; Jonathan had an off button object.

“Ha!” I thought. “He’s running this CGI program on his laptop, in the conference center, and there’s no way that Second Life can reach his web server through all of the layers of NAT and connection sharing going on here!”

Then he said, “How many of you have heard of reverse tunneling?” ssh -R opens a port on the remote host that forwards to the local host. With a quick connection to his web server in Australia, he switched back to his SL client, clicked on the off button, and the lamp turned off.

It had been shining in my eyes all morning anyway.

I just read an article at Linux.com about the OS habits of Linux users. The author of the article asked Linux Torvalds about his habits and found he exclusively used Linux. Torvalds said, “I don’t use either [Windows or Mac OS X]. OS X is kind of pointless (pretty much anything it has, Linux can do better) and Windows offers stuff that I don’t much care about (mainly games — and I’ve got games machines for those).” Before I comment on this any more than I have, let me just say that I have been a near-exclusive Linux desktop user since 2001. I love Linux and I still think there are some areas where Linux dominates. However, to Torvalds, I say, “hogwash”. He can get around careful scrutiny by his choice of words “pretty much anything it has” and “Windows offeres stuff that I don’t much care about”.

But the spirit of what he’s saying seems just dead wrong. And it’s an attitude that’s pervasive among many Linux enthusiasts. Whether Torvalds has the zealotry I’m about to discuss is irrelevant. It’s just a launch pad for me to address this attitude. The attitude goes something like this. “Linux is an awesome OS. We’ve come so far in a short amount of time. We have everything anyone would need. And bling to boot. Linux is ready for the desktop. In fact, it’s ready to take over the desktop.” There is a lot of truth in what both Linus said and my characterization of the Linux zealot.

Truth 1: Anything that Windows and Mac can do, Linux could do. Notice my choice of words. I said that Linux could do anything that Mac and Windows can do. But the sad story is that Linux is not currently doing a lot of what Mac and Windows is doing. Please, please, please someone show me wrong on this! Please show me a DVD authoring application on Linux that is as easy as iDVD is on Mac. Or a video editing application as easy as iMovie. Please! Yes, I know that there is wine and you can often get Windows apps running on Linux. But 1) it’s hit or miss and 2) most of the apps that I’ve gotten to run in wine look…..let’s just say “bad” to sound polite.

Truth 2: Linux has made considerable strides in the past few years. The desktop looks spectacular (both kde and gnome). There are tons of top notch applications available for free download. Hardware recognition and support works better than it ever has. I don’t think this one needs to be shot down, so I’ll leave it as it is.

Truth 3: Linux has the bling. In my opinion, Linux is actually winning the bling war. Just check out Beryl/Compiz/Fusion. It rocks. For bling, it is (my opinion, again) unrivaled. And there are some pretty cool productivity enhancements, too. But let’s not confuse bling and even cool productivity enhancements with a usable desktop (not that Linux has an unusable desktop). They aren’t necessarily the same thing. Just because Linux has bling doesn’t mean that its applications are well integrated with one another. Or that the applications work well on their own.

The point of this rambling is that all three of Mac, Linux, and Windows do some things well and other things not so well. Personally, I don’t like getting on Windows. It feels square and wooden. But it does some things pretty well. And I just bought a Mac a few weeks ago and I’m really enjoying it. I’m not at all ready to say that neither Windows nor Mac is really competitive with Linux. Conversely, I’d say that each of them spank the other two in some areas and don’t do so well in other areas. So, can we please let the zealotry die? Please?

Update: I’m closing the comments on this blog post because of recurring blog spam. If you want to carry the discussion on, please email me and I’ll post a new entry.

PyAtl is switching to Plone as time is short and we want a CMS that everyone can edit. A few people are are working on getting Plone configured and we are looking at using the gmail authentication plugin and the forum plugin.

All of this has got me thinking? What is the story on Plone and/or Zope? There is all this talk of Django/Turbogears/Pylons, etc., but what about Zope and specifically Zope3 and Plone? In the younger crowd you almost never here anyone talk about anything related to Zope and I wonder why?

So, what is the dope on Zope? Can you turn Plone into the next myspace? How hard is it to learn Zope3?

I’ve seen at least two mock object libraries for Python (here and here). But I wonder, what is the benefit of using a simpler mock object over creating your own dummy class? It seems that if I created my own dummy class (a class which implements the same interface that it is attempting to “mock”), I would have tighter control over the behavior of the thing as well as have a nicer re-use experience. It seems that mock objects are typically defined on the fly and then thrown away. I’m sure you could re-use them, but if they are really intended to be use-once-then-throw-away, it may be a little harder.

I guess what I’d really like to see is something between mock objects and dummy classes, something where you define a dummy class to be used as one of your application classes, but which also contains the convenience methods that the two mock libraries have. Suggestions, anyone?

I’ve been carrying around an interest in text processing for several years now which began with my work with EDI. Even though I don’t work with EDI and my job doesn’t revolve primarily around text processing, I still maintain an interest in text processing in general and processing EDI specifically. I created the project ediplex using Novell forge probably two years ago, around the time I wrote this article for DevX. ediplex back then was specifically an EDI processing engine with hopes of converting EDI to other formats pretty easily.

Over time, ediplex has evolved. A goal that I had for ediplex even from the beginning was the ability to easily define new EDI file formats. In its inception, it only supported X12, which is primarily a North American standard. But I had hopes for supporting EDIFACT and TRADACOM, which are more in use in Europe.

Which leads me to today. The latest incarnation of ediplex doesn’t support EDI. Not yet, anyway. What it does is allows users to create custom document definitions which describe what a document’s header and footer should look like. It also allows users to create custom handlers to allow the engine to feed them with data for a specific document type. The latest rendition is in early alpha, but it looks like a document is being passed all the way from its input to its handler. If you’re interested you can `bzr branch http://bzr.ediplex.org/trunk/` and start poking around. (This requires the Bazaar version control client.)

The architecture for ediplex is layered, but pretty simple. The first layer is the input layer. This layer gets input from somewhere (file, socket, whatever) and passes data to the scanner, which is next. The input layer was designed to allow users to create their own custom types of input receivers as they see fit. The next layer is the scanner. While this layer can certainly be replaced and customized, that shouldn’t be necessary. The scanner receives data from the input receiver and determines which document type the text should be passed off to and passes it off. The next two layers are the document definition and the data handler. I combine there here, because they are combined in the ediplex code. The document definition doesn’t do much except for describe a new document type and tell the scanner if a certain string of text matches its definition. The handler is intended to be extremely customized. When it receives data, it gets to do whatever with it that its little heart (and its coding master) desires.

So, if you’re in the market for a text processing engine, check out ediplex. I don’t have a license statement in the source tree, but will soon. I’m strongly leaning toward the MIT license, but am also considering GPLv2. Questions, comments, flames welcome.

I’ve been listening to The Linux Action Show podcast for a few months now and really enjoy it. I’ve tried other Linux podcasts, but they seem focused on the noob level. Can anyone recommend a good Linux podcast for experienced Linux users?

I’m pretty good at Perl, so I hear a lot of comments about programming language syntax. Many of them are fluff around the old argument that “I don’t like to read punctuation.” Many of them bring up the silly idea that an ideal programming language syntax should be so intuitive that people who’ve never used the language before should be able to understand programs written in the language.

That’s a ridiculous argument.

Next week is OSCON, so there will be plenty of wonderful programmers in Portland. We’re taking advantage of this to host a Beautiful Code discussion panel at Powell’s Technical Books, just across the river from the conference.

The panel will consist of the Beautiful Code contributors Karl Fogel, Greg Kroah-Hartman, Simon Peyton-Jones, and Andy Oram. Ward Cunningham will moderate, and I’ll chip in where I can.

I have my copy of the book already and I’m trying to figure out what to say that isn’t already in there… I’m really looking forward to hearing what Ward and the other panelists have to say.

Even if you’re not an OSCON or Ubuntu Live attendee, you’re welcome at the panel. I hope to see you there.

For anyone that plans to be in the Atlanta, GA area on Aug. 9th, I hope you can attend the PyAtl meeting. We have Juan Pablo who is a Django instructor for the Big Nerd Ranch, traveling in for a special Django presentation. This should be very exciting as he is a true pro. I also invited our local Atlanta Ruby group to attend as well. I think it would be great to have more Python/Ruby crosstalk as the languages have so much in common.

For our second meeting, we have up and coming author, TurboGears committer, and Elixir co-creator, Jonathan LaCour. He will be giving a presentation on SQLAlchemy + Elixir.

I was a bit bummed out when I moved from Los Angeles about a year and half ago, but it turns out that Atlanta, GA is one of the Python powerhouses cities in the United States. At this point, we have the largest meetup group, although I realize not every group uses meetup. I know quite a few of us are working on books, open source projects, and commercial projects in Python. If you live around Atlanta, GA area, program in Python and haven’t yet attended, your missing out!

If your city is a Python powerhouse, let me know!

It’s been some time since I don’t follow closely Lua development. But I try to keep updated with what’s going on. The announcement of LuaPOD 0.1 caught my attention (due to the gathering of three technologies I find quite interesting).

XSRF (Cross Site Request Forgery) is a huge security problem affecting most web applications. There have been a lot of articles written about XSRF, including the useful XSRF FAQ I linked to earlier.

There are quite a few free and commercial web application security assessment tools and static code analysis tools in the market today. A few commercial security assessment tool vendors have published white-papers about the importance of discovering XSRF vulnerabilities, yet their own products do not have the ability to assess for XSRF. I think there are multiple reasons for this, and here are my preliminary thoughts:

People who read my blog regularly know I’ve been researching what happens on mailing lists and in other forms of free online documentation. I now have a sort of portal or home page for the resulting articles. I’ve just published the most recent one, How to Help Mailing Lists Help Readers (Results of Recent Data Analysis). I hope to put up some other interesting experiments besides articles in the next stage of my work. I’ll be speaking about this research at O’Reilly’s Open Source convention on Wednesday, July 24.

I program in C very reluctantly. I don’t hate the language, but it occupies a curious niche between assembly language (where you can do absolutely anything, if you’re willing to write it yourself, and eval is trivial) and a true high level language (where you can do absolutely anything, you don’t have to write it yourself, and eval is available for everything else). Yet it’s ubiquitous, it has a lot of libraries, and it’s probably the best way to write reasonably efficient code that has to run on plenty of platforms.

Because I’m writing a lot more C code lately (and of that, finding and fixing a lot of bugs), I’ve spent a lot of time using the GNU debugger GDB.

As a reluctant programmer in general, I spent many years happily debugging with print statements, and then plenty of years debugging with comprehensive test cases. When you’re writing a virtual machine and your test cases are all in a high level language, you don’t always have that luxury, especially when you have segfaults.

I already knew the value of backtraces, breakpoints, and printing the value of local variables. Then I forced myself to learn a few more tricks to make the most of the debugger. For example, breakpoints can take conditions. That is, you can write break src/exceptions.c:59 if exception->type == exception_class_NULLACCESS. Learning that alone paid off several times over.

The other feature I forget after a couple of months away from marathon debugging sessions is that p can dereference a pointer to a struct. That is, if you have a Coord pointer in the variable coords, use p *coords to see a serialized version of the struct and its contents. Handy!

I could talk more about using up and down to walk up and down the call stack after a breakpoint, but even learning only three or four useful commands has already cut out hours of debugging time in the past month. (I even found myself wishing for a better debugger in one of the HLLs I was working on.)

Thanks to all of the contributors to GNU GDB and its ecosystem; you’ve made it easier for me to write further free software.

URI Use and Abuse written by my good friends Billy Rios, Nathan McFeters, and Raghav Dube (affectionately known as “baby Dube”) exposes how web browsers and applications fail to sanitize URIs leading to remotely exploitable conditions.

Billy started the ball rolling (after deriving inspiration from Thor’s Safari URI handling disclosure) when he discovered a remotely exploitable vulnerability in the firefoxurl handler. An example of his this can be exploited in IE is available from Billy’s disclosure: Click on this from IE to spawn cmd.exe (remote execution). Note: cmd.exe will spawn regardless of any IE or Firefox dialogs.

Large, complex, cross-platform applications with multiple developers sometimes have bugs. Some of those bugs never appear on your own machine; they lurk for a while until someone else builds and tests the software on a different platform, in different circumstances.

Tracing that bad behavior back to a particular checkin can be frustrating, even if you have a huge smoke farm that rigorously tests every configuration of every checkin on every important platform.

Parrot meets all of those criteria, except for the huge smoke farm. (Smokers welcome.) When I want to pinpoint a regression to a likely checkin culprit, I use a binary search. Will Coleda’s App::SVNBinarySearch promises to automate that process. Here’s what I found.

I just noticed a post on Grig Gheorghiu’s blog that mentions that Pownce is built on the Django web framework. Here is an interview with Leah Culver, the lead developer for Pownce, and here is Leah’s blog post on Pownce. For anyone not aware, Pownce is the new brain child of Digg creator Kevin Rose. Pownce is supposed to be a better Twitter and/or Jaiku.

In the spirt of health competition, and because I selfishly want to know what blogs to fill my Google Reader account with, what are the Top Ten Python Related Blogs? Also, how many RSS feeds should I take? What is manageable?

What are the Top Ten Python Blogs you read either “old school” or through an rss feed? Here are ten I happen to read in no particular order but feel free to order yours by whatever political statement you choose:

Tornado of Testing Titus
Dynamic Diatribes of Doug
Glitsy Glyf
Enigmatic Ian
Jazzerific James
Jewels of JJ
Jonathan’s Juleps of Joy
Ricky the Tricky Raccoon
Natural Selection Niemeyer
Jumping Jeremy Jones

IBM announces an interesting initiative today that will make it easier for open source programmers and other small coders to put together a range of software, including Web Services. The initiative is just a subtle procedural change, but a welcome one for people with little time and tolerance for bureaucratic red tape: IBM’s wide range of royalty-free patents are now available without formal licensing.

OK - so I haven’t done a “recipe of the week” in a while. But does titling the post “{{whatever}} of the week” mean that I’m going to do one of these every week, or does it mean that I promise not to do more than one per week? :-)

Anyway, I was googling around to see if a certain type of utility existed and I stumbled across this recipe for something called Pyline. Basically, Pyline allows you to pipe text to it and use Python syntax to manipulate what it will output, specifically at the word and line level. Here are a couple of examples from the recipe:

Print out the first 20 characters of every line in the tail of my
Apache access log:

tail access_log | pyline “line[:20]”

Print just the URLs in the access log (the seventh “word” in the line):

tail access_log | pyline “words[6]”

Good work Graham Fawcett. This is a useful little utility and the code is pretty brief. So, this is the recipe of this week.

Here is a press release on Dr. Dobb’s regarding the Storm ORM.

And here is a discussion on reddit about Storm.

It appears that Gustavo Niemeyer created Storm when SQLAlchemy didn’t exactly meet his needs (and after contributing some code to the SA project).

From the Dr. Dobb’s press release:

“Storm is particularly designed to feel very natural to Python programmers, and exposes multiple databases as stores in a clean and easy to use fashion.”

May SQLAlchemy and Storm feed off of one another, provoke one another to higher levels of excellence, and live peacefully with one another.

A couple of weeks ago, Noah Gift and I signed a contract with O’Reilly to write a book on Python for System Administrators. We’ll be covering topics ranging from creating command line utilities to processing text to interacting with databases to SNMP to a bunch of other fun stuff.

Noah just stumbled across Storm, an ORM created by Canonical (the folks who brought us Ubuntu) and has blogged about it.

Question one for the readers: is Storm something you’d like to see covered in the book?

Question two for the readers: is there something you’d specifically like to see in the book Noah and I are working on (taking into consideration this is a Python book for system administrators)?

Your thoughts are graciously welcome.