Although I have forthcoming posts that involve 2D drawing with dojox.gfx and rendering editable data with the really awesome dojox.grid enhancements that are landing in version 1.2 of the toolkit coming up later in the month, I just rediscovered a “note to self” (for about the tenth time) suggesting that I should write a quick blurb on the dojo.date module; right now seems like a good time as any to go ahead and check that box.
During the next few months, I will be presenting a brand-new talk titled "Suddenly Psychic: Knowing Everything About Everyone" at various conferences around the world. I will be presenting it with Akshay Aggarwal, a good friend of mine. Akshay and I have enjoyed researching the business, security, criminal, social, and psychological implications of this topic, and we look forward to sharing our research with you.
Currently, this talk is scheduled debut at the Microsoft Blue Hat Conference [v8] in October, followed by Hack in the Box in Kuala Lumpur.
I thought I would officially announce that Mike Orr and I are writting a book for Manning on Google App Engine.
It is a bit strange to be working on a new book, when the first book I wrote isn’t even released yet, but Mike and I plan to finish this book around December08/Jan 09, which is a very aggressive writing schedule. We will be releasing PDF’s of each chapter approximately every month until the book is finished. Stay tuned for more.
Yesterday’s How to Patch Perl 5 explained the big picture of how to add a new feature to a dynamic language with a virtual machine. Now it’s time to discuss the technical details.
Jesse Noller is championing the addition of the processing module to the standard library. We’re making extensive use of processing at work now, so I can say it is an extremely simple API for spawning and managing tasks in the background. Passing data between processes using the processing library is as easy as with standard threads.
As I have written before, I’m excited about this library and I’m looking forward to having it available everywhere without any extra effort on our part.
Perl 5 change #33858 adds three new features from Perl 6 to Perl 5.12. With recent interest in implementations of dynamic languages on various virtual machines, I thought it interesting to discuss how a feature comes about and how it works — even if you’re a very happy user of Perl or another dynamic language with no desire ever to look inside your favorite implementation, the details can be enlightening.
Here’s how it happened.
Although cookies almost seem like a prehistoric concept in web development, they’re a well understood commodity that still serve useful purposes. As such, I wanted to write up a quick column that demonstrates a common pattern involving cookies and JSON that you may find useful from time to time.
contextlib module contains utilities for working with context managers and the with statement.
traceback module contains functions for producing error messages with stack traces.
I’ve spent several hours optimizing Parrot over the past few months. In particular, I’ve concentrated on the build process for Rakudo (Perl 6 on Parrot), as it exercises a lot of parts of Parrot. We don’t yet have accurate numbers on the improvements, but rough figures show that the parts of the build process I’ve optimized will be about twice as fast as they were three months ago, despite Rakudo having grown tremendously since then.
Some of this comes from luck, some comes from a deepening knowledge of Parrot internals, a lot of it comes thanks to Callgrind and KCacheGrind, and some of it is experience. My instincts are improving.
I recently communicated 3 security issues in the Safari browser to Apple.
Apple let me know that they will fix 1 of the issues I reported. I will not discuss the vulnerability Apple has promised to fix until they release the fix because it is a high risk issue affecting Safari on OSX and Windows.
I let Apple know that I’d like to discuss the 2 issues they won’t be fixing with the security community and they let me know they are fine with it.
If anyone was interested in a great Google App Engine project, I would love to see a community blog/speaker registration tool. Jeff Rush mentioned something like this a couple of PyCons ago, but now there is the technology available for free with Google App Engine. Basically, it would be cool to have a google app engine app that allowed organizers to book meetings and plan them, then post about the meetings, and finally “book” speakers that happen to be traveling to that city.
Currently there is this blog, but I find it difficult to post the data there, plus our meetup site, plus by email…etc. Making this process easier would be awesome.
On a side note, is there a chart somewhere that graphs what open source web application components are working and what isn’t on google app engine. For example:
Templates:
Genshi
Django
Mako
URL:
blah
Frameworks:
blah
Here is an application on Google App Engine Application I wrote for an upcoming PyAtl Talk, and an upcoming O’Reilly Online Article: http://greedycoin.appspot.com/
Quick notes: Really liking the datastore API. I also liked the Django templates even though I have touched them in over a year and a half. I am looking for Google App Engine consulting or contract work…anyone..anyone :)
After reading a recent press release about System Center Operations Manager 2007 being able to “manage Unix/Linux”, the first thought that goes through my head is WTF? Who in their right mind would touch that product?
After thinking for a bit, I then realize this is the exact same corporate strategy Microsoft has pursued with Active Directory. Release a trojan horse into a corporation by making an inferior, arguably broken, operating system, Windows, that won’t work with anything else, or follow the same standards, and then release a steaming pile of bandages, duct tape, glue, and well…poo, and make everyone authenticate against it while charging an expensive licensing fee.
We have active bot nets that rival NASA in pure computing power due to boneheaded Operating System design, yet Corporate America should have Microsoft manage Unix and Linux. Ha, ha, ha, I am rolling on the floor laughing. For anyone has had the “pleasure” of dealing with Active Directory and the politics that goes on with that, I shudder to think of this product. May God have mercy on your souls.
References:
Post Script: This is a FACT, of the 600 million computers on the internet 100-150 of them have been a part of a botnet at one point. These are primarily Windows computers. This is a complete failure by Microsoft of Epic proportions.
“Operating systems like Microsoft Windows, meanwhile, still made it too easy for criminals to infiltrate them, the experts said.”
The Cloud Computing buzz is everywhere. The concept of grid computing on the Internet to provide elasticity and virtualization of resources is quite appealing, and hence there has been a lot of academic brain-storming going on recently that has given rise to abstract ideas on how cloud computing is destined to change the way technology resources are deployed and used.
Amazon’s EC2 isn’t abstract - it’s real. And it’s very impressive.
functools module includes tools for wrapping functions and other callable objects.
I am going to make an effort to solve classic computer science problems in Python on a regular basis, and blog about them. It seems like a good way to have some fun. This may be a regular blog series.
Because I am going to implement a Coin Changing Restful Web Service using Google App Engine for a presentation I am doing for PyAtl next month, I thought I would show three approaches that I could think of. In doing a google search for “Python Greedy Coin Change”, I didn’t find anyone that had written one and posted the code, so I figured I would make one easily googable. The full source code is here, along with unittests:
Python Greedy Coin Algorithm Source
Approach one was to just use conditional statements, which was pretty tedious to write. Approach two used a while loop and was much shorter. Approach three used recursion, which was also quite a bit shorter. My friend Toby beat me to the recursion solution, so I have to thank him for that one. It would be interesting to see how many other ways I could solve the problem, I think trying to use only functional programming could be a fun twist for example.
I put all three in a commandline tool which make it easier to run and test things out. A couple interesting things to point out about Greedy Coin Changer.
1. You need to get rid of decimals and only work with whole numbers. int(amount*100)
2. Divmod is a wonderful built in function that divides and returns the number and a remainder as a tuple.
Method 1: Conditional Statements
def make_change_conditional(self):
"""Greedy Coin Match with Conditional Statements
Return both number of coins and remainder
>>> c = Change(.71)
>>> c.make_change_conditional()
(2, 21, 2, 1, 0, 0, 1)
>>>
"""
quarter, qrem = divmod(self.convert,25)
#initialize values
dime, drem = 0,0
nickel, nrem = 0,0
penny = 0
#if remainder is dime or higher
if qrem >= 10:
dime, drem = divmod(qrem,10)
if drem >= 5:
nickel, nrem = divmod(drem,5)
if nrem >= 1:
penny = nrem
elif drem < 5:
penny = drem
#if remainder is nickel or higher
elif qrem >= 5:
nickel, nrem = divmod(qrem,5)
if nrem >= 1:
penny = nrem
#if remainder is penny or higher
elif qrem > 0:
penny = qrem
return quarter, qrem, dime, drem, nickel, nrem, penny
Method 2: While Loop
def while_loop_change(self):
"""Greedy Coin Match with While Loop
>>> c = Change(.71)
>>> c.while_loop_change()
2 quarters
2 dimes
1 pennies
"""
coin = self.coins.pop()
num, rem = divmod(self.convert, coin)
self.printer(num,coin)
while rem > 0:
coin = self.coins.pop()
num, rem = divmod(rem, coin)
self.printer(num,coin)
Method 3: Recursion
def recursive_change(self, rem):
"""Greedy Coin Match with Recursion
>>> c = Change(.71)
>>> c.recursive_change(c.convert)
2 quarters
2 dimes
1 pennies
[1, 0, 2, 2]
"""
if len(self.coins) == 0:
return []
coin = self.coins.pop()
num, new_rem = divmod(rem, coin)
self.printer(num,coin)
return self.recursive_change(new_rem) + [num]
Feel free to point out corrections, or alternate solutions
Summary of Problem
Given an arbitrary amount of change, say 1.34, determine the correct amount of change to give using a greedy match, which uses the highest coins first. With US coins, 25,10, 5,1, greedy match will lead to the lowest coins possible.
References:
Google App Engine Version
Greedy Algorithm
Coin Changing Algorithm
Greedy Coins
Jeremy and I are turning in the final draft of our book on “Python for Unix and Linux Systems Administration” on Monday, April 16th. But, if you are a sysadmin and have a really tricky problem you need solved shoot us an email ASAP and there is a chance we will solve it for you. Yes, this is free consulting, but you have to act very quickly!
Some examples could be:
1. Need to process 3TB’s of data everyday.
2. Need a solution to manage virtual hosting.
Tell us the specifics and we will do our best to tackle it as one of our case studies.
Vim is incredibly powerful, but it has the downside of a steep learning curve. Once that is surmounted, however, it’s easy to do a lot of powerful tricks with it. Many of these are things that one sees in full-blown IDEs. My setup provides auto-completion, test suite management, build management, source control integration and a variety of other useful tricks. One of the things that I really appreciate about vim is its handy filtering ability, but most vim developers don’t seem to be aware of it. I’ll explain one way of using it, with a primitive java2perl filter.
I have cut things down to the essentials as I finish up the book I am working on
The last column introduced some of Dojo’s AJAX machinery where we fetched content from a server using the dojo.xhrGet function. One thing that wasn’t mentioned, however, is that there is an underlying abstraction called a Deferred at play. Since the entire Dojo I/O subsystem uses this abstraction, it’s important to understand exactly what it buys you.
If you’ve never used multiple dispatch, you’re in for a treat. We’ve had it working in Parrot for years, but Jonathan Worthington just added the basics of MMD support to Rakudo (Rakudo is an implementation of Perl 6).
Why do you want multiple dispatch? Here’s a clever little example:
class Thing {}
class Rock is Thing {}
class Paper is Thing {}
class Scissors is Thing {}
multi sub defeats(Thing $t1, Thing $t2) { 0 };
multi sub defeats(Paper $t1, Rock $t2) { 1 };
multi sub defeats(Rock $t1, Scissors $t2) { 1 };
multi sub defeats(Scissors $t1, Paper $t2) { 1 };
my $paper = Paper.new;
my $rock = Rock.new;
say defeats($paper, $rock);
say defeats($rock, $paper);Download the new Parrot release next Tuesday, 15 April 2008, then type:
$ perl Configure.pl
$ make
$ make perl6… and you too can play with this in your own code.
Update: Note that multiple dispatch is different from static signature-based overloading and pattern matching in that multiple dispatch works correctly even when you don’t know the specific types of the invocants at compile time. Imagine that I threw a rand() call in there, for example.
Last night I went to my first Open Solaris User Group meeting. It was located at the Sun Office in Alpharetta, Georgia, and I learned quite a bit about the new roadmap for Solaris. The new roadmap at first glance, seems to incorporate many aspects of open source development combined with an enterprise Unix mindset.
Just like Ubuntu, there will be 6 month releases of Open Solaris, and 18 months of support. The new distribution in May will be released on one CD, just like Ubuntu. In addition, like Fedora and Red Hat, the community releases will eventually make their way into the actual enterprise Solaris release. One very interesting fact about SUN is that they will be the only vendor in the Unix/Linux Operating System game that makes hardware and participates in this new Open Source development model of frequent releases with support. This does appear to give them some sort of an advantage in making a stable enterprise Operating System.
On the virtualization front, Sun has quite a few buns in the oven. They have a partnership with Microsoft such that the hypervisor for each respective server will run the other’s guest OS. They have this new LDOM technology, which I have gotten to play, and can vouch is very cool! They also have a new upcoming x86 virtualization product coming out as well.
One of the other reasons I attended the meeting was inquire about the use of Python in Open Solaris. One use of Python is in their package management system, or pkg, in Open Solaris. Apparently, the packaging spec is still underway and in active development to some degree. If you are interested in contributing you can follow the link.
In a final note, when I mentioned the book Jeremy and I were writing, “Python For Unix and Linux Systems Administration”, one comment was, why another language? It is a very good point that I would love to hear a response to from other Python developers. I can think of a few responses to that question, especially as it pertains to Perl and Bash, but I would be curious what other people would say in response to the question of, “I already know Perl and Bash, and maybe C and Java too, why should I learn Python?”.
Just an FYI, the video podcast series, “Spotlight on FOSS” is scheduled to air in exactly one week from today from what I hear. We have a background article in which we explain some of the “fun” of arranging an interview over the internet and talk a bit about Python use at Canonical as well. The shortened show we produced is about 7 minutes, but at some point we may release a longer 20 minute version. This video would have been released much sooner, but Jeremy Jones and I have been pretty busy with getting our book on, “Python for Unix and Linux Systems Administration” done by April 18th, and it took a bit to get everything through the proper channels etc.
Our guest is Mark Shuttleworth, yes, that Mark Shuttleworth, and it is a powerful interview. We also cover Ubuntu with a quick screencast of some of the highlights of the OS.
It is slightly old news to some that Google has gotten into providing Cloud Computing.
The interesting part to Python Developers is that the runtime environment uses Python. All I can say, is way to go Guido!
If you want to see a real application using this cloud check out Huddle Chat.
At PyAtl the May meeting is dedicated to talking about Cloud Computing with Python, so if you are in the Atlanta area I hope you can make it. In addition, PyAtl, and some other people in the WSGI community are holding a WSGI Sprint this summer, where we are going to be developing with Cloud Computing and WSGI! If you are business in the Atlanta area and would like to get involved in sponsoring or contributing please contact me.
If you work in an media industry you might often have a problem of a common directory in which duplicate files get placed constantly. The duplicate files could be fonts, mp3 files, quicktime files, or whatever. One thing you could do is run a tool I wrote liten in a script via supervisor.
I recently added config file support, so you could setup an appropriate config file for liten, and then include it in another script with a sleep command:
#!/usr/bin/env/python
import time
from subprocess import call
time.sleep(3600)
call("liten.py --config=myconfig.ini", shell=True)
Tell supervisor to auto-restart this process and then you will have the duplicates cleaned out of that common directory every hour. This is a piece of cake with Supervisor and if you use it in this manner it can act as a more intelligent cron. Consult the manual for more details.
There is also an OS X Leopard Package for Liten available.
operator module contains functions that perform the same operations as man of the built-in operators.
At the 10 year anniversary of Mozilla’s rebirth as a F/OSS project, Brendan Eich offered a short history of the development of JavaScript. In particular:
The big debate inside Netscape therefore became “why two languages? why not just Java?” The answer was that two languages were required to serve the two mostly-disjoint audiences in the programming ziggurat who most deserved dedicated programming languages: the component authors, who wrote in C++ or (we hoped) Java; and the “scripters”, amateur or pro, who would write code directly embedded in HTML.
Whether any existing language could be used, instead of inventing a new one, was also not something I decided. The diktat from upper engineering management was that the language must “look like Java”. That ruled out Perl, Python, and Tcl, along with Scheme. Later, in 1996, John Ousterhout came by to pitch Tk and lament the missed opportunity for Tcl.
Of greater interest may be Brendan’s thoughts about code sharing and canonical URLs for the millions of identical bundles of JavaScript shuttled across the Internet every second.
I presented Bad Sushi: Beating Phishers at their Own Game (with Billy) at Blackhat Europe (Amsterdam) 2008 last week. I always enjoy doing this talk, and the feedback was quite positive. For more information, check out Nate’s coverage of the conference over at ZDNet’s Zero Day.
I’ll be presenting the Bad Sushi talk at Microsoft’s BlueHat conference in May this year. I’ll be apartment hunting and visiting friends in the Seattle area the last week of April, right before the conference, so if you happen to be in Seattle at that time just let me know!
This week’s installment of Dojo Goodness demonstrates dojo.xhrGet, an easy way to sprinkle some AJAX into your application and one of the biggest staples in Dojo. (In case the “xhr” part of that API call is a curve ball, it simply refers to the XMLHttpRequest object.) The xhrGet function streamlines the work entailed in issuing a GET request to the server, is included in Base, and is intuitive to use in that you pass it some descriptive keyword arguments; the default behavior is to issue an asynchronous request.
At PyAtl, the Atlanta Python User’s Group, we will be having a special presentation on Object Oriented Programming, “Object Oriented Python: From the Basics all the way to Voodoo”.
Jonathan LaCour, a Turbogears, and Elixir, developer will be giving a talk on Metaclasses. Drew Smathers will be giving a talk on MRO, and I will be giving a 10 minute talk on properties.
Everyone will be meeting at 6PM, at Six Feet Under before hand, and you can find directions there here.
There are still a few 10 minute talks we could squeeze in, so if you want to prove your Object Oriented chops, then grab something off list and volunteer to give a presentation on it. I would personally like to see a presentation on Mix-Ins, Static Methods and Class methods, and Operator Overloading.
You can RSVP for the event here.
Apparently, if you learn the hot, new, buzzworthy, Grok Python Web Framework, it helps you meet women. Grok was built on top of Zope, and I used it to build my website, and I think it is rather sexy. I have all the buzzwords, AJAX, Google Mashup, Python, Grok, JQuery, although I might need to do something with Comet..hmmm.
collections module includes container data types beyond the builtin types list and dict.
A dose of animation can give your app that extra bit of pizzazz that it sometimes needs to impress the customer/ladies/gentlemen/whatever-floats-your-boat, so I thought it might be a good idea to quickly showcase some of Dojo’s animation facilities.
If you’re just now tuning in, you can click on my mugshot or this link to get to the first two posts of the ongoing “Dojo Goodness” series that I’m writing to promote my upcoming book, Dojo: The Definitive Guide, which is available on Amazon as well as the O’Reilly catalog.
I arrived at PyCon 2008 on Thursday and attended the Eggs, Buildout, and Virtualenv tutorial given by Jeff Rush. It was quite good, and I would recommend going to any tutorial by Jeff Rush as I have been impressed by his ShowMeDo screencasts, and how thoroughly he prepares for his presentations. His slides for the tutorial are available at the python.org wiki.
Next, I attended the Generator Tricks for Systems Programmers Tutorial by David Beazly. This was my favorite technical talk at PyCon and I learned quite a bit. It happens that his book on Python is one of my favorite all time python books, so it was great to learn from him in person. If you get a chance to do a training session with David, do it, it. You will get more than your money’s worth.
Finally, I attended the web testing tutorial with Titus and Grig. I learned quite a bit about testing web applications, and I am glad I attended.
On Friday, I gave a talk on Creating Agile Unix Command Line Tools With Python, you can download the slides and source code for the presentation there, or at my personal site here. I was a bit surprised at the turnout for the talk, as the room was completely packed. The talk went pretty well, although it would have gone better, if I would have had network access to demonstrate discovering a subnet in a few seconds. I think the source code has some really cool stuff in it, so hopefully people download it and play with it.
I should add that my time leading up to and during PyCon 2008 was pretty crazy. The rough draft for the book I am working on with Jeremy Jones was due on Monday of PyCon, and then on Friday I had my talk. Finally, Brandon Craig Rhodes and I were working on an article on ZODB that we finished on Monday. It was a great lesson in managing stress to get all of this done, attend PyCon, and hang out with friends. Whew, glad this week is over though! I turns out that I have written 14 articles this year, in addition to writing a book, and having a full time job. What was I thinking!
The talks that I enjoyed the most were:
Supervisor: Chris McDonough and Mike Naberezny.
Supervisor is really an interesting tool, and I found out it was based on something Guido originally wrote, which makes it even cooler. I love it when people write tools that are immediately useful, and are simple to understand and use.
Managing Complexity: Matt Harrison
Managing Complexity was an educational talk that exposed me to several ideas that were packaged in a novel way. More than anything, I brought away from the talk that simplicity is the key to reliable software. While I personally enjoy solving the most complex problems I can find, I also highly value the simplest possible solution to a problem. I suppose we as software engineer’s still need to learn that All other things being equal, the simplest solution is the best”.
Nose: Jason Pellerin
The Nose talk was a little hard to hear because Jason had a cold, but he had one of the best quotes when he was asked to compare py.test to nose. He said,” py.test is a like a Belgian Ale, and nose is like bud light”. I again am very interested in starting to use nose more than my brief experiments, because it is simple, and designed to be simple.
Testing OLPC: Titus Brown
Titus gave a very entertaining talk on testing OLPC, and came up with a cool XML-RPC testing harness that detected lines of code that were executed. Apparently he wrote this during PyCon, in some “spare time”, when he wasn’t getting free dinners from my book editor, Julie Steele :)
Due to my crazy schedule, I did miss a few talks that I really wanted to attend, but for some reason missed out on. One in particular was Using Grok To Walk Like a Duck by Brandon. My wife arrived on Saturday, and on a spur of the moment JJ convinced us to grab an authenticate Chicago sandwich. The sandwich was good, but we didn’t make it back in time. Oh, well, I can always force Brandon to do this talk for PyAtl.
Random Thoughts
I had a wonderful time at PyCon this year, and would highly recommend it to anyone considering whether to go next year. Sure, there were a few things that could have been better, like not having to endure a guy in the room next door pulling a Fear and Loathing at PyCon Hotel. At one point, the “scented” smoke coming from next door was so bad, I wondered if there was somebody burning a campfire of dried hemp in their room. Not sure, why PyCon was the ideal spot to “hot box” a hotel room for a week, but whatever dude.
I also was a bit bummed about the wireless problems during tutorials and talks, but hopefully we can improve on this next year. Wireless is a tricky thing to get right, so I am sure everyone was doing their best.
I did not attend any of the Lightening talks this year, so I can’t comment on whether they were good or bad. It turned out that every time there was a lightening talk, I met somebody interesting to talk to, and didn’t get a chance to see them. I agree that half the fun of PyCon is meeting other people in the community, sharing ideas, and catching up.
A met quite a few interesting people including Eric Dahl, the CTO of Zenoss. Having used Zenoss, I will say it is a great product, and now having met Eric, I can say he is a very sharp guy. Zenoss has been doing some extremely innovative things with Python, and is a very good example of how Python can be used in the Enterprise. In addition, to having a great SNMP monitoring solution, Zenoss has also been doing some innovative work with Python, SAMBA, and WMI. If you don’t currently use SNMP to monitor your network, or if you are curious about what a Python solution would look like, download a virtual machine, and let it discover your datacenter in a few minutes.
I also got to talk with Guido, who needs no introduction, Alex Martelli, Steve Holden, Ian Bicking, Jeff Rush, Kevin Dangoor, Ben Bangert, and many other people for the first time. Jeremy and I got to work with Jullie Steele, the editor for our book, and it was a pleasure to meet her in person for the first time. I think our book is in good hands.
Brian Dorsey the creator of Noon Hat was fun to hang out with and we chatted quite a bit. It is always cool to meet new friends. I also got to chat it up with Mike Orr a bit at breakfast on Tuesday morning, and that was a blast as well.
For the first part of the week, I got to hang out with Grig, Titus, and JJ, who I email quite a bit during the year, so that was also pretty cool to see them again after a year. Overall, it was a great experience, and I can’t wait to catch up with everyone again next year.
Summary
The main thing I am going to take away from this PyCon is to embrace simplicity.
As announced earlier on my personal blog, I launched an open source project on Google Code called “loghetti”. It’s written in Python, and is a foundation for what I hope will become a very flexible tool to help admins (myself included) get whatever data they need out of their Apache logs.
Here are a couple of examples of stuff it can do:
Get a list of all of the 500 errors:
./loghetti.py –code=500 access.log
This will send all matching lines in access.log to STDOUT. To get a bit more complex:
./loghetti.py –ip=192.168.1.2 –code=500 –month=11 –day=21 –urlbase=index.php –count access.log
This will *not* return the lines that match all of those rules - but rather a simple count of the matching lines. This request is a somwhat typical support scenario. You have a client at 192.168.1.2 reporting 500 errors they received on some arbitrary date, when trying to reach your intranet’s home page. It’s not unusual in a support role to have the client say “it happened like, a million times”. Of course, –count will dutifully report that it happened 4 times (for example), which is likely closer to the truth.
Ok, one more example, because I happen to be a fan of this feature:
./loghetti.py –urldata=foo:bar access.log
This causes loghetti to parse the query string, and return lines where the query parameter “foo” matches argument “bar”. In other words, lines that look something like this:
http://www.yourdomain.com?stuff=things&foo=bar&this=that
There are billions of features I’d like to implement, but I figured since the tool is useful to me already, it would likely be useful to others, and maybe others can help get features that might help them implemented more quickly.
Let me know your thoughts!
The 4Suite.org projects are moving from CVS to Mercurial. I’m still quite happy with Mercurial having used it for several newer projects. One last test I anted was to convert the bigger projects from CVS to see how smooth the transition might be.
datetime module includes functions and classes for doing date parsing, formatting, and arithmetic.
Here is the presentation I just gave a few minutes ago at PyCon:
Slides: Using Python to Create Agile Unix Tools
Source: Using Python to Create Agile Unix Tools
References:
IBM Developerworks Article: Using Python To Create Unix Command Line Tools
In my last post, I ran through the fundamentals of getting Dojo loaded into the page via AOL’s Content Delivery Network and showed just how easy it is to query the DOM with dojo.query. In this post, I wanted to briefly show off some out-of-the-box Dijit (Dojo widget) goodness with a simple login page. Take a look at the sample page below that uses the TextBox and Button widgets to create a great looking form with virtually no effort required.
A quote from Steve Jobs during the iPhone SDK Press Conference last week:
If they write a malicious application we [will] track them down and tell their parents.
In other words, the iPhone applications will need to be digitally signed by Apple, and the developers will be required to register with Apple. It will be interesting to see what kind of information developers will be required to provide to Apple to register. Will they ask for the developer’s credit card number? How will the developers authenticate their identity with Apple before they are allowed to submit their applications to be included in the store inventory?
Brandon Craig Rhodes, a local PyAtl, Python wizard, has released an incredibly easy to understand screencast on Buildout. Since I “strong armed him”, into making this screencast, it is only fair I blog about it
Background: Buildout is a tool that Jim Fulton, a.k.a, the Zope Pope, of Zope Corporation has developed. While not directly aiming to solve world peace, it perhaps will play a role in the future, as people will be less angry about application deployment and will have more time for making love and music.
I just heard about Cobra. Too busy to talk about it, but posting the slides from the SoCal Piggies meetings
http://cobra-language.com/docs/papers-etc/Cobra-Socal-Piggies-2008-02-Slides.pdf
Since I’ll suddenly have copious free time on my hands once I turn in my final book manuscript this weekend, I decided that it might be helpful to start a short column on some of the fundamental Dojo building blocks. Although I certainly won’t be able to give this column the same “definitive guide” coverage that you’ll get in the book, my hope is to show you some tools that will increase your Dojo awareness and get you excited about some of the things that Dojo can do for you.
I presented Bad Sushi: Beating Phishers at their Own Game with Billy Rios last week at the Black Hat Briefings in DC. The best part of the experience was the opportunity to talk to people in the audience after the presentation, and to hear their perspectives on the subject.
Here is what others have to say about the presentation:
Black Hat Lifts the Cover Off ID Theft Phishing Networks [eWeek]
The myth of the Ninja Hacker [CNET]
Black Hat D.C. wraps up [CNET]
Black Hat, Day 1: Cracking GSM and skimming ATMs [ZDNet, thanks Nate]
Black Hat: Honor Among Thieves?
Here is some additional coverage:
Bank scammers scammed, says security researcher [The Register]
World of Phishing Exposed (podcast)
Researchers Expose "Stupid Phisher Tricks" [Dark Reading]
How phishers work — the inside story [Guardian]
Most Phishers Clueless, Say Researchers
If you were unable to attend the presentation, but would like to get a glimpse of the material, please read the on-line interview with Billy and me that I linked to earlier: Interview with Nitesh Dhanjani and Billy Rios, Spies in the Phishing Underground.
Since the presentation, Billy and I have discussed new material applicable to this presentation, and so I think we will be continue to present updated versions of this talk at future security conferences.
I just found out about a really interesting use of Python, that Red Hat Emerging Technology is developing called func. This was in a recent article at Red Hat Magazine, who I happen to write for as well btw.
We will be covering some examples of func in our book. At a high level though, func allows you to administer machines with python and python scripts. Here are a few examples from their front page:
import func.overlord.client as fc
client = fc.Client("*.example.org;*.example.com")
# package controls!
client.yum.update()
# service controls!
client.service.start("acme-server")
# hardware info!
print client.hardware.info()
# etc ... etc ..
I really like this example too, of rebooting all machines running http:
results = fc.Client("*").service.status("httpd")
for (host, returns) in results.iteritems():
if returns == 0:
fc.Client(host).reboot.reboot()
It is great to see Python use exploding at Red Hat. The cobbler/PXE boot project is also quite cool.
I practice annoyance driven development. I set my threshold of annoyance low such that everytime I feel frustrated by a technical limitation, I notice consciously. My intent is not to find technology endlessly frustrating (though that happens sometimes), but so that I can identify the next most important thing to fix.
For example, Parrot has a large test suite. Several of those tests exercise the source tree as a whole, checking for copyright notices, Subversion ID strings, and metadata properties. I call these non-functional tests, because they exercise externalities of the project, not features of the code. Having accurate copyright notices and repository metadata (especially “Make sure these files have the proper platform-specific line endings”) is useful… but analyzing thousands of files in dozens of directories isn’t instantaneous.
Because we have several contributors, we attempt to keep all of our tests passing on all of the platforms to which we have regular access all of the time. (Exotic platforms like Windows aren’t always so fortunate. Porters wanted.) To achieve this, committers must be able to run the test suite before checking in changes.
Everything so far is obvious. What wasn’t immediately obvious to me was that there’s a threshold beyond which people will not run the entire test suite.
Help Net Security has posted an interview with me and Billy Rios titled Spies in the Phishing Underground.
If you enjoyed the interview, and if you want more details and screen-shots, check out our talk at the Federal Black Hat Briefings 2008 [February 20]. The title of the talk is Bad Sushi: Beating Phishers at their Own Game. Following is a brief description:
Matthew Garrett did some quick calculations to figure out how much electricity the world wastes thanks to Adobe’s inefficient Flash player. Now I don’t use Adobe’s Flash player (partly because I’m no fan of poorly-coded proprietary software, but also partly because Adobe lies about its Linux support), but I’ve noticed that Flash on the machines of other people seems to make fans run, and Gnash eats up a lot of CPU too.
I realize that there’s probably little chance that Adobe cares that releasing their source code under an open license would allow them to support Linux without the little “but 32-bit x86 only” caveat they occasionally slap on their download pages, but would the argument that a more efficient Flash player would stop wasting electricity and save a few pretty trees go any further?
I’m sure a couple of hours with debugging symbols and Powertop and the source code could improve things.
Tired of calculating symbolic and octal formats in your head? Download a dashboard widget that does the calculation for you. This is one sweet dashboard widget!
UNIX Permissions Calculator Dashboard Widget
My Links:
noahgift.com
My O’Reilly RSS Feed
There is a good entry on Adam Leventhal’s Weblog, about Apple crippling DTrace’s ability to trace ITunes. I suppose I will unfortunately need to put this in my not cool category. So far Google has set the Gold standard for Corporate “coolness”. I wish Apple can lean more toward this direction, even though I still love Apple.
One of the common complaints against Perl is that it’s “write-only”. For many Perl programmers, this is regrettably true. Perl, by design, allows you to get things done in a quick and dirty manner. It’s an explicit design goal which allows, amongst other things, the famous “one liners” in Perl which get so much done so fast. However, this freedom comes with a price and that’s a heavy price. Newer Perl programmers often write excruciating code, but experienced Perl programmers write code that is relatively easy to read, once you understand the language. There’s a huge difference between reading code and understanding a language. But when does a language go too far?
I think it is extremely important for an organization to account for the reality of doing business (Risk based approach compared to the purist mentality of securing everything) when strategizing an information security plan. It is true that an individual who has a habit of perceiving security issues as purely a technology problem without understanding the business reality is likely to make bad security decisions.
However, I think some people in corporate security take this argument too far and end up awarding critical roles to individuals that do not have the appropriate skill-set and mind-set. More often that not, this happens when organizations responsible for information security misunderstand the argument to mean that you only need to probe for the understanding of business fundamentals and process management when recruiting for talent. Depending upon the criticality of the role awarded, this can deem disaster.
On the IPython list we have been struggling to get the system Python on Leopard to work with readline, as it is not included with Leopard. Ludwig Schwardt create a readline egg that is easy installable. The instructions for doing this are on the IPython wiki here.
Thanks Ludwig, IPython works on the System Python for Leopard, that means Dtrace support too!
Reference:
IPython
IPython Wiki Readline Instructions Leopard
Easy Install
My Links:
noahgift.com
My O’Reilly RSS Feed
My last two blog posts on egg-related topics had a title prefix of “easy_install tip”. This post is related, but since it’s handled with setuptools rather than easy_install, I’m prefixing it accordingly.
Have you ever wondered how various packages you install put scripts into your path, such as into /usr/bin? If they’re using setuptools to define their package, then they may be using a script entry point. Here is an example taken and modified from a toy setup.py I have sitting around:
entry_points = {
'console_scripts': [
'my_wonderful_script = my_wonderful_module:my_wonderful_function',
]
},
If you run
python setup.py install
, it will create a script named “my_wonderful_script” in your scripts directory. On Linux, this is typically the same directory that the Python executable itself is in. If you’re on Windows, this is a directory that looks something like C:\Python25\scripts. When you run the generated script, it will call `my_wonderful_function` from the module `my_wonderful_module`. This is something that is really easy to setup and can come in very handy. Next time, I’ll write about how to control where stuff goes when you easy_install it. I guess we’ll be back to the “easy_install” tip prefix.
I just released 0.1.3 as a python 2.5 egg here. Liten is a tool that determines duplicates on a file system by performing an efficient md5 checksum algorithm, so it is very reliable. I also added an entry point to the egg install, so it will install to the scripts directory of any *nix Operating System. The easiest way to install is to just:
easy_install liten
Reference:
Liten Project Page
My Links:
noahgift.com
My O’Reilly RSS Feed
The threading module lets you run multiple operations concurrently in the same process space.
Ian Bicking just created a Google Group for Virtualenv here, and a bug tracker at launchpad for virtualenv here. I also have slides from a talk at Pyatl here
Links:
noahgift.com
My O’Reilly Feed
Virtualenv Google Group
Virtualenv Launchpad Bug Tracker
Virtualenv Package Information
Virtualenv Slides
Another article of the series “Yet Another Perl 6 Operator”
Perl 6 introduces a new scalar data-type: the “junction”. A junction is a single scalar value that can act like two or more values at once.
example a value which acts like any(1,2,3) 1 or 2 or 3 all(@vals) all members of @vals at the same time one(<moe curly larry>) one of the three stooges none(@bad_guys) none of the listed bad guys
The operators '|', '&' and '^' are now junction constructors, providing a syntactical complement to the functional variants any, all, one and none.
$a | $b any($a, $b) $x & $y all($x, $y) $me ^ $you one($me, $you)
I spent a little time this weekend writing a mashup portal using the Grok, Python web framework, for my personal domain. Grok is based on Zope 3. I used Zope Template Pages with the Google AJAX FEED API , to aggregate both my O’Reilly Feeds and my Personal Blog Feeds into Web 2.0 goodness. I also threw in a little bit of JQuery which interacted quite nicely with Grok. I attached a link to the site I developed at the bottom, if you are curious to see what the AJAX Feed API looks like. You will obviously see this blog posting, which is meta-cool, or meta-blog, or meta-something…
So what did I learn from Zope? Well, it was super easy to do AJAX with the default ZPT, or Zope Page Template engine. In fact, it was quick and fun, and one of the least painful experiences I have ever had writing a web application. This is not your grandpa’s Zope, that is for sure. Things went so much more quickly and smoothly than I expected, that I had time to work onTurbogears, Django, and web.py projects as well.
I also like the Grok slogon, “Now even cavemen can use Zope 3″. I am 6′2″ 235, and once worked as a bouncer, in College, with the “Iceman”. I kind of look like a caveman, and my wife often says I act like a cavemen too. Well, this caveman gets Grok, and gives it a clubs up! As a side note, I will be releasing a screencast sometime this week on using AJAX techniques with Grok.
Links
noahgift.com
The weakref module lets you refer to an object without preventing it from being garbage collected.
A running joke in the Perl 6 world is that we’ll release a stable Perl 6.0.0 by Christmas. We just won’t tell you which Christmas.
As many community-developed projects have noticed, long blocks of holidays can be very productive for contributors. Both Parrot and Perl 6 on Parrot have made a lot of visible progress in the past couple of weeks.
Another article of the series “Yet Another Perl 6 Operator”
This article is not about some set of Perl 6 operators, but rather about what happened
to Perl 5 filetests operators. Short answer: They are not operators anymore.
Where programmers were used to write
# good ol' Perl 5
if ( -e $filename ) { print "exists\n" }
they will now use pair methods that may be expressed as methods or smart patterns.
if $filename.:e { say "exists" }
# or
if $filename ~~ :e { say "exists" }
I have been learning so many amazing things from Zope that I decided to create small tips as they crop up. Recently I was introduced to buildout do to my work on setting up several Plone 3 sites. Buildout allows you to define eggs within a simple config file and then perform on the fly package mangement by rerunning buildout. It is truly amazing, and worth a look!
Links:
Noah Gift Blog
osxautomation
I try to spend at least an hour a day most days doing cardio. One way I spend the time is by watching Jeff Rush’s insanely great videos at ShowMeDo. I watched the IPython series to prepare myself for a recent article I wrote on Net-SNMP and IPython.
Links:
Noah Gift Blog
osxautomation
Are you a sysadmin and know Bash?
If you know Bash, and I mean even as little as being able to throw a few lines together into a script, you can learn Python. I put some example Bash/Python scripts in svn up at Google Code. There are five scripts in total, two are the exact same program written in Bash and Python, and the last script is an example of a complex Unix command line tool that embeds Bash into Python.
http://python4bash.googlecode.com/svn/trunk/
Links:
python4bash
Noah Gift Blog
osxautomation
Just before I started my job at the BBC, one of our developers committed code which reduced our test suite run time from an hour and twenty minutes down to twenty-two minutes. One of my first tasks was to improve that. However, improving performance begs the old question of “cpu or developer performance?” Both are equally important, but I’ll just talk about making the tests run faster. Right now, it looks like we’re on track to get our test suite to run in under ten minutes. Here’s how we did this.
Another article of the series “Yet Another Perl 6 Operator”
In a previous article , we introduced the reduction operators (like '[*]' and '[~]') which produced list operators from infix operators (like '*' and '~').
There is a variant of the reduction operator that operates over its list argument producing
all intermediate results along with the final result of the ordinary reduction.
[\+] 1..5 # (1, 3, 6, 10, 15)
which is equivalent to
([+] 1), ([+] 1, 2), ([+] 1, 2, 3), ([+] 1, 2, 3, 4), ([+] 1, 2, 3, 4, 5)
Another article of the series “Yet Another Perl 6 Operator”
Binary '=>' is no longer just a “fancy comma”. In Perl 6, it now constructs a Pair object that can, among other things, be used to pass named arguments to functions.
my $pair = (one => 1); $pair.isa(Pair) # Bool::True $pair.key # 'one' $pair.value # 1
Another article of the series “Yet Another Perl 6 Operator”
We already have seen two Perl 6 meta-operators in articles of this series: namely, the
negate and the reduction operators. These are two of the five standard meta-operators of the language. What makes meta-operators interesting is how Perl automatically generates new operators from others (user-defined or builtins) with some straightforward semantics derived from the transformation of the base operators.
This time, we approach mutating operators, which are a shortcut for typical assignments where the assignment target and the first operand are the same variable.
my $s = 'foo'; $s x= 3; # $a = 'foofoofoo' my $x; $x //= 'default'; # $x = 'default'
Another article of the series “Yet Another Perl 6 Operator”
And that’s time to take a look at another of the Perl 6 meta-operators: the reduction operator.
By surrounding with square brackets an (associative) infix operator, a new list operator is created.
[*] 1..10 # that's 1*2*...*10 = 10! [~] <m oo s e> # 'moose' - [~] is basically Perl 5 join [,] 'a'..'e' # <a b c d e> - [,] is a list builder
Another article of the series “Yet Another Perl 6 Operator”
If you are wondering how processing the lines of a file will look in Perl 6, the answer is something like this:
my $h = open '<', $filename;
for =$h {
...
}
(Yes, we need error handling yet. I just ommitted the details for brevity.)
Let’s say you customized your Zsh prompt by editing ~/.zshenv
#customize prompt:
PROMPT=$'[%n@%m][H:%B%!%b][J:%B%j%b]> '
You should see something like this:
[root@bigbadunixbox][H:2487][J:0]>
If you want to see all 2487 lines of your Zsh history buffer do this:
history 1 | less
If you haven’t already seen it, check out Michael Trier’s new podcast “This Week in Django“. The first episode was good, so I’m looking forward to listening to the second.
Subscribed!
The zipimport module can be used to import and run Python code found inside ZIP archives.
Damien Seguy just sent me more stats on the versions of PHP in the wild. From their analysis of PHP Statistics for November 2007:
With support for PHP 4 ending in a couple of weeks, hopefully the 50% or so of PHP installations running PHP 4 or earlier will take the opportunity to migrate. (See the PHP 4 to PHP 5 migration guide.)
In a related note, I really like the plush PHP elePHPant.
Another article of the series “Yet Another Perl 6 Operator”
Perl 6 provides an operator 'X', the cross operator, which combines its list operands into a sort of cartesian product of these arguments.
1,2 X 3,4 # (1,3), (1,4), (2,3), (2,4) 1,2 X 3,4 X 5,6 # (1,3,5), (1,3,6), (1,4,5), ..., (2,4,6)
The more I write WSGI code, and read WSGI posts like this one on the Repoze blog, the more I wonder if Phillip Eby didn’t invent Python’s own Howard Roark in WSGI, when he wrote PEP 333.
Zope Corporation was founded in 1995, according to the, about Zope,page on their website. This means there are a heck of a lot of Python programmers who know a heck of a lot about Python Web Development. If you look at say, Ruby on Rails, which deserves a great deal of respect for what it has accomplished, you will notice it was released to the public in 2004, there is about a decade of experience difference there. In terms of computer science, this is an eternity.
Python now has mod_wsgi, a Zope TM that works inside of WSGI with any other WSGI application, people that spend every waking minute developing an ORM, and new hybrid, WSGI specific, Python web frameworks like Pylons,and,Grok. Not to mention incredibly mature full web applications, like Plone 3.0. Let’s also not forget Deliverance, which makes “skinning” multiple WSGI applications, and/or products like Plone trivial. In plain english, all of this stuff works together! As Ian Bicking explains, it is just a bunch of tubes. At this EXACT moment, people have completed the rest of the tubes, that connect the rest of the technologies in Python!
Python Web development reminds me of a very large battleship, which was slowly turning toward the shore, all the while loading the guns with large shells. Well, the ship has turned, the guns are loaded, and now, it is time to see what Python web development can really do in 2008. The perceived weakness of many different frameworks in Python, has now turned into a strength with WSGI, and these host of other technologies all working in harmony. Plus, lets not forget this is Python, perhaps, the most human readable language in existence.
Python has the potential to become the dominant Web Development language in the next couple of years, due to its massive, yet growing, pool of experienced developers, WSGI, the incredible, battle tested Standard Library, tested and proven scalability, with a plethora of concurrency solutions, readability, and outstanding leadership of the core developers. I suppose, armed with knowledge of these new developments, I wonder why anyone would not use Python Web Development for projects in the next two years?
Update: Here is the PyAtl/Repoze Talk, which discusses WSGI/Repoze/Deliverance.
Update: To see why Plone is such a big deal, watch this link, Better Web Application Development, and why having Plone combined with WSGI is almost an unfair advantage for Python
I was just commiserating with David Wheeler about a problem he had with mod_ssl on Mac OS X. The .dylib extension on shared libraries seems “arbitrary” in his words, and I’ve wrestled with it in cross-platform code a few times myself.
Then I realized where so many of my frustrations with Mac OS X came from as a developer:
Unix circa 1986 via NeXT is different from Unix circa 1998 or 2008 via Linux.
I’m happy to stick with POSIX when I want software to run somewhere outside of the nice cozy GNU/GCC/Linux/glibc universe, but those NS* functions just don’t quite feel right, you know?
Another article of the series “Yet Another Perl 6 Operator”
The syntax of an if-then-else expression in Perl 6 is composed by the conditional operator.
say "My answer is: ", $maybe ?? 'yes' !! 'no';
The expression above is equivalent to that, which uses the if-then-else statement within a do.
say "My answer is: ", do {
if $maybe {
'yes';
}
else {
'no';
}
};
Another article of the series “Yet Another Perl 6 Operator”
The design of Perl 6 includes some unification mechanisms to bring some extra power to syntax and developers. One of such artefacts is the notion of meta-operators. With them, it is possible to construct augmented operators from existing ones.
The first of such meta-operators we’re going to look at is the quite simple '!' negation prefix.
Another article of the series “Yet Another Perl 6 Operator”
In Perl 6, you may construct ranges with expressions like
$min .. $max $min ^.. $max $min ..^ $max $min ^..^ $max
and even
^$limit
If you read my feed, you probably saw that I changed the name of the project mentioned in SAP’s Composition on Grails. After I published that, my contacts at SAP said “You know, we had been meaning to change the name.”
If you’re interested in the project, you should definitely follow Will Gardella’s weblog, starting with Composition on Grails - New Version, New Name. He answers some of the most common questions he’s heard over the past few months about the project.
I realize that SAP doesn’t want to offer official support for languages other than Java and ABAP, but every demo I saw there with interaction between services over WS-* makes me think that the company could spend $50k in tech marketing very well by posting bounties with the appropriate F/OSS communities to beef up WSDL and SOAP support in languages such as Perl, Python, Ruby, and PHP such that they can be first-class consumers of SAP middleware services in the same way as Groovy now is.
Again, it doesn’t have to be official support, and it’s just the price of a full-page ad in a decently-read magazine, but it would be a shame not to take advantage of these rich ecosystems with an architecture that already supports such a thing….
Another article of the series “Yet Another Perl 6 Operator”
Among the new Perl 6 operators, there is the handy operator '//', known as defined-or or the default operator. This novelty was anticipated by the introduction of this syntactic bit in Perl 5 (see the upcoming 5.10 release) — so you won’t need to wait for Perl 6 to start using it.
# dor.pl use 5.010; print "arg: '", shift // "?", "'\n"; $ perl dor.pl one arg: 'one' $ perl dor.pl "" arg: '' $ perl dor.pl arg: '?'
Most people forget that python is both procedural and Object Oriented. If you find yourself wanting to use a procedural, Bash style of programming with Python, take a look at the global statement.
It lets you do something like this, because without the :
In [34]: x = 1; y = 2
In [35]: def func():
....: global x, y
....: print "X = %s, Y = %s, in func" % (x,y)
....:
....:
In [36]: func()
X = 1, Y = 2, in func
Here is an even better example, based on the comment by Ant, as the top example was too vague, probably because I was trying to make the example too "tiny" :) That is a challenge, to make a tip tiny, yet explicit.
In [46]: x = 1; y = 2; z = "out here"
ln [47] def func():
....: global x,y,z
....: print "X = %s, Y = %s, in func. Z says: %s " % (x,y,z)
....: x = 1000;y=1000;z="in here"
....: print "X = %s, Y = %s, in func. Z says: %s " % (x,y,z)
In [48]: func()
X = 1, Y = 2, in func. Z says: out here
X = 1000, Y = 1000, in func. Z says: in here
I should also say, thanks to Tres, who showed me this tip to begin with!
The last 7 days, have been very crazy for me. Last weekend, I ended up in Los Angeles, for a couple days, and was able to grab some Cuban Pastries:
Before hopping back on a plane for Atlanta:
Back in Georgia, I felt very much at home when I spotted some familiar local scenery, a guy with a mullet in a camaro (with a killer “G-Force” bumper sticker), next to a man in a costume in the middle of traffic, asking for money:
Next, it was a Tuesday visit to the monthly Atlanta-Plone meeting. Where we discussed the upcoming Repoze Sprint/Visit:
On Thursday, we met with Tres and Chris, who happened to write supervisor, and they gave a tremendous talk on WSGI, Repoze, and Deliverance, that blew the PyAtl crowd away.
One of the more dramatic, show and tell, pieces, was a local demonstration of their “theme trac like Plone trick”. The crowd was blown away, when Tres and Chris stole, borrowed, pick your favorite word, the pyatl plone 3.0 site, and themed a localhost trac instance. We also saw a great debugging middleware WSGI tool, that “leaked” objects in the WSGI stack. WSGI is truly an incredible technology, and I am so excited about it, I almost can’t sleep.
Next on Friday, we hunkered down at Georgia Tech, and starting playing with Repoze a little more:
One silly idea that came up after a few beers at lunch, was writing the simplest possible WSGI application using the WSGI spec from Pep 333. By using Ian Bicking’s pythonpaste, Tres was able to walk me through setting up the most simple possible WSGI application. We used string substitution and pickle, and gave birth to A******Glue, AGlue, for short. AGlue is just a proof of concept, with a funny name.
If you are use virtualenv, and pythonpaste, it is quite simple to make a little web application using WSGI. You really only need to create an /etc directory in your virtualenv, that includeds a .ini file, such as this:
Step 1: Create a .ini file
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8080
[app:aglue]
paste.app_factory = aGlue.app:factory
path = %(here)s/../var
[pipeline:main]
pipeline = egg:Paste#evalerror aglue
Step 2: Next make some simple model.py like this:
class Book(object):
"""A book object"""
def __init__(self, ISBN, title, reviewer=None):
self.ISBN = ISBN
self.title = title
self.reviewer = reviewer
Step 3: Finally, make a app.py, or controller:
import os
import pickle
from paste.request import parse_formvars
from model import Book
template = """
<html>
<body>
<p>
<a href = "http://pyatl.org">pyatl.org</a>
</p>
"""
row = """
<p>%(title)s <form method="post"><input type="hidden" name="index" value = "%(index)d">
<input type = "submit" name = "delete" value="delete">
</form>
</p>
"""
epilogue="""
<form method="post">
<input type="text" name = "ISBN">
<input type="text" name = "title">
<input type="submit" name = "submit" value = "add">
</body></html>
"""
def middleFinger(environ, start_response):
"""Why did you use this, punk?
"""
form = parse_formvars(environ)
if form:
if 'submit' in form:
book = Book(ISBN=form['ISBN'],title=form['title'])
books.append(book)
saveList()
elif 'delete' in form:
index = int(form['index'])
del books[index]
saveList()
print form
page = [template]
for index in range(len(books)):
book = books[index]
page.append(row%{'index':index,'title':book.title})
page.append(epilogue)
status = '200 OK'
response_headers = [('Content-type', 'text/html')]
start_response(status, response_headers)
return [''.join(page)]
def saveList():
file = open('/tmp/persistant.db', 'w')
pickle.dump(books,file)
file.close()
def factory(global_config, persist = '/tmp/persistant.db',**local_config):
global books
books = []
if not os.path.exists(persist):
saveList()
else:
file = open(persist)
books = pickle.load(file)
return middleFinger
With that little bit of code, you get something like this:
One thing I learned from the last few days, is that Ian Bicking is amazing! Between virtualenv, and pythonpaste alone, it is an incredible, how many tools he creates to help other Python programmers. Tres and Chris, are also equally amazing, and I would recommend trying to get them to come to your local user group for a Repoze/Deliverance talk too!
There will be a video posted this week of their talk this week on YouTube, and I will also upload a more refined version of AGlue, to the cheeseshop in a few days.
First off, the “I” in the title does not refer to me; it refers to my coworker Jenny Walsh. Jenny was recently setting up a new Macbook Pro with Leopard on it and ran into a snag with PIL. Running setup.py for PIL for the first time said that she had no jpeg support. So, she installed libjpeg and tried again. This time, it said that she had jpeg support, but then it threw this error:
ld: in /Developer/SDKs/MacOSX10.4u.sdk/usr/local/lib/libJPEG.dylib, file is not of required architecture for architecture ppc
She tried compiling libjpeg and followed suggestions she had found from across the interweb including compiling as a “fat binary” (intel and ppc), but she still had the same level of unsuccess as before. Finally, in a fit of hacking frenzy, she removed all “-arch ppc ” from
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/Makefile,
re-ran setup.py for PIL, and all was right with her world again.
In trying to diagnose and help her with her problem, I stumbled across
this
blog post. They didn’t seem to have any trouble getting PIL and libjpeg installed on leopard. The compiled libjpeg as follows:
wget http://www.ijg.org/files/jpegsrc.v6b.tar.gz tar zxvf jpegsrc.v6b.tar.gz cd jpeg-6b/ ./configure make sudo make install-lib
I tried the same thing on my laptop (which runs Tiger and not Leopard, btw), and it worked for me. But this just took my PIL installation from saying that it did not have jpeg support to saying that it did. Jenny was having a different problem. So, if you are having a problem with PIL blowing up when trying to set it up, you may want to give Jenny’s solution a spin. But make sure to make a backup copy of the Makefile.
The series “Yet Another Perl 6 Operator” is back with this brand new article
In the article on coercion operators, we got to know the prefix operator '?' which converts values into Bool::True or Bool::False. Like it happens with '~' for strings, '?' is recurrent for boolean operators.
In Perl 6, the usual infix boolean operators are:
?& - and ?| - or ?^ - xor
I was in Silicon Valley last week, and caught part of a demo of Composition on Grails. This is a project within SAP built on top of SAP NetWeaver Composition Environment.
I decided to change my plans, and had lunch with Will Gardella and his team. (Will gave the demo, and his team is building Composition on Grails within SAP.)
I’d never considered SAP or its business very interesting, because I misunderstood what SAP actually does. Though I very happily left the world of the Enterprise (twice, first as a system adminstrator and then as a consultant who wrote and maintained business software), and though SAP’s choice of technologies (ABAP, Java, WSDL, WS-*, shared-everything) are different from my preferences, it took just a few minutes for Will’s demo to convince me that I had overlooked something very interesting.
The BaseHTTPServer module includes classes which can form the basis of a web server.
Patrick Michaud is truly earning the Mozilla Foundation’s Perl 6 on Parrot Grant. He’s just published a Perl 6 on Parrot Roadmap for December 4, 2007, which describes the progress the project has made in the past couple of weeks.
In particular, he and Jerry Gay just switched the Perl 6 compiler in languages/perl6/ from the old compiler toolkit to the new compiler toolkit. This sounds like a small thing, but it’s actually a very large step.
As Jeremy and I get more down the road of finishing our book on Python for *NIX systems administration, working title, we are both going to start dishing out some meatier nuggets of Python that we expose in our book. I have a background in Feature Film Animation Pipelines, and I thought I would share an interesting Python module that Python programmers in the Animation world might enjoy. The module is called pyinotify, and it “monitor’s filesystem events with Python under Linux”. Sound cools right, well, it is, so lets get a little background.
Inotify went into the linux kernel in release 2.6.13, and, according to Wikipedia, “Inotify uses an API that uses minimal file descriptors, allowing programmers to use the established select and poll interface”, in plain english it notices changes to the filesystem and reports those changes to applications. In animation pipelines, generally, files need to make a round trip between CGI, and Editorial. The Animators are given a scene to create, and very large, raw image files, like 16 bit TIFFs, are created. The files generally need to be processed in many different ways depending on where those images need to go. They may need to be converted into a HD Quicktime Movie for viewing on a 2K or 4K, stands for thousands of pixels, digital projector, or they may need to be converted into a compressed format like MXF, or DNxHD, a variant on the MXF file format that Avid has developed. If the files are to get sent to the editorial department for editing, then they will need to be processed with a timecode, or keykode, value embedded into the metadata of the file.
Embedding either a timecode, or keykode, is mandatory, as it allows image files to be assembled automatically according to a shot list, or else they would need to be visually arranged by hand, which just wouldn’t work in an animation pipeline. One of the reasons why it wouldn’t work, is that the image files could be cut out of a scene, and it would be an incredible waste of money to animate a frame or many frames, that were cut out in editorial. This is just one of the reasons why embedded metadata in files is so important in animation. Of course things need to be kept track of, and that is why using something like SQLAlchemy could make a lot of sense. I am actually building a metadata management system into open source tool I am creating called Liten, and, when I get more time, it will use SQLAlchemy.
All of this rambling background material, leads me back to Python, and Pyinotify. Pyinotify, by talking the Linux Kernel API, can watch a directory or a whole filesystem, for the moment that say, a Maya Artist, has exported a sequence of frames to the “shot tree”, or file server, in plain english. At that point, when Pyinotify notices these changes, it could begun to process these files and perhaps move them to a High Speed Fibre SAN like Avid Unity, or XSAN, that is built for playing back HD media files.
Lets take a look at how that might work:
I threw this “toy code”, together in about an hour or so, and all it does is pretend to do things when a file is added to the /tmp directory and it closes. I might get around to actually making a useful tool this weekend and adding threading, etc.
If you would like to check this code out, I put up a Google Code Project here. These are some pictures of my “toy code” pretending to do things, when I create a file in a directory, I am watching. Thanks to author of pyinotify for making this so easy to work with!
import os
import sys
import optparse
from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes
class PClose(ProcessEvent):
"""
Processes on close event
"""
def __init__(self, path):
self.path = path
self.file = file
def process_IN_CLOSE(self, event):
"""
process 'IN_CLOSE_*' events
can be passed an action function
"""
path = self.path
if event.name:
self.file = "%s" % os.path.join(event.path, event.name)
else:
self.file = "%s" % event.path
print "%s Closed" % self.file
print "Performing pretend action on %s...." % self.file
import time
time.sleep(2)
print "%s has been processed" % self.file
class Controller(object):
def __init__(self, path='/tmp'):
self.path = path
def run(self):
self.pclose = PClose(self.path)
PC = self.pclose
# only watch these events
mask = EventsCodes.IN_CLOSE_WRITE | EventsCodes.IN_CLOSE_NOWRITE
# watch manager instance
wm = WatchManager()
notifier = Notifier(wm, PC)
print 'monitoring of %s started' % self.path
added_flag = False
# read and process events
while True:
try:
if not added_flag:
# on first iteration, add a watch on path:
# watch path for events handled by mask.
wm.add_watch(self.path, mask)
added_flag = True
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
# ...until c^c signal
print 'stop monitoring...'
# stop monitoring
notifier.stop()
break
except Exception, err:
# otherwise keep on watching
print err
def main():
monitor = Controller()
monitor.run()
if __name__ == '__main__':
main()
Sound fun? if so, let me know.
The SocketServer module is a framework for creating network servers. It provides base classes for handling TCP, UDP, Unix streams, and Unix datagrams and supports both threading and forking servers, depending on what is most appropriate for your situation.
Need to monitor remote disk usage? Use snmpdf
1. Start and configure snmpd (edit /etc/snmp/snmpd.conf):
rocommunity superSecret
disk /
2. Run the command on a local machine and be amazed:
snmpdf -v 2c -c superSecret localhost
3. Setup and use SNMP v3 on a real machine you admin.
Do you have a thousands of lines of debug code slowing you down…use __debug__ and optimize it out with -O command line option.
My colleague Jim Shore has just released a transcript of a short talk he gave called Does It Work? Are We Done? Is It Right? (Keep It Light!). I share Jim’s thesis that there are three questions we need to answer in software development if we want to achieve any sort of success. In Jim’s words:
So I want to answer these quality questions, “does it work, are we done, is it right,” without incurring waste… and ideally, I’d like to do it in a way that makes me faster, not slower.
His talk specifically addressed testing, especially functional testing, but in working with him I’ve found that asking those questions of any part of a software development process is incredibly valuable.
For the past few days I’ve been one of several people helping Titus Brown set up the Python Software Foundation’s portion of the Google Highly Open Participation(TM) contest. GHOP is an extension of Google’s Summer of Code project, for students not yet in college. The goal of the contest is to attract young people to open source, and teach them how to participate. Check out the FAQ for more details.
The inspect module provides a variety of functions for introspecting on live objects and their source code.
Are you an IPython nut too? Here is your chance to help out testing this Thanksgiving weekend. In between bites of Turkey, and pecan pie, download the latest 0.8.2 version, dev egg here, and report any bugs to Trac
The urlparse module provides an interface for splitting up Uniform Resource Locator strings into their parts.
I’ve been digging into setuptools and easy_install lately, so you can expect to see more of these tips in the coming weeks.
One of the really cool features of easy_install is that you can install packages from the Python Cheese Shop, which is a web-based repository of Python packages. For example, in order to install IPython using easy_install and the Cheese Shop, you would simply type `easy_install ipython`. It’s all fine and dandy for open source projects to put their source code out there for the world to look at, but what if you work on (gasp) closed source code? Do you have to miss out on all this easy_install remote repository goodness? Not at all.
First, you have to setup a web server that will serve up your packages. I am using lighttpd on my laptop, which misses the point in practice since i could just as easily install the eggs locally rather than by pointing at http://localhost, but it works for example purposes. I have lighttp serving up a directory that contains a repository directory called “repo”. So, http://localhost/repo will give a directory listing of all the packages I can serve up.
Second, you need to build your packages and put them in a directory that your webserver can serve up, the “repo” directory in my case. I have 2 packages, foo v0.1 and bar v0.1, that I made into eggs. The filenames are bar-0.1-py2.5.egg and foo-0.1-py2.5.egg. In this example, foo has a dependency on bar. Since easy_install handles dependencies so well, I would expect an easy_install of foo to resolve the bar dependency and install them both. Step three will prove this assumption either correct or incorrect.
Third, and finally, you simply tell easy_install to look at your local repository for packages. Here is the command and output from installing from my local repository:
root@lanik:/usr/lib/python2.5/site-packages# easy_install -f http://localhost/repo foo Searching for foo Reading http://localhost/repo Best match: foo 0.1 Downloading http://localhost/repo/foo-0.1-py2.5.egg Processing foo-0.1-py2.5.egg Moving foo-0.1-py2.5.egg to /usr/lib/python2.5/site-packages Adding foo 0.1 to easy-install.pth file Installed /usr/lib/python2.5/site-packages/foo-0.1-py2.5.egg Processing dependencies for foo Searching for bar==0.1 Best match: bar 0.1 Downloading http://localhost/repo/bar-0.1-py2.5.egg Processing bar-0.1-py2.5.egg Moving bar-0.1-py2.5.egg to /usr/lib/python2.5/site-packages Adding bar 0.1 to easy-install.pth file Installed /usr/lib/python2.5/site-packages/bar-0.1-py2.5.egg Finished processing dependencies for foo
The “-f” flag tells easy_install where else it can look for packages. That’s it!
Last night teams were given a green flag and set off on the 44 hour race. All teams completed the HPCC benchmarks by 1:00am and results were handed in to the judges. Teams were then given data sets for GAMESS, POP, and POVRay, and will spend the rest of the available time working through them.
Arriving back about 6am, we found a few team members in the ‘understudy’ position (crashed out on the floor, sofa or under the tables). About half were in the midst of a team change, so there were a few bright eyes available. One team was reported to have to pull a team members fingers off the keyboard and send them back to the hotel for some sleep (and to stay within the rules).
About 12:20 the convention center experienced loss of power, and the Cluster Challenge teams’ equipment all suffered a hard crash. Not all came back fully, and the teams are working on it. Some times real world things happen and we learn from it: at least one team is now using checkpoint features in some of the applications so they don’t lose much runtime should they have this issue again.
Visitors to the event are enthusiastic, and are having excellent interaction with the teams. Kudos to the teams for taking the time to answer visitor questions while they are in the middle of this race.
Brent
After almost a year of preparation by the committee, a week of on-site infrastructure setup, and 2 days of on-site prep for the teams: we are finally at race day!
Tonight at about 8pm at the SC07 conference in Reno, we will officially kick-off the first attempt to build and race cluster-based supercomputers on the conference floor. The room is huge (20,000 ft^2) and beautiful, being decorated with banners, posters and even an airplane courtesy of the Reno Air Race Association.
If you are at SC07 and in the technical program, we invite you to visit us in hall 1 @ 8:00pm for the start and to cheer on your favorite team.
Brent Gorda
The Cluster Challenge at SC07 is in the preparation phase, with all 6 teams and their hardware on site in Reno Nevada. In a little over a day, the teams will begin a 44 hour computational competition (called the SC07 Cluster Challenge).
We have 4 teams from the US: Purdue, Indiana, University of Colorado, and Stony Brook
The University of Alberta from Canada is here, as well as the National Tsing Hua University from Taiwan.
Yesterday all the teams made it to the convention center and found their equipment. The energy of the students is huge at this point, and it is creating a level of excitement that is bound to grow throughout the week. The committee is rushing to complete last minute items and ensure preparation.
Stay tuned…
The pprint module includes a “pretty printer” for producing aesthetically pleasing representations of your data structures.
The SuperComputing 2007 (SC07) Cluster Challenge invites teams of students to Reno mid-November to compete in a demonstration of talent, technology and entry-level supercomputing. The activity seeks to highlight the gains in hardware performance, ease of use of clusters and the power and availability of simulation software. A half-rack (a full rack is about the size of a household refrigerator) of modern day servers, or “pizza boxes” is competitive with the number one system on the top500 from only 10 years ago!
The Cluster Challenge will be held in Reno the week of November 10-16, 2007 as part of the annual Super Computing conference. Each of the six international teams of undergraduates have designed their commodity cluster and are supported by their chosen vendor partner. Competition rules are simple: teams may use a peak of 26 amps on a 30 amp circuit. They will use their system to run the HPC Challenge benchmarks and three open source applications: POP, GAMESS, and POVRay.
The challenge starts Monday night during the gala opening where attendees attend to preview the exhibit hall and socialize with food and drink. At the start, teams will run the HPC Challenge benchmarks and post their results. Once they have done this, they will be given access to the data sets for the applications and will have the next 40+ straight hours to complete as much of this workload as they are able. At 4:00 pm on Wednesday, teams will be judged by a group of high performance computing experts lead by Jack Dongarra (founder of the Top500 website).
With the movement toward multi-core, it is clear that future performance gains will come from parallelism. The scientific community, with over 20 years of experience, has produced quality (and open) simulation software such as POP and GAMESS. Other software that might be thought of as not parallel, such as POVRay, can harness parallelism by running a copy on each processor.
We believe that computational simulation on this scale has attained a level of capability and accessibility wherein it is now a critical tool available (and soon to be necessary) to enhance the competitiveness of industry. By inviting undergraduates to design, build and then compete with these systems, we intend to demonstrate this. Through the selected applications we will show that interesting and useful simulations can be done on this modest amount of hardware.
It’s been crazy over the last couple of months with the book and with starting a new job, so I’ve been slack on blogging. Well, I’m trying to get back in the saddle again, so I’ll briefly mention something here that I only discovered recently.
If you are working on code that you are planning to release as an egg, it can be a huge pain to go through the code->bdist_egg->easyinstall->test cycle, slightly less of a pain to do code->setup.py install->test, and slightly less of a pain still to do export PYTHONPATH=/path/to/my/project->code->test. But there is a better way. If you do “easy_install develop”, then you can just enter into a code->test cycle. This option puts a .pth file in your site-packages directory and points to the set of code you’re working on. This keeps you from having to redeploy your code every time you want to test something. It also keeps you from having to remember to put your current working code in your PYTHONPATH. It’s just a really friendly option for easy_install that will make your life just a little easier. Thanks, PJE!
The shutil module includes high-level file operations such as copying, setting permissions, etc.
Google’s Webmaster Tools site provides a reporting feature to let you see who is linking to you. Unfortunately, the report is backwards from the orientation I want to read it.
I happen to enjoy going to a local grocery store that employs self service checkout. I find it very convenient to get a few items this way. This morning I went to get a few bottles of water, and some coffee, but was a little surprised to see this:
Since, I don’t use Windows in my household, it has been quite some time since I have seen a Blue Screen of Death, and it brought back some painful memories. This was kind of funny so I thought I would share it. (By the way, I snuck this photo on my iPhone).
It does bring the question to mind though, where are all of the OS X, and GNU/Linux, kiosks? I think Beryl or Cocoa both make very nice looking kiosks. I know I would probably use a grocery store just because it had a Linux or OS X kiosk. Anyone working on something like this?
A few people who read my blog are already aware that I was working on a little pet project to develop a command line interface to a Google Spreadsheets document that is being used by some of the administrative operations team at Python Magazine. The long and short of it is that I finally had about two hours to work on it yesterday, and got my little utility to the point where it’s actually useful to me: I can now add rows to the spreadsheet without opening a browser.
What’s even better is that it’s not specific to any particular spreadsheet. You can use it for any Google Spreadsheet you want. This code is 99% cut-n-pasted from Google’s examples and docs. The rest is the result of a bit of poking and prodding to figure out how to get at data elements that aren’t documented and I couldn’t find examples for. So I wanted to post it here so anyone could have it. It requires you to install the Google Data Python Client.
Save this to a file, execute it, and it’ll prompt you for your gmail account credentials, then show you all of the spreadsheets you have access to. Pick one, then pick a worksheet within that spreadsheet, and it’ll dump the contents, or (if you use the -a option) it’ll prompt you to fill in values for each column of a new row.
This is quick and dirty code, not production quality, and I probably do some un-Pythonic things in there somewhere, but a buddy told me he was having some issues with Google’s documentation too, so I figured that making more code available for people to read might help.
Enjoy!
#!/usr/bin/env python
try:
from xml.etree import ElementTree
except ImportError:
from elementtree import ElementTree
import gdata.spreadsheet.service
import gdata.service
import atom.service
import gdata.spreadsheet
import atom
import getpass
import string
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-a", "--addrow", action="store_true", dest="addrow", default=False)
(options, args) = parser.parse_args()
gd_client = gdata.spreadsheet.service.SpreadsheetsService()
gd_client.email = raw_input('nEmail: ')
gd_client.password = getpass.getpass()
gd_client.source = 'pymag-test-1'
gd_client.ProgrammaticLogin()
def PromptForSpreadsheet(gd_client):
# Get the list of spreadsheets
feed = gd_client.GetSpreadsheetsFeed()
PrintFeed(feed)
input = raw_input('nSelection: ')
return feed.entry[string.atoi(input)].id.text.rsplit('/', 1)[1]
def PrintFeed(feed):
for i, entry in enumerate(feed.entry):
if isinstance(feed, gdata.spreadsheet.SpreadsheetsCellsFeed):
print '%s %sn' % (entry.title.text, entry.content.text)
elif isinstance(feed, gdata.spreadsheet.SpreadsheetsListFeed):
print '%s %s %sn' % (i, entry.title.text, entry.content.text)
else:
print '%s %sn' % (i, entry.title.text)
def PromptForWorksheet(gd_client, key):
# Get the list of worksheets
feed = gd_client.GetWorksheetsFeed(key)
PrintFeed(feed)
input = raw_input('nSelection: ')
return feed.entry[string.atoi(input)].id.text.rsplit('/', 1)[1]
def ListGetAction(gd_client, key, wksht_id):
# Get the list feed
feed = gd_client.GetListFeed(key, wksht_id)
return feed
def AddRow(columnfeed, spreadsheet, worksheet):
# take the columnfeed.entry object and prompt for a value for each column
# Build a dict from the resulting column:value pairs.
dict = {}
for key in columnfeed.entry[0].custom.keys():
dict[key] = raw_input("%s: " % key)
gd_client.InsertRow(dict, spreadsheet, worksheet)
spreadsheet_id = PromptForSpreadsheet(gd_client)
worksheet_id = PromptForWorksheet(gd_client, spreadsheet_id)
columnfeed = ListGetAction(gd_client, spreadsheet_id, worksheet_id)
if options.addrow:
AddRow(columnfeed, spreadsheet_id, worksheet_id)
else:
for attr, val in enumerate(columnfeed.entry):
for key in val.custom.keys():
print "%s: %s" % (key, val.custom[key].text)
print "n"
As I mentioned in Debugging GC Problems in Parrot, being able to find a failure as soon as possible helps with debugging. With a new runcore for GC debugging, we’ve found several memory problems and fixed most of them. (Some require rethinking certain design decisions.)
Here’s the process to finding and fixing bugs of this type.
Leopard uses dscl, directory services command line, tool to manage most things important. Here is how you switch to Z-Shell:
sudo dscl /Local/Default -create /Users/ngift UserShell /bin/zsh
Memory problems can be difficult to find and fix. I’m a huge fan of Valgrind, but it only works at the C level.
The Parrot virtual machine
allocates memory with malloc and releases it with
free in a few places, and Valgrind is indispensable for making
sure that those match.
Unfortunately for Valgrind, Parrot provides garbage collection, and two of its fundamental data structures rely on the correctness of the garbage collector. Some of the weirdest, and most difficult bugs to solve within Parrot are bugs in our implementation of garbage collection, usually where active objects get collected too soon.
Memory problems are often very unportable. Not only can one program demonstrate a problem where hundreds of other programs run without problems, but one operating system differs from another. Perhaps 64-bit platforms work just fine, but 32-bit platforms have a nasty segfault. Maybe the particulars of how you compiled the program may change memory layout sufficiently to avoid the problem.
Debugging GC-related problems reported on that one platform you don’t have access to is time-consuming and difficult. I discovered another way recently. Now reproducing GC problems in Parrot is possible across platforms, making them much easier to diagnose–and once you can diagnose a problem, you’ve done most of the work to fix them.
One of the hazards of working with multiple computers is keeping data synchronized between them. I know I should keep Joey’s advice about Keeping Your Life in Subversion, but configuration files aren’t they only offenders.
For example, my personal web site is completely static. I generally update only a single YAML file when I need to make a change, then run a short script to rebuild the affected pages and update them on my server. scp is great for transferring files once in a while, but there’s a better option for frequent updates: rsync.
I’m also slowly migrating users off of a machine being decommissioned to a new machine, and there’s an hourly cron job which uses rsync to copy all of their files between the machines. Because rsync transfers only the differences between the files, it’s efficient on network bandwidth (even if it still must perform a lot of file IO to traverse their directory trees). It’s a program I could have written myself, very badly, but I don’t have to, thanks to the Samba hackers. Best yet, I’m not even sure all of the places I have it running as part of scripts or projects; it just does its job and stays out of the way (at least once you skim the man page for the correct option… my one gripe is that I never use it manually often enough to remember exactly every flag I need in certain situations.)
Thanks to everyone who’s contributed to the project for making this useful tool.
The commands module contains utility functions for working with shell command output under Unix.
The nice folks at Apple have given us a workaround to get IPython tab completion to work on Leopard. I post the exact details here if you are curious.
Basically you manually import readline…
ln [1]: import readline
In [4]: readline.parse_and_bind (”bind ^I rl_complete”)
A while back, Jeremy and I asked about topics people would like to see covered in the book we are writing about Python for Systems Administration. I would like to take this one step further as ask if there are any essential open source libraries we should cover in our book. An example of this would be the unbelievably useful IPy Module, which my co-worker Doug introduced me to a while back. Since then, I have found the module indispensable.
Also, if you have done something really cool with Python and Systems Administration, send me an email, and I will see if I can give it some coverage in the book. Even if you plan on doing something cool, then make a google code project and send me a link.
The itertools module includes a set of functions for working with iterable (sequence-like) data sets.
I’ll be speaking at the hack.lu 2007 security conference in Luxembourg on October 20, 2007. My talk is titled Breaking and Securing Web Applications. The conference agenda is here.
The shlex module can be used to create mini-languages using simple syntaxes like the Unix shell. It is also handy for parsing quoted strings.
Adrian Howard asked on
PerlMonks is there any way to
access the contents of a block eval?. After catching an
exception in Perl, can you see the source code that threw the exception,
without using a source filter?
I wrote many of my parts of Perl Hacks by trying to do impossible things that no one had ever done before. (I don’t know that no one had ever done them, but I’d never heard of them before, which was close enough.) Because everyone thought it was impossible, I decided to try.
The difflib module contains several classes for comparing sequences, especially of lines of text from files, and manipulating the results.
How many open source relational databases can you name? My friend Gabrielle recently sent me the links to two of them. However, if you’re like most technical people, you probably don’t know any — just as I didn’t until recently. I can already imagine many of you saying “bulls**t”, what about MySQL and PostgreSQL?” (to name just two), but those are just databases, not relational databases. Noted (and controversial) database experts Chris Date and Hugh Darwen, building on the work of the founder of relational theory, Edgar F. Codd, have tried to educate people about the actual relational model but with little success. It’s a strange world of relations, relvars, attributes and other things which sound familiar, but when you look closely, they’re not.
Another article of the series “Yet Another Perl 6 Operator”
In the the last article, we’ve seen some of the usual relational operators in Perl 6 and their enhanced syntax through chaining (which allows expressions like a < b < c).
Another kind of comparison operators are those that, instead of true/false returns, identify the relative order between its operands: before, equal, or after.
Another article of the series “Yet Another Perl 6 Operator”
As expected, Perl 6 supports the usual comparison operators. This includes the numeric comparison operators:
== != < <= > >=
(where '!=' is a short for '!==', the negated version of '=='). These operators convert their terms into numbers before comparison.
The string comparisons operators are here as well.
Another article of the series “Yet Another Perl 6 Operator”
In Perl 5, we expect values to DWIM (”do what I mean”) in various contexts. For example, if we use a string containg “42″ as a number we expect it automagically act as a number. Perl 6 keeps this tradition of DWIMmery and introduces several new explicit coercion operations.
? to get booleans + to get numbers ~ to get strings
Michael Feathers addresses the question “How do you test private methods?” in The Deep Synergy Between Testability and Good Design. (If you’re stuck in a language with a fetish for compile-time access control on the part of the library writer, you have my sympathy.)
The answer is more interesting than the question. As Michael points out, the desire to test private methods extensively often indicates that he has too much behavior in the class. Other comments suggest that the barrier to creating new classes is too high–we don’t do it often enough.
I’ve often suggested that a major benefit of TDD is that it encourages better design, not only because of YAGNI but because you immediately have to use the API you’re designing. If that’s painful, or awkward, you may need to refactor. Testability is one part of the aesthetic necessary to write good code.
Issue 13 of [IN]Secure Magazine is now available. It contains my article: Social Engineering Social Networking Services: A LinkedIn Example (originally a blog post, but now with cool graphics). Download it here.
Do any of the iPhone users out there subscribe to video podcasts yet, or watch TV or Movies regularly on their iPhone? I am currently watching the internal release of the Pilot for “Spotlight on Free and Open Source Software”, and it looks quite good on an iPhone. More on that separate topic a bit later….
Another article of the series “Yet Another Perl 6 Operator”
Perl 6 has two repeat operators: one for replicating a string/buffer and the other for replicating lists.
String repeat 'x' takes a string as the left argument and the number of times to replicate as the right argument.
$string x $count my $s = 'a' x 3; # 'aaa' my $empty = 'foo' x 0; # '' my $n = 2; my $dots = '.' x ($n - 3); # '' because ($n-3)<1
Another article of the series “Yet Another Perl 6 Operator”
Today’s operator is a very simple one, the string concatenation operator.
my $a = 'ab' ~ 'c'; # 'abc' my $b = 'def'; my $c = $a ~ $b; # 'abcdef'
Perl 6 has an operator Z, named zip, to interleave elements of two or more arrays.
my @a = 1,2 Z -1,-2; # (1,-1),(2,-2)
The zip is one of the list generating operators that gives the language some flavor of functional programming. This gets further as the usual semantics for lists is to get lazy generation, which means easy/efficient handling of large lists (and also the extreme case of infinite lists).
You surely heard about the upcoming Perl 6 language. This language will be endowed with a set of features so rich that every Greek and Trojan is eager to see a full working implementation. Among these features, there are operators, many of them. Perl 6 was even said to be an operator-oriented language, with a yet larger diversity than Perl 5 already has.
I’ve used Vim for most of a decade now, having completely failed to understand Emacs, then going through a vi tutorial on an HP-UX 9.x box and watching how fast text editing can be. I’m proud that the Unix command-line is my IDE. I have plenty of great tools like Perl, grep, find, ack. I’m quick and productive.
I’ve known about Ctags, and I used it once or twice, but never really took advantage of it until last week.
I’ve used vim -t tag to open my editor to a function’s definition, but that seemed a little bit clunky. If I were in a Vim session already, I’d have to background the process, launch the new editor, and remember to switch back and forth between the two processes. (screen makes this easier, but it’s less of a benefit for short-lived processes.)
Then I had a thought, and looked it up in a Vim reference to confirm. Ctrl-] on an identifier jumps to the definition of that symbol. Ctrl-t jumps back. This works without having to start a new Vim process.
Thirty seconds of research bumped ctags from something I use once in a while to an indispensible tool for programming and debugging. Thanks to everyone who’s contributed both to ctags and to the Vim integration!
At our last PyAtl meeting we talked about Testing. I did a short presentation on Doctests, and Toby Ho did a talk on Test Driven Development and Behavioral Driven Development in Python.
I got inspired and stayed up until 4 AM on Friday/Saturday morning writing tests for the deduplication tool I am working on,Liten. You can grab a copy of it from the cheeshop as well. As a result, there is now version 0.1.2, which has pretty much full, doctest, and, unittest ,coverage. I also added the ability to search by bytes, KB, MB, GB, and TB and a –quiet option. I think I am test infected now.
I am also getting excited about this project. I hope to get the reporting a lot nicer in the next month when I release 0.1.3. I also am mulling over the best caching system to use, as I am not completely sure if I want to cache previous searches via a shelve or an ORM with sqlite. It looks like, I will probably have a deletion mechanism ready in 0.1.3 too.
I went camping with my family a couple of weeks ago. True to form for an ironic universe, the day before I left, I received a hard copy of a book manuscript to review, with final edits due my last day on the beach.
I finished everything but a sixty-page chapter before we left. I knew that final chapter would only take a couple of hours, and I knew that I could actually review the hardcopy on the trip… and if I could get web access, I could type my findings and then mail them in one quick shot. Sure, that would take a couple of hours out of my previous vacation time, but we’d meet the book deadlines and everything would be good.
That was my assumption, anyway.
I was working with some XUL that was eventually to be part of a Firefox extension, but at the time loading the XUL directly into Firefox. Didn’t take long until I stumbled into the infamous “Error: uncaught exception: Permission denied to create wrapper for object of class UnnamedClass”. Time to break out XULRunner. I hadn’t played with XULRunner since Ubuntu days and it was enough of a pain to set up just right on Ubuntu. On Mac, it’s a right headache.
So, I made good on my promise to start learning Ruby. I do python programming in my day job and want to get better at Test Driven Development, Test Enhanced Development and Behavioral Driven Development, etc, in Python. Basically, I want to take a very similar language to Python, like Ruby, and strictly start the “testing” way. I am not sure how many other people have done something like I am doing, but it would be fun if a Ruby person did the same thing with Python. It might be neat to exchange notes.
These are the reasons why I think my theory will work so far:
1. I have no ego with Ruby. I know I suck, so it is ok to be slow at first while I am only doing Test Driven Style Development. Basically, I won’t be tempted to just crank out some code without tests.

2. I don’t have any friends yet that use Ruby. This again allow me to bring in any preconceived notation of how I am supposed to do things. I will be an empty vessel.

3. There are things in Ruby that I can learn and apply to Python.

4. It is just as fun to program in Ruby, as it is in Python. Lets face it Dynamic Languages are fun!
If any Ruby experts have any advice on what is the proper way to do TDD or BDD in Ruby, please let me know. I DON’T KNOW ANYTHING. Ahh…that is fun to say. Any advice or guidance is appreciated. I suppose this is almost like a cultural exchange visit. I just learned about BDD, and rspec and they seem interesting.
Oh, and I visited the local Ruby meeting a couple of nights ago in Atlanta, GA, and found it to be quite fun and light, yet still interesting. I plan on going every month now. One of the most interesting things I noticed is that when there is only one web framework, it kind of focuses the discussions and everyone knows what you are talking about. I have to admit, this seems like a very refreshing environment for a Newbie to a language, as no matter who you ask, they ALL know ROR.
If you are in the Atlanta area and interested in learning Ruby, then I recommend visiting:
ATLRUG
unittest module, sometimes referred to as PyUnit, is based on the XUnit framework design by Kent Beck and Erich Gamma. The same pattern is repeated in many other languages, including C, perl, Java, and Smalltalk. The framework implemented by unittest supports fixtures, test suites, and a test runner to enable automated testing for your code.
optparse module is a modern alternative for command line option parsing that offers several features not available in getopt, including type conversion, option callbacks, and automatic help generation. There are many more features for to optparse than can be covered here, but hopefully this introduction will get you started if you are writing a command line program soon.
Update: excellent tip from Eric Larson.
I’m a long-time CVS head, and I recognize CVS is quite showing is age, but I’ve never had much of a taste for the pundit’s heir apparent Subversion. I could go on about why, but I think it all boils down to SVN’s not fitting my head. I’ve been thinking of VCS and hosting options for my FOSS projects, and have discussed some of this in the community. I’m leaning heavily towards Mercurial (with Bazaar a close second). I decided to set up Hg on my MacBookPro so that I can play with it, and so that I can see if I can start doing local work on my machine in Hg and then push to a centralized SVN repo as a bit of insulation from SVN. Herein a few notes on the matter.
Most of the code that I’ve been writing for the book has been getting its own unit tests. I’ve been working on a chapter on networking for the past week and a half and have written a little code for the chapter. One of the challenges of writing tests for something like networking code is that there are so many variables which may influence a suite of unit tests. For example, if my unit tests rely on hitting some Google webserver and I encounter problems, trouble shooting questions may include: is my router acting up, is my ISP acting up, am I failing to get DNS resolution, is that particular server down, have they changed the URL for this resource, etc.
So, for the purpose of testing, I decided to bypass the socket module in this case and handle everything locally. I created a faux socket class fleshed out with the methods that I needed. I then monkey patched my module under test with the new faux socket class. All attempts to connect to a real socket actually “connected to” a fake socket from which I could totally control the behavior.
I’ve been reading Beautiful Code, picking out chapters here and there as I have time. While reading Brian Kernighan’s explanation of Rob Pike’s regular expression program from The Practice of Programming, I had an idle thought. “Hey, that’s a highly recursive program with complex behavior suitable for didactic purposes.”
Of course, Kernighan says that almost verbatim in the text. He also says “It’s a nicer example than Yet Another Fibonacci Sequence Generator.”
So I ported it to Haskell. I don’t promise it’s necessarily great Haskell, and I wouldn’t consider it entirely beautiful, but it appears to function.
I have been meaning for the longest time to scratch the personal itch of creating a robust command line tool to analyze duplicates on a file system. There are a few scripts floating around in various languages and the problem is not all that difficult to solve, but I went the whole nine yards and wrote a reasonably cool command line tool that uses md5 checksums to detect duplicates. A report is generated in addition to stdout dup messages in CSV format, so you can manually look through dupes and decide what you want to do with them.
Liten can be downloaded from the cheeseshop: http://cheeseshop.python.org/pypi/Liten/0.1a
GoogleCode: http://code.google.com/p/liten/
I have a rather long list of things to finish, like threading, daemonizing, caching ORM backend, way more unittesting etc. Give it a whirl and let me know what you think…
Oh, and thanks to the following people that I bugged with dumb questions like I usually do :)
Titus Brown, Shannon Behrens, Rick Copeland, Jeremy Jones, Scott Leerssen.
Why the Lucky Stiff’s fast, enjoyable Hpricot library makes hard Rails View tests effective and fun. Hpricot is a deep and useful HTML parser with a wide, flexible interface. It supports many clever systems to read and edit HTML. When we put it to work in Rails functional tests, it offers lots of different ways to solve hard problems.
I’m pleased to bring the Python Module of the Week series to ONLamp. The goal of the series is to cover one of the Python standard library modules each week with a brief description and examples.
I will post one or two of the popular past episodes here, but for the complete archive check out the main feed. Future posts will be posted to both sites.
Maybe you know how to drive a car. Maybe you can fix the carburetor. You probably don’t assume that you know how to design the car. However, programmers who can write huge SQL statements and understand the output of an EXPLAIN statement often assume that they can design a database. Admittedly, unless we’re comparing Oracle 9i with the 1967 Volkswagen Beetle, databases usually aren’t more complex than cars, but a properly designed database can be very hard to come by yet it’s the key to a solid application.
This is part 4 of an N part series on rewriting my podgrabber application. Here are links to part one, part two, and part three. In part 3, I outlined my strategy for synchronizing between mediaStores. This post will update that strategy slightly to show how I’m now handling threading.
For the curious, the code lives in a Bazaar repository at http://bzr.podgrabber.org/trunk/
The SyncManager now takes a taskManager in its constructor.
class SyncManager(object):
"""This is a concrete implementation of a syncronization manager which is
intended to be subclassed if necessary.
A SyncManager connects two mediaStores with filters and processing steps.
It should be able to copy files from the fromStore to the toStore, exclude
any files which were filtered out, and execute any processingSteps along
the way.
"""
def __init__(self, fromStore, toStore, copyFilters, deleteFilters, preProcessingSteps, postProcessingSteps, taskManager):
self.fromStore = fromStore
self.toStore = toStore
self.copyFilters = copyFilters
self.deleteFilters = deleteFilters
self.preProcessingSteps = preProcessingSteps
self.postProcessingSteps = postProcessingSteps
self.taskManager = taskManager
self._init()
And on copying a file, the SyncManager pushes the request to the task manager:
def syncCopy(self):
for mediaFile in self.getCopyList():
print "ADDING MEDIA FILE", mediaFile
logger.info("Copying file %s" % mediaFile)
self.taskManager.addCopyFile(mediaFile, self.toStore, self.preProcessingSteps, self.postProcessingSteps)
Here is the task manager code in its entirity:
from Queue import Queue
import thread
import threading
import time
import logging
logger = logging.getLogger("podgrabber.syncTaskManager")
class Shutdown(object):
pass
class CopyWorker(threading.Thread):
def __init__(self, q, fileDict):
self.q = q
self.fileDict = fileDict
threading.Thread.__init__(self)
def run(self):
#print "Running copy thread", self.getName()
logger.info("Running")
while 1:
logger.debug("Blocking while pulling items from queue")
mediaFile, mediaStore, preProc, postProc = self.q.get()
if type(mediaFile) == Shutdown:
#print "Break"
logger.info("Shutting down")
break
logger.debug("Retrieved items from queue")
for preProcessingStep in preProc:
mediaFile = preProcessingStep.process(mediaFile)
logger.debug("Retrieving file %s" % mediaFile)
mediaStore.addFile(mediaFile)
logger.debug("Done etrieving file %s" % mediaFile)
for postProcessingStep in postProc:
mediaFile = postProcessingStep.process(mediaFile)
self.fileDict[mediaFile][1] = time.time()
class TaskManager(object):
def __init__(self, numCopyThreads=5, numDeleteThreads=2):
self.copyQueue = Queue()
self.deleteQueue = Queue()
self.numCopyThreads = numCopyThreads
self.numDeleteThreads = numDeleteThreads
self.threadList = []
self.fileDict = {}
for i in range(numCopyThreads):
#thread.start_new_thread(self._copyFile, ())
copyWorker = CopyWorker(self.copyQueue, self.fileDict)
copyWorker.setDaemon(True)
copyWorker.start()
self.threadList.append(copyWorker)
for i in range(numDeleteThreads):
pass
def addCopyFile(self, mediaFile, mediaStore, preProc, postProc):
self.fileDict[mediaFile] = [time.time(), None]
self.copyQueue.put((mediaFile, mediaStore, preProc, postProc))
def addDeleteFile(self, mediaFile, mediaStore, preProc, postProc):
self.deleteQueue.put((mediaFile, mediaStore, preProc, postProc))
def _copyFile(self):
while 1:
mediaFile, mediaStore, preProc, postProc = self.copyQueue.get()
if type(mediaFile) == Shutdown:
break
for preProcessingStep in preProc:
mediaFile = preProcessingStep.process(mediaFile)
mediaStore.addFile(mediaFile)
for postProcessingStep in postProc:
mediaFile = postProcessingStep.process(mediaFile)
def _deleteFile(self):
pass
def shutdown(self):
for i in range(self.numCopyThreads):
self.copyQueue.put((Shutdown(), None, None, None))
for i in range(self.numDeleteThreads):
self.deleteQueue.put((Shutdown(), None, None, None))
for t in self.threadList:
t.join()
Basically, the task manager creates a queue for copies and one for deletes and a number of threads for each operation. When the sync manager passes the request to copy files to the task manager, it is a non-blocking call. The downloading/processing of each file happens N files at a time, depending on how many threads you’ve allowed to be active. The defaults are 5 for copying and 2 for deleting.
So far, this seems to be working pretty well. The only thing that I see that could use some immediate improvement is to either thread the downloading of the RSS feed(s), or to use Doug’s feedcache, or both. I’m going to try to work on trying to get that supported over the weekend.
The next area of functionality addition is that of creating a GUI. From the feedback I received in a post today, I’m going to have to check out wxPython.
Does anyone have any tips on GUI building on Mac using Python? It looks like PyGTK for Mac isn’t totally there yet. PyQT for Mac looks really out of date. Is building a Cocoa app using PyObjC my best bet? The more native, the better. The fewer dependencies, the better.
For everyone who read my A Taste of Haskell, A Taste of C, and whose eyes glazed over when I told Nat that Haskell functions only take one argument, John Goerzen explained what I meant in Haskell manipulates functions with the same ease that Perl manipulates strings..
If you watched A Taste of Haskell - Part I, Simon Peyton-Jones answers my question about parenthesizing expressions about halfway through; that’s when I was trying to explain this feature to Nat and made his brain hurt. (I realized halfway through SPJ’s answer that I had forgotten about associativity.)
If none of this makes sense to you and it still all sounds somehow cool (and once you get past the syntax and avoid some of the math, it’s seriously cool), check out Adam Turoff’s Why Haskell?, Pure Functions in Haskell, and Monads, for a great introduction to the language for LAMP programmers.
You could do a lot worse than to grab a lazy Saturday afternoon and work through the code. It’ll expand your mind in very good ways.
XPath is a complex query language that provides substantial benefits. It treats XML as a database, permitting queries as powerful as SQL SELECT. This post shows how to use a nifty visual explorer, XPath Checker, to write aggressive and accurate queries. Then we install these queries into test cases using assert_xpath.
I spent some time the past couple of days wrestling with HTTP authentication for a WSGI atom store implementation. It’s well known to be a tricky topic, and I went through a lot of bother, especially trying to figure out a Python WSGI library for HTTP auth that was clean, well-documented, and sported a flexible API that worked well across framework choices. I ended up using httplib2 for the client and decided on AuthKit on the WSGI server. I generally like to test my WSGI Web components with CherryPy, Pylons and raw Paste. In this case I didn’t get around to Pylons, I was able to get raw paste working well with code along the lines of the bundled example http://authkit.org/trac/browser/AuthKit/trunk/examples/authorize.py. I was never able to get things working with CherryPy, and I’m not sure why. I ran out of time to debug further. It seems CherryPy doesn’t like the HTTP status line sent by httplib2 with a www-authenticate response to a 401 using Digest authentication. I think this might have nothing to do with Authkit.
One problem I found with AuthKit is that I had to manually place ez_setup.py in the PYTHONPATH before the install would work. I might be doing something wrong, but this is not a problem I’ve had with other packages.
What Python tools do you use for your HTTP auth needs?
I’ll be at the Black Hat briefings in Las Vegas this week. In addition to the briefings, I enjoy going to Black Hat to reconnect with old friends, and to make new friends in the security industry. It is also a delight to meet people who read my blog and to have the chance to hear their thoughts and philosophies.
If you will be there as well, and would like to catch up, please send me an email.
When a test case calls methods that write new records to a
database, sometimes the test needs to fetch those records back and
inspect them. This post develops assert_latest, an
assertion that detects newly created records.
This post introduces developer tests that constrain
exceptions. Our platform, as usual, is Ruby, yet these topics
apply to any system. We will extend assert_raise()
for more control over program faults.
The best developers write tests to keep their projects on track. Tests should cover every aspect of a program, and should take special care with program details that are sticky, hard, and mysterious. Exception handling is a murky topic, because when a program fails its control flow might not be obvious. Many a program has failed in the field because nobody in the lab tested all its error paths. Our test cases must ensure that faults make our programs degrade gracefully, not derail.
Billy Rios has let me know about another vulnerability he has found along with Nate McFeters. Here are the URLs, which when clicked from Firefox running on Windows should spawn cmd.exe and calc.exe in order to demonstrate remote execution flaws in Firefox:
I’ve seen at least two mock object libraries for Python (here and here). But I wonder, what is the benefit of using a simpler mock object over creating your own dummy class? It seems that if I created my own dummy class (a class which implements the same interface that it is attempting to “mock”), I would have tighter control over the behavior of the thing as well as have a nicer re-use experience. It seems that mock objects are typically defined on the fly and then thrown away. I’m sure you could re-use them, but if they are really intended to be use-once-then-throw-away, it may be a little harder.
I guess what I’d really like to see is something between mock objects and dummy classes, something where you define a dummy class to be used as one of your application classes, but which also contains the convenience methods that the two mock libraries have. Suggestions, anyone?
I’ve been carrying around an interest in text processing for several years now which began with my work with EDI. Even though I don’t work with EDI and my job doesn’t revolve primarily around text processing, I still maintain an interest in text processing in general and processing EDI specifically. I created the project ediplex using Novell forge probably two years ago, around the time I wrote this article for DevX. ediplex back then was specifically an EDI processing engine with hopes of converting EDI to other formats pretty easily.
Over time, ediplex has evolved. A goal that I had for ediplex even from the beginning was the ability to easily define new EDI file formats. In its inception, it only supported X12, which is primarily a North American standard. But I had hopes for supporting EDIFACT and TRADACOM, which are more in use in Europe.
Which leads me to today. The latest incarnation of ediplex doesn’t support EDI. Not yet, anyway. What it does is allows users to create custom document definitions which describe what a document’s header and footer should look like. It also allows users to create custom handlers to allow the engine to feed them with data for a specific document type. The latest rendition is in early alpha, but it looks like a document is being passed all the way from its input to its handler. If you’re interested you can `bzr branch http://bzr.ediplex.org/trunk/` and start poking around. (This requires the Bazaar version control client.)
The architecture for ediplex is layered, but pretty simple. The first layer is the input layer. This layer gets input from somewhere (file, socket, whatever) and passes data to the scanner, which is next. The input layer was designed to allow users to create their own custom types of input receivers as they see fit. The next layer is the scanner. While this layer can certainly be replaced and customized, that shouldn’t be necessary. The scanner receives data from the input receiver and determines which document type the text should be passed off to and passes it off. The next two layers are the document definition and the data handler. I combine there here, because they are combined in the ediplex code. The document definition doesn’t do much except for describe a new document type and tell the scanner if a certain string of text matches its definition. The handler is intended to be extremely customized. When it receives data, it gets to do whatever with it that its little heart (and its coding master) desires.
So, if you’re in the market for a text processing engine, check out ediplex. I don’t have a license statement in the source tree, but will soon. I’m strongly leaning toward the MIT license, but am also considering GPLv2. Questions, comments, flames welcome.
XSRF (Cross Site Request Forgery) is a huge security problem affecting most web applications. There have been a lot of articles written about XSRF, including the useful XSRF FAQ I linked to earlier.
There are quite a few free and commercial web application security assessment tools and static code analysis tools in the market today. A few commercial security assessment tool vendors have published white-papers about the importance of discovering XSRF vulnerabilities, yet their own products do not have the ability to assess for XSRF. I think there are multiple reasons for this, and here are my preliminary thoughts:
URI Use and Abuse written by my good friends Billy Rios, Nathan McFeters, and Raghav Dube (affectionately known as “baby Dube”) exposes how web browsers and applications fail to sanitize URIs leading to remotely exploitable conditions.
Billy started the ball rolling (after deriving inspiration from Thor’s Safari URI handling disclosure) when he discovered a remotely exploitable vulnerability in the firefoxurl handler. An example of his this can be exploited in IE is available from Billy’s disclosure: cmd.exe (remote execution)cmd.exe will spawn regardless of any IE or Firefox dialogs.
Large, complex, cross-platform applications with multiple developers sometimes have bugs. Some of those bugs never appear on your own machine; they lurk for a while until someone else builds and tests the software on a different platform, in different circumstances.
Tracing that bad behavior back to a particular checkin can be frustrating, even if you have a huge smoke farm that rigorously tests every configuration of every checkin on every important platform.
Parrot meets all of those criteria, except for the huge smoke farm. (Smokers welcome.) When I want to pinpoint a regression to a likely checkin culprit, I use a binary search. Will Coleda’s App::SVNBinarySearch promises to automate that process. Here’s what I found.
OK - so I haven’t done a “recipe of the week” in a while. But does titling the post “{{whatever}} of the week” mean that I’m going to do one of these every week, or does it mean that I promise not to do more than one per week? :-)
Anyway, I was googling around to see if a certain type of utility existed and I stumbled across this recipe for something called Pyline. Basically, Pyline allows you to pipe text to it and use Python syntax to manipulate what it will output, specifically at the word and line level. Here are a couple of examples from the recipe:
Print out the first 20 characters of every line in the tail of my
Apache access log:tail access_log | pyline “line[:20]”
Print just the URLs in the access log (the seventh “word” in the line):
tail access_log | pyline “words[6]”
Good work Graham Fawcett. This is a useful little utility and the code is pretty brief. So, this is the recipe of this week.
The 5.9.5 release of Perl was announced today and may be downloaded at your nearest CPAN mirror. This is a release from the development branch of the Perl interpreter, also known as “perl-current, bleading edge perl, bleedperl or bleadperl”.
I just got myself an iPhone and I’m extremely pleased with it. I think it’s the best cell phone on the market - a sheer pleasure to use.
The purpose of this post is to alert new iPhone customers about a security vulnerability in AT&T/Cingular’s Voicemail system that has not been fixed for more than a year. I first wrote about this on February 1, 2006: Exploit Cingular Voicemail Vulnerability via Caller ID Spoofing. As soon as I got my new AT&T/Cingular number, I tested for this vulnerability and I can confirm that it still exists for new AT&T/Cingular accounts (atleast for iPhone customers). I can’t force AT&T / Cingular to fix this issue, but I can tell you about it so you know what to do to protect yourself from this vulnerability.
My involvement with the Wild West side of Python came somewhat accidently. I am helping organize PyAtl and on June 14th we had an incredible meeting! My company Racemi gave a mind boggling demo of our datacenter management tool that is written in all python. Our FlagShip Product Dynacenter allows any OS, including Windows to move around to different hardware in the time it takes to warm reboot…go Python! Finally, Google gave two presentations, one on Cross Site Scripting Attacks and one on Twisted. We also officially launched the PyAtl website that night which is running the bleeding edge Turbogears stack of Sqlalchemy,Genshi, and Toscawidgets. My friend Alberto Valverde is in charge of Toscawidgets and the concept is really awesome! If you haven’t met Alberto yet, you should, he is one of those rare exceptionally helpful, yet insanely smart people.
Here is where the the fun started…
I invited Mark Ramm and Jonathan Lacour to come to our meeting and talk about Turbogears. Mark and Jonathan mentioned that on the way up to the meeting they had a crazy idea. How about building Turbogears on top of Pylons? They announced an experimental sprint the next weekend and this is where things got wacky!
Rick Copeland, Jonathan, Mark,Mike Schinkel, and myself met at Jonathan’s house and started to experiment. We ran into an initial snag with understanding the pylons controller and I called up Shannon Behrens, another friend, who is insanely smart and incredibly helpful. Shannon works on the Pylons trunk and asked him how we would mount Turbogears on top of Pylons. After he got over the “you want to do what!”, he helped us with some good advice. At some point we all went to get some Pizza, then came back to watch Jonathan and Rick go into the “Zone”. After they came up for air, a controller was working and Frankenstein was born..mu ha, ha, ha, ha!
It was 1 in the morning by the time we all quit, but Mark, Jonathan, Rick and I decided to meet at Panera the next day at 1PM to finish it off. A little more work was done the next day, but part of the day was spent just hanging out and talking shop which was pretty cool as I hadn’t met Mark or Jonathan before. It turns out Mark and I have a bit in common as we both grew up on a “Ranch type compound” for parts of our lives, we both have been SysAdmins, and we are both writing a Python book right now. Mark is a really fun guy to hang out with for anyone who hasn’t met him yet!
So, after the weekend was over with I started to hear about some of the excitement. I emailed my most educated friend Mr. Phd from Caltech Titus and mentioned maybe he could contribute with some Twill stuff for TG2. I talked via email a little with Kevin Dangoor and noticed his big announcement.
Apparently, people were really fired up about the collaboration between Pylons and Turbogears. Lets face it, I am very excited that all of these smart people are working together! It now seems that some momentum in the battle for the perfect Python Web Application has shifted, as Pylons and Turbogears have the 800 lb Gorilla of ORM’s in SQLAlchemy, and they have Toscawidgets which is about to come into its own.
I have written several small web applications in Turbogears and Django and I like both. Currently Turbogears and Pylons don’t have a way to graphically manage the database like Django’s admin tool and the API isn’t as stable, but from what I hear this is about to change…..
I do get the impression that many people in the Turbogears/Pylons world feel left out and a common heard rallying cry is that Django has a “Not invented here attitude”. Whether this is true or not, I learned this past week that if smart python programmers feel they aren’t apart of the fold, they are capable of creating an uprising and doing just about anything!
I will close with this comment, Ian Bicking, who wrote paster which I think is pretty sweet, mentioned in a fairly famous post that it would be great, but unlikely that Pylons and Turbogears would merge, yet the impossible happened and the two frameworks are closely working together. May I suggest an equally implausible scenario? What if Django, Pylons and Turbogears worked on developing an interchangeable API? Is this impossible…you tell me!
If you read up on the Model-View-Controller (MVC) design pattern, you might find yourself a bit confused. In fact, I found myself confused by it when I first started reading about it, because there are plenty of resources out there to describe it, but so many of them seem to have different flavors of MVC and different diagrams explaining how data flows that it’s no wonder that programmers are bewildered about it. Fully believing that I don’t want perfect to be the enemy of the good, I’ll show a few practical implementation details of one way of looking at MVC, primarily focused on the Web needs.
Somehow I missed James Carr’s TDD Anti-Patterns late last year. I’ve perpetuated almost every one at least once. If you’re new to testing, browse the list, think about each entry, and watch for it in your own code.
I actually like fixing bugs and optimizing code. It’s satisfying to simplify a piece of code while making it perform better and use less memory.
I’m a terrible guesser, though. I guess about where bottlenecks are correctly perhaps half of the time, and likely less often than that. To make the most of my available time, I need a good profiler.
I’ve tried to use GNU gprof, but the requirement to recompile all of my software specifically to use gprof was too much. Instead, I use Callgrind.
Callgrind works like Valgrind (and it’s part of the Valgrind tools now). Run your program through Callgrind as normal (valgrind --tool=callgrind program_name opt1 ... optn); it will collect statistics about the run. Then run callgrind_annotate on the output and see a nice report about where your program spends its time. Though this is only the most basic of Callgrind’s features, it’s often sufficed for me to find and fix true bottlenecks.
If you need more features, I hear that KCachegrind is a useful visualizer of Callgrind’s statistics. I haven’t used it enough to discuss its value.
Because of Callgrind, I spend more time optimizing bottlenecks than I do finding them. Thanks to its developers and all contributors!
One of the sub-projects of Pugs is a series of Perl 6 sanity tests which define a minimal set of useful Perl 6 features. The idea behind those tests is that a Perl 6 implementation which can pass the sanity tests supports enough features so that it’s possible to bootstrap the rest of Perl 6 in that minimal implementation.
The Parrot project recently borrowed those sanity tests for the Perl 6 on Parrot implementation. (I work on Parrot in part because I believe that Parrot’s compiler tools are much more suitable for building compilers and languages than anything else I’ve ever used.)
Though I spend more of my Parrot time these days applying submitted patches, fixing bugs, and refactoring code, I try to make time for new development. I heard that we almost had all of the first suite of sanity tests passing and decided to see if I could improve the situation.
I’m a Vim bigot, but I reluctantly leave my comfortable off-white-on-black terminal windows for a web browser once in a while. Sometimes I even have to type more than a word or two in a textarea.
I used to grumble every time a site provided a tiny little text box for entry, or when I wanted to make an edit and left a row of
Now I use the It’s All Text Firefox extension and, before I let annoyance creep up on me, flick my mouse to a little blue box at the bottom right corner of the text area. With one click, up pops Gvim.
Suddenly, it’s worth typing more than a sentence into a web form again.
Thank you to all contributors to It’s All Text for making textareas usable.
If you’re still writing your own authentication for your websites, you may want to get with this program. Have a look at the bottom right of this page:
www.buxfer.com/index.php
Yep, almost everybody has one of those accounts these days, and more and more of those users are getting tired of endlessly multiplying username/password combinations.
I realize that Microsoft tried to do some Passport service in the past, but you can base your website’s auth on any site or combination of sites, even if they don’t have an API. As Tony Stubblebine has put it: “A login form is an API.”
I had been fighting with some silly old-fashioned servlet code. No need to say, I was not getting the upper hand. Never having been a web developer, I was baffled with why my POST request never got the contents I expected it to.
I spend a lot of time searching text files. More accurately, I spend a lot of time searching nested directories of text files. For source code, I know I should use Ctags, but I’ve never quite made the switch.
For plain text files (books, articles, stories, weblog entries, notes, contracts, et cetera), I’m still a GNU grep fan.
I spent a few hours in the past week editing a book manuscript and producing well-formed and valid DocBook XML. (I wrote two books in DocBook XML. While it’s a great file format for producing a book, it’s a face-stabbingly hateful format for actually writing a book.) Unfortunately, the conversion process to DocBook revealed some problems in the source material. In specific, certain links from one part of the manuscript to others were invalid.
I needed to find and fix the dangling links in all fifteen book chapters, spread out in several dozen individual files. Grep and a little bit of command-line magic made the task much, much easier. I ended up with the pattern:
vi $( grep -l 'L<refactoring_strategies>' ?_*/*.pod)That is, search all of the .pod files in directories whose names start with one character and an underscore. For all of those files which contain a link to an anchor named refactoring_strategies, print their names. Open that list of files in Vim.
I still had to edit plenty of text, but finding only the files I needed saved me a tremendous amount of time. Throw in grep’s -r (recurse into subdirectories) and -i (use a case-insensitive match) switches, and I’m very happily productive.
Thank you to everyone who’s contributed to grep and GNU grep through the years. Your work helps me work, every day.
Google Gears, as you may have heard, is a browser extension that lets you develop applications that can run offline. If you haven’t already, try out the sample applications to get a feel for the functionality Google Gears has to offer. You can even use it to read Google Reader offline.
It’s a good idea to brain-storm in the possible security implications of Google Gears because it facilitates web code to act upon the user’s local disk (sand-boxed with the browser’s same origin policy). I’ve spent a few minutes looking at the architecture, and here are my initial thoughts:
My first patch to Perl 5 was a quick and dirty tiny feature enhancement. It also broke a couple of tests. That small act of public humiliation reinforced what I already knew was a good practice; automated testing is an important part of creating software that works.
I spent a couple of years chasing two goals. First, to create tools of such quality and ease of use that there’s no reason not to write good tests for Perl code. Second, to add tests so that we could immediately identify regressions and track them down to specific checkins.
Today, the core Perl 5 test suite (as of the most recent snapshot leading to Perl 5.8.9) has 121873 assertions. It could use more, but those tests cover the language and core libraries. Modern CPAN distributions are incomplete without tests written in the modern style, and test coverage and quality are topics of wider understanding and discussion.
I can’t imagine relying on a piece of software that I can’t verify with automated tests.
Analyzing the JRuby test suite, a recent weblog post from Christian Neukirchen stunned me. The most complete test suite for Ruby is the JRuby test suite and it has only 2747 assertions.
I know Ruby’s simpler than Perl, but it’s not that much simpler.
Yes, this is a Movable Type weblog. Yes, I accessed it through Firefox. Yet I hate typing in little text input boxes.
I’m writing this (surprisingly post-modern) entry in Vim, a tool I find almost indispensible. It seems like every time I use a computer, I end up wanting to use Vim.
I’ve used Vim to write all of my articles and all of my books. (I’ve used Vim to edit all of the articles and books I’ve edited too. I even wrote a two-line wiki in Vim, which I use to, among other things, keep track of the list of free software projects that deserve my thanks.
Speaking of software, I don’t know how I’d program without Vim. I’ve tried a few IDEs here and there, and there are some nice features, but the ability to work with text rapidly and quickly, through powerful muscle memory, keeps me using Vim.
So to Bram and all of the other developers and contributors to Vim, especially those who’ve shared a few of their secrets which I’ve incorporated into my personal corpus of productivity, thank you. Your work is indispensible.
Perl has a long history of copious documentation through the Plain Old Documentation format. This applies to much of CPAN, not just the core modules and documentation.
The perldoc utility is the main way to view this
documentation. It has more features than people imagine (it was worthy of
Hack #2 in Perl
Hacks), but it’s a command-line tool only. Even for a CLI fan like me,
sometimes hyperlinks are nice.
Pod::POM::Web
is a CPAN distribution which turns all of the POD on your system into
browsable, linked HTML. I use perldoc all the time; could
anything displace it in whole or in part?
In my last post on the topic of rewriting my podgrabber utility, I promised to post the rewrite-code-in-progress to a Bazaar repository. You can branch from here if you’re interested. In this post, I’m going to discuss the paradigm I’m following for getting files from a webserver, pulling them onto a computer, then onto a portable media device.
In the current version of podgrabber, there was a concept of a download manager which would take a URL and save the file to a particular directory. This download manager was built with a small amount of extensibility in a very clunky way. I looked at the URL in order to determine how to download the file. After getting the files from the webserver to my computer, a single function would synchronize files between my computer and my portable media device.
This approach works, but it doesn’t provide a cohesive approach to the problems. It also isn’t very extensible. In order to come up with new file sources (such as FTP) would probably involve a lot of cut and paste and an ever-growing download method. And synchronizing downloaded files to anything other than some MP3 player that shows up as a USB disk drive would prove quite painful.
It is sometimes humbling to look back on code you’ve written. It is particularly humbling when you see a piece of your own code and wonder, “What was I thinking?” I’ve been going through this sort of programmer-introspection induced humiliation lately with my podgrabber project.
podgrabber began its life as a really simple Python script which would go through a set of RSS feeds, figure out what to download, then pull them in. I then decided to make it a little more interactive and allow the user to specify which of the undownloaded podcasts to download. A little coding, a little hacking and it was so. Somewhere along the way, I created a simple sync script to get the podcasts synced up on my MP3 player. (The MP3 player I had was a Sandisk Sansa e130 which showed up as a USB drive under Linux.) The final addition to podgrabber was a GUI (built in pyGTK) which would consolidate all of these features.
The GUI podgrabber has been pretty functional. I can add, remove, or update podcast feeds. I get a list of new podcasts and can select to either download all of them or just some of them. And I can go through a list of podcasts which are currently on my hard drive, delete them, and the next time I run “sync“, the old ones will be removed from my MP3 player and the new ones will be added.
These days, the perl6-language@perl.org mailing list is haunted by some passionate discussions, spawning long (often too long) threads. Just yesterday, someone posted yet another “Why Perl 6 has not such and such features” message which sparked a lot of discussion. After a while, someone detached one of the points of the original message and asked why Perl 6 still goes with explicit line termination with the semicolon. And then came an answer by Larry Wall (and they are always worth reading).
These sorts of things are almost never for a single reason. Some of
it is my prejudice against dangling syntax, and perhaps prejudice
against anything resembling Fortran. Some of it is not wanting to
distinguish different kinds of whitespace any more than we already do.
Some of it is simplicity of parsing, both for the human reader as
well as for the computer. I think if I had to pick one reason,
though, it’s that it allows the parser to understand the intent of
the writer much better and hence give more useful diagnostics when
something seems to be going wrong. Much more than other languages,
Perl depends on the prohibition against two terms in a row as a kind
of “self-clocking” mechanism to disambiguate programmer intent, and
not requiring a semicolon between the final term of one statement
and the first term of the next statement would tend to weaken that,
especially when the term starts with a prefix operator that could be
mistaken for an infix.
Last night at the Atlanta Python Meetup, I became aware that there are folks who have been around Python for a while who are unaware of the Python Cheese Shop. The “Cheese Shop” (named after the Monty Python skit) is intended to be a central repository for libraries and applications written in Python. Couple this with easy_install and you have a quick, easy, and clean way to install libraries and applications to your Python installation. By issuing the single command `easy_install ipython`, you will have the latest IPython installed on your system.
What’s happening in the front of Perl 5 development? I bring you some rumours that cheer me up.
There are nice markup languages out there. One of these is Textile. When writing the basic stuff, it lets you write uncluttered code (even with attributes, classes, ids). And when needed, you can jump the gun and write raw HTML.
You find Textile implementations in many programming languages, including the four P languages: PHP, Python, Perl and Ruby (the P language with R :).
Textile looks to me like a stripped HTML variant, one that gets rid of all those angle brackets. It makes me feel lighter. Such a piece of text seems just right to me.
h1. Header
A paragraph with a *bold phrase* and an _italic phrase_.
"Nice looking quotes"! Pretty ellipsis... and beautiful rendering of 2 x 2, one(TM), two(R), and others. I also enjoy footnotes[1], @pieces of code
in monotype@, and %{color:red}more%. Try to paste that code into the "Textile development page":http://textile.thresholdstate.com/ .
fn1. Cool, isn't?
Who uses Textile? Do you feel the same as I do? Or am I wrong and these distinguishing features are just too common?
Update: fixed link mistake pointed by Tj.
And this is my first blog entry for O’Reilly. It’s such a horrible thing to do something for the first time. I wish I started it by the second one and I hope you enjoy the topics here. Maybe I even try an introduction some other time.
Alligator Eggs is a very cute puzzle game with a surprising mathematical model beneath. I wish it explained how to indent my alligators, though. They keep going out of alignment.
Hey, it’s that time again. If you have an idea that you think would be beneficial for the Perl community, please check out our Call for Proposals. Get paid for open source work!
I should listen more to Brian Aker. He gave me a tip at least two years ago that I should visit Linuxfest Northwest. He was right.
This free conference expanded to two days this year, with nearly a hundred talks. The latest estimate I’ve seen is that around 900 people attended. That’s impressive!
There are four talk slots per day, so if you go all weekend you can only catch eight full talks. I heard about everything from building a persistent file system on a USB drive for Ubuntu to LiveJournal’s scaling strategy to the open sourcing of Second Life’s code to the history and intent of copyright. Imagine a computer lab full of people from age three to sixty-plus all building their own Linux-capable thumb drives.
I didn’t expect that, and I’m supremely glad I went.
Of course, the Linden Lab and Silicon Mechanics-sponsored party in the American Museum of Radio and Electricity didn’t hurt, either. (A working theremin! A 75 year-old car phone! Free food and drink!)
If you’re in the Pacific Northwest next year (or can be), go to this conference.
Thank you to the founders, volunteers, sponsors, speakers, and attendees. I’ll be back.
A few days ago, April 24 2007 to be exact, I performed a search on apple.com and out of curiosity, performed another search with HTML characters to see if they would be echoed back into the HTML. In other words, I was trying to see if apple.com’s search feature was susceptible to XSS (Cross Site Scripting). I found one attack vector and immediately alerted product-security@apple.com. A XSS issue on apple.com is of significant risk because it can be exploited by attackers to steal data from users that are signed on to apple.com.
On April 25, 2007, I received a thank-you email from Apple letting me know that they were investigating the issue. The email also stated: “Because of the potentially sensitive nature of security vulnerabilities, we ask that this information remain between you and Apple while we investigate it further” and included a case number.
The PEP submission deadline for Python 3000 has just ended. One idea really caught my eye: Traits/Roles instead of ABCs. (ABC means abstract base class.)
I fiddled with the ideas that would become roles in 2003 and 2004 before Allison Randal handed me a copy of the Smalltalk Traits paper and asked “Is this what you were trying to tell everyone?” Since then, we added them to Perl 6 (see Apocalypse 12).
I’m very pleased to see Collin Winter’s proposal (and his reference to Roles: Composable Units of Object Behavior. The concept is very powerful and–if applied throughout the entire system–provides a different and more flexible way of thinking about types and objects and classes.
I still stand by my warning about redesigning a programming language, however much in the spirit of good-natured ribbing. Integrating powerful new concepts cleanly may require and inspire further changes.
I attend more than my share of conferences. Because I travel for work, it’s important to take my laptop with me–and there’s little more convenient than open, unfettered wireless Internet access. (Wireless power that doesn’t cook my innards is the only thing that comes to mind.)
The easier it is to find an open wireless network and join it, the better my life is. (Good wireless support is one of two things I though Mac OS X did well; see Switching Back to Desktop Linux.) Fortunately, I found WiFi Radar and life is easy.
For example, I connected my System 76 laptop to the Portland airport’s free wireless several months ago and haven’t thought of it since. I had some free time before a flight yesterday, so I booted my laptop and went to connect to the network, only to find that my system had already joined.
The fewer pieces of infrastructure I have to keep in my head, the better. If selecting an ESSID and entering a key once is sufficient to let me update code and fetch and send e-mail, great. Thanks to all WiFi Radar developers and contributors for making that process easy.
Preface: if you love XML, that’s fine. I’ve nothing against the technology per se, but it’s not always the best tool for the job.
I’ll be in Copenhagen next weekend for the Nordic Perl Workshop giving a talk about multi-language test suites. This will be based on the work done with TAP::Parser and it will contain a brief discussion of TAP (the Test Anything Protocol), a protocol which is almost 20 years old and is gaining in popularity.
One question I’m sometimes asked by those not involved with TAP is why we don’t use XML for our test results. This is a brief attempt to answer that.
“The current pugs implementation is just translating to the old form underneath, so it’s not surprising it’s a bit off. That’s the sort of thing that happens when the language designer gives the language implementor whiplash. However, I rather suspect the interpersonal metaphorical meaning was lost on the physicist/comic who decided that the 3rd derivative of position should be called ‘jerk’. :)”
– Larry Wall, in ‘What should file test operators return?’
“developers shouldn’t live in fear of $^O”
– Jerry Gay, in ‘Use of English pragma’
I believe that a programming language should never crash, even given bad input. There may be cases where it reports obscure syntax errors that are difficult to understand, but crashing is unacceptable.
One way to make sure that there are no crashes is to feed your parser as much invalid input as you can imagine and check that you only ever get syntax errors. (I suppose another way is to write formal proofs for your parser, but even then you may have bugs in your implementation.)
To do that, you need a large corpus of valid programs and a way to generate a large corpus of mostly-valid programs that aren’t quite right.
I installed Algorithm::MarkovChain and set to work.
“I don’t think that it’s possible to make this non-conformity a fatal heresy :-(
(gcc –spanish-inquisition)”
– Nicholas Clark in ‘[perl #42110] [PATCH] Returning values from void functions’
This week saw the introduction of the Perl 6 Microgrants. Read more about them in ‘Perl 6 Microgrants. Now accepting proposals.’
One of the weirdest portability problems I’ve ever encountered is dealing with platforms with case-insensitive filesystems. Files I expected to be there mysteriously weren’t. Files I didn’t expect to be there mysteriously were.
Case-sensitivity in programming languages confuses novices too, especially on insensitive platforms. The capitalization of certain words matters within the language where it doesn’t on the filesystem.
Usually that only leads to mysterious error messages. Though a novice may not know enough about the language to decipher the message yet, there’s no silent failure. Unfortunately, there’s a case in Perl where there are silent failures: pragmata.
I feel awkward admitting that I don’t use primarily mutt as my mail client (I do use it when I ssh into various machines), but I’m a big fan of KMail.
I’ve never found a perfect mail client, but I’ve used KMail off and on as my primary client since 2000, and it’s definitely my favorite.
I strongly prefer IMAP, and only two clients I’ve tried have handled IMAP at all decently: Mail.app on Mac OS X (just about the only Mac OS X application I might reasonably miss) and KMail. My most recent flirtation was with Evolution, but as my mail folders grew larger and more numerous, its performance went from adequate to abysmal and, finally, unusable.
When I switched back to KMail, I set up my IMAP connections as cached IMAP connections, and my problems went away.
I have a few small gripes and questions that I really should ask on a #kmail or #kontact IRC channel sometime, but it’s only when something goes seriously weird that I even have to think about my mail program. That’s the sign of a quality program.
Thank you to all KMail and Kontact contributors and developers!
Way back in December, Tim Janik wrote State of the Gtk+ Maintenance.
Like many large projects, Gtk+ is an essential part of many free software desktops. Like many projects, it’s also grown organically. Like many projects, it has far fewer core developers than you might expect.
It can be difficult to mentor new developers through the process of making a first contribution to becoming full-fledged developers. That doesn’t happen very often, in my experience. What I appreciate most about Tim’s message is that he explores all of the necessary issues to improve the Gtk+ project to the point of making that goal more reasonable.
I’ve long believed that having source code available under a free license is no guarantee of utility, much less attractiveness to potential contributors. Thoughtful project governance and management is, perhaps, an order of magnitude more likely to make one project more attractive and sustainable.
I’m not a huge advocate of getting type checking into Python, but this is an interesting recipe. By applying a decorator, you can specify what types the input and output of a function/method should be. I like having the flexibility to be able to do this if I want to. (As an aside, one benefit for me of working with code that has explicit input and return types is knowing for certain what type of thing a method/function takes.)
The recipe can be found here. It’s not revolutionary and may be less relevant when signature annotations make it into Python 3000, but it’s really interesting to look at now. (Less relevant meaning in the immediate knowledge of what types something takes and returns by reading it in a Python 3000 signature-annotated method. I know that there aren’t plans for typechecking in py3k.)
Both Twitter and Jott authenticate users by their phone number. Twitter does this by validating users based upon the source of SMS messages sent to the phone number 40404 (US), and Jott does this by trusting the incoming Caller ID when someone calls 877-568-848. From a security perspective this means the following:
I’m working my way through Graham Hutton’s Programming in Haskell. Despite the fact that I’m not a mathematician (and I still believe that Haskell has syntax only a mathematician could love), it’s an accessible computer science text. It’s nicely clear, too, despite its relatively short length.
Most of the exercises are good too… except for one stumper in the list comprehensions chapter.
Due to a lack of free time, I will be unable to continue creating the Perl 6 summaries after April. We are looking for a volunteer or group of volunteers to take over the task.
Please contact me if you would like more information about what the job entails (contact information can be found at the end of this post).
I’m not a fan of procmail. I like what it can do, and I deeply respect people who make it work and especially make it work for them. I use Email::Filter instead, and I’m extremely happy with it.
I get a lot of mail, and plenty of it comes from mailing lists. To direct mail from a new list to the proper folder, I only need to add a single mapping of the list ID to the folder name. A little Email::Filter program takes care of everything else. I haven’t needed to touch the program in years. I also don’t have to remind myself of how procmail works every three or four months, when I need to change a rule. I rarely even notice it’s there.
Thanks to rjbs, the Perl Email Project, and everyone who’s contributed somehow to procmail and Email::Filter. You’ve made my life much easier.
“Q: Can February March? A: No, but April May…”
– Larry Wall’s r14313 log message for a date correction
TAP::Parser is the intended replacement for the venerable Test::Harness module. The intent is to clean up the code in such a way that writing custom test harnesses and supporting new TAP features is possible. (I’ve hacked on Test::Harness:: Straps; it wasn’t the easiest programming task I’ve ever tackled.)
I added TODO tests to Parrot’s test tools a while ago, to make it easier
to distinguish expected test failures from accidental failures.
Unfortunately, Test::Harness displays very little information
about TODO tests that passed. In TAP terms, these are bonus tests.
The programmer expected them to fail, but they actually passed, so they
need further investigation.
It would be nice to collect information on skipped, TODO, and bonus
tests in the normal test run. Though I could write a harness via
Test::Harness::Straps, I decided to try
TAP::Parser instead. Here’s what I discovered.
This week’s recipe can be found here. It shows how to use closures to perform various sorting operations. Before you click on the link, let me point out the same thing as one of the commenters of this recipe: this recipe is pretty much obsoleted by mixing operator.itemgetter with list.sort(key=foo). I thought this recipe was interesting, though, because it showed a really good use for closures. Having a concrete idea of how a certain feature can be used sometimes helps to use the feature in other ways. Hopefully that will be the case here.
We had a situation come up at work the other day where we seemed to be receiving some spurious data from a data provider. The data providing process connects to one of our processes on some specific port and sends a relentless stream of data. Said spurious data uncovered a bug in our process which was causing it to die abnormally. My first action was to get tcpdump to show me what was going on. But the results were just wrong. I suspect the incorrect results I was seeing were caused by the antiquated version of tcpdump running on an antiquated FreeBSD machine and trying to view the results on Wireshark/Ethereal on a recent Ubuntu box.
So, I figured a logging proxy help. So I whipped one up using Twisted. It worked pretty well. I know I didn’t get everything right since I don’t regularly use Twisted. Basically, every connection that is made to the proxy from the data provider initiates a client connection to my server process. That connection also creates a log file on disk with a name that identifies where the connection came from. Each piece of data that is sent from the data provider is logged and forwarded on to my process. What it doesn’t handle properly is my process going down. I didn’t spend enough time to figure out exactly how to attach a reference to the server piece of the proxy onto the client piece.
Enter the recipe of the week. Just ten days ago, this excellent recipe was either submitted or updated (I can’t tell which). This recipe contains code for a proxy server which would fit my needs and log a hexdump of the received data. I haven’t tried it in the context of what I was trying to do, but given the testing I did with it, it looks like it would work quite nicely. The only thing I would change is the format of the logging. For my purposes, I’d still need to have a raw log of the transmitted data. But this is a great recipe that shows an example of a working proxy in Twisted.
Thinking of trying the Drupal open source content management system? It’s a powerful platform, but the learning curve can be steep, even if you’re already comfortable with its underlying technologies: PHP, MySQL and CSS. As the volunteer webmaster for the Monterey County (California) Democrats, I’ve gotten deeper into this stuff than I ever anticipated, and believe me, I know that learning curve well. Here’s a list of some of the top gotchas. Some of them are just plain good web development practice, but they become especially important with Drupal, and even more so if you’re using the CiviCRM contact relationship management module.
I have a love/hate relationship with GNU Make. Yes, it’s picky about syntax and it’s difficult to write cross-platform Makefiles (though that’s not really GNU Make’s fault), but a make utility of some sort is mostly ubiquitous across the free Unix-like platforms.
When I need to compile a project written in C or C++ (or when I want to automate certain system administration tasks, such as remembering to update my Postfix files when I update them), I use GNU Make. It does a difficult job without much thanks or thought. I suspect that its maintainers, like me, would like to see a cleaner and friendlier replacement sometime in the future, but for now, its ubiquity and its power are definite advantages.
Thanks to everyone who’s contributed to make and GNU Make!
While the idea of circumventing the privacy offered by Tor via DNS, Flash, and Java (applets) is nothing new, HD Moore’s “Torment” Tor server hack has made news at Securityfocus and ZDNet. Although I’m not quite sure why this big news now all of a sudden, it does have positive side effects for the Tor project (see my opinions below).
In yesterday’s installment (Testing FizzBuzz in Parrot), I explained a test file for testing multiple Parrot implementations of the FizzBuzz problem. I also promised to show two different ways to solve the problem in Parrot. The test framework requires both approaches to take a single integer describing how many FizzBuzz elements to produce and to return an Array-like PMC containing the FizzBuzz strings.
A recent discussion on interviewing programmers (in hopes of finding clueful ones) brought up the FizzBuzz challenge. Can you write a program to print the numbers from one to one hundred, printing also “Fizz” for multiples of three, “Buzz” for multiples of five, and “FizzBuzz” for multiples of three and five?
This ought to take no more than a few minutes for a developer with any proficiency in a language. I decided it would be fun to write it in Parrot’s PIR. There’s the straightforward procedural way, the array overloading way, an object-oriented way, the coroutine approach, and the generator technique.
I chose the first two, but I also decided to work entirely with test-driven development, even though this is normally the realm of a SpikeSolution–I thought that might be more interesting for everyone.
Snow.
Wind.
Snow.
Wind.
Snow.
Repeat.
What is Colorado? That’s correct. I’ve been in the Winter Park area for about two months now, and I can only laugh at myself for thinking that my home, in the Piedmont of North Carolina, has a real, bone-freezing winter. Don’t get me wrong, our climate can stir up some pretty rough winters, but the wind chill here has gotten so low, to the point where you just laugh about the surrealism of it all. I’ve got my better Brazilian half to keep me warm, and some new cryptovirology research to keep my thought process in a relatively thawed out state. So, let’s talk about fish.
Well, not a real fish, but the sea-faring, fin-bearing creature I’ve chosen to use as a naming convention for cryptoviral functions. Science tells me I have a whole load of names to choose from, so I’ll be fine for a while. Without further pointless ado, allow me to introduce Mackerel, a family of cryptoviral functions. Here’s the preliminary abstract for a paper that will be presented at Security Opus, an information security conference in San Francisco, with technical lectures being held from March 19th through the 21st:
“Mackerel is a family of symmetric cryptovirus constructions that allows up to IND-CCA2 and INT-CTXT security; they’re based around the AES in CTR mode (IND-CPA) for preserving confidentiality and CMAC-AES (SUF-CMA) for preserving integrity. The optimal configuration (IND-CCA2 and INT-CTXT), “King Mackerel,” employs two 256-bit symmetric keys, for encryption and authentication in the Encrypt-then-Authenticate (EtA) composition, and claims a 128-bit security level. All functions operate in the Troutman mode of information extortion (TIE), a slight variation of Young and Yung’s information extortion attack [1]. While Mackerel requires its own set of intrinsic analyses, it takes advantage of the analytical scrutiny of the AES; as such, the security of Mackerel reduces to that of the AES. Mackerel is based on original research conducted by Troutman, in [2]. Mackerel is in the final stages of preliminary cryptanalysis, of which will support Mackerel in a standalone paper, set to appear in Spring ‘07, along with a complementary protocol for ensuring fairness via game theory.”
[1] A. Young, M. Yung, “Cryptovirology: Extortion-Based Security Threats and Countermeasures,” IEEE Symposium on Security & Privacy, pages 129-141, May 6-8, 1996.
[2] J. Troutman, “Examining Misimplemented RSA and Strengthened Authentication for Variations of the Cryptovirological Information Extortion Attack,” Duke University (TIP), July 24th, 2006.
So, as you can see, research has gotten as far as receiving a cool name - well, a name, at least. As of right now, Mackerel has taken on a completely standardized approach, by using AES. However, Mackerel is merely a shell, of sorts; that is, encryption and authentication functions, and their parameters, are largely arbitrary. As such, Mackerel can be configured for various trade-offs between efficiency and security. The paper will focus primarily on the most conservatively secure configuration, dubbed “King Mackerel,” which is IND-CCA2 and INT-CTXT secure. The algorithm specifications and design rationale paper will be available during, or shortly thereafter, the conference. It follows that a complementing game theoretical paper, outlining the Troutman information extortion mode of operation (TIE) for Mackerel, is set to be completed by the Spring of ‘07; in late June, it will be presented in a guest lecture at Duke University.
In the meanwhile, I’ll be investigating other niche environments for Mackerel, both software and hardware, as well various other structural possibilities and applications for cryptovirus design. As always, I’m quite interested in any feedback - criticism included. Recognizing insecurity comes before understanding security, so the more folks looking at cryptovirology, the better. Until next time, I’ll be dreaming of warmer days, when I’m back in the South, away from 70mph wind gusts, incessant snow, and -30F wind chill. It’s all good, though.
Long live thermal underwear and down feathers (and a future excursion to Ipanema).
Tchau.
“‘Course, if someone goes ahead and adds the Y combinator, one must naturally begin to wonder what the YY combinator would be… :-) “
– Larry Wall
“Obviously it generates a function so anonymous that it can’t even refer to itself. I call it the depressed existentialist solipsist operator.”
– chromatic, in ‘Y not’
The mini-CPAN, a smaller version of the Comprehensive Perl Archive Network that includes just the latest versions and excludes a few big things, is now about 700 MB on my machine. That means that it can’t quite fit onto a single CD, at least without removing parts of it. What should go though?
I’ve been playing with GrandPerspective, a Mac OS X utility to show a tree map of a directory to easily show where the big files are. Here’s the map for my /MINICPAN:
The big files represented by the tan section in the lower left are BioPerl, Most of the other big boxes are parrotin various releases, but from different authors (so maybe my minicpan script needs to recognize the multi-author situations to remove old versions.). This would probably be a cool movie of an animated window, but I don’t know how to do that just yet. :) Now it’s easy to find the big files and remove them (although a du -s can do this too, but it doesn’t have the pretty picture).
Sure, it’s an oldie, but I use GNU bash every day and have no idea how I’d get my work done without it.
Not only do I have a reliable stable of shell scripts (my favorite is xt, which fills my screen with appropriately-placed XTerms in the current directory–it’s great for programming !), but I use bash completion to make the tab and space keys more powerful.
When a full shell script is too much, Damian Conway’s realias trick has saved me an immense amount of time.
Simon Myers’ Power Shell Usage is full of wonderful bash tips and tricks. I’m a big fan of Ctrl-r and $!, for example.
There are dozens of bash features I’ve never even heard of and might one day use. The level of customization and flexibility from a really good command line makes it almost painful to use a GUI sometimes. Fortunately, I don’t even have to think about what I’m doing. Yes, bash is that good.
Thank you to all of the shell developers and documenters and contributors who keep the command line flexible, powerful, and useful.
Remember that the European Perl Hackathon will be held next weekend, from 2-4 March, 2007 in Arnhem, the Netherlands. Registration is open until Thursday, 1 March. For more information, please look at the hackathon website.
Allison Randal and Jonathan Worthington will be coordinating the Parrot/Perl 6 portion of the hackathon.
“> Errrr … I’m the one who needs the tutorial, not the one to write it.
“That makes you a prime person to capture the questions it needs to answer! You can’t evade the Responsibility Ponies that easily.”
– chromatic, responding to James E Keenan in ‘What Skills Do We Need to Finish Parrot?’
“: This mornings up date proposed
“Now the da rn spam fi1ters are chang.ng my spelling to look like sp*m. Yeah, that’s the 4icket… :)”
– Larry Wall, in ‘Enhancing array indices’
“Take the longest token, lie down and if the unease persists, write some code…”
– Brad Bowman, in ‘DFA/NFA context is non-local’
I use HTML Tidy in a well-tuned shell alias that cleans up HTML from articles and weblogs before I post them. We use a subset of XHTML on the O’Reilly Network, and this wonderful utility turns poor HTML (especially converted from word processor files) into valid XHTML. It’s simple to parse that with an XML parser to transform into something useful and clean.
I’ve even used it on hand-written HTML just to make sure things were correct. It’s a great utility I use almost without thinking. Thank you, developers of and contributors to HTML Tidy!
While I was just creating a blog post on this site, Firefox decided to die horribly. No worries, right? I’m running Ubuntu Edgy and Firefox 2.0.0.1. It has crash recovery. It even preserves the text which you’ve typed in text areas. I guess it would have, too, had it decided to come up. Instead, each tab which was opened prior to the crash just sat and spun until each of them appeared to totally freeze and I became too impatient to wait any longer.
I figured Firefox had to keep that session information somewhere, so I began rummaging through my Firefox session folder (~/.mozilla/firefox/{{random text string}} on Ubuntu). Two files which showed promise were sessionstore.bak and sessionstore.js. I opened the .bak file and looked for some text that I had been typing in and to my (almost) surprise, I found it.
I saved off the section that I was interested in to its own file. It had been reformatted slightly. All the spaces were %20. The urllib module in the Python standard library has an “unquote” function, so a little ` urllib.unquote(open(”/tmp/blogpost.txt”, “r”).read())` at an IPython prompt fixed my text right up.
The 0.10 release for a project called “pycallgraph” just popped up on the Cheeseshop. This library is supposed to allow you to create graphviz diagrams based on your application’s call tree. This sounded pretty interesting, so I decided to install it and see how well it worked.
I did an `easy_install` of the pycallgraph package and followed the usage directions on the pycallgraph main page. I have a simple little “main” module whose contents I’ve listed below. Basically, main calls mod1 which imports and calls mod2 which imports and calls mod3. Here is the code for `main.py`:
import mod1
import pycallgraph
pycallgraph.start_trace()
print mod1.mod1("Some Text String"), "\n"
pycallgraph.make_graph("callgraph.png")
The pycallgraph output was a little less than desirable:
I thought that pycallgraph was confusing itself and not filtering its own activities, so I took a stroll through the source code to see if there was a “filter” option. That’s when I noticed a `stop_trace()` function. When I changed my original script to look like this:
import mod1
import pycallgraph
pycallgraph.start_trace()
print mod1.mod1("Some Text String"), "\n"
pycallgraph.stop_trace() ##HERE'S THE stop_trace() CALL##
pycallgraph.make_graph("callgraph_clean.png")
here is the output that was created:
I can see this utility coming in handy.
Chad Fowler and Kevin Barnes recently considered the idea of The Big Rewrite:
I’ve most often seen the desire for The Big Rewrite where Technical Debt is so high that developers believe that throwing away the existing code and starting over is cheaper than fixing the problems in the code. It’s technical bankruptcy.
Of course, if (like most developers) you can’t actually read source code, your bankruptcy threshold will be awfully low–but that’s another rant.
I very much appreciate how both Chad and Kevin analyze the other costs and drawbacks.
“For the first time in the two months I’ve been working on Parrot, ‘make test’ completely succeeded — and with some TODO tests passing, to boot!”
– James Keenan, sharing the good news in ‘All tests passing!’
“…I think you’re the path toward sanity.”
– Larry Wall, in ‘Patterns’
I don’t know when it popped up, but as I’ve been working on a new Django project, I’ve been noticing the newforms documentation page. Today, I decided to check it out. I am quite impressed with what newforms do. The “old style” “automatic” form handling in Django revolved around database Model object and “manipulators” (which you could create from a Model). Form handling was closely tied to your database model. Indeed, form handling was closely tied to the Django ORM.
New style “automatic” form handling revolves around a Form object. Said Form object is apparently decoupled from the Django ORM. (I suspect this was a necessary step in order to integrate SQLAlchemy and any other non-Django ORM.) Here is an awesome little tutorial on using newforms.
Some of the benefits of using newforms are
If you can’t tell, I’m really worked up about newforms. Django just got a whole lot cooler. I’m sure I’ll write more on this later.
Remember, the next Parrot bug day is 13 January, 2007. Join the rest of the team at #parrot (irc.perl.org) to work on closing as many bugs as possible before the next Parrot release.
If you’re like me, you never want to lose a command. I’m constantly searching back through them to find out just what those command line flags were, what the esoteric command is (and where it’s located), and most of all: what in Tcl’s name did I do last month when I installed foobazzulator. First thing to know: control-r.
The Adobe JavaScript execution bug recently discovered is a huge security issue for any organization that serves PDF files via its web servers.
“Grrr, otta profefreed my onw righting occashionaly.”
– Larry Wall’s commit message for r13508–one typo correction of many
Installing free software on Windows can be unnecessarily difficult; that’s a significant barrier to attracting more developers for free software (especially those with Windows experience). I’m pleased to see that Alberto Ruiz has created an all-in-one PyGTK installer for Python and GTK development on Windows.
Plenty of projects need similar tools. Good work, Alberto!
“Short answer: absolutely.
“Long answer: emphatically not.”
– Larry Wall, in ‘RAII in Perl6/Parrot’
Have you wanted to start playing with Perl 6 but find yourself wondering what to write? I use Pugs, a Perl 6 implementation being written in Haskell and have been tremendously enjoying Perl 6. Like many, I’m impatient, but the work on Perl 6 has been progressing quite well and I’m quite keen to see the alpha. However, if you’re like me, you probably do better with a new language by actually writing something in it. Well, not only do I have something for you to write, you can actually help out the Perl 6 effort!
Recently I stumbled across 99 Problems in Lisp, which was in turn apparently borrowed from 99 Problems in Prolog. I’ve started 99 Problems in Perl 6.
“With the little sense of smell I have left, this smells like INTERCAL’s ‘COME FROM’ statement to me…”
– Larry Wall, who has a cold, in ’supertyping’
I recently needed to filter and process some Atom feeds. I know enough XML that I could process them with my own SAX filter, but this seemed like a better opportunity to use the XML::Atom module. Fortunately, it was very easy.
Disclaimer: I have programmed in both Ruby and Python, but not enough to be familiar with their conventions, so the following could be a serious misunderstanding on my part.
Any Python or Ruby programmers out there able to explain why these languages default to integer math?
$ python -c 'print 7/2' 3 $ ruby -e 'puts 7/2' 3
“Your faithful Dynamic Environmentalist”
– closing signature of Bob Rogers in ‘RFC: Proposal for dynamic binding’
Quite a few months have passed since my last post, as cryptography-related endeavors, and family matters, most importantly, have occupied about all of my time that’s available for occupying. As far as cryptovirology research is concerned, results are coming together, and what began as a single paper a couple of years ago, has branched off into separate avenues of design philosophy. One particular branch deals with the development of a cryptovirus that is efficiency-oriented, with an emphasis on speed and reasonable security trade-offs. The original cryptoviral information extortion attack, by Dr. Adam L. Young and Dr. Moti M. Yung, was built on public-key cryptography and an on-board random number generator, with the latter proving to be the bottleneck, responsible for consuming over half of the time the attack takes (i.e., roughly 11 seconds). A couple of years ago, I looked into stripping these two components and playing around with different security trade-offs. Roughly speaking, what I ended up with is a structure that uses a symmetric block cipher and block cipher-based MAC, to satisfy confidentiality and integrity requirements, while sending the keys in - yes, you’re reading correctly - plaintext.
The reasoning behind this is practical. The cryptovirus needs a carrier. All that is necessary is for the carrier to drop the cryptovirus onto the host, and for the cryptovirus to perform its operations. If the carrier is fast and discreet enough in getting the cryptovirus to the host, this should compensate for the plaintext keys that are riding shotgun with the cryptovirus. We’ve ditched asymmetric cryptography, and already regain over half of the attack’s operational time, by generating key material beforehand, so we’re already looking at an attack that takes less than half the time of the original. By using fast implementations of AES, we can achieve a completely standardized approach, complete with tight security bounds. As of now, I’ve chosen AES in CTR mode, which is IND-CPA secure, as well as CMAC-AES, which is a SUF-CMA MAC; if used in the Encrypt-then-Authenticate composition, this renders IND-CCA2 security and achieves INT-CTXT, for some constructions of this attack. The key to understanding all of these acronyms will grace the end of this post. The important thing to know here is that I’ve just described the basic setup for a cryptovirus that is just as stout as any good, defensive system that uses the same cryptography.
For those who aren’t familiar with the idea of a cryptoviral information extortion attack, here’s how it goes. We have an adversary and victim. The adversary desires some resource that the victim has; we’ll call this “H.” (I’m following the naming convention of the original attack by Dr. Adam L. Young and Dr. Moti M. Yung.) To get this resource, he’ll need some leverage, so he designs the cryptovirus to look for some critical data on the host victim; we’ll call this, “D.” When the cryptovirus finds D, it encrypts it, using AES in CTR mode, overwriting the original. When it finds H, it computes a MAC on it, using CMAC-AES. (This implies that the cryptovirus houses two symmetric keys: one for encryption and one for authentication.) At this point, both keys have been overwritten in RAM, and the cryptovirus instructs the victim to send H, along with the corresponding MAC, to the adversary. Upon receiving this from the victim, the adversary will compute a MAC on H using the authentication key (remember, the keys are precomputed, so he has a copy) and CMAC-AES. If the MAC he computes matches the MAC that accompanied H, then H is valid. Otherwise, the victim tried to send some bogus substitute, H’. Because the victim hasn’t any knowledge of the authentication key, he can’t compute a valid MAC on a bogus file to send in place of H.
Theoretically, if the victim cooperates, the adversary sends the encryption key used to encrypt D, such that the victim can decrypt it, thus restoring the critical data. The problem here is that the trust model is based on a relationship between a victim and adversary. Traditionally, we look at channels between Alice and Bob, where trust is implied. However, there is no balance of trust between a victim and adversary. I’ve addressed this issue, theoretically, with an arbitrated, game-theoretic protocol (i.e., a trusted third party, who treats the exchange, between the victim and adversary, in terms of a non-zero-sum or zero-sum game.) I’ll leave this for another rainy day, though. Now that you have the contextual gist of the attack, I’ll pose my questions.
For those of you in charge of policies for handling adversarial attacks, how would you go about handling a cryptoviral information extortion attack? The obvious defense is a good back-up, of course. However, it seems plausible that with enough a priori information of the system (i.e., insider attack), a cryptovirus could be tailored to attack prior to a scheduled back-up. For information that is archived in relatively short time intervals, perhaps the cryptovirus’ fast execution time can increase the viability of this tailored approach. After all, we’re looking at seconds.
Some might be quick to respond, “You should just ignore the adversary and never give him what he wants.” Failed attempts at cryptoviral information extortion attacks have been reported, where the “H” that the adversary demands is in the form of a monetary payment. If this demand will cost more than the data, being held for ransom, is worth, then sure, that’s a good argument for ignoring it. However, what about when this isn’t the case, and it’d be worth it to pay the ransom? Others might retort, “But what if the adversary doesn’t follow through with his end of the bargain?” Theoretically, it’s in the best interest of the adversary to play fairly. Keep in mind that attacks are often reported and publicized. If the media reports reflect the adversary as someone who never pays up, future victims are less likely to take the risk. However, if the reports reflect an adversary who always holds up his end of the bargain, future victims may give in, if statistics show them that there’s a promise of hope in recovering their data. Again, this is just one theory.
Oh, and here’s some alleviation from the acronym jungle you had to venture through, to get to this point:
If you want a friendlier treatment of introductory cryptoviral extortion, that even a kid might enjoy reading, I’ve written some things about it here. Pardon all the thinking-out-loud, but now that I’ve laid out the fundamental context, it’ll be much easier to get to the point with future research. Until then, have a good one and stay warm, if it’s winter where you are. It’s downright cold, here in the South, tonight. Almost 10 degrees. Here’s hoping that this research is as fruitful as my fireplace is toasty.
“…
On the sixth day of Christmas my true love sent to me,
Six versions of Perl,
FIVE LANGUAGES COMPILING!
Four bytes of bytecode,
Three POST nodes,
Two ASTs,
And a Partridge with a parse tree.”
– Jonathan Worthington, ‘Naming PAST-pm compiler tool chain’
Fritz Mehner’s amazingly useful perl-support Vim plugin supports Perl::Critic. (Okay, it’s done so for a year, but he just mailed me to make sure that I knew.) Fantastic!
See the perl-support Vim plugin screenshots or the perl-support Vim plugin help file for more information.
Thanks, Fritz!
USB drives are popular Christmas gifts. They are portable, relatively inexpensive, and extremely useful to store electronic files. My graduation gift from the Tufts Computer Science department in 2004: a 32 MB USB drive. My gift for a security presentation last year: a 128 MB USB drive. Now, you can get a 1 GB USB drive for roughly $20.
The prevalence of electronic data and portable devices, including USB drives and laptops, have spawned a huge data security problem. Devices are easily lost or stolen, and the data is unencrypted. I’m sure we all remember the story of the missing laptops at the Department of Veterans Affairs (VA) earlier this year with thousands of personal records. Or the news of US Army USB drives being sold on the streets of Afghanistan. It is hard to read the news these days without seeing a story on a data security breach somewhere.
Some USB drives will come with data encryption software while many will not. Do yourself or someone else a favor and encrypt the data on the device. TrueCrypt will accomplish this, and it is not difficult. Todd Ogasawara mentioned this last week on his list of open source software for Microsoft Windows. Here are the steps to create an encrypted data volume on your USB drive on Windows:
So now you have created an encrypted volume. But what about privacy on surfing the Internet? Look at the mess that occurred when AOL released the search data for over 500,000 users –yes, the searches can be traced back to the user. One piece of software that will allow you to surf the Internet anonymously is Torpark (again, open source). It is a portable version of the Firefox browser with Tor. Download Torpark and install onto your encrypted volume. Run the Torpark program (e.g. Z:\Torpark 1.5.0.7\Torpark.exe). The Tor connections will be established and a customized version of Firefox will load. Torpark is currently available only for Windows.
There are businesses that sell encrypted/privacy USB devices, or “computers on a stick.” Now, you can build one on your own. Remember, this is not a foolproof solution to the data security or privacy problems. For general computer users, this is a good start and good practice. One can still (attempt to) crack the password to your secured volume. But isn’t having the data encrypted using some universal method better than absolutely no encryption at all? You are still not completely anonymous on the web using Tor (e.g. you reveal your identity on forms). But it does protect the transport of data from one computer to another pretty well, and that is important.
I missed this trick a while back, but for everyone who has to program in Python and misses actual working closures, close over a container variable or use a function attribute. jjlinuxland’s Modifying a Counter in a Closure and the comments explain more.
Now to play with lambda to see if this trick makes them more useful….
“…of course [that] can’t be a bug as there are no specs ;)”
– Leopold Toetsch, in ‘[perl #40968] [BUG] :multi doesn’t seem to work right’
“Sadly, the hallucinogens are essential, not external.”
– Mark J. Reed in ‘List assignment question’
I recently had need (actually, more of a want thing) to mount a remote server from my laptop. The server in question has NFS running, so I figured it would be easy to mount my home directory on the server to some location on my laptop. It was simple. But when I mount it, all the files in my home directory were owned by some user which does not exist on my laptop. I thought I had tackled this problem in the past and found a way to map IDs from one system to another system when you NFS mount. Googling around turned up “idmap”, but I didn’t find clear documentation on how to configure it to do what I wanted, particularly when I only own the client side.
Then a friend recommended I look at sshfs. (Thanks, Michael!) Basically, if you have SSH access to a server, you can mount a directory on that server and access it locally. All I had to do on my Ubuntu laptop was to install the “sshfs” package. sshfs uses FUSE, so it installs all the FUSE dependencies it needs.
The command I use to mount the remote server looks like this:
sudo sshfs {{user id}}@{{server hostname}}:{{desired remote share}} {{desired local mount point}} -o idmap=user -o allow_other -o uid={{local user id}} -o gid={{local group id}}
This will not only mount the remote share, but will resolve any user id mismatches with the ones you specify with the uid and gid options. The performance is pretty good when both machines have a good network connection between them. But when I mount a directory on my server at home (meaning over a DSL connection), it lags noticeably. I think this is a great option for mounting a remote system when all you have is SSH access to it. It even works well when you don’t feel like fighting through user id/host mismatches. If anyone has a client-side only fix for the NFS user id mismatch, I’d love to read about it!
Rails fans are understandably proud of the magic metaprogramming facilities of Ruby, the database introspection capabilities of ActiveRecord, and the fact that the most basic model class is only two lines long (at least in every tutorial I’ve seen).
I say that’s two lines too many.
Here’s how to have zero-line model modules in Perl — as many as you want. (If you have a complete CRUD application, you can use the same idea to generate RESTful controllers, too.)
The three most common types of security vulnerabilities in 2005 were cross-site scripting (16.0%), SQL injection (12.9%) and buffer overflows (9.8%). So far in 2005, buffer overflows has lost the #3 place to PHP remote includes.
The good news is that Perl has long had capabilities in the language and its most common libraries that effectively shut down many of these attacks.
It’s not surprising that buffer overflows are on the way out. Perl programmers have long been able to not worry about buffer overflows. Dynamic strings mean no buffer overruns. Fortunately, all the new dynamic languages like Ruby, Python and PHP have dynamic strings as well, leaving only C and C++ programmers having to worry about the size of their malloc buffers.
Where Perl shines in web security is with its built-in “taint mode”. When taint mode is enabled, all data from an external source, such as from a web input form, is assumed to be untrusted and tainted. If a user types in her name, the resulting string is marked internally as tainted. Most of the time, this effect is invisible.
Perl will print out the the user’s name, because no matter what $name is, it doesn’t present a security risk. However, consider this common rookie programmer mistake.print "Hello, $name, glad to see you.\n";
$dbh = ... code to make a database connection ...;
$dbh->do( "insert into visitors (name) values ('$name')" );
That works fine for values of $name like “Bob Smith”, but consider a string like:
Your SQL expands out into'); drop table visitors;
insert into visitors (name) values (''); drop table visitors;')
That results in three statements, separated by semicolons: One inserts an empty value in the “visitors” table, the second deletes the “visitors” table, and the third a syntax error. The effect is that one well-crafted string from a miscreant means you’ve lost your data table. The possibilities are endless.
Of course, DBI handles the safe way of doing SQL calls, using placeholders:
The data is passed to DBI, but entirely separately from the command. The command is not created using tainted data, so is safe for DBI to execute.$sth = $dbh->prepare( "insert into visitors (name) values (?)" ); $sth->execute( $name );
SQL injection prevention is just the beginning of the value of taint mode to Perl programmers. Tainted data also can’t be used for executing system commands or reading source code, as in the PHP remote include exploits. For a more thorough discussion of how taint mode works, and why you want it on in every web program you write, see the perlsec documentation for Perl with perldoc perlsec, or online at http://perldoc.perl.org/perlsec.html
I hope that other dynamic languages continue to borrow Perl’s features and add explicit taint-mode checking to their bags of tricks. Modern web development demands it.
I wanted to get a Django site working under Python 2.5 the other day using Sqlite. I downloaded the source for Python 2.5 (even though 2.5 is in the Ubuntu repository), compiled, downloaded the Django 0.95 release, installed it, and tried to create a new blank database. Error. Here is the exact traceback:
Traceback (most recent call last):
File "manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
After digging for a few minutes, I realized that I didn’t have the Sqlite header files on my system when I compiled Python. I added the libsqlite-dev package on my laptop (which is running Ubuntu), did the make && make install dance, and tried it again. No error.
Had I taken the time to read all 1291 lines of the README in the Python 2.5 release, I would have read this around line 760:
Building the sqlite3 module
—————————To build the sqlite3 module, you’ll need the sqlite3 or libsqlite3
packages installed, including the header files. Many modern operating
systems distribute the headers in a separate package to the library -
often it will be the same name as the main package, but with a -dev or
-devel suffix.The version of pysqlite2 that’s including in Python needs sqlite3 3.0.8
or later. setup.py attempts to check that it can find a correct version.
Now all is well with the world.
“…problem 2 is probably just me being confused (though I’d love an explanation, from @leo ;-)).”
– Jonathan Worthington, in ’set_pmc_keyed_int delegates to set_pmc_keyed…?’
Alan Coopersmith reported that he has integrated DTrace probes into X.org. This is great news; it means that OpenSolaris (and soon, Mac OS X and FreeBSD) developers can profile the X server very easily. It will likely lead to performance improvements.
DTrace is one of the best technologies to come out of Sun in a long time, and it’s one of the top features of OpenSolaris. I heard that it only took a dozen or so probes to instrument MySQL effectively, too.
“That fuzziness is classic $Larry. Some of the rest of @Larry can be more *mumble*matic.”
– chromatic in ‘where constraints as roles’
I have a little project called Parrot::Embed. It’s a Perl 5
extension that makes Parrot
available to Perl 5 programs.
Parrot makes a shared library called libparrot. The actual
parrot executable is usually just a little program which uses
this shared library. This is very handy for my extension; I can use all of
the public functions in the shared library myself.
Actually building this code is trickier than it should be, however.
I stumbled across a message on the Ubuntu Devel mailing list this weekend which I found pretty disturbing. Here is the body of the message. This mail thread is commenting on a recent Slashdot article on the pain of upgrading from Dapper to Edgy. The thing that disturbed me was this comment:
Although it is very difficult to diagnose problems from blog and forum
posts (hence the analysis below is probably wrong, incomplete and
unhelpful) I think a large number of problems fall into the following
categories:* Using apt-get dist-upgrade rather than upgrade-manager
- Could this be reduced by emphaising on the release notes, on
ubuntu.com and in the support channels, the correct way to upgrade?
- Could apt be patched to give clearer warnings that dist upgrading
could break your system, and recommend that the user run upgrade
manager instead. In fact, just run update-manager when the user tries
this, whilst siulanousy taking their pony away from them
The replies in the rest of the thread never refuted that using upgrade-manager (I assume he’s meaning update-manager) is the “right” way to upgrade. However, at least one person replied back and stated what I was thinking, namely that it’s absurd to officially discourage (or prevent!) Ubuntu users from upgrading their systems by `apt-get dist-upgrade`. I’m still digging around to see if update-manager is the official means of upgrading a system.
If some of the pain that I experienced is because of doing a dist-upgrade rather than update-manager, then someone needs to do a better job of making the community aware of how they should be upgrading their systems. I didn’t run Automatix or Easyubuntu. I didn’t have Beryl/Compiz installed and definitely didn’t have the Beryl repositories in sources.list. And I didn’t have any binary video drivers installed which I had downloaded from the vendor. These are some of the other factors contributing to a painful upgrade, so it must just be my use of “apt-get dist-upgrade”. This was a pretty vanilla install I was upgrading from. I would have expected it to go smoother.
If you are upgrading from Dapper to Edgy, beware. I’ve read way too many tales of pain and woe from the above referenced Slashdot article to discount possible problems with this upgrade. You may want to try running update-manager with a “-c” flag (checks if a new distribution release is available). Actually, first backup your data. Then update. Or just install from scratch.
“IMHO, @Larry got overly precise in the above S02 quote: s[More precisely] = “Usually”"
– Jonathan Lang, in ‘where constraints as roles’ (was: ‘how typish are roles’)
Fredrik Lundh just posted an excellent overview of the “with” statement. If you haven’t been watching, the “with” statement is new in Python 2.5.
Frustration 1: When attempting to `apt-get dist-upgrade`, one package (mzscheme) refused to uninstall properly. The /var/lib/dpkg/info/mzscheme.prerm failed to stop the little scheme webserver (I guess it’s a documentation server or what-not), so I had to delete that prerm script and it uninstalled fine.
Frustration 2: xorg didn’t like “ati” as my video driver. X failed to start and upon examining the X logs, it seemed to not like that particular driver. When I changed “ati” to “fglrx” in my xorg.conf, X started right up.
Frustration 3: When attempting to install from the desktop CD, I couldn’t get past trying to partition my drive. Here is the hard drive layout I wanted to go with:
Here is how I wanted to lay out the partitions:
And you can see the “No root file system” error I was getting. This is the same laptop that I’m using right now. I have Windows XP installed to sda1, / on Ubuntu to sda2, swap to sda3, and /home to sda4. And I wanted to format sda2 and sda3. Why is this not working? I guess I’ll try to use the alternate install CD tomorrow. I’m sure someone will point out something I’m doing that’s just plain stupid, but I’m not seeing it at the moment.
Frustration 4: I attempted to install to my son’s laptop (an antiquated Compaq Presario 1200) this evening and it seemed like it would go well. Until it installed to 64% and just sat there. Twice. I guess I’ll be trying the alternate install CD on that laptop as well.
I’ll update more later.
“The | notation is mentioned in S012:1029, by the way. Obviously you still haven’t quite memorized all the synopses. :-)”
– Larry Wall, in ‘class interface of roles’
According to PHP Eats Rails for Breakfast, PHP is more dominant than Python or Ruby because a survey shows that there have been more lines of code written in PHP in the 3000 surveyed projects.
Of course, if a metric as broken as SLOC really suffices for advocacy these days, it will only take a handful of web apps written in assembly, COBOL, and AppleScript to leave PHP far, far behind as the new hotness.
“The whole point of reserving these namespaces is not to prevent users from misusing them, but to ensure that when we eventually get around to using a particular block name, and those same users start screaming about it, we can mournfully point to the passage in the original spec and silently shake our heads. ;-)”
– Damian Conway, on POD specifications
Reviewing software for security bugs is a highly recommended best practice. There are various techniques for doing source code reviews, one of them being “static code analysis” which (in most cases) involves the use of a ‘grepping’ (pattern matching) tool along with a database of patterns that indicate potential security flaws. There are disadvantages to static code analysis: high rate of false positives and the inability to detect logic errors that may lead to security bugs. That said, static code analysis tools can be used to perform a quick first pass on the source code to detect bugs that can be easily identified by a grepping technique (”low hanging fruit”). Some of the free static code analyzers (security) are: Flawfinder, RATS, and SWAAT.
Some of you might recall chromatic’s review of Class::Trait, a module which implements traits (PDF) in Perl. He gave a couple of good examples of how one can use traits and I figured a follow-up on one of my many “real world” uses of traits and how it has simplified my code.
“When I first read ‘Warnock applies’ on things in p6 summaries a year or so ago, I thought it was some really energetic programmer who went around and applied patches as soon as people posed a question.”
– Carl Mäsak, on Warnock’s Dilemma
While listening to Perlcast’s interview with Pragmatic Andy Hunt, Andy said “Bugs tend to clump together.” I’ve said that many times myself — it seems to be true — but I never asked why.
“And here I thought you were a responsible, law-abiding citizen… :P “
– Jonathan Lang, commenting on Larry Wall’s
self.HOW does Hash
Many programmers feel they have bragging rights if they’ve written large systems. This isn’t always fair as many times a quick twenty-something line program might save the day and programmers who can crank them out shouldn’t be undervalued. Be that as it may, sometimes we need to write large systems and we need to know how to do it. But what if you’re just writing a small system? What’s small? And as many of us know, small systems stick around and often grow. While rules which affect larger systems don’t always seem as important on small systems, it’s fair to say that if you want your small systems to be able to grow to large systems, it doesn’t hurt to start with sane rules.
If you really want to make something a habit, find a way to do it
without thinking about it. I like to automate the things I value so I never do
them incorrectly, incompletely, or infrequently. Thus Test::Perl::Critic
allows you to add customizable Perl::Critic tests to your test
suites, so you can ensure that you’ve followed local style.
Most technical writing I’ve read isn’t very good. One of the reasons the Head First series works is because applying a little cognitive science to the process of writing forces writers and editors to think from the audience’s point of view now and then.
The mini-rant How Not to Teach Database Design uses an existing article to show mistakes common to much technical writing. Even if you don’t know or care about databases, read and think about the post if you might someday write documentation, an article, a tutorial, or even a set of instructions in your weblog or a wiki or an e-mail somewhere.
Don’t worry; they’re not grammatical rules you’ll never remember. They’re just five tips that will help you craft better prose.
“my Yellow sub marine { @we.all.live }”
– Larry Wall, concluding a series of Beatles references in the grep thread.
Everything that I have built in Django so far has been pretty simple. Perhaps “limited scope” is a better choice of words. Each project I create tends to only contain one user-created app. There are a couple of sites that I’d like to build which could share some code between them. So, I decided to look into using multiple user-created apps and seeing how things fit together. My primary interests in this exercise were the url configuration and template inheritence among applications.
I created a project named “mproject” and, for the sake of simplicity, three apps named “main_app”, “foo_app”, and “bar_app” under the “mproject” directory. In addition to the autogenerated files, I created a “urls.py” for “foo_app” and “bar_app” and one html template for each of the three apps (named “bar_main.html”, “foo_main.html”, and “base.html”). Here is a tree view of the directory structure:
.
`-- mproject
|-- __init__.py
|-- bar_app
| |-- __init__.py
| |-- models.py
| |-- templates
| | `-- bar_main.html
| |-- urls.py
| `-- views.py
|-- foo_app
| |-- __init__.py
| |-- models.py
| |-- templates
| | `-- foo_main.html
| |-- urls.py
| `-- views.py
|-- main_app
| |-- __init__.py
| |-- models.py
| |-- templates
| | `-- base.html
| `-- views.py
|-- manage.py
|-- settings.py
`-- urls.py
I added the following three lines to the INSTALLED_APPS section of my settings.py file:
'mproject.main_app',
'mproject.foo_app',
'mproject.bar_app',
This is how applications are “plugged in” to your Django project.
I added the following to the project-level urls.py:
(r'^foo/', include('mproject.foo_app.urls')),
(r'^bar/', include('mproject.bar_app.urls')),
This says to use everything in the foo_app.urls url configuration, but map it by prepending “foo/” to all those entries. Likewise for bar_app.urls and “bar/”.
main_app/views.py is empty. Since this is a simple example, the main application isn’t doing anything.
main_app/templates/base.html:
<html>
<head>
<title>{% block title %}Main Base Title{% endblock %}</title>
</head>
<body>
{% block content %}
Unset Content
{% endblock %}
</body>
</html>
bar_app/urls.py:
urlpatterns = patterns('',
# Example:
# (r'^mproject/', include('mproject.apps.foo.urls.foo')),
# Uncomment this for admin:
# (r'^admin/', include('django.contrib.admin.urls')),
(r'^main/', 'mproject.bar_app.views.main'),
)
I only created one mapping for this app. “main/” will map to the function mproject.bar_app.views.main. But, since the main url config is including this url config, “/bar/main/” will map to the mproject.bar_app.views.main function.
bar_app/views.py:
from django.shortcuts import render_to_response
def main(request):
return render_to_response('bar_main.html', {})
This simply links the “main” function with the “bar_main.html” template.
bar_app/templates/bar_main.html:
{% extends "base.html" %}
{% block title %}Bar Title{% endblock %}
{% block content %}Bar Content{% endblock %}
Here, I’m extending base.html, which Django will pick up from the main_app application. I’m overriding the title and content blocks so that this template gets to fill in its own content.
Here are the details on foo_app, but without explanation since I’m doing the same thing here as in bar_app.
foo_app/urls.py:
urlpatterns = patterns('',
# Example:
# (r'^mproject/', include('mproject.apps.foo.urls.foo')),
# Uncomment this for admin:
# (r'^admin/', include('django.contrib.admin.urls')),
(r'^main/', 'mproject.foo_app.views.main'),
)
foo_app/views.py:
from django.shortcuts import render_to_response
def main(request):
return render_to_response('foo_main.html', {})
foo_app/templates/foo_main.html:
{% extends "base.html" %}
{% block title %}Foo Title{% endblock %}
{% block content %}Foo Content{% endblock %}
A couple of really practical uses for this come to mind. First, this is an easy way of separating logical pieces of your application. This approach makes it really easy to separate your site into its logical pieces. For example, you could have a blog, main static content, user reviews, photo gallery, etc. each in their own applications. This should lead to a less cluttered views.py file (even though you should be able to do something similar by separating things out and creating a “views” directory and putting logically separated code there).
Second, this makes for really nice sharing of applications among projects. Why not create your own customized user registration app and use it in each of your projects? Or a blog? Or photo gallery? (Or what have you.) All you have to do to re-use code is add an entry to the INSTALLED_APPS section of your settings.py file and create an include in your urls.py file.
I remember hearing or reading someone say that the pluggability of Django apps is one of its under-touted features. I couldn’t agree more. Something that I had previously avoided because I didn’t need it is turning out to be an incredibly powerful feature.
I’ve uploaded the project and app files here for anyone interested.
This summary has been posted early because I will be going to OSCON Europe on Sunday. Next week’s summary will include any threads which were not included this week.
I love to program in Perl. It is so effective for system administration tasks, but we all know that. Several weeks ago, I resurrected a project I did when I was a graduate student. Back in 2003, a course called “Perl Programming Practicum” was offered at Tufts by Professor Alva Couch, and the first assignment was to write a program to report system resource usage by user. It is similar to the ps command. Snapshots are sorted by greatest CPU-time use, and then by greatest memory use. You can use this tool to yell at the biggest CPU hogger. I made a number of changes to it, including logging the information on processes and its dependencies to a database (yes, on large systems, the database might get really big really fast). That way, I can create an interface to run system usage reports on the web.
I recently released the source and my changes to the new Google Code Project Hosting site. The program is called syshogs (originally named “pigs”). I wanted to see the differences between hosting a project though Google and SourceForge. The obvious difference I noticed with Google was no review process to host my project. Unlike SourceForge, you have to wait at least 24 hours before your project becomes official. You cannot register a project name in Google that is already registered in SourceForge. The one thing that I am still adjusting to is Subversion. I have experienced some kinks with checking-out and committing with https:// (remember if you are a developer, you better use https:// for both). With regards to the project page, it is very lite in Google: project home, issues, source, and administer –no frills. Project pages in SourceForge can be very heavy: lengthy message board and project details, and administration functions are hard to find. Finally, I can now see the source code directly in the project’s Subversion repository. All in all, Google’s Code Project Hosting is good, especially for hosting small projects.
I’ve seen references to ShowMeDo in the past, but just haven’t taken the time to check them out. I have been poking around there for a few days and really like what I see. From the main page of the site:
This site is dedicated to showing demonstrative videos produced by our community. For many subjects seeing something done is the most effective way to learn, whether it’s peering over an expert’s shoulder while they explain how to program or watching while someone grinds beans and makes coffee. Please watch, learn and perhaps share your knowledge back.
They have a smallish (but growing) number of tutorials available from debugging Python using the IPython shell to making the perfect cup of coffee. One of the really interesting features of the site is voting on a desired tutorial topic. Currently, there are requests for tutorials on TurboGears, Django, and IPython among other topics. Currently, the top voted request is for
I’d like to see Django worked-example videos for building e.g. a ‘newspaper’, wine-store, blog and other practical applications.
The tone of the tutorials that I’ve watched and the overall feel of the website is very community friendly. I think this site has tremendous potential for helping spread information to knowledge seekers on a potentially limitless range of topics. This could be a very interesting site to keep an eye on. Check it out and vote for a topic you’re interested in.
James Shore is my friend and co-author of The Art of Agile Development (in progress). In between writing furiously, becoming a father any day now, and running Cardmeeting, he still has time to discover new insights and write them for the world.
Technical Debt refers to all of the design decisions and shortcuts you make that may eventually cause you to do more work to resolve them later. Managing technical debt is one of the most basic (and most challenging) parts of any project.
Jim’s new essay on Voluntary Technical Debt demonstrates Technical Debt on a real project, as well as an advanced technique: deliberately incurring technical debt for short-term gains. Please note Jim’s estimate that he and Dave will spend more time reducing their debt than if they’d never incurred it. (Yet also note that Jim, Dave, and I all voluntarily took out mortgages to buy our houses. Sometimes it’s worth it.)
Jim Thomason has an excellent writeup at Perl Monks called The History of a Templating Engine. It’s almost a rite of passage for Perl programmers to write a simple templating system, and novice programmers somehow universally fail to understand why writing yet another templating system is an exercise in pain, frustration, and usually unmaintainable, unremarkable code. Jim’s account of the evolution of his system and the lessons he learned is, hopefully, a great bucket of cold water to cool those youthful urges.
Damien Seguy from Nexen just sent some interesting statistics on PHP Versions in Use in August 2006. I’ve ranted before about the use of obsolete, unsupported software way past its expiration date, but it’s nice to see PHP 5 adoption continuing to grow (nearly 10% of all servers running PHP, with PHP 6 on the way) and PHP 4.4 poised to overtake all other versions as the most used.
Perhaps the most useful piece of information for PHP developers is that targeting the feature set of PHP 4.3.9 will produce software that runs on three out of four servers surveyed. If backwards compatibility is important to you, that’s good to know.
“I once saw an overfilled waterbed that was almost as tall as I am. I would have called it PHP, but it didn’t explode and throw cold water all over the house.”
– chromatic
A significant new part of Perl 6’s object model and type system is the addition of roles. Part of their origin is an implementation in Smalltalk (there called traits). They also solve some systemic problems of other OO systems.
Why are they useful and how do they work?
“My school’s punch card machines were in the same room as the TRS-80 Model I (”THE COMPUTER ROOM”). These kids today with their hula hoops and fax machines and intarwebs…”
– Chip Salzenberg, arguing in favor of lines in excess of 80-characters.
I updated my MBTA subway map created using the Google Maps API: http://www.cs.tufts.edu/~mchow/mbta_gmap. In addition to now using version 2 of the API, the most significant update is the location-based search. Thanks to the Google Maps API geocoder, you can now type in a street address, and the closest subway station will be returned with approximate distance (a black polyline from your address to the station will also be displayed). For example, try ‘161 College Avenue, Medford, MA’ –the Computer Science department at Tufts.
Now if you are really brave, you can also get a list of the 5 closest subway stations based on your IP address. In many cases, it is inaccurate. I am using hostip.info as my IP geocoding source. If you know of a more accurate source (preferably free), please let me know.
When I’m working on a Django project, I typically open all of my python files (settings.py, urls.py, views.py and models.py) in a single vim instance. All of my projects to date have been simple and only have a single app. When I start working, I fire up a terminal session and type:
vim urls.py settings.py {{app}}/views.py {{app}}/models.py
Being the lazy type, I just created a zsh function to do that for me:
djvim() {
vim settings.py urls.py ${1}/views.py ${1}/models.py ${1}/templates/*html
}
Now, all I have to type is “djvim {{app}}”. This will also grab all of my htmlish files, which I usually have on a separate virtual desktop altogether (obviously in a separate vim instance), but I’m going to try this approach and see if I like having everything all together. But the main point is less typing to start working on a Django app, which is a good thing.
Over on SUN’s ZFS web site, you’ll find an interesting little article about ZFS snapshots…. (notice the screenshots) ;-)
http://blogs.sun.com/roller/page/timf/20060523
I think you’ll find it similiar enough to confirm that OS X Leopard will have ZFS as it’s main file system. All I can say is “Awesome” Read more about ZFS to see why I’m so excited about OS X having the most advanced file system ever.
Since I took so long to write this several people beat me to the punch on posting. (I started it on 8-8-2006 and got sidetracked with some work things and didn’t post it) The following is my favorite although I don’t agree with the consclusions he comes too.:
http://arstechnica.com/staff/fatbits.ars/2006/8/15/4995
And here are some more…
More about ZFS available at these links:
http://www.sun.com/2004-0914/feature/
http://www.opensolaris.org/os/community/zfs/whatis/
http://blogs.sun.com/roller/page/ahrens?catname=%2FZFS
Screencasts of ZFS in action available from this page:
http://www.opensolaris.org/os/community/zfs/
These days it seems like everybody and their dog wants to create something like Ruby on Rails. While there are a number of competitors out there, the only one I have any experience with is Catalyst, written in Perl. I had a commit bit on that project once and it looks really promising, but it’s “some assembly required” (or at least, it was last time I checked). It has an overwhelming number of options and tries very hard to not tie your hands at all. It has a great dispatching action, but in the end, if you’re new to Catalyst, or worse, new to Perl, the number of different options and how to hook them together can seem daunting. Once you get them working, though, Catalyst is pretty fun.
For various reasons, I’ve moved on from that project, but the idea of “just get it done” is very appealing to me. Since I’m primarily a Perl programmer nowadays, when I saw that Jifty had been released, I was pretty excited. I know Jifty’s primary architect, Jesse Vincent, and have a lot of respect for him, so that helped me be excited even though Jifty is yet another web application framework. (It’s also worth noting that Jifty is put out by the makers of RT, a very popular request tracker, so these folks know a good thing or two about progamming.) However, Jesse might not like me mentioning Rails and Jifty in the same blog post since Rails isn’t even mentioned in the Jifty docs. To be fair, Jifty is not a port of Rails and doesn’t pretend to be. It’s just your basic AJAXified full-stack continuation-based Web Application Platform. Hmm, how can I make this not boring? I know, I’ll write this blog entry while I work through the tutorial. Something has to break, I’m going to do something stupid, or some other problem will arise.
I’ve installed and played with xgl and compiz in the past, but decided to not keep using them for various reasons. I recently installed and set up xgl/compiz again, and will probably keep them as my primary X environment. I configured xgl/compiz so I can select them from my list of sessions in gdm. Here are my experiences so far.
Setup was relatively painless. I installed the xserver-xgl and compiz packages from the Ubuntu repository. I then created a file /usr/bin/startxgl.sh with the following contents:
#!/bin/bash
Xgl :1 -fullscreen -ac -accel xv -accel glx:pbuffer &
DISPLAY=:1
gnome-window-decorator &
compiz gconf decoration wobbly fade minimize cube rotate zoom scale move resize place menu switcher &
xmodmap /usr/share/xmodmap/xmodmap.us &
exec gnome-session
And created a file /usr/share/xsessions/xgl.desktop with the following contents:
[Desktop Entry]
Encoding=UTF-8
Name=Xgl
Comment=Start an Xgl Session
Exec=/usr/bin/startxgl.sh
Icon=
Type=Application
I didn’t need any modification to my xorg.conf file. The last time I tried xgl/compiz, I recall needing something like this my xorg.conf:
Section "Extensions"
Option "Composite" "Enable"
EndSection
When I tried putting those lines in this time, it was unusably slow.
Performance initially seemed abysmal. Rotating the cube of desktops around, switching between apps, maximizing apps, and doing the expose-like “scaling” were all miserably slow. I noticed that I had “direct rendering: No” when I ran glxinfo. I assumed that this was the cause of the poor performance. Since xgl is designed to make use of OpenGL, I figured that direct rendering could only help. I then looked around the web and found indications that xgl doesn’t use direct rendering. One of my co-workers (thanks, Tony!) recommended that I muck around with the speed settings in gconf-editor. I tried it and performance is mostly tolerable now. Hopefully, this is something that will be improved as the project presses forward.
Video playing performance is still not on-par with my non-xgl X server. I can’t play a 960×544 xvid encoded avi full screen (1920×1200) without it getting really choppy. I’ve tried vlc, xine, totem, and mplayer. All of them play it fine at its native resolution, but not at full screen.
One of the drawbacks to using compiz is that I lost some of the metacity goodness I’ve gotten used to. For example, many of my keyboard shortcuts just don’t work. Some of the standard ones do, like Alt-F4 to close a window. I have Ctrl-Alt-b and Ctrl-Alt-e mapped to launch firefox and sylpheed respectively, and those still work. However, Alt-r to launch the “run application” dialog and Alt-t to launch a terminal don’t work. But, keyboard shortcuts for some things are configurable in gconf-editor, so I can see that Alt-F2 is the compiz-defined shortcut for launching “run application”. So, for tasks that have a keybinding in compiz, I can either adapt to their “new” keybindings, or I can reconfigure them.
“scale” seems a little buggy. As I mentioned earlier, “scale” is xgl/compiz’s expose-like (from Mac) functionality. When I scale back all applications on all desktops and then click on one to bring up, I often do not receive the desired result. When it doesn’t do what I think it’s supposed to do, it just brings back up the application that was already in front. Sometimes “scale” is just touchy and you have to keep your cursor over the desired application for a few seconds to make sure it is highlighted, and then it will work. Other times, even if you pause, it still doesn’t work.
Anyway, I think this is making great progress for *NIX desktops. Hopefully all of this will be hammered out by the time Ubuntu Edgy ships.
In a recent post on the Perl QA mailing list, David Golden showed off two lovely Vim bindings to load Vim’s configuration file in a buffer for editing, then to reload that file in the current Vim process.
",v brings up my .vimrc
",V reloads it -- making all changes active (have to save first)
map ,v :sp $VIMRC<CR><C-W>_
map <silent> ,V :source $VIMRC<CR>:filetype detect<CR>:exe ":echo 'vimrc reloaded'"<CR>I had to modify this to replace $VIMRC with the path to my configuration file (~/.vimrc). It worked beautifully.
It’s much easier to experiment with writing new bindings in Vim itself, rather than through the ex interface. It’s also immeasurably faster to be able to reload the configuration file itself, rather than quitting and restarting the program.
This tip has already made me much more productive. Thanks, David!
There is a misconception floating around that Django isn’t good at doing Ajax, or that other frameworks do it better. It’s true that DJango doesn’t bundle an Ajaxy library, but it doesn’t mean you can’t easily integrate one into your application. Istvan Albert posted an excellent example of using the MochiKit and Prototype javascript libraries to build an Ajaxy example application using Django. This is an example to show the code for using both libraries; it isn’t a live demo. It’s really helpful to get the source code and see what is going on under the covers. It’s really slick.
I’ve been spending the last few months driving the technical side of preparing Socialtext Open, the open source version of Socialtext’s core wiki product. I’ve been working with other Socialtext developers to get a huge codebase that only needed to be tended by internal developers into a form that can be easily downloaded and installed by anyone. It’s been a daunting task.
As we came down to the wire, preparing for today’s big release, I read Nat’s Radar article “Opening the source” about how much more than mere licensing will change when open sourcing a project. It prompted me to write a list that I posted to my internal development blog.
Socialtext Open is going to change everything about how we write code. Even if you’re not one of those directly responsible for getting Socialtext Open out the door on Monday, after that day, we’ll see a huge shift in how things happen. It won’t happen overnight, but it’ll happen.
Off the top of my head:
- We’re going to have a larger dev team.
- We’re going to get a tenfold increase in potential distractions.
- What once were nice-to-have features may become mandatory.
- Socialtext Open releases will become big events.
- Little arcane bits of knowledge stored in our collective brains, like the maintenance of /etc/aliases.deliver, will need to get documented in a central place.
- We’re going to have two bug tracking systems to maintain: our existing RT system, and one on SourceForge (not RT) for the open source community.
- Our visibility will increase tenfold.
- Each of us will have to know more about our products than we did before, because we will all become the face of Socialtext.
- Design decisions will have to be documented for the world to see.
- FAQs will be VERY FAQs.
- It won’t just be the business folks checking us out, but now also the code folks.
- Things that we’re used to, that have grown organically over time and that we’ve lived with for a while, may be held against us when held up to new eyes. (”Why do you guys use both TT and Mason? That’s stupid!”)
I know there are plenty of others. If you’ve gone from a closed to open source product, I’d love to hear your additions to the list.
Plagger is a great tool for doing interesting things with RSS feeds. It’s written by Japanese Perl Monger Tatsuhiko Miyagawa (with help from a large supporting cast). If you want to get a feel for what it does then the slides from the talk that Miyagawa gave at YAPC::NA are a good start. Over the next couple of months he’ll be giving similar talks at YAPC::Europe and OSCON.
I haven’t yet got fully to grips with all of the cool things that you can do with Plagger, but what I have been doing is to use it to build blog aggregation web sites (which people have started calling “planets”). I reckon that it currently takes me about five minutes to set up a new planet - and most of that is finding the list of feeds to aggregate. Recently I’ve set up planets for my local area, UK MPs’ blogs and a “personal planet” that contains links to various web feeds that I create. Yes, I realise that they all look the same, but I don’t really expect people to visit the web sites as they all produce both RSS and Atom feeds that you can add to your favourite feed reader.
All in all, Plagger looks like a very useful tool if you’re doing any kind of work with RSS feeds. And I think I’ve barely scratched the surface of its power.
I’ve been posting on my progress around the podcast grabber I’ve been building. You can find a couple of my back blogs here and here, and some preliminary source code here.
My latest work in the podgrabber saga has been to convert the GUI from using Glade and PyGTK to hand-coded PyGTK. See the blog posts above for more details. I have now converted over everything except for a simple configuration dialog. The old GUI view Python module (using Glade) weighed in at 419 lines of code. The new GUI view Python module (using hand-coded PyGTK) weighs in at 477 lines of code. This is really interesting to me because the Glade way didn’t have any definitions of GUI code in it; all of that was stored in a .glade file. The new way has both the GUI definitions as well as the event handling code. I’m surprised that it’s only a 58 line difference in size (or about 14% increase).
After making the switch to hand-coded PyGTK, I’m starting to wonder what the benefit was to using Glade. Maybe it provided me, an inexperienced GUI person, with some sort of a security blanket as I tromped through unfamiliar territory. I don’t know. This is totally subjective and maybe a fallible comparison, but my code feels cleaner in the re-vamped raw PyGTK module than the module using Glade. It could be because I’m copying and pasting a lot of code from the old module and am refactoring it as I go. Or it could be because I feel more in control of the whole process, so I’m asserting myself to structure things better. Anyway, I thought this was an interesting comparison.
I’ve been reworking my podcast grabber. When I started putting a GUI frontend on it, I decided to use Glade because I lack any skill whatsoever in GUI development. (Glade is a GUI builder application which stores the structure of an application in XML. It is also a library which interprets the XML file and allows your application code to interact with the GUI you built.) After using Glade to build a usable interface with most of the functionality I wanted, I felt like I lacked the desired level of control over my GUI. It has also been a little bit of a hassle to layer objects in deeper containers than what I would have expected. For example, I have a couple of TreeViews side by side and I wanted them to be embedded in an HPaned (horizontal pane) so I could easily resize them. In order to put my TreeViews inside of an HPaned, I had to add another row to my main VBox, add an HPaned to it, cut and paste each of my TreeViews into it, and then delete the now-empty VBox row. This wasn’t a deal-breaker for me and there is probably an easier way to do it, but it was a little bit of a nuissance.
I have been reading docs and playing with building a GUI app by hand using PyGTK. In my reading, I came across the concepts of Actions, ActionGroups, and UIManagers in PyGTK (actually, they’re gtk constructs which have PyGTK wrappers). You can find some documentation around Actions and ActionGroups in this section of the PyGTK tutorial. Basically, you create an Action for different things a user might want to do such as quit, open a file, edit configuration settings. These actions may have different ways the user can accomplish them such as a button, a menu item, or a keyboard accelerator. You attach an Action to a “proxy object” such as a button or menu item.
The UIManager provides a way to create menus and toolbars using an XML description. Some documentation for the UIManager can be found here.
Actions seem a much more code-maintenance-friendly way of associating actions with objects the user is going to interact with as opposed to the alternative: create an object (such as a button), specify all the attributes of it such as label, tooltips, etc, and (with Glade) create event handlers by name and associate the event name with some function. I guess the alternative isn’t so bad, but it’s nice to see a library provide facilities to make code management a little nicer.
There appears to be a steeper learning curve to building a GUI by hand than using something like Glade, but I’m hoping that it’ll pay off in a tighter level of control as well as ease of maintenance. I’ll post back as I make headway with coding PyGTK by hand.
I’ve been having a problem getting audio while playing Flash videos on my Ubuntu Dapper powered laptop. Many times, if I want to ask a ninja or find out how emo Hope is, I see moving video, but I don’t hear anything. The (pain in the rear) workaround has been to kill Firefox, “killall esd“ from a terminal window, restart Firefox, and try to watch the video again. While this has worked, it’s been a huge pain. I just decided to google for “flash esd ubuntu” and stumbled across a Ubuntu launchpad page. Something that I just tried which seems to have fixed my problem is the following suggestion from Daniel Carrera:
sudo ln -s /usr/lib/libesd.so.0 /usr/lib/libesd.so.1
sudo mkdir -p /tmp/.esd/
sudo touch /tmp/.esd/socket
And I am happily asking a ninja and seeing how emo Hope is. Now if Hope would just release a new podcast…..
use.perl.org reader amoore’s first journal entry, How to Sneak Testing into Your Development Team is an excellent personal retrospective of improving the quality and test coverage of an existing, large, untested application. I particularly like his (her?) second and third suggestions — especially racheting up quality.
Not every language or environment has the same tools as Perl (particularly the cannot-say-enough-good-things-about-it Perl::Critic), but these suggestions can work with minor modifications in many environments.
David Golden has just announced the release of Vanilla Perl build 4. This is an important project, as the completed version will provide Windows users with a standard Perl as well as a freely redistributable compiler, making it possible to install CPAN modules with the same ease as on a standard Unix system.
(There’s much more to post from YAPC::NA 2006; I’ll get to it through the week.)
I’ve been working on wrapping PyGTK around a podcast grabbing Python script that I wrote. You can take a glance at the code that actually does the work here. The posted code, particularly the configuration management, is really tightly coupled to all other parts of the app. Also, the download manager expects much more information than it should in an odd data type comprised of a dictionary and an elementtree object. I’ve removed some of those warts in the version I’m currently working on.
Anyway, I’ve been working on wrapping PyGTK around the above-mentioned script and have finally gotten something that sort of works. One of the minor annoyances, though, is that when I click on a button that tells the app to show me which podcasts are available for download, the app freezes up until it has figured out every last available download. Even though I have code to update the status bar in the method which handles this button press, the only way I can figure out if anything is going on is to look at the terminal window I started the podgrabber app from.
Since I have the app somewhat functional, I decided to fix this blocking annoyance. The reason that this GUI app is behaving like this is that when the application is launched, it is being controlled by a single thread of execution. When you click on any button, that single thread of execution has to run code to perform any actions associated with that button click event. Subsequently, all other activity handling is blocked until the main thread returns from handling the current request.
So, a fix is to allow the main thread of execution to spawn a new thread to handle the button click event. Here is the code that currently executes when the “show downloads” button is clicked:
def OnShowDownloads(self, widget):
status_bar = self.wTree.get_widget("mainStatusbar")
self.downloadList.clear()
for feed_url in self.controller.get_available_feeds():
feed_name = self.config.Feeds[feed_url]["name"]
mode = self.config.Feeds[feed_url]["mode"]
status_bar.push(1, feed_name)
for download in self.controller.get_download_list(feed_url):
dl_url = download["enclosure"].attrib.get("url", "NONE")
try:
length = int(download["enclosure"].attrib.get("length", "0"))
except ValueError:
length = 0
file_type = download["enclosure"].attrib.get("type", "UNKNOWN")
title = download.get("title", "No Title")
self.downloadList.append([feed_name, dl_url, length, file_type, mode, title])
status_bar.push(1, "%s - Done" % feed_name)
Adding threading support here was pretty easy. I simply had to add “gtk.gdk.threads_init()” at the beginning of the program, create a threading decorator, apply the threading decorator to the event handler, and strategically sprinkle “gtk.threads_enter()” and “gtk.threads_leave()” throughout the affected code. Here is the decorator that I created:
def threaded(f):
def wrapper(*args):
t = threading.Thread(target=f, args=args)
t.start()
return wrapper
And here is the slightly modified handler method:
@threaded
def OnShowDownloads(self, widget):
gtk.threads_enter()
status_bar = self.wTree.get_widget("mainStatusbar")
self.downloadList.clear()
gtk.threads_leave()
for feed_url in self.controller.get_available_feeds():
feed_name = self.config.Feeds[feed_url]["name"]
mode = self.config.Feeds[feed_url]["mode"]
gtk.threads_enter()
status_bar.push(1, feed_name)
gtk.threads_leave()
for download in self.controller.get_download_list(feed_url):
dl_url = download["enclosure"].attrib.get("url", "NONE")
try:
length = int(download["enclosure"].attrib.get("length", "0"))
except ValueError:
length = 0
file_type = download["enclosure"].attrib.get("type", "UNKNOWN")
title = download.get("title", "No Title")
gtk.threads_enter()
self.downloadList.append([feed_name, dl_url, length, file_type, mode, title])
gtk.threads_leave()
gtk.threads_enter()
status_bar.push(1, "%s - Done" % feed_name)
gtk.threads_leave()
I’ve been thumbing through the Perl Hacks book lately. Some of the “Productivity Hacks” are (nearly) applicable to Python as well as Perl hackers. For example, most of “Hack #5: Autocomplet Perl Identifiers in Vim” applies directly to Python vim users. I’ve been using the CONTROL-N thing for years now. Basically, if you have a “word” in a file that you are vimming, you type the first couple of letters of it, hit CONTROL-N, vim will match and complete that already-entered word. But, as chromatic pointed out, it won’t match (in Perl) “words” that look like “Foo::Bar”. Nor will it match (in Python) “words” that look like “foo.bar”. The solution to this for Perl is to add “set iskeyword+=:” to your .vimrc. The solution in Python is to add “set iskeyword+=.” to your .vimrc. Actually, I added this to my “.vim/ftplugin/python.vim” so that this would only get loaded when I’m editing a Python file. Now, I have greater joy in my vimming experience.
Anyway, this was a great hack that was easily modified to work for Python. There are other useful hacks that are very applicable to Python folk. I don’t know how many of these I want to give away for free, though. I might make you go buy the book :-)
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainability.
Here’s what I learned about programming, refactoring, and 30-day projects during this series.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is a race against the clock to get just one single test running against the live, not a mocked, database. Can chromatic do it? His reputation is on the line!
Perhaps my keenest frustration with Perl 5 is its lack of a clean metaprogramming facility. Stevan Little’s Class::MOP has filled in much of the gap. Here’s what I learned after playing with it for an afternoon.
I’ve been trying for the last couple days to use MySQL’s full-text indexing with accent-insensitivity with no success.
From what I know, when you’re using accent-insensitive strings it doesn’t matter what is the case of the accented characters. For example, café and CAFÉ (coffee, in portuguese) are considered the same.
Well, it just doesn’t work that way. I’m using the utf8 charset and the utf8_unicode_ci collation. I created a fulltext index on two columns and it will return different results when using different cases of the same accented character.
Does anybody have any clues? I’m about to create two lower-case columns specifically for the purpose of indexing. Although I really don’t like this solution, I think it’s the only way to make it work as supposed.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to get just one single test running against the live, not a mocked, database.
Greg Law of Undo Software bent my ear the other day about his company’s new release of UndoDB, a piece of software that adds reversible debugging to GDB.
That is, when running your program under UndoDB, you can set a breakpoint at the point of failure and rewind and replay the program’s state repeatedly. For trivial bugs this is perhaps nice, but for difficult bugs, whether arduous to reproduce or impossible to find, being able to move back and forth between a known good state and a known bad state can be invaluable to fixing the bug.
Like the also impressive Coverity static analyzer, UndoDB is not open source, but it is available freely for non-commercial use. Hopefully we will see improvements to the underlying tools as well as improvements in existing software from better development tools.
I seem to have spent a large part of the last year ranting about people who don’t use database metadata properly. Usually I’m talking about things like primary keys, foreign keys and unique indexes, but recently I’ve come across a far more basic level of idiocy - using the wrong data types.
Currently part of my work involves taking data dumps from one database and loading it into another database (and then generating some reports from the data). Some of the fields involved contain dates. So being a sensible programmer I’m storing them in DATETIME columns in my database. It’s simply a case of working out what format the other database uses as its default export format for datetime values and converting that to my database’s default import format.
I’m creating yet another website in Django, mostly for fun. One of the things I wanted this website to have was an RSS feed. I glanced at the documentation for generating a feed from Django and was initally put off because it looked a little cumbersome. I then began investigating my other options, like generating the feed manually, and decided to give the Django syndication doc a fuller read. I’m very glad that I did. The reason that Django’s RSS generation looked cumbersome at first glance is because it is featureful, flexible, and has every option you will probably need.
In order to create an RSS feed, all I needed to do was create a “feeds” dictionary in my url.conf:
feeds = {
'articles': ArticlesFeed,
'blog': BlogFeed,
'podcasts': PodcastFeed,
}
a URL pattern:
urlpatterns = patterns('',
##lot of URL configuration omitted here
(r'^feeds/(?P
)
and a few Feed objects. Here is an example of one:
class BlogFeed(Feed):
title = "Blog Title"
link = "/blog/"
description = "My random thoughts and musings on a variety of subjects."
def items(self):
return get_items("blog")
And everything just works. I tested my feeds with Blam and Sage and they work just fine. I also created a feed with enclosures and both iPodder and a custom podcast grabber I created had no problems with the feed. I can’t imagine this being easier.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is adding a SQLite backend.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to test the new Workspace code.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is removing around a thousand lines of code while improving test coverage. Really? Really!
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is fixing the NodeBase tests.
LAMP technologies traditionally include Linux, Apache, MySQL, and, by happy coincidence, one of three “P” languages: PHP, Python or Perl. Of course, this is changing a bit. Many folks have long preferred PostgreSQL over MySQL because prior to MySQL 4.1 (or 5, depending upon your point of view), MySQL was pretty much a sophisticated toy, but LAPP probably doesn’t sound as cool. Others prefer Ruby to Python, PHP and Perl, but prior to Rails, many folks assumed that switching to Ruby wouldn’t be a good career move.
But the question that some folks wonder about is whether or not Perl6 will take its place along the “P” languages. The loudest objection I hear today are that Perl6 is taking too long and that it’s too complicated. Well, as anyone who’s programmed C++ can tell you, complexity is not a guaranteed barrier to language adoption. However, while Perl6 is very feature-rich, the common usage of the language is easier to learn than Perl5. As for “taking too long”, were I a betting man, I’d be willing to bet that we’ll have an alpha in 2007, and probably in the first half rather than the second. Given that I’ve been paying attention to Perl6’s development and I know some of the designers and implementors, I feel very comfortable with this prediction. But if Perl6 really is everything the designers hope it will be, why should we bother? Admittedly, I love Perl and its quirks, but I readily concede that those quirks are something of a liability. Not so with Perl6.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to unentangle the database code from the nodebase code. This is the biggest and most valuable refactoring yet.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to figure out what to do next — and to plan for reshuffling the database code.
Imagine you’re programming an application and you run across the following (pseudocode) method:
double foo (int num) {
return someNum/num;
}
Any programmer who’s been programming for more than, oh, 3 days, is going to ask “what happens when you pass a zero?” What’s remarkable about this is not just that the original programmer forgot to consider this case, but that many times the original programmer will have all sorts of fascinating arguments about why they deliberately ignored this case. Why is it that many serious programmers would be aghast at these arguments but somehow accept them when it comes to programming a database?
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to finish porting all of the node tests and to clean up the mess!
A few years ago I was trying to explain to a Java programmer what closures were and after listening to me, he merely replied “I’d use an object for that”. After he started working with Python, he began to change his mind after encountering Python’s somewhat limited lamba expressions. Once you understand closures, they become an indispensable tool for making code smaller and more flexible, but trying to explain closures to someone is about as daunting as explaining objects to seasoned COBOL programmers.
The Sapir-Whorf hypothesis basically asserts that thought is constrained by language. Some have interpreted this to mean (incorrectly) that we cannot understand ideas that we do not have words for. However, it might be better stated that we have difficulty understanding ideas that we do not have words for. Programming languages support this hypothesis.
I’ve been exploring the social networking space lately and have come to one solid conclusion. They’re driven by fun. I belong to a number of social networks and despite a few having a purpose they don’t really have any activity there because they’re boring. Utility has its purpose, but zzzzzz. I don’t care if “Peter Pumpkineater wants to be your friend” if there’s nothing to do otherwise.
I know this is hardly a shocking revelation to most, but if it’s common knowledge than why are some networks still so ho-hum?
Anyhoo, I -just- setup a social networking site for alphas on Ning (it was perfectly suited). The fun element is to indulge in discussions across a lot of alpha topics (mac, apps, gadgets, games) in one place.
Naturally with the aplomb of mac news this week I set up groups for Mini, MacBook, MacBook Pro, & OSX apps. Feel free to start new group topics. I’d like to join them (Second Life?) when my MacBook arrives.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to port the remaining node tests as quickly as possible. Then it’s on to greater things!
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today I ported several test files to the new format… and discovered a nasty bug in my SUPER module. Oops.
Jeff Croft’s Django for non-programmers article is a good introduction to one of the top Python web development toolkits. You can build a weblog in thirteen lines of declarative Python — and no SQL.
Of course, the “non-programmer” has to be capable of putting those nine lines of code in the right place with the right customization, but that’s not awful at all. I’m certainly impressed.
This does raise one important issue (no, not “does the world need yet another weblog program” — it doesn’t): “how do people get Django installed on cheap hosting providers?
That is a subject worth much more thought.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
With nodegroup ported, I can port the nodeball tests. How well does inheriting from other test classes work? Not as well as I had hoped.
I don’t know how I missed this until now, but Noel Llopis and Sean Houghton have a length weblog posting containing their GDC 2006 paper Making Better Games with Test-Driven Development.
There are several good ideas: parts stick out for me. First, the quote:
TDD is a development methodology, not a testing one.
We are hoping that as TDD becomes more common in the games industry, middleware providers will make their APIs more TDD friendly and even ship with their unit tests.
Imagine then how useful TDD can be on a full engine developed with TDD from the start.
We have found that counting the number of unit tests is a really good measure of progress.
There’s also a good discussion of three ways to test graphics code.
(Next year I have to go to GDC.)
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is finish porting the nodegroup tests. I finally made a decision about how to handle parent test methods.
Working with complex, nested data structures takes practice and patience. It
helps to be able to visualize your data. Data::Dumper is
one of the oldest and most widely used modules because it does what it says –
it serializes a Perl data structure to its equivalent Perl code.
It’s not a perfect module, though. Its default output is a little verbose (if customizable), it can use a lot of memory, and it can be slow. It also doesn’t handle complex references well.
Data::Dump::Streamer
is a newer alternative that works better in some cases. Here’s what I learned
from playing with it one afternoon.
Ubuntu Dapper Flight 7 came out earlier this week and I decided to do a fresh install on my laptop. Part of the reason for a fresh install is that I’m ready for a new Ubuntu to install and automagically configure my system as a few things have gotten unconfigured and I can’t seem to re-configure them properly. Another reason is that I’d like to get Windows on this laptop as well as Ubuntu and the easiest way to do that is to repartition the hard drive and do a from-scratch install, anyway.
When I booted up after installing Dapper, my laptop was set at a proper 1920×1200 and it looked great. The video was configured to do direct rendering, but was using the xorg ati driver rather than the ATI proprietary fglrx driver. The 3d performance was pitiful and the quality of playing videos was poor, so I decided to try the fglrx driver. It only took a one-line change to my xorg.conf file (changing “ati” to “fglrx”) and it just worked. Another thing X related that just worked without any configuration on my part was the Synaptics touch pad scroll area. I’ve gotten this working in the past, but only with some hacking on the xorg.conf file.
Of course, none of the video files I tried to play worked since their respective codecs don’t come installed with Ubuntu. I Installed win32codecs and VLC was able to play everything I tried. xine is having some trouble seeing any codecs I give it, so I’ll keep plugging away at xine and use VLC in the meantime. Or maybe I’ll just stick with VLC.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to continue porting the nodegroup tests. This is a great way to test the inheritance of test methods.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to start porting the nodegroup tests. This will take a couple of days. It’s painful, but it’s good for me. That’s what I keep telling myself.
Update: Being my first entry here, I didn’t realize that my listed name would be “Curtis Poe”. Everywhere else it’s “Ovid” or “Curtis ‘Ovid’ Poe”. Sorry for the confusion.
Since this is my first blog entry for O’Reilly, an introduction seems to be in order. If you’re involved in the Perl community, you probably know who I am. For those who don’t, I’m one of the authors of the new Perl Hacks book, along with chromatic and Damian Conway. I also sit on the Perl Foundation steering committee and I run the Perl Foundation grant committee. I also have a moderately popular CGI Course online and a fair amount of code released on the CPAN.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task demonstrates how easy it is to port further tests. The new system works.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today the porting of yet another node test worked almost flawlessly. Am I getting good at this?
Patrick Michaud joined the Perl 6 project in 2004, after the team put out a call for a compiler pumpking. Since then, he’s ported Perl 6 rules to Parrot, written an operator precedence parser, and has a working Perl 6 compiler. In addition, Patrick is an active contributor to the design of Perl 6 and the implementation of Pugs. Recently, he agreed to answer Perl.com’s usual interview questions.
Google’s second Summer of Code has started to accept student applications since I started to write this. The project has expanded and matured and the list of mentoring organizations has grown.
I’ve volunteered as a mentor for the Perl Foundation this year. In particular, one project has my eye — an improved, nearly-automatic native call interface generator for Parrot and Perl 5. We’re looking for one student with some C and Perl abilities as well as an interest in parsers, compilers, and shared libraries. I don’t want to give away the secret too much, but if this project works out it could completely change the nature of language interoperability.
The Perl Foundation’s Summer of Code site has more details about the entire project. TPF’s SoC Project Ideas is a great list of potential projects. If you’d like to flip bits, not burgers this summer, dig through the ideas from any of the mentoring organizations and apply. We’d love to have you.
(Mentors and mentoring organizations, feel free to post your top project ideas here; it’s fun to see what other projects need and want.)
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to port another node tests to the new system. Unfortunately, the order of porting matters quite a bit… problems abound!!
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is to finish porting the nodetype and setting tests!
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is starting to port a child node to Test::Class. Hooray!
In a post to the Perl 5 Porters mailing list, Nicholas Clark (maintainer of the stable version of Perl 5.8) talks about how he feels as if he is running hard to stand still. There are a lot of good changes going into the development version of Perl (what will be 5.10), but making sure that those changes are suitable for the stable version — as well as keeping new core dual-lived modules up to date.
No one maintains Perl full-time and very few people hack on Perl as paid contributors. Is this the case for other languages?
(It’s unfortunate that there’s so little publicity about the new features and day-to-day development of Perl. LWN.net does a great job of describing Linux kernel development every week. I sometimes wonder if a regular developer-level overview of new features, bugfixes, and such would be valuable. If so, I can probably devote resources to it. The same goes for other languages we cover on ONLamp.com….)
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is finishing the port of the parent node to Test::Class. Hooray!
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is still more work porting the node tests to Test::Class. When will it all end?
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is, yet again, continuing to port tests from the procedural form to the Test::Class form.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Today’s task is continuing to port tests from the procedural form to the Test::Class form.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
Several weeks ago there was a notable bit of controversy over some comments made by James Gosling, father of the Java programming language. He has since addressed the flame war that erupted, but the whole ordeal got me thinking seriously about PHP and its scalability and performance abilities compared to Java. I knew that several hugely popular Web 2.0 applications were written in scripting languages like PHP, so I contacted Owen Byrne - Senior Software Engineer at digg.com to learn how he addressed any problems they encountered during their meteoric growth. This article addresses the all-to-common false assumptions about the cost of scalability and performance in PHP applications.
My on-again-off-again problem with suspend to RAM on my laptop is now on-again. I’ve been getting really tired of shutting down my laptop to transport it between work and home, as well as just keeping it running all the time. I decided today to dig in just a little bit and see if I could figure out what was causing my laptop from properly suspending, or more to the point, resuming properly from suspend.
Here’s a little background for anyone who doesn’t know my story. I purchased a Dell Inspiron 9300 less than a year ago and installed Ubuntu on it. At various times, I’ve had most everything working well. When I recently upgraded to Dapper, suspend to ram broke. It seems to suspend fine, but when it comes out of suspend, I was getting a nasty orange or purple noisy screen. Today, I stumpled across an acpi howto that mentioned trying to turn off optional devices. I decided to try to shut off my wireless cart (Fn-F2) and see if that would help. Surprisingly, it did. I then decided to leave the wireless up and put “rmmod ipw2200″ in my suspend script. That worked, too! So, my wireless card has been the guilty party in keeping suspend from working. Now that I know what the problem was, I can live with this as a solution.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
From the mailbag, fellow PDX.pm hacker Eric Wilhelm has a vision of creating a publicly-readable Subversion repository consisting of all of the modules on the CPAN. Though TPF declined to fund his work, he’s put up his current version at svn4cpan along with some notes on what could happen. If you’re looking for a project, here’s one that could have amazing benefits.
Python 3000 has been a joke for so long now that it’s funny to consider that it will eventually materialize into something real. Guido just posted some thoughts on very real implementation details for Python 3000. The topic at hand was adaptation. Basically, Guido has killed PEPS 245 and 246 which are interfaces and adaptation respectively, and has posited the use of generic functions as an alternative approach.
I’m curious to see how Guido and the other Python big-brains envision everyday use of any kind of protocol-based approach for type management (or whatever you want to term the topic at hand). From one of Alex Martelli’s posts, I can see how protocols could be really cool for library, specifically the standard library, developers. You create a library funtion, state that it can handle this protocol of thing, and it (not-so-)magically just does the right thing with various types that are passed in. For example, you can create a function that really expects a file-like object, create a protocol that adapts a string to a StringIO object, and now it can handle a file or a string. At least that’s how I understand things so far.
As I’ve been watching the discussion around protocols, I’ve assumed that this was going to be Python’s way of gently (and optionally) enforcing type “safety” or requiring function/method call arguments to be of certain types, but it appears to be much more than that. I really like the idea. I’d just like to see more day-to-day use cases rather than just the theoretical discussions that I’ve read thus far. From PEP 246, there is a question “What benefit does this proposal provide?”, to which the answer begins “[t]he typical Python programmer is an integrator” and seems to sound like protocols will just “make stuff work” if a sensible type of object is passed into a function or method. So, maybe that’s the “common, everyday use case” that I was looking for. Maybe it just makes stuff work easier and integrate better. How protocols and adaptation fit into my original “type safety” thoughts is becoming less clear. I was really hoping Python would incorporate optional type checking at some point. Maybe this protocol approach can provide a foundation for type checking. Or maybe that’s an entirely different discussion.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
This 30-day project explores the refactoring of a legacy system. The Everything Engine is an aging software project that powers Perl Monks, Everything 2, and a few other websites. It suffers from poor design and maintainiability. Learn what it’s like to look over the shoulder of an experienced developer as he refactors, redesigns, and updates the code.
I’ve been noticing that my CPU utilization has been higher than I had been used to with Breezy. Running “top“ only showed me that a lot of CPU was being used, but not a process that was using it. I suspected beagle was doing something funky, so I killed it. That didn’t help. I really didn’t think that was it, but it didn’t hurt to try. It felt like something lower down that beagle out to be running. I stumbled across a bug report mentioning the same problem. The problem is in the 686 kernel and can be resolved (hopefully only temporarily) by reverting to a 386 kernel. This appears to have resolved my problem.
Now I’ve started beagle back up and he’s happily sniffing through my system, taking all the CPU time and has my CPU at a temperature that I’d be comfortable if my chicken were cooked to. Hopefully he’s just re-indexing some stuff.
I just ran across this blog post about getting NetworkManager to work under Dapper. This post mentions a couple of problems. The first has to do with an icon cache which prevents NetworkManager from even starting. I wasn’t having that problem. I was getting nm-applet in my “systray”. The next problem mentioned is that /etc/network/interfaces should only contain references to the lo interface. Oops. Guess that was modified somehow during the upgrade process. I guess this problems should only occur to people upgrading to Dapper from a previous Ubuntu. I would hope that a new install of NetworkManager would fix /etc/network/interfaces.
Since the release date for Ubuntu Dapper was set back 6 weeks, I decided to not wait until June to update my system. Until very recently, I had been running Ubuntu Breezy. You can read about my experiences with installing Ubuntu on my laptop in this article. On Friday, I updated my /etc/apt/sources.list to point to “dapper” rather than “breezy”. I then executed
sudo apt-get dist-upgrade
and let the upgrade process begin. Downloading and installing the packages took about two hours.
When I rebooted, everything seemed pretty normal. The bootup sequence looked a little different. It didn’t hang on “setting up network devices”, which it typically does when I’m not physically jacked into an ethernet port. It lost my previously selected GDM login screen. No big deal. I’ll reset that so I don’t have to look at the Ubuntu standard dirt colored one. Logging in worked just fine and I was pleased that at least my desktop background and window borders had survived the upgrade. But NetworkManager couldn’t find my wireless network card. Weird. This still isn’t a huge deal as gtkwifi does a great job managing my wireless connection. I’m sure NetworkManager will just need a little tweeking to get back to its previous splendor.
One thing that I really am disappointed with is that suspend to RAM is now broken. After an ongoing battle with Breezy, I finally got suspend working. In Dapper, neither my custom suspend script nor the builtin suspend functionality work properly. When I come out of suspend, my screen displays purpleish noise. Guess I’ll be hacking on that over the next few days.
Other than those two negatives, everything is running great. Everything feels a little more responsive. And it seems that my system is using a little less RAM than it was before. I blogged a week or so back about my previously non-good experience with beagle search. I am glad to say that so far, beagle has been behaving itself with regard to memory consumption. And I’m pleased with how it works.
When Dapper final is released, I plan on doing a full reinstall. Maybe it’s my experience with Windows that makes me even consider it. Or maybe I think things will run better if I give the installer a chance to find and configure my hardware. Regardless, in June, I’ll have a clean system.
My home office network connection went out for a while yesterday. Instead of taking the day off to rest and relax, I asked fellow Perl 6 and Parrot hacker Allison Randal to walk me through building a compiler for a little language with Parrot’s compiler tools.
Though we didn’t finish the project before the fairly useless telephone company fixed the problem, I learned a lot about how the compiler tools work. Here’s what I know now.
Dave Cross has posted a short series of slides entitled What’s Wrong with ORM. Though they’re slightly Perl-centric, they discuss weaknesses of many ORM systems of other languages too — especially not taking advantage of the power of the database.
Jifty::DBI is yet another database abstraction layer, with object-relational mapping and (more importantly to me) schema generation. It’s part of Jifty, the new web development toolkit from Best Practical (creators of RT) and it’s a re-imagining of DBIx::SearchBuilder.
Despite its history, the public distribution is fairly new and has some rough patches, mostly related to documentation and understanding. I experimented with the latest stable release (0.16) for a couple of afternoons. Here’s what I learned.
Perl has a concept called “DWIM” (do what I mean). Greg McCarroll has prototyped a idea that he calls “SWIM” (search what I mean) where he monitors the speed that you type words in a Google search box and automatically quotes words that seem to have been typed together.
There’s a fuller explanation in his blog and you can also play with the prototype.
Now we just need someone to turn it into a Greasemonkey script.
There’s a rule of thumb that’s been passed down to me from earlier generations of systems administrators that I will occasionally forget. I’m always sorry when I do. Not only that, but I’m also often sorry when another administrator forgets it, because it makes helping them very hard (and I like helping other admins where I can). Here’s the scoop:
Let’s say an employee has just left your firm. Let’s say, for giggles, that he was a webmaster who maintained several LAMP-driven applications. Well, you’ll need to change a few passwords on a few databases, in a few files for the applications, maybe change a pg_hba.conf file here, a slapd.conf there, etc., etc. Well, don’t do any of this until you perform this essential step: without making ANY CHANGES to the configuration or anything else, simply shut down and then restart the service.
In a perfect world, this is a waste of time. It is not a perfect world. Suppose the ex-employee made a last minute change and figured it would be reflected after the next downtime - a downtime they will now not be present to see. Suppose the employee was disgruntled and planted a present there for you. The point isn’t to determine whether malice was involved or not, the point is that you, as an administrator, need to get to a place where you’re confident that the service is still healthy.
If you DO NOT restart the service before making the requisite changes, and the service doesn’t restart properly, well, how can you be sure the service wasn’t borked before you made your changes? And so how are you going to determine where to start troubleshooting? Looking at your own changes may or may not be a waste of time. Who knows?
Well, I hope this has helped someone. Share your war stories about a time when this bit you!
KinoSearch is a CPAN distribution that’s a Perlish port of the powerful Apache Lucene search engine. (In one sense, it’s a competitor to the earlier Plucene project.) Here’s what I learned from playing with it one afternoon.
I’ve used wikis for years. I’ve even written a few. I’d use one myself to manage my notes and ideas, but I hate typing in textareas and don’t want to launch a web server and new browser tab whenever an idea strikes. Ideally, I could just type a quick note in Vim, as I always have a command-line somewhere. Here are the three lines of code it took to make a single-user Wiki in Vim.
My wife’s store has been running in production for a few weeks now without much event. As with any website/webapp, enhancements and bug fixes are inevitable. I’ve already installed one new release which fixed a Javascript bug. I’m currently working on another release which will add some additional content and a couple of areas of feature enhancements. I’ve also been refining my release management procedures which includes a more formal test environment.
Which brings me to the topic at hand. I recently created a test environment at my hosting service. The test environment consists of a different domain name (subdomain, actually), a different directory within my home directory for placing application files, and a new database instance. When I installed the latest version of my code from Subversion, I encountered an error that I could not get past and which took me several hours to track down. (As a side note, it’s amazing how “obvious” a solution is once you’ve solved it.)
It turns out that in my test environment, all “decimal” types from the database were being pulled into my application as strings. When I performed a math calculation on them such as adding what should be a decimal to an int, I would get a type error. If only it were that obvious at first. The first error that I saw was a pickling error. And it wasn’t the pretty traceback that Django typically gives. This looked more like a documentation page generated from Python source, only nastier. I couldn’t figure out why I was getting a pickling error. I thought at first that it didn’t like me putting an instance of a custom class in session memory. (By the way, the pickling error appeared to be happening when Django was trying to stuff something into session memory.) That turned out to not be it. Somehow, through changing code, I was able to get a better error message and diagnose the problem.
This odd behavior is only happening in my test environment, and in neither of my primary development environment nor my production environment. Here is a query in my production environment showing the proper behavior:
>>> from django.models import store
>>> p = store.products.get_list()[0]
>>> p
The Gosling Design
>>> p.price
Decimal("25.00")
And here is the test environment showing bad behavior:
>>> from django.models import store >>> p = store.products.get_list()[0] >>> p The Gosling Design >>> p.price '25.00'
Just for the sake of getting everything working properly, I had to go through the code and explicitly wrap decimal database lookups with a call to Python’s decimal.Decimal(). The only thing I can conclude is that my hosting service created the new database instance on a different version of MySQL. So, it really could be either a Django or a MySQL thing.
© 2011, O'Reilly Media, Inc. (707) 827-7000 / (800) 998-9938 All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. |
|
