Related link: http://openqrm.sourceforge.net/

There’s an interesting new distributed management project called
openQRM. It’s currently around 73 on
SourceForge, and has been up in the top 50 recently. It was released
under a modified version of the Mozilla Public License by
Qlusters,
a company that was founded by openMosix developer Moshe Bar and that
I’ve repeatedly met with at LinuxWorld Expos. I just talked to a
colleague of mine, William Hurley, who recently took a job as Qlusters
CTO.

According to Hurley, what distinguishes openQRM from the many other
available network and cluster management tools is that it lets sites
continue to use the architecture and software they currently have with
minimal disruption. openQRM developers have already created plug-ins
for Nagios, Xen, and VMWare ESX. Integration with other software or
home-grown scripts at each user site should be fairly easy. Now that
the core technology is open source, the team is hoping to pull in more
developers from the community, particularly to support FreeBSD, OS X,
and other platforms.

openQRM is important because of the proliferation of servers run by
small organizations on small budgets, made possible in recent years by
free software running on cheap hardware. The bottleneck now, as many
TCO analysts point out, is system administrator time. According to
Hurley, without good management tools, system administration
can add $7000 to $20,000 to the annual cost of each box.

openQRM offers automatic failover for servers under its control. In
fact, you can set up automatic failover from one openQRM managing
server to another, so there’s no single point of failure anywhere in
your architecture. The system also supports diskless servers, which
lowers cost and rates of failure.

I find this product interesting also because it reflects the
continuing move to free software by commercial vendors. Qlusters
started out as an entirely proprietary product, but because they
lived in a Linux environment and ran on Linux systems, they felt they
wanted to become more a part of the free software community. I had a
long meeting on this topic with VP Fred Gallagher at the most recent
LinuxWorld, which I wrote up in a

blog from that conference. Since then, they’ve acted on this
impulse. Hiring Hurley and releasing openQRM are both significant
steps toward supporting the free software movement.

Qlusters itself will market a variety of proprietary plug-ins and
management tools on top of this open platform. The license allows
other companies to develop proprietary products on top of openQRM. The
only stipulation (and the only modification Qlusters made to the
Mozilla Public License) was that commercial vendors have to
acknowledge they’re using Qlusters’s openQRM.

I spent two days this past week at Microsoft Search Champs, a
conference where invitees make suggestions for their search tools and
other MSN offerings. Microsoft paid for everything and picked our
brains concerning a lot of different topics, some under NDA and some
public.

Why would I do this, and why would they want me there? I’ve been
associated with the free software movement for at least ten years. But
while I value openness, I also value functionality. If you browse my

articles and blogs about Microsoft,
you’ll find about as many positive references as negative ones. I
appreciate new solutions and technologies from all vendors, and I
think one company’s success will provide a model and a motivation for
others to move forward.

Furthermore, Microsoft is around to stay, and people who make a living
in the computer world have adapted to it. Every professional and
aspiring professional I know, both programmers and system
administrators, have learned their way about both Microsoft systems
and Unix-like systems. A host of projects such as Samba and Mono set
this accommodation in code, as I discussed in my article

Can the Samba Story be Retold?

Finally, I have seen evidence–this week and other times–that there
are many different attitudes toward the open source movement and
transparency in general at Microsoft, and believe that I could have a
positive impact by going there, partly to argue for more openness on
their part.

Some highlights of Search Champs

All the tools and sites we looked are part of

Windows Live,
a next-generation combination portal, social network, and information
site. Microsoft hopes that people who use MSN and competing networks
will move to Windows Live.

My favorite feature is
gadgets,
which are like the mini-applications you can add to your tool bars on
many operating systems, but with more real estate and therefore more
features. You could probably create an extremely powerful interactive
Web page in half an hour or so by adding gadgets for maps, RSS feeds,
message boards, and so forth. The key value will come if Microsoft is
successful in making it easy for non-Microsoft developers to create
and contribute gadgets.

For years, web designers and programmers have put together rich sites,
but gadgets can encapsulate the most popular features and makes it
something you can throw together as easily as an RSS feed or bookmark
list. It won’t look wildly creative, probably (it depends on how much
extra sweat you want to put into it), but it will look consistent with
other useful sites. If Microsoft can persuade a few alpha bloggers to
switch to this system, it may become a de facto standard.

The current crop of new Windows Live features seem consistent with
what I see as Microsoft’s two general strengths: attractive interfaces
and elegant integration.

For instance, their Windows Live Local combines maps, aerial views,
and 45-degree-angle photos, all very easy to reach. (My family and I
were a bit spooked to see our neighborhood, so lifelike were the
photos.) You could just click on two points of a map, and driving
directions are generated. (Some attendees asked why they don’t present
public transportation options too.) Then you can select a point and
see a photo of a particular traffic intersection so you can recognize
and navigate it. You can also drag a site to a scratch list that is
saved between sessions.

So these local features are an incremental improvement over what we’ve
had so far–incrementally better enough to make a difference for many
people.

Microsoft’s plans for Windows Live are also based on building
communities. This means persuading users to share personal
information. Productive citizens of Windows Live will have rich
identities, so that they can find other people with similar interests.

The last attempt by Microsoft to leverage user information was
Passport, which we all know didn’t go very far. Passport is still the
ID system that lies behind personal identity on Live. But the intent
now looks a lot different.

I think Passport failed because its core promise was for Microsoft to
guard very sensitive data for its users, such as phone numbers and
credit card information. Supposedly, when you buy something online you
could have Passport automatically transfer such information to the
corresponding party at the appropriate moment. People didn’t trust the
whole environment for online security–even if they could trust
Microsoft’s security, another point of contention–enough to place
their information in Passport’s hands for such hard-to-monitor
purposes.

But the new identity doesn’t involve credit cards so much as who your
pets are and what music you listen to. Microsoft certainly hopes
you’ll share a couple key pieces of demographic information with them
(age and gender) to help them target ads. But for the most part, what
you build up as online identity is not what you’d share with a vendor,
but what you’d share with neighbors and school chums.

The Live developers are working on lots of other interesting
things–multimedia search, a classified ad site, and more–but I’ll
leave it up to others to introduce them.

The value of openness

I certainly took the opportunity to press my philosophy at the
conference. Drawing on debates where I live in Massachusetts, I
complained to Microsoft managers that some of Microsoft’s supposedly
open formats (such as the XML format for Office) were encumbered by
all sorts of small but ominous restrictions, including the threat of
exercising patents. These cumulatively make potential users and
competitors afraid of Microsoft acting against them.

In addition to pointing the managers to the

groklaw analysis
of the legal labyrinth Microsoft erected around its Office XML format,
I also pointed to

critical coverage
of their assertion of patent rights on the FAT filesystem. (The
supposedly novel technique they patented looks to me like just a
variation on the familiar idea of file attributes stored in a parallel
location to the files.) And I did not omit mention of the

absurd Slashdot tug-of-war
over Microsoft’s Kerberos enhancements, which not only broke compatibility
with other Kerberos implementations, but were described in a document
you had to license just to read.

My point to management–and you have to remember I was talking to
developers here, not the lawyers or other managers who thought up
those legal forays–was that such activities create bad feelings among
many of the people they want to attract: the amorphous “information
loving” community of artists, academics, lawyers, and so forth. They
make developers worry, because if developers have to cede a substrate
to Microsoft and just build on top of that substrate, nothing prevents
Microsoft from coming along later and taking over the new layer they
just built.

And if such maneuvers do anything to help Microsoft’s business model,
it’s the wrong business model. (I limited my complaints to legal
issues, and did not want to load on yet more by talking about business
practices.) I think the very existence of Search Champs shows there’s
movement toward more openness at Microsoft, a pull against the more
controlling elements.

I was by not means the only one of the 57 invitees to have such
sympathies. I heard plenty of discussion of both Macs and GNU/Linux
systems. MSN managers declared they wanted their site to work on all
these systems. (I have tried live.com out a lot on Linux, using both
Firefox and Konqueror, and find it works fine.) On the bus to the
Redmond campus, I heard a possible solution to my problem getting Ext3
filesystem support compiled into the Linux kernel. Another attendee
told me bluntly, “There’s no reason for major sites to use anything
except open source software” and cited Lawrence Lessig as one of his
most inspiring influences. Several people (including Microsoft staff)
brought up
Creative Commons
approvingly, and DRM came in for a lot of criticism.

The Justice Department subpoena

I would have liked to spend more of the sessions discussing
Microsoft’s legal activities and lobbying, but another policy debate
upstaged it. Over the past two weeks, press reports revealed that the
U.S. Justice Department subpoena’d MSN and other sites to hand over
large amounts of search data, and MSN complied. The public, already
rubbed raw by the revelations that George W. Bush and the NSA ignored
laws to carry out widespread wiretapping, reacted with fury to MSN. In
our sessions at Search Champs, the MSN managers succeeded in
justifying their actions and winning us over, but they made some
promises to communicate better in the future.

A

semi-official Microsoft explanation
starts a Web page with valuable list of comments. At Search Champs, we
heard even more clearly that Microsoft negotiated hard with the
Justice Department and insisted on stripping out IP address
information. Furthermore, what they handed over was merely a list of
terms and the number of searches on them; no term could be correlated
with another term or with an IP address.

MSN managers came away with some rough guides for handling future
challenges. First, the major search sites should talk to one another
and come up with a common policy for handling government and research
requests. Second, they should publicize what requests they get and how
they respond.

What it felt like to be there

A corporate junket is a new experience for me, and I don’t know
whether I would have understood how it felt like in advance if I’d
heard about it from others.

On the one hand, Microsoft pampered us in almost every way, from the
cars they sent to pick us up at the airport to fine food and gift
certificates. They plied us with liquor before and after meals. We
were all lodged at the W Hotel, which is famous for a particular look
and feel. For instance, all the hallways and common areas are dim–not
romantic dim, but suspended-reality dim. In Seattle, which is
naturally dim most of the time, this is no enhancement. The hotel
follows up the theme with fancy functionalist designs along Central
European designs, leading to some electronic equipment that’s almost
impossible to operate. And then there’s the background music you hear
everywhere, which can perhaps be described as Third World New Age
technopop. One day they threw in ten minutes of the Adagio from
Mahler’s Fifth Symphony between two technopop selections.

Some of us would have felt more comfortable had they lodged us at
something equivalent to the YMCA, fed us on burgers, and made us take
public transportation. But we probably wouldn’t have felt guilty
enough to work as hard as they wanted.

On the other hand, the Microsoft building we were in was much like
other buildings used by high-tech companies, and spending two days
there was much like sitting around talking to any computer programmers
about any topic at any time. The meeting appeared highly structured
when the schedule was presented to us, but in fact it was fairly
informal. And they always followed up meetings by telling us what a
fantastic job we were doing–just for reacting to their work out of
our personal experience.

I’m not sure who learned more from the whole event. I certainly
learned a lot about people in the field as well as technology, and I
appreciate all the money and effort Microsoft put in.

Related link: http://groups.google.com/group/turbogears/browse_frm/thread/7528362868b88299/15f…

The “eggs” that Kevin Dangoor (the creator of TurboGears) is referring to here is the package file format which TurboGears uses.

I’m not exactly sure what all the numbers are, though. He mentions that TurboGears eggs have been downloaded over 20,000 times and that overall there have been over 100,000 eggs downloaded. I guess he’s referring to downloads of other packages which TurboGears has a dependency on and which are hosted at turbogears.org.

Congratulations, Kevin! That is quite an accomplishment. I’d be interested to also see how many SVN checkouts there were. I’ve been running off of SVN for so long I can’t remember the last egg I downloaded. I suspect there are plenty of other folks in that same boat.

Related link: http://ipython.scipy.org/

From the changelog, this looks like mostly cleanup and bug fixes rather than a major revamping. I just downloaded IPython using easy_install and it brought me up to 0.7.1. It complained about a conflicting version already installed, so I gave it the -D flag so it would delete the conflict. I should’ve looked more closely at where the conflict was before letting it just “fix” the conflict. It said the conclict was in a plain IPython directory and it installed this one to an ipython-0.7.1-py2.4.egg directory and updated a .pth file. Aahh. ipython-0.7.1-py2.4.egg contains an IPython directory, so I must’ve installed whatever version that was from source. It looks like easy_install did the right thing.

From what this message says, it looks like IPython will be getting a pretty major overhaul soon. The SVN branch for the overhaul is named “chainsaw”. That message also mentions that Ville Vainio will be taking over maintenance of the 0.7.x development line, thus freeing up Fernando Perez to work on the overhaul. This can only be a good thing.

I can’t say enough good things about IPython. For anyone not familiar with it, it’s a powerful replacement for the standard Python interactive shell, as well as a customizable shell which can be used for pretty much anything. People have even made it their default system shell. I highly recommend it.

Related link: http://www.walterzorn.com/index.htm

Here are some fundamental, well made JS goodies. From the site:

JavaScript Vectorgraphics Library

Graphics capabilities for JavaScript. Routines to draw inclined (oblique) lines, ellipses, circles, rectangles, polylines, polygons. Elements which actually aren’t available through HTML.

Drag’nDrop & DHTML Library

A DHTML JavaScript Library with extended yet easily understandable DHTML API. Provides also Drag & Drop functionality for layers and images …

Tooltips with JavaScript

A cross-browser solution for javascript-created tooltips (information boxes close to the mouse pointer) that works even on Opera 5 and 6. The appearance of these tooltips can be customized in multiple ways (color, border, shadow etc.). The tooltips may contain plain text as well as HTML, for instance images etc.

Rotate Image

An experimental JavaScript Library to rotate images dynamically by arbitrary angles. Just a demonstration - it’s strictly advised against using this unpromising JavaScript experiment on a website!!

I’ve put these four JavaScript libraries under the LGPL (Lesser General Public License, http://www.gnu.org/copyleft/lesser.html ). You may use them for free under the terms of the LGPL and of my copyright.

Online Function Grapher

Written in JavaScript. Draws function graphs directly into the browser window - no download, no plugins required.

Related link: http://shiflett.org/archive/184

Last month, I discussed Google’s XSS Vulnerability and provided an example that demonstrates it. I was hoping to highlight why character encoding consistency is important, but apparently the addslashes() versus mysql_real_escape_string() debate continues. Demonstrating Google’s XSS vulnerability was pretty easy. Demonstrating an SQL injection attack that is immune to addslashes() is a bit more involved, but still pretty straightforward.

For the impatient, here’s the code:

<?php 

$mysql = array(); 

$db = mysqli_init(); $db->real_connect('localhost', 'myuser', 'mypass', 'mydb'); 

$_POST['username'] = chr(0xbf) .                      chr(0x27) .                      ' OR username = username /*'; $_POST['password'] = 'guess'; 

$mysql['username'] = addslashes($_POST['username']); $mysql['password'] = addslashes($_POST['password']); 

$sql = "SELECT *         FROM   users         WHERE  username = '{$mysql['username']}'         AND    password = '{$mysql['password']}'"; 

$result = $db->query($sql); 

if ($result->num_rows) {     echo '<p>Success</p>'; } else {     echo '<p>Failure</p>'; } 

?>

The full explanation covers this in a bit more detail, but the basic idea is that addslashes() can be tricked into creating valid multi-byte characters out of invalid ones. Whenever a multi-byte character ends in 0×5c (a backslash), an attacker can inject the beginning byte(s) of that character just prior to a single quote, and addslashes() will complete the character rather than escape the single quote. In essence, the backslash gets absorbed, and the single quote is successfully injected. This opens the door for SQL injection attacks.

The moral of the story is to use mysql_real_escape_string(), bound parameters, or any of the major database abstraction libraries.

Related link: http://blog.chris.tylers.info/index.php?/archives/17-How-to-Rollback-Package-Upd…

Fedora Core 4/5 uses yum for package management. yum is build on top of rpm, and pirut, pup, and yumex are graphical interfaces built on top of yum. Together, these tools provide a simple-to-use, powerful package management system.

One of the least-known secrets about rpm is that it can rollback (undo) package changes. It can take a fair bit of storage space to track the information necessary for rollback, but since storage is cheap, it’s worthwhile enabling this feature on most systems.

Here are cut-to-the-chase directions on using this feature:

1. To configure yum to save rollback information, add the line tsflags=repackage to /etc/yum.conf.

2. To configure command-line rpm to do the same thing, add the line %_repackage_all_erasures 1 to /etc/rpm/macros.

3. Install, erase, and update packages to your heart’s content, using pup, pirut, yumex, yum, rpm, and the yum automatic update service.

If/when you want to rollback to a previous state, perform an rpm update with the –rollback option followed by a date/time specification. Some examples: rpm -Uhv –rollback ‘9:00 am’, rpm -Uhv –rollback ‘4 hours ago’, rpm -Uhv –rollback ‘december 25′.

Have you used package rollback?

Related link: http://www.djangoproject.com/

When TurboGears came out, I was pretty excited about it. I was able to quickly throw together a digital photo management application for my wife. I was also able to quickly build her an online store for her new business. As is common with any technology, I encountered a number of problems while building my wife’s store. There have been issues with the Kid templating system which still appear to be unresolved. Maybe they’re fixed, I don’t know. I modified my code to bypass what was causing the errors I was seeing - at a loss of functionality and flexibility. Another issue I’ve seen is with CherryPy deadlocking on what appears to be session file cleanup. I could switch over to database session storage, but I’m not sure I want to go that route. If I stayed with TurboGears, though I might not have to switch; I just read on the CherryPy list that this issue may be fixed.

Serendipitously, I recently started looking at Django. All of these issues and my glance at Django really got me thinking about how TurboGears was put together. I have had almost no problems with the TurboGears code itself. Actually, I can’t really think of any problems I’ve had with core TG code. The problems I’ve had have been with the underlying components - specifically CherryPy and Kid. I began to wonder if I might be better off with a unified solution rather than a solution made of components from separate projects. So, out of curiosity as much as technological motivations, I began porting my wife’s store to Django. I may be totally wrong about unified vs. multiple project component based, but my thoughts are at least reaffirmed by this blog post from David Heinemeier Hansson (the creator of Ruby on Rails).

The store consists of a product catalog, a shopping cart, a shipping calculation algorithm, and a payment system using PayPal. The most logical first step was to migrate the database model. From a user perspective, both SQLObject and Django take a common approach. A class corresponds to a table and class attributes (each with a certain declared type) correspond to columns in that table. The database migration was pretty simple. I did do a minor overhaul in how I group products together. I still have a little work to do on that, though. One huge plus to Django is the pre-built admin interface. With a total of two extra lines of code per class/table, you get a beautifully usable, customizable adminstrative interface to your database. I haven’t utilized it much yet, however. In these initial stages, I find it easier to populate the products into the database through a script. But I think when the site goes live, this will be a huge feature in managing orders.

The next thing I did was to write a couple of quick pages to display the items in the product catalog. At this point, I knew that I was just experimenting and wasn’t committed to using Django yet. I wanted to see how Django would run in my production environment, which is under FastCGI.

Let me take a small pause to say that much of my recent trouble with TurboGears would have been caught earlier if I had deployed it incrementally to my production hosting server (or probably to a comparable environment not on the hosted server). So, I blame myself for not catching the problems earlier. I did try at one time to get the store running under FastCGI on one of my own servers in a comparable manner to how it’s run on Dreamhost (my hosting service). When I couldn’t get it running in a timely manner, I decided to not pursue it any further. In hindsight, this was obviously a mistake.

So, back to my tale. I let this minimalistic site run for a day or so to see if I could feel comfortable moving forward with Django. Everything checked out well. There were no anomylous errors. Both the admin interface and my product listing pages were displaying consistently. (However, the admin interface is devoid of images or a stylesheet. I’ll figure that out soon. I think it has to do with how I have my static/media mapped. I’m sure it’s not a problem. Famous last words, right?) At this point, I became fully committed to porting the full store over to Django.

Next, I set up a base template that each page would extend. This will be a nice addition to the site since I was unable to use this same feature in TurboGears. This is one of the problems I generally hinted at above regarding the Kid templating system. I also fleshed out some of the static pages so I mostly had the same look and feel and content. That was a snap.

I decided that I should next start building up the code around the product catalog, getting all the images and product details displaying properly. Again, this was simple. Django’s templating system is really simple and straightforward to use.

This is about as far as I’ve gotten. I’m hoping that I’ll have the site totally migrated by next weekend. I’ll post back with progress and thoughts on Django. So far, it’s working pretty reasonably.

Related link: http://www.cs.tufts.edu/~mchow/excollege/s2006

I have returned to Tufts University to teach a new course Introduction to Game Development this semester. I am excited and fortunate for another teaching opportunity. Last year, I taught “Security, Privacy, and Politics in the Computer Age” at Tufts University, and it was a tremendous success. Teaching the course was a most rewarding and flattering opportunity for me. My course evaulation was very good. The students appreciated the applicable value of the course, and it gave them an exposure to the “tech culture” (most of the students were non-technical). Many of the students expressed that they wanted more technical content. Finally, the Tufts Experimental College asked students what courses they would like to see in the future, and many said a course on game development.

My experiences with computer graphics, networking, databases, software engineering, HCI/user interfaces, and algorithms, will all come in handy. My past development of several small games will certainly be valuable as well. When I was a Computer Science student at Tufts, most of the courses offered were theory-based and very few implementation-based. I always questioned the value of what I was learning, and how could I put everything that I learned together. That is the beauty of game development: it requires all facets of Computer Science. I wished that such a course was offered to me when I was a student, and this is a major reason why I am teaching the course back at my alma mater. Already, my course is filled. Several students said that they appreciate that I am teaching such a course at Tufts.

I will be using Java in the course, not C/C++. Why? Two reasons: portability and cost. I do not have a computer lab for the course, and not all students have Windows PCs. Most of the Java development tools, including the SDK and Eclipse, are free as in free beer, so students can do their work from their PC in their dorm room. Many students said that they know C/C++ so I’ll spend two days giving a Java crash course. And yes, I know that Killer Game Programming in Java will be vital resource for me and my class.

I welcome any insights or concerns. All the lectures, assignments, examples, and resources are available on the course website at http://www.cs.tufts.edu/~mchow/excollege/s2006/. Please feel free to follow my course online.

It used to be that compiling a custom Linux kernel was almost a necessity. Something you just had to do if you wanted a working system. These days, with loadable kernel modules and better hardware support in the vanilla kernel, I find kernel patching and custom configuration less “necessary” but often still desirable.

I’m curious. Why do Linux users these days configure and compile a kernel? To increase performance or hardware compatibility? To add filesystem support or enable experimental features? Just for the fun of it? An attempt to have absolutely the smallest kernel image possible? An attempt to build a highly portable kernel? My reason usually boils down to getting a new piece of hardware to work fully.

Also, what problems do most people have when compiling a kernel? Migrating to new kernel versions? Patching the kernel? Getting an initrd image to load? My biggest three problems are knowing which options I must enable, finding those options in menuconfig, and knowing the name of module I just compiled.

Please, write a comment and tell me why you find it necessary or desirable to compile a kernel, and the most annoying parts of the process.

Related link: http://orsn.org/

From the ORSN FAQ:

A root server has a reference data base of all of the TLDs released by the ICANN (Top level Domain) e.g. DE, AT, CH, COM and many others.

The ORSN serves as a alternative for the existing root-server network since February 2002, which is coordinated by the ICANN. In contrast to the root servers of the ICANN, the ORSN servers should predominantly be placed in Europe. The maximum number of ORSN root-servers will be 13.

Until now, the administration is done by the USA and/or the ICANN. Therefor, a large number of root-servers is located in America. A loss or the modification of the root-server information could result in serious consequences for all other countries concerning their internet use. It is for example possible to stop a whole country from using the internet. In practice, this scenario didn’t happen so far but it can’t be excluded either.

It appears to be a local, independent ICANN root mirror and fail-safe. It subscribes to ICANN’s TLD policies, yet it reserves judgement over what it might mean for ICANN to ‘fail.’

When I first got into Open Source many moons ago, the advocacy movement was a thriving and vocal part of the community. Most of the movers and shakers back in the day were advocating the use of free and open software at work, to their friends and to their local community via LUGs and other groups. Back then, advocacy was a key part of the community, not only in showing existing computer users this alternative software, but also advising disadvantaged people for whom free software could really open up the doors to skill, employment and potential.

Recently it seems this community-driven advocacy effort has petered out somewhat, and there are far fewer people talking about, conducting, exploring, refining and pushing Open Source advocacy. What is surprising is that advocacy is certainly still going on. Within Open Source organisations as well as LUGs, community groups, IRC channels, forums and mailing lists there are countless people discussing and pushing the Open Source message. As I have written about a number of times in previous articles, advocacy is an artform that needs a reasoned, measured response, and is something that can certainly be buffed and refined. In other words, a stronger community could not only help spread Open Source further, but refine the quality of the message that is being pushed to develop an increased understanding in Open Source and free software.

So what can we do? Well, I would love to see more and more people getting involved in advocacy. To help push this a little further, I have set up Planet Advocacy. Like every other Planet site, Planet Advocacy collects together the blogs of those people who are involved in advocacy in some way. Planets have proven an ideal mechanism for developing ideas and communication between different people. It provides a one stop shop for the cutting edge.

With the site still shiny and new, I am looking for people to add to Planet Advocacy. If you think your blog would be an interesting addition, get in touch with me and tell me how you are involved with advocacy and also include a 62×80 hackergotchi .png of your face. To be clear, you don’t need to work as a professional advocate to get on Planet Advocacy - if you are advocating Open Source in your spare time, you are more than welcome. Planet Advocacy is really only the start. It would be great to see more articles, case studies, discussion groups and public meetings. The people are out there, we just need to share our experience and ideas.

Advocacy is an important component in the Open Source community. There are thousands of companies, charities, schools and people for whom Open Source could make a real and tangible difference - intelligent advocacy can help bring them over to us. There is nothing quite so satisfying as seeing someone get as excited about Open Source as we all were when we first started out. Helping to bring new people over not only extends and improves our community, but it spreads the underlying principles of Open Source such as choice, openness and collaboration. Lets see what we can achieve…

What do you think? Would you like to get involved? Do you think increased advocacy is worthwhile?

Related link: http://www.apple.com/support/downloads/bonjourforwindows_readme.html

I had to fire up my Windows box today, and I wanted to get some files off of my Mac. That’s not a big deal because they can see each other on the network, but from Windows I need to know the IP number of the machine I want it to look at.

I don’t do anything fancy to give names to my machines on the home network since most of them are Macs and find out about each other through Bonjour. Since my main Mac is called “buster”, from other Macs I get to it as “buster.local” whereever I need a host name. It’s all very nice and happens without me thinking about it.

I figured someone had probably made this available for Windows, and indeed, Apple has Bonjour for Windows. Now my Windows box is a bit more useful. Thanks Apple!

Related link: http://www.turbogears.org/bugbounty.html

If I’m reading the link I’ve referenced right, if you submit a patch to the TurboGears project (a web development framework for the Python language) for one of the bugs with a “Develix” keyword and the patch gets accepted, Develix will reward the patch submitter with one year of free hosting.

I assume that it’s somehow important enough to Develix that these specific bugs get fixed that they are willing to set a bounty so that someone will fix them. If this is the case, it’s a win-win for the community. Develix gets some pertinent bugs fixed for a relatively low cost, the patcher gets free hosting, and the community gets the same bugs fixed, as well.

Related link: http://conferences.oreillynet.com/cs/os2006/create/e_sess/

For the first time, the O’Reilly Open Source Convention will feature a Microsoft Windows track. The focus of this track will hopefully capture the growing momentum behind projects like Mono, mojoPortal, iFolder, TomBoy, F-Spot, Banshee, NHibernate, NAnt, and Nunit (just to name a few).

I believe that this new track addition recognizes this growing momentum and seeks to share it with the broader open source community. Of course a lot of the momentum is due to the ongoing success of Mono. Mono is now on release 1.1.13, which marks a feature freeze point for Mono in preparation for Mono 1.2. Windows.Forms is the only piece left before they officially move to version 1.2 of Mono. Their aim is to release Windows.Forms functionality that implements the .NET 1.1 API. From the news available, the most visible missing pieces left are Multiple Document Interface (MDI) and a few RichTextBox features.

I’m writing to encourage the various project communities from these projects to submit proposals. It is important for us to respond, now that we have been given an opportunity. Think of it this way. As much as we might enjoy contributing to a project, I think most would agree that they get even more enjoyment when people use their software. The O’Reilly Open Source Convention offers the opportunity to expose your projects, your software, and yourself to a wide audience of perhaps some of the most influential people in the software industry. I can’t think of any other event that offers the same opportunity. It simply is “the” place for the open source community to meet up and connect face to face.

With the recent announcement of Mono’s inclusion in the next release of Fedora Core, it is clear that .Net related open source is growing. Now is the time to share your knowledge and love of .Net with the world, don’t hesitate, navigate to the Submit a proposal page, and take the first step!

Will this new Windows track make you attend the O’Reilly Open Source Convention?

As I’ve posted before, my wife wanted me to build her a website. Initially, I planned on building it using Plain Old HTML. It was going to be a plain storefront and customers would phone in orders. Then she decided that it would be more convenient if they could upload their images to us rather than email them. CGI would work perfectly for that. Then, we thought that maybe a store catalog and integrated shopping cart would be cool. I started digging into PHP for that. I shied away from TurboGears because I thought hosting would be a problem. After looking around, I decided that hosting was a non-issue, so I built her site in TurboGears.

I settled on Dreamhost for hosting because of price and FastCGI support. FastCGI is one of a handful of methods for deploying Turbogears in a hosted environment. FastCGI has been a source of frustration for me during this process and I don’t expect the frustration to go away any time soon. It just seems really quirky.

I finished my wife’s site yesterday. We did a final walkthrough of the site and I did a few finishing changes. I then began the “deployment to production” process last night. I followed the instructions on the “Installing TurboGears on Dreamhost” wiki.

Thus begins my frustration. Copy my files over. Not a problem. Modify the tg_fastcgi.fcgi script. Not a problem. Make a couple of changes to my TurboGears config file. Not a problem. Drop in a .htaccess file. Not a problem. Test that tg_fastcgi.fcgi runs properly from the command line. Not a problem. Point my browser at my site and get it to kick off the FastCGI process(es). Hmmm. It looked like it was trying to start something. I saw CPU utilization increase, but not on any process I had access to view. Then after what seemed like forever, as if by magic, there were maybe a dozen tg_fastcgi.fcgi processes running. That was liberating. The site was running. And it was pretty snappy, too.

There didn’t appear to be any obvious problems. Except when I needed to change something, then I had to “killall“ the tg_fastcgi.fcgi processes so the change would take effect. FastCGI is apparently more finnicky starting up right after you’ve just killed it. I again saw some unknown process eat a little CPU and then there were entries appearing in my log file that looked like this:

[Wed Jan 18 08:09:17 2006] [error] [client ] FastCGI: incomplete headers (0 bytes) received from server “/home/(my account)/(my domain)/tg_fastcgi.fcgi”

And then, after a while, it just magically came up.

Again, no obvious problems. Except when I added an item to my shopping cart. When I went to view my cart, there was nothing there. Then, when I clicked “View Cart” again, there was my item. Click again and it’s gone. Click again and it’s there again. Round and round we go. I’ve created a magical disappearing-reappearing shopping cart! Cool! Wait. Not cool. Customers won’t like that. Neither would my wife. I figured that the problem may be caused by the multiple tg_fastcgi.fcgi processes not sharing session data properly. Aarggghh. I switched over from using RAM as session storage to file-based session storage. The problem immediately went away.

Then I started getting 500 errors and entries in my log file that look like this:

server.log: self._lockFile(lockFilePath)
server.log: File “/home/(my account)/lib/lib/python2.4/site-packages/CherryPy-2.1.1-py2.4.egg/cherrypy/lib/filter/sessionfilter.py”, line 345, in _lockFile
server.log: raise SessionDeadlockError()
server.log:SessionDeadlockError

And 500 errors in the browser. And an unusable website. There appears to have been a bug entered against CherryPy which was supposed fix this problem. Maybe I hit a corner case. I don’t know. But it looks like another session-oriented issue. Maybe FastCGI isn’t playing nicely with the session storage files.

So, I have a web application which is difficult to modify quickly because FastCGI doesn’t appear to have a nice “restart” option. (If someone knows of one, I’d appreciate you posting it here. I found a reference to giving a “killall -USR1“, but I really don’t want to try that right now. The server is running OK for the moment). It seemingly randomly spews 500 errors and has session deadlocking issues. There is also sometimes a significant lag during the first request after there have been no requests for a while. The site has been (mostly) fun building. Deployment has been a beast, though.

I’m not blaming Dreamhost or TurboGears or FastCGI or CherryPy or anything else. I’m just venting a bit. It’s good to do that every once in a while. I guess tonight I’ll start trying to find solutions to the relevant problems.

Related link: http://gplv3.fsf.org/

The General Public License covers some of the most important software
in widespread use: Linux, MySQL (dual-licensed by the vendor), Samba,
and many other modern packages, not to forget the suite of compiler
tools and command-line utilities from the Free Software Foundation,
for which the GPL was originally designed.

That’s why hundreds of people came to hear Richard Stallman and Eben
Moglen (law professor and general counsel for the FSF) lay out their
proposed new version of GPL on Monday. And why the audience included
world-renowned leaders from many free software projects–even some
projects such as Apache that aren’t covered by the GPL.

After the dramatic convocation
I reported on,
where the veil was lifted on the hitherto secret draft of the GPLv3,
the packed hall at MIT thinned out over the next day and a half. Most
people got what they needed at the opening session: they found that
the draft opposed patents and Digital Rights management, as expected,
but that it made no drastic changes in reaction to these threats or
other changes in the software field, and that the draft was graciously
accommodating to Application Service Providers (who could have
expected their trade secrets to come under attack), to software under
non-GPL licenses, and to companies acting in good faith to propagate
and make a business from GPL-covered software.

These attendees probably also realized that further work on the GPL
was going to descend into detailed textual analysis requiring both
sophistication and dedication. And most people were ready to leave
that to the committees set up by the FSF.

But don’t fade into the background. It’s easy to view the GPL and the ongoing
discussion about it
through the nicely designed
website,
including a Javascript-driven
comment area.
Moglen has urged the public to stay involved. And open source
proponent Bruce Perens offered several reasons to follow the upcoming
year of GPL discussion:

• This is a rare chance for the public to make law.

• His committee found many problems with the draft; it will need a lot
  of work.

• Feedback will be taken seriously. Bruce’s committee, at least,
  promises to review comments carefully.

So try visiting the site from time to time and take at least an hour
to look at where discussion is heading. Don’t expect to reverse the
philosophy behind the endeavor–Bruce believes “the intent of the
document is sound”–but help to avoid unexpected harm.

Behind the GPL version 3

So what’s in the GPLv3? The actual
draft license
is not that hard to read, and a
rationale document
helps to explain it. Still, I feel it worthwhile to summarize some of
the more interesting points:

Compatibility with other free licenses

The drafters made changes that allow programmers to combine
GPL-covered code with code from other projects. The Apache and Eclipse
licenses were explicitly mentioned as compatible. This outreach is
particularly praiseworthy because those two projects offer key support
to Java programmers, and some others in the free software movement
have sometimes expressed distrust of Java. This change should help
everybody work together.

Patents

These are mentioned four places in the draft. The goals here are
modest: essentially, to force programmers to relinquish patent-related
controls if they use free software. If they have patents on free
software, they must give a patent license to anyone using it. If they
have cross-licensed patents or otherwise gained rights to use patents,
they must help spread this protection to the users of their software.

Digital Rights Management

The goals here are also modest: to make sure free software and DRM are
not used together–in short, to prevent freedom from being used
against itself. First, users are forbidden from closing off access to
works through encryption or authentication keys. (This doesn’t cover
legitimate uses of encryption and authentication for privacy
purposes.) Another clause attacks the notorious “technical
circumvention” measures in the Digital Millennium Copyright Act and
copycat treaties and laws, ruling out the use of GPL-covered software
to carry out the measures.

Tracking infringement

Previous versions of the GPL had built-in termination of the license
if a propagator infringed on it. This minimized the need for copyleft
holders to police users, but it placed a burden on vendors and other
users trying to build systems on free software. They might infringe
unknowingly and have the carpet pulled out from under them at any
time.

Version 3 requires the copyleft holder to notify an infringer within
60 days of the occurrence. The new clause provides protection for
people trying to build a business. It also demonstrates a confidence
by the drafters that the free software community has matured enough to
invest the necessary resources to check up on users.

Provision for additional restrictions

If copyright holders want to go further than the GPL in trying to open
up software–by requiring Application Service Providers to reveal the
code running on their servers, or to retaliate against patent
holders–they are explicitly allowed to do so. These clauses allow
people to experiment with their own solutions to what the Free
Software Foundation sees as problems, but for which it currently does
not see effective remedies.

There are many, many more details. The drafters have learned over the
years which clauses of the GPL have created confusion or prevented
people from doing useful things. The license has also been drafted
with more care to making it applicable in different countries.

Zak Greant, a volunteer who answers licensing questions for the FSF,
told me he is happy with the new draft, finding it both clearer and
more comprehensive. While he currently has to refer people to a FAQ or
other ancillary documents to answer questions, he estimates that 70%
of the questions now could be answered by the legal document itself.
I hope the preceding list makes you curious enough to check out the
official FSF site.

And Beyond

All that said, I took away from the conference a pessimistic
impression that the GPL is not the battlefield where the information
struggles of our day will be resolved. The drafters made no
suggestion that they had solved the problems of patents, DRM, or other
threats to user’s control over information. On the contrary, they used
the conference as a forum to call for political action on these
threats.

The looming collision between the control-obsessed entertainment
industry and today’s dynamic communities of programmers and modders
will be carried out in the social realm more than the legal one. The
law may produce some of the carnage, but it will mostly come along to
clean up the debris after the victory of one side or the other.

If the public turns against Digital Rights Management–if they even
understand what it is–they will do so because of outrageous missteps
like the recent botched Sony CD controls. Even during this highly
publicized incident, it was nearly impossible to find a teachable
moment concerning the importance of user control over computer systems
and software.

I hope FSF spokesperson Peter Brown is right in saying that we have a
great opportunity to explain the benefits of freedom to the public
over the coming year. I also sympathize with his claim that one must
use the term “freedom” instead of focusing on “open source.”

But opponents of the “open source” terminology always caricature the
term and its supporters. Those who pushed for open source have
promoted its ethics and community benefits just as free software
proponents have. The virtue of “openness” as a general principle is
powerful, and has brought people out on the streets in many countries.

I admit that the words “open source” do not slam the ethical challenge
down on the table the way the word “freedom” does. But “open source”
has helped free software spread to far more places in business and
public organization. Now many more people have something to defend
when the free software proponents warn them they’re in danger of
losing it.

I think FSF knows that it needs allies; that’s why the proposed
license demonstrates so much conciliation and coalition-building. In
addition, popularity of Samba, and the presence of Samba project
leaders at conference, shows that the free software movement has
accepted the need to co-exist with a non-free world, at least for a
while.

The FSF has reacted to the encroachment of outside control by trying
to exert forms of control all their own. They have often been
criticized for this, and I don’t want to rehash the flames wars here.
But after the license assails software patents and DRM, it goes on to
impose a ban on “works that illegally invade users’ privacy.” This
makes some sense in context (because some forms of DRM snoop on users)
but one wonders when the FSF is going to stop.

Why not keep going and ban the use of free software, for instance, to
promulgate racism? The obvious answer to this question is that it’s
hard to define what constitutes promulgating racism, and that banning
it would lead to encroachments on other activities that are
beneficial. But the same dilemmas dog the FSF as it tries to fend off
patents and DRM.

The other way to approach free software is the old BSD way of throwing
open the doors and allowing proprietary vendors to enfold the software
into closed products. Proponents of the BSD approach have made a
strong argument: if the free software movement is really a superior
way to treat software and its users, the free versions of the software
will ultimately win out over the proprietary ones. After all, who
could turn down the free software promise of open source code, a
community of experts to turn to for support, and a stream of new
features that will automatically interoperate on different systems?

History seems to bear out this argument, but once again, I’m not
writing this to revive an old debate. I’m putting it here to show that
the fate of free software depends on the reactions of the general
public.

It’s good for programmers to have a choice. For those who
feel it safer to require the unencumbered freedom of what they’ve
produced, the GPL should be as robust and usable as possible. The year
2006 is our year to make it so.

Related link: http://perlcast.com/2006/01/17/interview-with-brian-d-foy-about-the-winter-2005-…

Josh McAdams of Perlcast interviews me about the latest issue of The Perl Review.

If you don’t want to listen to me (I sure don’t!), Perlcast also has interviews with Slash-programmer Chris Nandor, Learning Perl author Randal Schwartz, Perl creator Larry Wall, and many other Perl names.

Related link: http://gplv3.fsf.org/

We got it just a few hours ago–the proposed GPL 3 license. Most of
the world got it from a web site, while a lucky few hundred of us got
it at a formal meeting at MIT,

Lots of observers wondered how Richard Stallman, Eben Moglen, and
their advisers would handle such hot issues as remote services (called
Application Service Providers in the 1990s) and patents. Surprisingly,
the license embodies both the conservatism and the room for
experimentation for which we can take U.S. law as a metaphor.

There’s a big right to innovate in law, as in everything else, in the
United States. The right to make law is divided among the national,
state and local governments. For instance, states vary widely in tax
schemes, health insurance provisions, abortion controls, environmental
protections, and other things. This latitude is important not only
because different regions have different needs, but because an
experiment in one state can prove whether something is a good idea,
and can then be adopted at the national level.

The designers of the proposed GPL took a similar open approach in
remote services and patent retaliation. On both issues, the proposed
GPL upgrade takes a middle ground.

Thus, it makes no change that would restrict remote services from
using free software. This is wise in my opinion, because no
reasonable observer would want to drive Google (for instance) away
from free software by requiring them to release all the code that
implements their ranking algorithms.

But the proposed GPL leaves an opening for experimentation: it allows
people to add clauses that would require remote services to propagate
their source code.

This means that if you think you have a smashing good restriction that
would help the public by encouraging remote services to share their
software, and you have a valuable program these services might want to
use, you can release your code under the new GPL and add in your pet
cause. If you strike it lucky and your software is so valuable that
services want it, they will comply with your restriction.

That means there’s a market for legal innovation in the GPL. If others
in the free software community decide your clause led to more benefit
than harm, they’ll start adding it to their own licenses. And
eventually, I assume, after several years of success, the guardians of
the GPL may incorporate your clause into a new version of the GPL.

Similarly, the GPL designers took a much more modest approach to
patents than many people expected. The GPL itself includes a handful
of limited clauses.

Thus, if you have a patent on any software you release under the GPL,
you are granting a patent license without encumbrances on everyone who
uses the software.

Furthermore, if you have a patent license yourself (obtained by
cross-licensing, for instance) on software you release under the GPL,
you have to “act to shield” all users of that software. This is a
vague clause that Moglen hopes to tighten up after discussion.

Neither of these clauses address the most common situations where
holders of patents swoop down to attack free software. But clearly,
Moglen’s years of research into patents have not persuaded him that he
can provide an effective defense against this in a license (or, I
imagine, in patent pools and other mechanisms).

But again, a clause in the GPL 3 allows other people to impose patent
retaliation. This can provide a legal prop for efforts such as patent
pools. We will see how well they function over time.

Meanwhile (as several people at the conference have stressed) we need
to continue to fight software patents on a policy level. In the
European Union, software patent proponents react to every defeat in
every legal forum by finding another legal forum to bring the issue
back to life.

In contrast to the tentative steps toward handling remote services and

patents, the GPL comes out very strongly against Digital Rights
Management, even the term for which Stallman objects to. (No law gives
a copyright-holder or broadcaster rights to impose the restrictions
that DRM usually imposes.) And the new GPL contains a complicated
clause targeted at DRM. As I read it, the clause requires the sharing
of any key that controls access, thus rendering the key useless for
such control and making access equally available to all.

The conference was buzzing long before the opening statements and has
been buzzing ever since, but I wonder how much more we’ll learn, or be
able to improve on the proposal, during the next day and a half. In my
opinion, Moglen did a stupendous job presenting the meaning and
reasons for the clauses. Thoughtful responses will take weeks or
months to emerge, and the proposal is open now for world discussion.

I also heard from Free Software Foundation staff that more conferences
like this one are being planned, one for somewhere in Latin America
and another in Europe. Stallman apologized for holding this conference
in the United States, explaining that they couldn’t arrange an
alternative and listing diverse ways that people were prevented from
visiting the United States (or refused to come and be subjected to
harrassment at the consulate or the airport).

Meanwhile, it’s a whale of a conference. The weather is cold outside
but the atmosphere is popping in the conference hall, which is full to
capacity. I think I’ve never been with so many people I know in one
place, including my own wedding. The tone is very constructive.

People who have always hated the GPL will show no new warmth to the
new version. People who have used the GPL, I predict, will move to the
new one. The changes are relatively conservative, in my opinion, and
the ones that take the most risk are doing so for causes that all of
the GPL’s supporters are united on. However, no one is forced to
move. If substantial projects stick to the GPL 2, it will represent a
failure to persuade on the part of Stallman and Moglen. But in this
matter there is always choice.

Related link: http://www.yapc.org/America/

Yet Another Perl Conference, North American edition, is in Chicago on June 26-28, 2006. They issued their call for participation during the black hole I call December.

Their website has the details of the submission process (i.e. where to send your email) and topics of interest. Since this is Chicago, you’re allowed to vote as many times as you like for your own submission.

Curiously, the first Ruby on Rails conference (RailsConf 2006) is in Chicago on June 22-25. The YAPC folks hope that they can get some of the Ruby people to stick around so they can have some sort of cross-language event.

Related link: http://mysql.he.net/doc/refman/5.0/en/charset.html

I’ve just finished one of the most difficult and tedious problems I’ve ever solved, so I have to share the solution here in a little tutorial of how I fixed this, even though I’m sure there are better ways, this is what worked for me.

THE PROBLEM - PART 1:
My old CD Baby MySQL database from 1998 was filled with foreign characters and was in MySQL’s default (latin1) encoding.
For years, customers and clients had been using our web interface to give us their names, addresses, song titles, bio, and many things in all kinds of alphabets.
I wanted everything to be in UTF-8. (The database, the website, the MySQL client, everything.)

QUICK DEFINITION : "FOREIGN CHARACTERS"
When I say "foreign characters" I mean not just Greek, Icelandic, Japanese, Chinese, Korean, and others shown at Omniglot, but also the curly-quotes, ellipsis, em-dash, and things described at alistapart.

START OF THE SOLUTION (THE EASY PART):
* - Found a few hours of downtime at 2am on a Sunday night.
* - Shut down the website.
* - Did a raw data dump (mysqldump) of the data to a regular text "dump.sql" file. (85 tables, millions of rows, an 8 gig dump)
* - Completely removed MySQL 3.2 from the system
* - Installed MySQL 5.0 (FreeBSD ports), making sure to use –with-charset=utf8 while compiling (see http://dev.mysql.com/doc/refman/5.0/en/charset-server.html)
* - Did a sed replace on the dump.sql file, changing all table types to utf8.
* - (Also changed from MyISAM to InnoDB but that’s a different story, and had no problem.)
* - Changed my HTML header Content-Type to charset=utf-8 everywhere
* - Changed /etc/my.cnf to default charset utf8
* - Loaded the dump.sql file, and turned the website back on.
* - Made sure it mostly worked, and went to sleep

THE PROBLEM - PART 2:
Some foreign characters were perfect. Others were a jumble : what should have been one quotation-mark turned into a series of THREE jumbly characters. Weird. Had to be fixed. No idea where to start.

FIGURING OUT WHAT’S WRONG (THE HARD PART):
* - Unless you want to do *everything* in a web browser, you need to get a terminal that does Unicode and can display foreign characters. I used uxterm. See http://czyborra.com/unicode/terminals.html
* - I learned about using the SET NAMES utf8 query, but when I did that almost everything turned into a jumble.
* - I could send the database a set names utf8 command, and SOME would work. Or I could do set names latin1, and the rest would work. I was stumped.
* - It took about 10 hours of frowning and furiously typing, but I found out that
— #1 : The MySQL server was using UTF8 encoding.
— #2 : The MySQL client was using latin1 encoding.
— #3 : Even if I got the command-line MySQL client to use utf8, the PHP client was still using latin1 encoding.
— #4 : Most of my data must have been put into the MySQL server with latin1 encoding, which is why it worked with latin1 encoding on the client when getting it out.

Seems I had some characters in latin1, some characters in UTF-8, some in the database as HTML equivalents (分) and some characters that were just a total mystery.

A TOOLBOX FOR SLEUTHING CHARACTER ENCODING PROBLEMS:

#1 - USE MySQL CHAR_LENGTH TO FIND ROWS WITH MULTI-BYTE CHARACTERS:
SELECT name FROM clients WHERE LENGTH(name) != CHAR_LENGTH(name);

#2 - USE MySQL HEX and PHP bin2hex
SELECT name, HEX(name) FROM clients;
Get the result back into PHP, and run a bin2hex on the string, compare it to MySQL’s hex of that same string

#3 - SEE IT IN BOTH ENCODINGS
$db->query("SET NAMES latin1");
$db->query("SELECT name, HEX(name) FROM clients");
(compare the string and its hex result from MySQL with the bin2hex from PHP)
$db->query("SET NAMES utf8");
$db->query("SELECT name, HEX(name) FROM clients");
(compare the string and its hex result from MySQL with the bin2hex from PHP)

For all those strings that looked perfect in LATIN1 encoding, here’s how I would fix them in the database:
$db->query("SET NAMES latin1");
$db->query("SELECT id, name FROM clients");
$hex = bin2hex($x[’name’]);
$db->query("SET NAMES utf8");
$db->query("UPDATE clients SET name=UNHEX($hex) WHERE id=$id")

That seemed to work, for most things.
Problem is, only SOME of the database was in latin1 encoding, so I had to use a few quirky ways, but mostly my own eyes, to fix only these things, and not accidently re-encode something that was perfect.

#4 - USE A HEX/UNHEX REPLACE FOR THE UNFIXABLE CHARACTERS
Imagine, after all that fixing, you found strings like this:

Let~!@s say ^|%What a nice house you~!@ve got here, don~!@t you think?^!%.

Who knows when or how this happened, but obviously ~!@ is meant to be an apostrophe, ^|% an open-quote, and ^!% a closing-quote.

I’d use MySQL SUBSTRING to find the 3 characters that needed replacing:
SELECT SUBSTRING(quote, 353, 3) FROM table WHERE id=1;

Once narrowing it down to the exact string, add a HEX() around it:
SELECT HEX(SUBSTRING(quote, 353, 3)) FROM table WHERE id=1;
… which would give you a result like C8035EF6BB92BF2

Then use that with MySQL REGEXP to find and replace all occurences in your database!
UPDATE table SET field = REPLACE(field, UNHEX(’C8035EF6BB92BF2′), "’") WHERE field REGEXP UNHEX(’C8035EF6BB92BF2′);

I set up some PHP arrays of all my tables, and all their text fields, to run this same query on everything in my database.
Then do it again for curly-quotes and other weirnesses.

A few times, I had no idea what a character was supposed to be (like the Icelandic and Gaelic ones) - so I had to go visit the artist’s website, and find their song titles or bio information spelled correctly there.

#5 - VALIDATE UTF8
Got the is_utf8 function from PHP docs to validate all the values in the database.
Doing this found a bunch of invisible problems, which only through hours of MySQL SUBSTRING and HEX revealed that there were invisible characters with HEX values of 00-19 scattered around my text fields.
I used the same solution as above to replace them:
UPDATE table SET field = REPLACE(field, UNHEX(’05′), ‘’) WHERE field REGEXP UNHEX(’05′);
I looped this inside an array of all hex values under 20.

#6 - CONVERTING HTML ENTITIES
Find HTML entities hidden in the database:
SELECT field FROM table WHERE field REGEXP ‘&#[0-9]*;’
Use the utf8_chr function from the comments of the PHP html_entity_decode page.
Use PHP preg_match_all to find the entities inside the string, and replace them:

function myreplace($string) {
preg_match_all('/&#(\d*)/', $string, $matches);
foreach($matches[1] as $num) {
$string = str_replace("&#$num;", utf8_chr($num), $string);
}
return $string;
}

Update the database with the returned result.

After all this (about 60 hours work over the last 5 days) I think it’s all done.

Phew.

LESSON LEARNED: KEEP EVERYTHING IN UTF-8, ABSOLUTELY EVERYWHERE, FROM DAY ONE. You’ll be glad you did some day.

Go ahead. Show how smart you are. Show how my 60 hours could have been done in 5 minutes with the language of your choice instead of my mess, above.

Fedora Core 5 Test 2 is expected to be released on Monday. This is the second of three installable ISO versions leading up to the eagerly-anticipated Fedora Core 5 release in mid-March. Although there are a few rough edges, this test release is shaping up very nicely (I’m running Rawhide, the package collection from which the test releases are built, on several machines).

What’s new in Fedora Core 5?

• Mono provides a .NET-compatible development and runtime environment. Poster-child Mono apps such as Tomboy, F-spot, and Beagle make their appearance on the Core desktop. Mono was a surprise addition made just this past week.

• Xen 3 provides improved (semi-)virtualization capability.

• Multi-category security (MCS) brings discretionary labelling to the Fedora SElinux implementation. This add new security options that are useful and friendly (and therefore likely to be used).

• Package management has been greatly improved, with Yum integrated into Anaconda (the system installer) and the new Pirut (graphical package administration) and Pup (graphical updater) tools.

• The X.org X Window Server has been updated to the modular 7.0 release and packaged for flexibility when installing.

• GCC has been updated to 4.1, which brings stricter adherence to some language standards.

• Gnome has been updated to 2.12, which seems more responsive than the 2.10 version used in FC4 (now that we’re used to having a Desktop menu, it’s being changed to System!)

• Various kernel, driver, and desktop enhancements mean that more hardware runs “straight out of the box”, and version bumps on many applications provide bug fixes and new features.

In three weeks, development will be frozen, and in mid-February we’ll see Test 3. Allowing time for a few rounds of bug fixing, the final release is tentatively scheduled for March 16. (My book, Essential Fedora Linux, should appear on shelves just a few weeks later).

If you have a spare machine and a couple of hours this week, you can grab the Test 2 ISOs and preview Fedora Core 5 for yourself. Just be sure to report any bugs so they can be fixed before the final release!

Which FC5 features are you looking forward to using?

Intel-based processor? Check.

Sufficient screen size? Check.

Sufficient graphics card? Check.

Sufficient default memory and hard disk size? Check.

Bonus: Energy Star compliant, built-in iSight camera, Superdrive –check.

Affordable price? A system starting with all of the above starting at $1299, that’s very good. Check.

Run Windows, Linux, and Mac natively on one box? Well, I am not sure yet. This is the only thing that is holding me back from running out to an Apple Store today and buy a new Intel-based iMac. Last year, we saw a preview of Windows XP running on an Intel-based developer Mac (and that was no April’s Fools Day joke). Now, there is news that Windows will not run on an Intel-based Mac because of Apple’s use the extensible firmware interface (EFI), not BIOS. I do not know what to believe.

It would be a blessing if one could run all three systems natively on one box. Right now, I have a dual-boot Linux and Windows Intel box, and an iBook. I use my iBook for traveling, and to manage my media (music and photos). I use the Windows partition to primarily play games (locked down from the network). I use the Linux partition for virtualization, research, and everything else.

I am sure that I am not the only one who wants such a system. Remember Scott Granneman’s article on SecurityFocus last year, his conversation with an FBI special agent? Recall:

“…Many of the computer security folks back at FBI HQ use Macs running OS X, since those machines can do just about anything: run software for Mac, Unix, or Windows, using either a GUI or the command line. And they’re secure out of the box. In the field, however, they don’t have as much money to spend, so they have to stretch their dollars by buying WinTel-based hardware. Are you listening, Apple? The FBI wants to buy your stuff. Talk to them!”

Any confirmations of triple-booting on an Intel-based Mac?

Related link: http://lists.canonical.org/pipermail/kragen-tol/2005-April/000772.html

Kragen Sitaker summed up “enterprise software” effectively in an email last year: Enterprise software is software that gets sold to a so-called enterprise. I’m finding the argument that some programming language or framework or whatever isn’t “ready for the enterprise” dull. Aren’t you?

Don’t get me started on “scalable”….

Related link: http://www.thesmokinggun.com/archive/0104061jamesfrey1.html

This comes to my attention from Publishers Weekly, which covers the book industry and offers breaking news. I haven’t delved deep into the topic myself, but rely on these sources, below.

From James Gets Frey-ed in Blogosphere:

…

It looks as though James Frey’s “haters” and “doubters” are coming out in ever greater numbers. The response to the damning expose published by The Smoking Gun (www.thesmokinggun.com) claiming key elements of the author’s bestselling, Oprah-backed memoir A Million Little Pieces (along with its follow-up, My Friend Leonard) were either exaggerated or wholly fabricated, has hit the media and the blogosphere with quite a thud. After PW reported Monday on The Smoking Gun piece, the major media outlets ran with the story in full force on Tuesday (articles appeared in The New York Times, The AP, Reuters and USA Today among others), while Frey’s fans and readers reacted to the scandal on various online message boards.

…

The cited expose is The Man Who Conned Oprah, which says:

…

Police reports, court records, interviews with law enforcement personnel, and other sources have put the lie to many key sections of Frey’s book. The 36-year-old author, these documents and interviews show, wholly fabricated or wildly embellished details of his purported criminal career, jail terms, and status as an outlaw “wanted in three states.”

…

This is an issue only because the book is sold as truthful non-fiction. The blogs suggest that many readers feel betrayed.

I almost find it humorous. Readers are upset that this crummy guy (”430 pages chronicling every grimy and repulsive detail of his formerly debased life”) might also be a con man (or, rather, a con man instead of the described crummy guy). If I had read the book, however, I might not find this funny at all. I understand the book is supposed to be about redemption — seems it was really about money. That’s a dirty trick. Or, rather, new fuel for my cynicism.

The capper would be for the author to now write a book about conning readers into buying his books — like Malcolm McClaren’s Great Rock ‘n’ Roll Swindle … Okay, make it a movie, not a book.

Today, Publisher Weekly reports: Frey Backs Book, Winfrey Backs Frey:

Although he acknowledged on the Larry King Show last night that he changed certain details in his memoir, A Million Little Pieces, James Frey insisted that the essence of his story—of how he overcame years of addiction—was true. That stance was endorsed by Oprah Winfrey who called the King show just before it went off the air to say she viewed the controversy “as much ado about nothing.” She said that while some details may have been altered, Frey’s message of redemption “still resonates with me and I know it resonates with millions of other people who have read the book.” Winfrey somewhat distanced herself from the uproar over the book’s authenticity by pointing out that she relies on publishers to vouch for the accuracy of their works. She challenged publishers to more closely examine what they classify as fiction and nonfiction.

…

… Geoff Shandler, editor-in-chief of Time Warner Books’s Little, Brown imprint, said, more than anything else, he was disheartened to see support for Frey’s work continuing, despite the now overwhelming evidence that key elements of it were greatly exaggerated and/or entirely fictionalized. “As someone who works on a lot of nonfiction…to hear folks say this is OK because that’s the nature of memoir, that’s upsetting,” Shandler said. “There is a huge difference between subjective recall and flat out fabrication,” he added, noting that those defending A Million Little Pieces by saying Frey did what any author does, embellish the facts to make for a better read, are essentially offering up “an ends justify the means argument.”

…

Among booksellers the consensus seemed to be that, if anything, the scandal might help the already-strong sales of A Million Little Pieces and its follow-up, My Friend Leonard. As Tom Steadman, owner of The Book Depot in Mayetta, NJ, put it, “the more controversy, the better it will sell.” Steadman pointed out that the first he heard of the scandal was from a customer who bought AMLP explicitly because of the media coverage it’s now getting.

…

So a fraud becomes a bestseller, and its expose becomes a PR event. That is disheartening.

The interesting part is that traditional media has advanced the fraud, and the online media (it appears) has advanced the truth. Mad crowds notwithstanding, might this be a trend? Or does social media simply like to criticise?

Update (1/17/06)

Here is a helpful report on the topic:

Why James Frey Doesn’t Get It by author Heather King.

…

Now that the accusations of lying have surfaced and I’ve actually read the book, I see the differences go even deeper. Drama is the movement from narcissism to humility, but Frey is exactly the same at the end of his story—minus the drugs—as he is at the beginning: an insecure braggart without a spark of vitality, gratitude or fun. “A ballsy, bone-deep memoir,” Salon.com called it, but for any alcoholic worth his or her salt, throwing up blood, puking on oneself, and committing petty-ass crimes in and of themselves couldn’t be bigger yawns. What’s gritty is the moment, knowing you’re dying, when the world turns on its axis and you realize My way doesn’t work. What’s ballsy isn’t just egomaniacally recounting your misdeeds; it’s taking the trouble to find the people you’ve screwed over, looking them in the eye, and saying you’re sorry. What’s bone-deep—or might have been if Frey had done it—is figuring out that other people suffer, too, and developing some compassion for them. Oprah speaks of “the redemption of James Frey”—but redeemed from what, and by whom? Sobriety, in my experience, isn’t the staged melodrama of sitting in a bar and staring down a drink to prove you’ve “won”—as Frey does upon leaving rehab. It’s the ongoing attempt, knowing in advance you’ll fall woefully short, to order your life around honesty, integrity, faith.

…

As John Cheever said, “I lie, in order to tell a greater truth.” But Frey lied to tell a lesser truth: he lied to make himself look like a hero. He lied—about the crimes he supposedly perpetrated and the tragedies that befell him—because on the one hand he wants the reader to feel sorry for him, and on the other he wants to be held in awe. …

Update (1/26/06)

Oprah takes Frey to task, publisher too:

James Frey and publishing standards were both in the hot seat this morning when Oprah Winfrey brought in the author and his editor Nan Talese to talk about the controversey swirling around Frey’s A Million Little Pieces. Saying she felt “duped” by Frey, Winfrey said she had allowed her feelings about the book and Frey’s strong relationship with her producers to cloud her judgement about the author when she called in to defend him on the Larry King Show earlier this month. “I made a mistake and left the impression that the truth does not matter, and I am deeply sorry about that,” Winfrey said.

…

Related link: http://www.djangoproject.com/snakesandrubies/

Snakes and Rubies was a meeting held in December to allow developers from the Ruby on Rails and Django projects to present their respective projects and discuss them in an open forum.

If you’ve been reading some of my posts recently, you’ll know that acceptance of diversity in the programming, especially open source, community is really important to me. Maybe that’s why I liked the “Snakes and Rubies” video so much. Or maybe it was David Heinemeier Hansson’s (of Ruby) hilariously opinionated comments during the Q&A portion of the forum. Or maybe it was getting to hear Adrian Holovaty talk about some of the ideas behind Django, what it’s been used for, and what “batteries” it has included.

Regardless of exactly what it was, the “Snakes and Rubies” video was a highly entertaining, extremely informative piece of media. It was well worth the time (3 hours!) I invested to watch it. Up until this video, I had not taken more than a cursory glance at Django. I’m glad this video came along to remedy that.

I’ve been using TurboGears since about a week from its public release and previously had been using CherryPy since around (I think) version 0.7, so those are the Python web frameworks I’m most familiar with. Django takes a completely different perspective from either CherryPy or TurboGears. Django tries to be an all-in-one project, while CherryPy is a generic web framework. TurboGears tries to be an all-in-one project as well, but TurboGears pulls in pieces from different projects (CherryPy, MochiKit, SQLObject, Kid), whereas Django developed its pieces in house.

At this point, many readers will invariably raise mental red flags against Django because it is violating the DRY (Don’t Repeat Yourself) principle because it isn’t re-using existing technology. (I know, they’re not repeating themselves as much as they are repeating work which has already been done. But it’s the spirit of the thing.) One of David Heinemeier Hansson’s comments during “Snakes and Rubies” was something to the effect that code re-use is overrated. That got me thinking. I’m not pronouncing anyone right or wrong at this point. I’m still pondering the thought. I’m merely raising the question, what if re-creating existing code sometimes isn’t as evil as we’ve been taught? What if the proliferation of web frameworks in Python is actually a good thing? I can think of at least a few reasons why it’s a good thing. I can also think of several situations where re-creating existing code can be pure evil, as well. Maybe I’ll blog about it if I can codify my thoughts and get it down into 0 and 1s.

Is “re-inventing the wheel” really as evil as “the experts” say?

Related link: http://www.djangoproject.com/weblog/2006/jan/11/091/

Django is a web application development framework with similarities to Ruby on Rails and TurboGears. I’ve been tinkering around with it over the past few days and am pretty impressed with it. It is definitely a framework worth considering for your next web project.

Related link: http://ipython.scipy.org/

The announcement is here and downloads are here.

Related link: http://www.cherrypy.org/

According to the link above, the vulnerability allowed clients to retrieve any file on the server’s filesystem which the CP server had priveleges to read by requesting URLs with “..”. If you’re running CherryPy, you probably want to update ASAP.

For those of you who don’t know what CherryPy is, it’s a web application development framework. From the CherryPy website,

CherryPy is a pythonic, object-oriented web development framework. It provides the foundation over which complex web-based applications can be written, with little or no knowledge of the underlying protocols. CherryPy allows developers to build web applications in much the same way they would build any other object-oriented Python program. This usually results in smaller source code developed in less time.

Related link: http://blog.chris.tylers.info/index.php?/archives/16-Its-Time-for-a-New-Power-St…

I was interested to read a report in The Register of Larry Page’s CES speech, where (among many other, more notable things) he advocated a new standard for electrical power cabling for PCs and peripherals.

I didn’t hear his speech, but the reports suggest that his ideas agree with one of my blog entries from a couple of weeks ago: that it’s time to get rid of the dozens of AC-to-DC power bricks that litter our homes and offices and adopt a new power standard, where a high-quality supply (I called it a ‘power hub’) would supply DC to all of our gadgets in a more efficient, organized, and reliable manner (more thoughts on the details).

If a UPS manufacturer (APC?) and some PSU makers got together to kickstart such a standard, I’m sure that makers of other equipment - hubs, routers, scanners, printers, DSL modems - would find it hard to resist shaving a few dollars off of their per-unit costs by leaving out the power brick.

If you’re as fed up with the tangle of wires under your desk as I am, let’s gently lean on the hardware companies to make it happen!

Do you agree that a new power standard is needed? What would it look like?

Related link: http://www.blueskyonmars.com/2006/01/05/turbogears-087-released/

Here’s what the post says TG 0.8.7 includes:

This update primarily solves installation issues and is not required otherwise.

* quickstart corrected to properly produce egg-info directories (previously, directories could be created with “-” when it should have a “_”). Note that setuptools 0.6a9 will warn you if you have a - in your egg-info directory name. Just rename the directory, and you’ll be fine.
* Installation issues that people may have had earlier are resolved in this setuptools update.
* version number set to 0.8.7 to reflect that this is considered the “stable” version of TurboGears vs. the current 0.9 code in svn.

Project Updates

* setuptools 0.6a9
* sqlobject 0.7.1dev_r1457 (updated to handle the setuptools change, but also includes other bugfixes)

But of course, you’ll want to be running the SVN version :-)

I made my first razr mod. I’m proud of myself, but if you’re already a phone hacker this post is going to seem really old and “been there done that”. However, along the way I had to track down quite a bit of information, and nobody really presented the complete solution or the whole story.

Let’s back up a bit. I have a T-Mobile razr V3. For some reason, T-Mobile didn’t enable (or disabled?) the “Email Message” menu item in the “Messages” menu. Everything still works, but you can’t email simply because you can’t ever select it as an option. Well, that sucks. My moblog stopped working when I stopped using my Nokia 3650 simply because I couldn’t figure out how to send mail from my new razr. I felt really lame telling people who asked about it that I was too dim to get email working on my phone.

Everything I needed to know to enable email was on the net, although it’s scattered all over the place. First, I had to figure out that I should Google for “razr” instead of “rzor”. I keep making that mistake, and so do some other people apparently since I got some interesting hits.

Once I figured out how to properly misspell “razor”, I found quite a lot of good information on Howard Forums, including Help getting razr email client enabled on t-mobile and SEEM edit for email client on T-Mobile V3 RAZR?. I’ll come back to those in a moment.

Before I could do anything, I needed a couple of things: a Windows machine, Motorola PhoneTools ($29.95 if you didn’t get it on a CD with your phone!) to allow the computer to talk to the phone, and Motokit 1.06, which I had to google as “motokit106.exe”. I finally downloaded it from http://www.kempsun.com/Motokit/motokit106.exe after finding many dead links and only getting that link from a dead page that Google had cached. Yep, I downloaded and installed a completely untrusted and highly probable exploit kit: if these kids are making mod-kits for phones, who knows what else they are doing. I felt adventurous so I went for it. Once I downloaded everything I disconnected the PC from the network just in case.

The Motorola PhoneTools installs easily and does its magic. I got a shortcut on the Desktop. Whoopee. I connect the phone to the computer with a USB cable and sync the phone book and calendar. Remember, I’m about to destroy my phone probably so I’m going to back up everything. The PhoneTools look like it has some other interesting features, but I’m not that curious.

The Motokit was a bit tougher to install since the file I found was set to use Chinese by default. Have I ever mentioned that on my Windows box I use the Gaelic version of Firefox, or that I don’t read Gaelic (yet)? I did this entire step without seeing any English. After a few tries I had the installer on my computer and I let it do it’s thing. It installed as C:\Program Files\MotoKit and that’s where I found the program (no shortcut showed up on the desktop). I launched the program and discovered that the “Option” menu item (third from the left if you’re still seeing Chinese) lets me select an alternate language. Motokit also comes with a US-English language file (in …\Motokit\Languages), so I can either select it with the menu or simply delete the Chinese language file (which a lot of sources suggest).

When I start up Motokit, I get a window with some icons across the top, a hierarchical tree in the left pane, and some info in the right pane. For some reason, my Windows machine doesn’t want to do screenshots today, so you’re stuck with my prose (unless I figure it out later or you want to see a very similar window at the bottom of this page). The middle icon across the top is the “Backup/Recover”. I backup everything before I do anything. Motokit will save each backup as a separate archive, similar to a system restore point. If I really mess up, I should be able to reload my phone with something that works. The backup takes a bit, so here’s a good place to get a cup of coffee.

Once I have the backup (and I verify that Motokit can restore it by actually going through most of the Recover process), I’m ready to add email to my phone. Under the “VSeries” folder in the left pane is “Menu Setting”. I select that and click the “Launch” button in the right pane. A pop-up window with a bunch of checkboxes and labels appears, and the checkbox next to “Email Messages” isn’t checked. I checked it and click one of the buttons: I think the language file somehow didn’t reach this point because I can’t read what the buttons say. However, I learn that one is “Cancel” and one is “OK” (duh). When I click the “OK” button, Motokit tells me that it’s going to restart my phone. This is were I either make my life really miserable, or I get an email-enabled phone.

My phone restarts and things seem a bit odd at first because my wallpaper doesn’t show up and the fonts seem a little funny, but 10 seconds later it’s all back to normal. I look in my “Messages” menu, and there’s “Email Msgs” at the bottom of the list. The first time I select it, it tells me I have to configure the email settings.
I have to enter a lot of information (quite annoying with the keypad), and most of it is easy to figure out. For the things that weren’t so easy, I got some information from another post in HowardForums, ALL T-Mobile Configurations. The POP3 (or IMAP) information comes from your mail provider.

Since I want to use the GPRS connection, in the ISP settings I choose GPRS (over CSD), and for the APN address I use wap.voicestream.com. I leave blank everything else under ISP settings. I discovered that if I don’t configure this portion, the phone will repeatedly prompt me to setup the email settings when I try to select “Email Msgs”.

From there I’m all set. I give it a try by sending a picture to my GMail account, and it gets there right away. Hey, that wasn’t so bad. It only took me two hours to get it working.

There is more I can do though, but I’m satisfied for now. On the phone, under “Email Msg Setup/Security”, I can enable SSL support (or so it says), and “Email Msg Setup/Protocol” I can choose between POP3 and IMAP.

Related link: http://www.blueskyonmars.com/2006/01/05/frameworks-matter-too/

The more I interact with Kevin Dangoor by reading his blog posts, getting answers to my questions on the TG list, or reading his answers to others, the more I like this guy.

Kevin responded earlier today to Peter Hunt’s post about Python winning the web. The gist of Peter’s post was that we as a Python community are spending a lot of time trying to create the dominant web framework, but we’re not going to win by doing so. We need to look at how we are already winning and focus on that, which is creating a number of not-neccesarily-connected, excellent applications. Peter’s post was pretty mild (opinionated, obviously, but mild) and seemed to be an attempt at genuinely expressing his thoughts. +1 for Peter for taking a potentially controversial topic and handling it quite reasonably.

Kevin disagreed with Peter in the same manner that Peter began - opinionated, but quite rational. +1 for Kevin for responding as he did. Here’s the funny thing, though. I’ve re-read Kevin’s post and he never (other than quoting Peter) mentioned “winning the web”. If I can extract an essence of the spirit of what Kevin replied with, this is it:

But, I like programming in Python. Lots of other people do, too. And I want to have a productive time programming in Python. So, there’s value in a framework like TurboGears.

I like this perspective. I always get the feeling from Kevin that he’s focused on excellence. He doesn’t seem to be out to “win” something. Here is a snippet from his blog post on What TurboGears is not:

If what you want is to pick and choose every piece of the application stack, then the web framework you want is CherryPy or, I guess, one of the other controller frameworks. If one tried to support everything in order to become the web framework “to rule them all”, you’d eventually end up with just Python (or some hideous layer on top of Python) and the current collection of fine components available for Python.

TurboGears is not going to go that route. Here are the overarching thoughts that go in to TurboGears’ evolution:

1. best-of-breed
2. one way to do it

He just seems to be focused on creating an excellent framework for his own use and for the good of the community. I’m pretty sure very little thought went into making TurboGears “better than Ruby on Rails”. And I get the impression that’s not a driving factor now.

Life isn’t always about winning. I think Kevin gets that. I would rather be doing something excellent for the sake of excellence than to be on the “winning team”. If I wanted to be on the “winning team”, I’d probably be learning Ruby right now. But I believe that TurboGears is a truly excellent piece of work which I’m excited about using (and hopefully improving). Maybe one day, the excellence will speak for itself and people will be drawn en masse to TurboGears. But, then again, maybe not. And if not, that’s OK with me. I’ve had a great time with it so far and expect to have a great time in the future.

So, is it wrong to want to evangelize and promote and market and try to gain a dominant share in the web framework space (or any other space)? I really don’t think so. I just think that a proper perspective is helpful. If we have something worth sharing, let’s share it. If we have something excellent, let’s share even more so. And we can hope for wider adoption for the sake of excellence being spread. And we can even hope for wider adoption for the same of building a strong community because that will benefit us as individuals.

I guess what set off this post is that when I hear talk of “winning”, I see the dividing lines being drawn and the religious battle beginning and the “us against them” attitudes rising. I don’t like that. Not a bit. It’s OK with me if not everyone uses the technology stack that I choose to use. They can produce excellence without my technology. A lot of it comes down to preference.

I don’t want to accuse Peter of this politicized “wanting to win”. I’m not sure that’s his intent. I’m guessing he meant that we have something excellent in Python and has a desire for others to see it and use it for the sake of excellence. I think that’s a great attitude. I’m with him if that’s the case.

And I believe that if TurboGears continues as it has, it will continue to draw people to it, because it is excellent.

Related link: http://deliciouspython.python-hosting.com/

I was browsing through del.icio.us and came across a Python API for del.icio.us. I thought it would link me back to an official del.icio.us project, but it took me to the link I’ve posted. It looks a little alpha, but if you have a need for such a thing, you might want to keep your eye on this project.

I have reported in detail, in a companion blog, about an
historic public forum on NSA wiretapping.
Here I’ll report on one technology-related aspect of particular
interest to me: the collusion of the telephone companies, which has
not been played up in the press.

All the warrantless wiretapping we’ve recently heard about required
help from the telephone companies and Internet service
providers. These companies knew they were not only aiding the
government in breaking the law, but were themselves violating terms of
service for their customers–and in the case of telephone companies,
also breaking the law. One law mentioned at the public form (and
submitted years ago by the forum’s moderator, Congressman Ed Markey)
forbids cell phone companies from revealing the location of cell phone
users–except with a court warrant.

In fact, the NSA wiretapping scandal represents one of the largest
conspiracies in recent years: a conspiracy between telephone companies
and the government to defraud Americans out of our Fourth Amendment
rights.

Pertaining to this is the issue of industry concentration–the death
of small phone companies and the mergers of larger ones into
behemoths–which was also one of the goals of the Bush administration,
pursued with determination by Michael Powell as FCC chair. Provisions
for competition set up in the Telecom Act of 1996, and enforced by
relatively even-handed regulations passed by earlier FCCs, were
systematically weakened and discarded under Bush. (For some history,
see an
earlier blog of mine.)

Admittedly, it’s hard for any company to buck a demand from law
enforcement. The PATRIOT Act’s secrecy provisions (when the FBI
approaches you, you can’t even publicize the very fact that they have
done so) leaves the impression that you’ll be prosecuted for going
public with government misbehavior, and thus contributes to the
growing unaccountability of government. A few Internet service
providers have done challenged illegal wiretaps, but not enough to
establish the pattern we now see in the wiretap scandal.
Overwhelmingly, the phone companies and ISPs just went along.

One might argue that the pressure would have been even stronger if
ISPs and phone companies were smaller, but size obviously hasn’t
helped them put up any resistance. Believe me, if we had an industry
of scrappy Mom-and-Pop providers like in the 80s and 90s, word about
this civil liberties horror would have come out sooner.

The 300+ seats were filled to capacity last night for an ACLU
emergency meeting on wiretapping, held in Lexington, Massachusetts. It
has been years since an event made me so angry–and even longer since
an event made me so inspired.

The moderator was Congressman Ed Markey, and the two speakers were
Marc Rotenberg, Executive Director of the Electronic Privacy
Information Center, and Carol Rose, Executive Director of the American
Civil Liberties Union of Massachusetts (ACLU-MA). All of them were
extraordinarily eloquent and to the point, having learned over the
years how to explain complex legal and technical points in precise and
constructive ways.

As far as I know, this is the first open meeting concerning the
scandal that’s rocking the country. There were many local touches,
such as being called a “Town Meeting” in an old New England tradition,
and some corny references to Lexington as the cradle of American
democracy. But Markey promised that many more such meetings will take
place around the country.

The panelists included no one from the other side–no proponents of
unmonitored wiretapping–but since George W. Bush has said that
discussing the issue is “shameful” and aids the enemy, they presumably
had nothing to say anyway. Not only were these forces absent from the
stage, they were also completely absent from the audience, judging
from the comments, applause, and the unanimously negative response
when Markey asked whether the person who revealed the wiretapping was
a criminal.

But never fear–the statements and justifications of the
administration were amply represented during the presentation.

I assume readers of this blog are well-educated on the issues, or can
retrieve the main background from other news sources, so I’ll just
list some of the less well-known aspects that came up.

The key demands that the public needs to make now are for
Congressional hearings to uncover what the NSA is doing and what the
administration told it to do, plus an independent special prosecutor
with subpoena powers to launch an investigation.

Rotenberg pointed out that we’re at an amazing historical confluence:
the greatest Constitutional crisis since Watergate in the form of this
wiretapping scandal, a debate about elements of the Patriot Act in
Congress, and upcoming hearings for a Supreme Court candidate whose
record suggests he doesn’t believe in the key doctrine at stake in
this issue–judicial oversight of the executive branch, particularly
in the area of civil liberties.

(Personal comment: I’ve worked with Rotenberg’s group,
EPIC,
for years, and I’ve found they have an amazing reach and depth. As a
separate plug, I’d like to note they were an outgrowth of an
organization I’m a member of,
Computer Professionals for Social Responsibility.)

Markey pointed to a strong libertarian strain among Republicans that
makes many Congressmen and Senators as disgusted as last night’s
audience is by Bush’s bypassing of oversight in wiretapping. Markey,
speaking for Congress, said “The President is going to be on the run
from the moment we get back to Washington,” drawing applause so
protracted that I wasn’t sure it would ever stop.

Strong applause also greeted a call for impeachment from the audience,
but Markey pointed out that the whole point of our protest is to
uphold the notion of a meaningful process, and that our process in
this case must start with investigations to establish the facts.

Markey also reminded us that a Congressional election is coming up in
ten months, and said that electing more Democrats would put even more
pressure on Bush and lead to hearings that determine the
truth. (Personal comment: I can’t say I have Markey’s confidence in
the Democratic party as a whole, as much as I admire his own record.)

It was repeatedly pointed out that the court Bush was
bypassing–Foreign Intelligence Surveillance Act (FISA)–is a
rubber-stamp court, turning down only 4 of the 19,000 wiretaps that
law enforcement has asked for since its founding in the 1978 law. They
act in total secrecy and hear arguments only from those requesting
wiretaps, not from any opponents. Law enforcement can even do wiretaps
first and ask permission later, if they feel it necessary. So why did
the administration feel it had to bypass even this fully captured
body?

Well, the FISA judges did occasionally question or call for changes in
wiretap requests. But the key problem was that the kind of wiretapping
the NSA is doing is so broad that it can’t even be represented as a
request for wiretapping. The NSA can’t say who they’re spying on or
why. They’re doing massive data mining–just opening up a funnel take
in everything they can and using heuristics to look for suspicious
patterns.

(Personal comment: Anyone with an interest in the subject has known
for eight years that this spying is going on. The project was called
Echelon and was first revealed by minor journalistic outfits in
1998. It was credited a couple years ago in the capture of an Al Qaida
operative in Afghanistan. So it’s a bit disingenuous for people in
positions of power to express shock over the New York Times
revelations, just as it is disingenuous to suggest that recent
revelations of torture under U.S. auspices are, well, revelatory.)

Speaking of revelations, the administration is taking intimidation to
a new level by claiming they want to criminally prosecute not only the
person who leaked the information about the NSA, but the journalists.
This has no precedent since threats from Richard Nixon during
Watergate. And it seems a lousy payback (my personal comment) for the
New York Times, which did Bush about the biggest favor any publication
ever performed for him by suppressing news of the NSA wiretaps prior
to the 2004 election. Rose said we should back the New York Times in
defense of independent media, while expressing to them our outrage
that they sat on this news of such public importance.

Rose pointed out that in Massachusetts we seem to have our own
mini-echelon. Governor Romney has set up something he calls a “fusion
center” that combines information from law enforcement and various
other sources of differing accuracy. No one knows exactly what it’s
supposed to do–the ACLU is trying to find out–but the mission seems
similar to the NSA stuff.

Rose raised the most philosophically significant question of the
evening (aside from Constitutional issues) when she pointed out that
the public has to discuss and understand the meaning of the vast
data-capturing and data-mining technology the NSA has apparently
developed. The technology clearly breaks all the assumptions that led
to our historic laws on wiretapping–so what should we do now? The key
point, so far as Bush’s behavior goes, is that any changes to current
practice have to be made by Congress. His administration decided they
couldn’t get Congress to approve blanket wiretapping, so they did it
in secret instead.

I have stressed the rhetorical and spectacular aspects of the
presentation, but much of the evening was devoted to a detailed and
cautious examination of eight claims by Bush and his supporters
(called “myths” by Markey) and the actual facts and legal precedents.

For instance, whereas Bush claims he is legally allowed to violate the
law, Rose and Rotenberg pointed to Supreme Court precedents and the
history of legislative bills going back decades, to show that the
Court and Congress both explicitly set limits to what Bush can do.

The limited information we have about the current NSA scandal, as I’ve
said, shows it’s a huge funnel. And the ACLU has a huge stack of
records documenting more conventional wiretaps on churches, anti-war
groups, environmental groups, and so on. All this gives lie to another
bush myth that the wiretaps target only terrorists–known “bad
people.”

What about the administration’s claim that they submitted their
law-breaking initiative to Congressional oversight? The facts around
this claim are very revealing.

What the administration did was lift the veil to a few senior members
of the House Intelligence Committee and to two FISA judges–but left
the information so classified that these leaders couldn’t even discuss
it legally with their own counsels. Some of the Congressmen expressed
opposition to the administration, but they couldn’t take it anywhere.

(Personal comment: this tactic of shredding the law’s oversight
provisions is typical of the conduct of Republican leaders for over a
decade. They preserve the flimsiest, superficial elements of laws and
procedural standards in order to undermine the law’s intent.)

But as I said, tonight’s meeting was inspirational. That’s because the
speakers (who know how to draw together and move an audience) left us
feeling that the tide is turning.

I believe that public mistrust over wiretapping is fueled by other
spectacular administration failures–part of the post-Hurricane
Katrina syndrome. Most Americans don’t trust the administration’s
promises to be protecting us and having our best interests at
heart. But the wiretapping issue in particular could be the spark to
turn the country back to some semblance of sanity.

Related link: http://kubuntu.org/

For years I have had an on-again/off-again relationship with desktop linux. My old flame: Debian running fvwm. Over the years, however, we have both changed. Where I used to enjoy tinkering, I am now impatient and want things to ‘just work.’ Desktop linux has thankfully developed along these lines. Goodbye, stock Debian and fvwm. Hello Ubuntu and KDE (Kubuntu). (Ubuntu is based on Debian.)

I first downloaded the Kubuntu live CD, burned it, and booted it on my laptop. It looked good. I especially liked that the antialiased fonts still looked good at small sizes. When I am coding, I use small fonts so I can see more code.

While downloading the Kubuntu install CD, I rummaged through my old hardware and assembled a so-so machine: an AMD K-6/2 450MHz CPU with 384MB RAM. Too slow? Well, let’s try it.

The install went well, except it inexplicably rebooted five times. Each time something different seemed to trigger the reboot. Bad CD media, perhaps? Bad memory? I especially like that it detected my hardware and spared me from configuring X.

Booting into Kubuntu for the first time felt good. The Mac-like admin layer is nice. The UI seemed choppy sometimes, but then my machine is under-powered.

First problem: I wasn’t online. The network configuration window made sense, except I couldn’t get my gateway setting to ’stick.’ A google search quickly yielded a helpful discussion in an ubuntu forum. Using the console I edited /etc/network/interfaces by hand and restarted the network service — fixed the old fashioned way. I was glad to see that the ubuntu community had me covered.

I set to work updating and installing packages. The Adept package manager made sense and worked well.

Setting the clock raised the same issue I saw trying to configure the network. The GUI let me change the time zone, but the change didn’t stick. Back to the console, man pages and Google. Turns out I was missing a symlink from /etc/localtime into /usr/share/zoneinfo. Fixed.

I hope to try my hand at some Linux desktop development. In particular, I would like to create a PDF manipulation tool based on my pdftk. Ubuntu feels like a good place to begin. I am also looking for a desktop my family can comfortably use — we’ll see.

I didn’t have much time to explain EVDO, a wireless broadband connection, in my last pos because I only had a few minutes to use the network before the flight attendants closed the door. Now that I’m on Interstate 5 on my way to Los Angeles (don’t worry, I’m not the one driving), I have a bit more time. There’s really not much I can explain: I put the PC card in my laptop, connect to the Verizon network, and I’ve got a connection at least as fast as my cablel modem at home. It’s unlimited airtime for $60/month.

Aside from a few deadspots along Highway 1 and going over the Grapevine (a big hill, or a little mountain), reception has been fabulous. Instead of spending time loading up my laptop with things to read during the trip, I just surf during en route. Where a Blackberry would allow me to check and send email, my EVDO card let’s me do anything I like, including chatting with Randal Schwartz and uploading patches to the stuff we’re working on.

I also have to put in a kind word for the people over at Booster-Antenna.com. They saw my last post and unsolicitedly offered to refund my money since I only wanted the software and didn’t need the service. Verizon offers the software for free, although they certainly didn’t make it easy to find. If I need service, though, I know where I’m going back to. :)

In the well-connected world of Hollywood, as shown in the TV series “Curb Your Enthusiasm”, what you say can have far-reaching effects. It’s a lesson techies should learn.

The HBO comedy series stars
Larry David as a talented, loud-mouthed TV writer in Los Angeles. Whenever
he feels wronged, he always retaliates against his perceived opressor.
Unfortunately, in Hollywood, everyone knows
everyone else, and his big mouth, easily-bruised pride and huge ego come
back to haunt him.
In one episode, Larry berates a woman at a movie theater over a relatively minor slight.
He feels safe in arguing with and belittling her because she was a stranger, and he seems to enjoy it. He makes little jokes and snipes, since he’s a pretty clever guy.
It wasn’t until after the movie that Larry sees his nemesis talking
about him to the wife of someone with whom Larry is to make a business
deal with the next day. The meeting quickly goes south, of course.

Whenever I see “Curb” using this plot device (about as often as Mr. Roper
misunderstands something he overhears on
“Three’s Company”),
I think of the parallels to the perils of the modern loudmouthed geek,
willing to disparage anyone and anything to anyone who will listen,
with no thought to the consequences. Losing a job like Larry isn’t just TV.
I’ve discarded resumes of potentially capable job candidates because of
what I’ve seen on mailing lists. It’s a very small world we’re in.

It’s surprising since sites like
Flickr and
LinkedIn
are all about
making connections between people. If you mention to Person A something
disparaging about Company X, it’s only a few quick jumps to Person
D who works at Company X, and shows it to someone in HR, or a future
hiring manager. You might not even have directed it to Person A, who
was merely a reader of the message forum you post to, as in the case
with my summarily-ignored job applicants.

Your comments don’t even have to be actively passed on. Google knows all.
You should expect everything you’ve ever written as a blog entry, or commented on someone else’s
blog, should be expected to show up on Google. Most mailing lists are
archived somewhere. People log IRC channel traffic, even if the rules
of the IRC network say that it’s not allowed.

The most damaging situations, however, are still those where people know
each other, and pass on their impression of you to someone else. “Hey,
Andy, I’ve got this resume from this guy named Steve Grumpo. Says he
knows Perl. Ever heard of him?” “Grumpo, Grumpo… Yeah, doesn’t he
work at FooCo? He sent this pissy email to the perl5-porters list a
couple of years ago griping about how the bug queue was getting handled.
Even when it was explained to him how things are, he wouldn’t shut up.
Here, let me find the archived thread for you…” I’ve had many
conversations along those lines in the past few years.

The problem comes from two sources. First, in our roles as techs
and geeks, we pride ourselves in being intelligent and always right.
When challenged, we feel that we must preserve our honor. Second, and even worse,
we forget that the people typing at the other end of the network is a
person with feelings of his or her own. If you want to be completely
logical about it, forget the other person’s feelings, and just think
of how well connected he or she is.

All of this can be easier said than done. I know I’ve pissed off a
few people in my life, and keeping my tongue in check is something I
struggle with whenever I feel slighted, whether in IRC or a mailing list.
Next time you feel a need to flame someone, or disparage anything, take a
minute to think about it and curb your enthusiasm for revenge, for your future’s sake.

What have you said that’s come back to bite you?

We were supposed to roll out my wife’s website last night, but it’s been delayed. The reason? My last minute adjustments and testing on MS IE showed that I’m seeing drastically different things between IE and Firefox. Different isn’t necessarily bad. In this case, I’m proclaiming IE as guilty of misbehavior. When I state that <img width=”970″>, I expect it to take 970 pixels across my monitor. That’s what Firefox does. But IE inflates it by (this is just guestimation here - I’m not on that monitor now) maybe 25%. The image in Firefox looks clean and crisp and the text that is embedded in the image has well-defined borders. Not so in IE. Pardon my Esperanto, but IE’s display of my page looks like rubo. (And if someone wants to correct my Esperanto, I’d be much obliged.) The embedded text is really fuzzy around the borders, the image is way larger than I intended, the quality of the image even looks lacking. To be fair, it all could have something to do with the funky scaling up of the image that IE is doing, but it’s hard to get a side by side comparison when IE is munging the page so badly.

This is my first experience of being abused by IE. I’m no web designer (or really a web developer) as I thoroughly warned my wife upon doing this for her, but I can’t believe that the terrible page display is all me. It looks at least tolerable in Firefox. It just looks awful in IE.

The biggest problem in IE is how it is mangling the words that my wife embedded in the image. We’ve tried jpeg and png. I’ve tried a variety of resolutions which are reasonable to put in a web page. We’ve tried some different fonts. I even tried mucking around with tables, using the image as a background, and positioniong <a> tags on top of the image where I want them. (Which is exactly what they were for - just links.)

Now, what I’m resorting to tonight is splitting two copies of the image (one “normal” resolution, one high resolution) up into the same number and same size, shape, and position of smaller images, throwing the “normal” resolution images into a table and replacing the pieces that contain text with the higher resolution images (and leaving the img width and height attributes at the “normal” values). IE will likely still munge my image, but at least it’ll have higher quality images in the text slots. Hopefully it won’t mess that up as well.

I think it’s ridiculous that I have to go through such gymnastics in order to get a page to display properly in IE, but I’m nearly certain that the majority of site visitors will use IE. Am I going about this all wrong? I really don’t like the idea of using a 900 pixel wide image as a banner, but my wife’s business is very image intensive, so she wants the site to represent her work (which is basically family-oriented custom graphic design).

Anybody have any Firefox/IE compatibility tips for graphically intensive sites? How about IE compatibility war stories?

Advocacy is a funny old game. Although it seems a loose and inexact science, developing as an advocate demands a range of communicative, philosophical and technical skills. Advocacy is not just about the message, but it is about the tone, colour and dynamics of the communication.

As a professional advocate of Open Source, I get email asking about how to advocate efficiently and with a high degree of success. Most of these emails come from enthusiastic members of our community, and ask for hints and tips about advocating well. From my experience, advocates need to take an inverse perspective - instead of asking how to add more tools to your armour, ask what vulnerabilities afflict your weaponry. There is without a shadow of a doubt a massive, cavernous, gaping hole in many advocate’s approach - offering an accurate message.

Defining reality

The act of communicating one desired product or concept over another comes in many different flavours and forms, with advertising, marketing, ideology and advocacy as prime examples. Its temping to assume that these different words are namebadges for essentially the same job, but the different disciplines have dramatically different methodologies and processes. Describing advocacy and advertising as the same science is simply wrong.

Advocacy is largely a skill that is dependent on experience and conscience. This dependence is what separates advocacy from many, but not all forms of advertising, and some forms of marketing. In advertising, it is not uncommon to advertise a product that few, if any of the staff have actually used. An example of this are tampons. I am pretty sure that every advertisement for tampons has not been exclusively developed by women, and even if it were, I am sure that not all of those women use that particular brand of tampons being promoted.

Aside from the experience of using a product, advertising also differs in the form and premise of the communication. Advertising tends to stick to specifics - products, services and brands. In our previous example, an advertising firm would not be promoting the benefits of tampons in general, but instead promoting that specific brand. As far as the advertisers are concerned, there is no benefit in promoting the generic subject as the consumer may simply choose a competitor’s product. The advertiser instead needs to hammer home the benefits of that particular brand, irrespective of whether the competitor’s brand is better or not. With this limitation, advertising and some forms of marketing really work in more of a vacuum; a vacuum populated by a limited range of issues.

Advocacy is entirely different. The role of advocacy is typically not to promote one particular incarnation of a concept, but the concept itself. As much as I love Ubuntu to bits, and choose to run it on everything I can, the reality is that Ubuntu is merely a vehicle for the bigger picture - Open Source and free software. This bigger picture is where the real advocacy happens, and it incorporates all the usual suspects such as availability of source code, more eyes on the code, preventing vendor lock-in, computing equality, lower cost yadda yadda yadda…

With this bigger picture, advocacy tends to revolve around a series of defined benefits that are present in the concept and not just the product. Part of the reason why we have Open Source advocacy in the first place is because the benefits outlined in the generic Open Source model can be implemented in any product that is Open Source - this makes the message easier to explain and easier to demonstrate with these products available. Luckily for us, the Open Source message ties up with the generic demands that most people make in their IT - they want a low cost of hardware and software, readily available functionality, more choice and strong stability. These concepts are not ticks on the side of a specific product’s box, but general demands for good, solid IT.

Advocacy is about conscience

Every advocate has an in-built system for deciding if the software and services they use meets the grade for their advocacy. Take security as an example. There are many, many security advocates and boffins out there, and these people fight for nothing more than good, sensible security and privacy in IT. These advocates demand easy access to security and a dedication to security principles from vendors. As advocates, the ethics of security are the driving factor, and each product will be run through the advocate’s ethical machine to see if it holds its own. Although the products may come in different shapes and sizes, the advocate runs each of them through the same machine.

This objectivity is key to successful advocacy, but this does not justify a zealous approach to your ideals. Take Mac OS X as an example. When Apple released Mac OS X, they pushed the fact that it had the power of UNIX and was based upon an Open Source core (Darwin). At every conference that I went to, I saw more and more Powerbooks appearing with people running Mac OS X. Even die hard fanatics of Open Source and free software were wooed by the Powerbook with its small form factor and sexy glowing apple light. The key point is not whether moving to Mac OS X was good or bad, but rather identifying why people moved to it. Instead of lambasting these people as traitors to the Open Source ethos, you really need to ask what were the attractive factors of Mac OS X and how did this satisfy the needs of the user. Remember, Open Source advocacy is not like Star Wars - there is no black and white division and there is no force. Even if there were, it is sometimes better to sit down with Darth Vader and ask why he is such a total shit instead of just giving him to the light saber treatment.

Hindsight is not always 20/20

Experience is something of a double edged sword, and can quite easily conjure up a view that conflicts with your objectivity. Always remember prior experience will be accented and embellished over time. Experiences such as how easy it was to do this, how hard it was to do that, how cool this new feature was or how that feature sucked are all run through a mental process. This process takes the experience, compares it with how much of a challenge it was to you, compares that with how much of a challenge you think it should provide to the user and then this little lot is combined with how much joy or pain you had with the experience. As an example, when I first set up a Smoothwall firewall, I thought it was insanely cool. The task was within my technical capabilities, fairly straightforward to perform and won me the geek kudos in my house. So, a simple, breezy experience, right?

The danger with experiences such as this is that they really can become distorted in your mind as the memory starts to fade. In that particular example, installing Smoothwall was fairly simple and I got a far larger win for the time and expense that I invested. While the investment/benefit comparison remains the same, hindsight forgets many of the original details that conflict with the overriding opinion concocted from the mental process just described. As such, when people ask you about firewalls months later, your immediate recollection has been distorted into a non-accurate description of what actually happened - the recollection fails to be as objective as it could be.

I see these kind of inaccurate experiences discussed all the time in the Open Source community, and it really does us no good. Objectivity and honesty are key attributes in the good advocate, and it is always important to try and assess how this objectivity and honesty is applied to the different aspects of your advocacy. There is nothing wrong with having opinions, and there is nothing wrong with swinging to a particular view, but it is essential that when advocating you have an accurate recollection and approach to your experience.

What do you think? What do you think of advocacy?