Related link: http://tim.oreilly.com/

One of the best speeches of the Accelerating
Change Conference 2003 was Tim O’Reilly’s talk about "Diffusion of
Memes in the Media". (Only talks, that stay in my head and keep my brain
churning for days earned the label "best".)

He told the story of how it came to be that O’Reilly is doing conferences:
The CEO of Borders told Tim in the early 90s, that Programing
Perl is one of the 100 most sold books of that year. Tim was very surpised,
he knew that the book was selling very good, but no one out there in the press
was talking about Perl.

It was a grassroots thing under the radar of most people. They realized (in
his talk Tim always refered to the O’Reilly team, he never claimed to have done
it alone), that something was missing for these Perl programmers: Somewhere
to meet and exchange ideas.

They started their first conference in 1997, which later expanded to the Open
Source Convention and other conferences.
Open Source is one of the memes that the O’Reilly folks helped to create and
spread. That was very important, because the "Free Software
Movement" was at that time in a corner being perceived as radical. By creating
the Open Source meme, the dialog shifted and all these new possibilities opened up.

I just like the way he operates in general, see his Weblog:

"In my introduction to the session, I outlined some of the key elements
that put technologies on my radar: hackability, being in line with some major
trend (such as the increase in ubiquitous networking), disruptive potential,
grassroots enthusiasm rather than top-down corporate promotion but still the
presence of professional practitioners and a possible business ecology."

Or his support of the Electronic Frontier Foundation
and the Internet Society, from an interview:

"In the case of both, I was motivated by the thought that ‘we need
a Sierra Club for the Internet.’ I loved the open, cooperative nature of the
early Net, and worried as I saw it become increasingly proprietary"

Tim O’Reilly often claims to only follow the Alpha Geeks. It is way more: O’Reilly
provides a platform with their books, the conferences and this online presence,
a fertile ground for new ideas to develop into whole new industries.

Tim and your tribe, keep up the good work.

I don’t know Tim’s birthday, but just pretend it is today and send him some
good words; even saints need to be reminded of their goodness once in a while.

While considering the Do Not Call list, I began to think about the ways of getting the data out to the various telemarketing firms and how it was likely they were using xml for the interchange of data.

Of course, this made me think that it is likely that the telemarketing folks likely have their own set of standard xml tags that they, and their spiritual brethren, the spammers, use…so I present to you, the XML <sucker> tag.

Tag: <sucker>

Usage:

<sucker>

<vulnerableto>

nigeriascam

</vulnerableto>

<vulnerableto>



mortgage

</vulnerableto>

<vulnerableto>



botox

</vulnerableto>

<vulnerableto>


vacation

</vulnerableto>

Alternative Usage:

<sucker vulnerableto=”enlargement”/>

<sucker vulnerableto=”clickingremoveme”/>

<sucker vulnerableto=”workfromhome”/>

Of course this only makes one think of DTD for the creation of windows viruses and what would more efficiently inspire people to continue to open insane attachments, but I digress, and I really don’t want to see rfcs posted about the “interchange of vulnerability data for use by bulk emailers”.

In which Chris contemplates XML tags that might be in use by jerks.

Information technologies are implicated in a worldwide and
world-historic crisis: falling employment. As the wealth of nations
increases, those who have lost jobs or had to accept menial ones over
the past three years are left with only a wealth of culprits to blame:
financial scandals, wars, tax cuts, stagnation, etc. But there is
little doubt that a large contributor to rising unemployment is rising
productivity, which in turn can be laid to advances in computerization
and communications. I can no longer avert my eyes from the
consequences of the field I have chosen, and no one else who programs,
administers, or promotes the use of computers can morally avert their
eyes either.

The gigantic combine of capitalism has always obsessively pursued
efficiency, and computers make the pursuit almost child play.
Capitalism has succeeded in sowing a cornucopia of innovation up and
down society. But capitalism is atrocious at distributing the fruits
of innovation. Each labor-saving device means the idling of thousands
of people, wasting their years of experience, rigorous training, and
practical insights.

People who work with computers remain fixated on efficiency. Every
week I hear the debates over whether businesses should use Linux or
Windows, the commentators always wrangling over which systems will
save the most money. I find this battle increasingly tiresome. I’m
more interested in finding the systems that will put more people to
work.

I have a sinking feeling that we can’t wait for the next upturn in the
employment cycle, as optimists would have us do. I sense that this
upturn may never come, unless people in a position to influence
innovation make a conscious effort to involve the worker. Anyone who
writes programs or plans system deployment should start thinking,
“What can I do to bring average people back into the process of wealth
creation?”

It is not my goal to place restrictions on investment or innovation;
it is only to present a new way of thinking that some people may find
stimulating. I am simply stretching a new canvas on which others may
spread their oils; I am not providing a frame for the canvas. Just to
illustrate what’s possible, though, I offer a few tentative
suggestions.

Write free software for individual industries

A lot of programmers are pounding their treadmills in the free
software movement in order to create pleasant desktop experiences and
improve general-purpose applications. These help everybody and are
worthwhile in themselves, but think how society might benefit if a few
hundred of these programmers took a trip down to small, local,
cutting-edge businesses and asked the proprietors, “What would you
like on your computers to make you more productive?” And think of what
would happen if the programmers went on to write industry-specific
software that solved immediate, felt problems and distributed it for
free.

Businesses can afford to pay for software. But small businesses cannot
pay as much as one would think, and specialized packages can be
incredibly expensive. Proprietary packages also suffer from
limitations, bugs, and lack of guarantees that they will meet user
needs. Free software opens more possibilities, and perhaps can drive
the expansion of job-creating businesses.

Make devices more responsive and easy to customize

Personal devices and cellular phones are growing in power and
complexity, particularly as Java applications become available, but
they still don’t provide the flexibility to augment the ordinary user
at work (as visionary Douglas Englebart first suggested in the
1960s). I would like a computer to plan ahead for me, track things
that are too much trouble for me to remember, and combine inputs to
suggest efficient courses of action. My desktop computer has software
to do some of that, but my cell phone does not. And soon I’ll be able
to have a dozen devices in my office with the hardware capability to
augment my intelligence–I’d like to have the software capability as
well.

In the previous item I suggested very specialized software. But very
generalized software on cheap, available devices can also be
liberating. I am reminded of the power that desktop publishing
brought to ordinary writers in the 1980s, a power that made a social
force out of the same Apple Computer that is currently doing
innovative things to make a mass movement out of another medium with
even more relevance and social impact–video. It’s nearly impossible
to overestimate the advances that users can make when they are
presented with flexible, open-ended technologies. Maybe it can make
more of them into productive members of society.

A key part of the solution is easy scripting languages. Current
languages always seem to develop tangled syntax; they look easy enough
for “Hello, world” applications, but as soon as you start to work with
real data and serious tasks into which you can sink your teeth, they
slap on braces and other indigestible characters. I want a scripting
language that is really simple enough for a kid to learn and powerful
enough to run a small business with.

Create a truly public key infrastructure

People have been trying to get corporate communications and
negotiations online for years, and probably the biggest beneficiaries
of such a move would be small businesses and individual contractors.
After all, who finds it hardest to pay travel costs and conference
room fees for expensive legal help?

The move online has been held up by deep and serious problems in the
processes for validating users and dealing with such issues as
certificate revocation and non-repudiation–social aspects of
security, or what I call the social infrastructure for
technology. But perhaps we’re asking too much. Perhaps the average
user could be happy with a less universal and less ambitious system.

When you want to contract with some professional or service, it might
be enough for you to verify that he or she is a dues-paying member in
good standing in some association. Individual associations could
provide authentication services for this. Perhaps a contract could be
sealed by the combination of recorded voice messages and a digital
signature on a computer file. We have to be flexible and creative.

Those ideas are here just to get people thinking. We don’t have all
the time in the world. Already, educated professionals are griping
about jobs moving to other countries, a form of heightened national
and racial tension that not only bring their own horrific consequences
but dampen the spirit of exploration that can raise everyone’s
opportunities. And meanwhile governments, businesses, venture
capitalists (what are you doing with all that money your pets in
Congress and the White House brought you, tails all awagging?),
universities, and NGOs seem paralyzed in the face of this economic
disaster.

There are precedents for this type of thinking. In the 1970s, a
movement called participatory design started in Scandinavia to develop
technologies that enhanced and strengthened workers’ skilled
contributions, instead of eviscerating them. I have written about
other elements of the program suggested here in earlier articles:

• 
  The Semantic Web: It’s Whom You Know
  

• 
  Applications, User Interfaces, and Servers in the Soup
  

• 
  Structure and Service: Illuminations From a Trip to the Forest
  

But a few good examples will promote change more than all the talk in
the world.

What can you do?

Related link: http://www.apacheweek.com/issues/03-09-26

Apache Week reports that the Apache web server’s market share has recently reached 100%, give or take some statistical wobble… at least, if you plot it by domain name. Finally, something good comes from SiteFinder.

Related link: http://qa.perl.org/phalanx/

Phalanx is a Perl QA project created to provide a solid testing
base for Ponie, the next version of Perl 5 that will be based on
the Parrot virtual machine. By increasing the test coverage of
Perl modules and Perl itself, we will make Ponie the best-tested
version of Perl ever. We also hope to involve members of the Perl
community who might not have contributed to Perl before.

The first phase of Phalanx will update the tests in
100 of the
most-widely used modules on CPAN. A Phalanx team member, or hoplite,
will pick a distribution and, with the agreement and cooperation
of the module’s author, start working on the improvements. There
may be one hoplite, or many, if the lead hoplite wants to bring
others into her part of the project. Along the way, the hoplites
will verify accuracy of the documentation, explore the depth and
breadth of the module, and make sure that everything that can be
tested is tested. Once changes are made, the lead hoplite will
feed her patches back to the author, who will update his distribution.
All changes are voluntary, and the author still retains full control
of his module.

There are three goals for Phalanx. The first is to provide an
excellent set of tests for the next version of Perl. Perhaps even
more important, we want to encourage participation from members of
the community who have never contributed back to Perl. It will be
easy to get involved, and your involvement can be as much or as
little as necessary. Prospective hoplites need not be part of
perl5-porters or any perceived “Perl cabal”, or even know about
Perl internals. Finally, we’re certain that we will uncover
undiscovered bugs, and we’ll identify them for the author, if not
eliminate them.

We hope that module authors will welcome the contributions of the
community, and that those who are interested in getting involved
with Perl and open source can make a contribution, no matter how
small. Even one .t file for a distribution, or updated documentation
for a function, can be a big help.

After we’ve had some success in the first phase of Phalanx, we’ll
expand the process to Perl 5 itself and the core modules. This
phase will be trickier, as we must work with the pumpkings and the
rest of perl5-porters. We hope to have enough good lessons learned
that it will be an easy transition.

Please visit the Phalanx website at
http://qa.perl.org/phalanx/.
If you’d like to help out, join the perl-qa mailing list, or email
me at andy@petdance.com. I look forward to hearing from you.

Have you worked on a project like this before, either in terms of QA or scale?

so, i recently took some time off. as American standards go, a rather long time
off (8 weeks). instead of sleeping for the entire time (as my wife expected), i
got turned on by the idea of attacking our home network/automation system and
making it actually work. after two different (and expensive)
contractors, it still didn’t behave the way i wanted. but hey, i’m a geek, how
hard can it be? hmmmmm…

why am i telling you this? after all, you can read about “what i did on my
summer vacation” on my home blog. the
reason is that when i’m not playing amateur home automation geek, my day job
for the last 2 years has been as dev-lead for the runtime in BEA’s WebLogic
Workshop. and horror of horrors, i didn’t use WebLogic Server or Workshop in my
summer quest. i used, gulp, Apache . in my defense, i stated upfront that i
would move to WebLogic Server as soon as i could. but when would that be?

this little fact did not escape the notice of
Byron Sebastian, my boss’ boss, and
“the man” when it comes to Workshop. one of his first emails to me when reading
my home blog was “when you try switching to WLS/W, please write up your
experience because others will likely find it useful”. the end result of that
email thread was this post

the first thing i had to figure out was exactly what i wanted out of WLS in
order to make the switch. why, after all, did i start with Apache in the first
place? after all, WLS is a J2EE AppServer that is also a full featured web-
server. so what’s the problem, just install it and away you go. well, not
quite…

you have to realize that the design center for WLS is the clustered, high-
throughput and heavily transacted IT environment, and the installer is
generally focused on this case. recently, with the introduction of Workshop,
the installer works better for the developer using the Workshop IDE, but still
needs work to deliver well outside these scenarios

but neither of these scenarios was really what i wanted. the latter came
closest in that it made development easy, but i wanted more. here’s the high-
level list of what “more” was:

• simple setup (100% question free)
  i didn’t want to have to answer a bunch of questions in order to get the basic
  system going. i’ve been burned too many times when trying to recover a working
  setup after crash or upgrade, that i try to run with defaults whenever i can
• start/stop via service/chkconfig
  like many, i’ve become addicted to the way *nix supports managing “services”.
  on my server and workstations at home, as well as my dev box at work, i run
  RedHat9. but OOTB (out of the box) WLS only supports the “service” model of
  management on Windows. i wanted to make WLS support the *nix service model
• listen on port 80/443
  the default install has WLS listening on ports 7001/7002 (http/https). i wanted
  it to run at the standard ports instead. on linux, only root can access port
  numbers 1024 and below. which by itself isn’t a huge issue, just start the
  server when you’re logged in as root. of course, this can be a big
  security issue. for example, this would mean that the server and any
  user code would be running with root privilege. if someone cracks your server,
  they would own your machine

  the solution is to start the server as root, and then once it’s running it
  needs to switch to another user. from that point on, it’s no longer root-
  privileged

• cgi enabled (Python by default)
  while Java is currently my “work” language, i use scripting languages all the
  time for various odd-jobs and tasks. depending on the task at hand, i tend to
  reach for either Bash or Python. everyone has their favorites, and enabling CGI
  scripts to run in a web-server opens things up to scriptors of all stripes.
  since i’m a Python addict, i wanted that enabled by default, but it’s trivial
  to enable others

one thing i really wanted, that i decided not to tackle, was to support
having an RPM-based install. i thought about it, but since i don’t understand
how easy/hard it is to create an RPM i took a pass

with this loosely defined set of goals, i downloaded WLS off BEA’s
dev2dev website and dug in. the result of
my efforts can be found
here. it’s a zip file containing:

• wlsInstall: a Bash script that installs everything
• silentI.xml: an XML file used during the first phase of the install
• silentC.txt: a text file used during the second phase of the install
• default: an exploded J2EE web-app used during the third phase of the
  install
• wls: a Python script that supports the “rc-script” standard for
  start|stop|status

i also need to mention that i wouldn’t have gotten anywhere if it
wasn’t for the help of Raj Alagumalai. he
answered a ton of questions with way more patience then i would have shown in
his position. he really knows WLS install and setup cold. thanks Raj!

i’ve verified that this stuff works on both RedHat9 and RedHat8. it’ll even
work on RedHat7, but with RH7 you’ll need to upgrade your version of Python.
i know it works with Python 2.2, not sure how far back you can go and still
have it work. there’s also no reason WLS won’t run on
Debian or
SuSE, but i haven’t tried them. if anyone finds they work, or makes fixes
so that they do, let me know

ok, onto using this stuff yourself. do the following:

1. download the zip file
   and into the same directory, download
   
   WebLogic Server 8.1. the server link provided here gets you most of the way
   through the dev2dev site, but there’s a few things you need to do manually:
   • on the page i’ve linked you to, you need to select the right OS. in the
     dropdown, in the 2nd section labeled “WebLogic Server Package
     Installer”, pick “Red Hat Enterprise Linux (2.1, 32 bit)” and then click on the
     “Select >>” button to the right of the drop-down
   • at this point, you will be challenged for a username and password. you
     can’t download without giving marketing its due <g>. if you don’t have a
     dev2dev account, getting one is trivial. there’s a register button on the login
     page that lets you sign up. once you’ve either logged in or registered,
     you’ll come back to the right spot in the download process
   • once past that, you’ll see a page asking if you’re downloading a server
     you’ve already paid for, or do you want to evaluate a server first. BEA now
     provides a “developer license” where you can get and run the server for free
     for 1-year, as long as you don’t “put it into production”. so just select
     “I would like to evaluate this product and click “Next >>”
   • you’ll now see a form asking some questions about what you want to do with
     the product, blah blah blah. answer however seems best, and then click
     “submit”
     please note that the download process you’re forced to go through is
     being streamlined and revamped, so that there will be less marketing/sales gorp
     to deal with. but for now, please ignore it as best you can
   • next up is the License Agreement. click “I Accept” when you’re ready
   • click on the download area that is the closest match to the region you’re
     in and your browser should prompt you to open or save. pick save. do not
     open it yet. when saving, make sure the saved name is server811_linux32.bin
     . that should be the default, but in case your browser did something
     funny…
2. unzip simple.zip. you want the sub-dir that’s created to be right there,
   next to the server binary and the zip file

   unzip simple.zip

3. move into the directory that was just created (should be named simple)

   cd simple

4. security issue: the management password being set during this install
   is weblogic, which is fine as long as your machine cannot be seen from
   the net at large. WLS supports management via the browser, and you don’t want
   random people on the web changing things on you if you can avoid it <g>. so
   if your machine will be seen on the Internet, you should change the management
   password in the silentC.txt and wls files in this directory
   before proceeding with the install. just search for the word “password” and
   you’ll see the values that need to change
5. the next steps require root-privilege, so switch-user to root (this will
   prompt you for the root password)

   su --command="cd $PWD" -

6. run the wlsInstall script

   ./wlsInstall

7. at this point, WLS is installed, you’ve got a default web-app, and the
   server is ready-to-run. you can start and stop the server using the service
   command

   service wls start

8. with the server running, you should be able to type http://localhost/
   into your browser and see the home page of the default web-app
9. if you want the server to start during boot, use chkconfig

   chkconfig --add wls
   chkconfig wls on

10. assuming you don’t want to have to be root in order to add content to your
    web-app, you can give all users on the machine the ability to modify the web-app
    with the following command

    chmod -R a=rwx /bea/user_projects/domains/wls/applications/default

    the directory listed here is the root of the default web-app

11. within the default web-app, there’s a directory named “cgi-bin”. they way
    i’ve configured things, you can place Python scripts with the extension ‘.py’
    in the cgi-bin directory and then hit them from the browser and they’ll run. so
    for example, if you put a Python script named ‘foo.py’ in there and then point
    the browser at http://localhost/cgi-bin/foo.py, you’ll see the results of your
    script. you can see what enables this by looking in
    /bea/user_projects/domains/wls/applications/default/WEB-INF/web.xml at the
    following lines:

        <init-param>
          <param-name>*.py<param-name>
          <param-value>/usr/bin/python<param-value>
        <init-param>
    

    if you want to enable other script engines, just add another section like this
    one with the appropriate file extension and script engine

well, that would seem to be that. a simple install, an easy way to control the
server and all on top of an industrial-strength J2EE server. cool, eh? and i’m
sure there are ways to make it simpler/easier, so if you spot any, i’d be very
interested in your input. btw, you can find the online doc for WLS 8.1
here

next up i’m going to spend some time building some samples and services that
will run live on dev2dev, which is BEA’s developer support site. i’ll be
posting the code for all of these, along with my experience in building them,
on this blog. and i will be developing all of them on top of the install
described in this post. make no mistake, this install will evolve, and it
wouldn’t surprise me if it ended up changing quite a bit over time. and i
should also make clear that anyone who would like to participate, from
criticism (you moron Rod, why did you do it that way?) through code, and
anything in between. in the end, i’m a utility junkie. i’m interested in things
that get used, that do something *useful*. we’ll see

one of the ideas i’ve got for the first online developer service to write would
be an “XML comparison” service. initially, you feed in two payloads, either via
file-upload or a web-service, and it performs an xml-wise compare and returns
the result. it could be extended over time to take a schema to optionally
perform a schema-wise compare. for example, one payload specifies a value, and
the other doesn’t, but the value supplied is equal to the default value
proscribed in the schema. from a schema-wise perspective, the two payloads
would be considered “equivalent”. what think?

did you find this useful? any ideas for improvement of the process? did it work for you (or not)?

Lest we forget: Samba is one of those projects that makes life better. Thanks to the fine work of the Samba team, you don’t need to run a windows machine to serve file and print to windows clients. I can’t express what a difference this has made in the quality of life for IT workers everywhere, so I wanted to take up O’Reilly Network weblog space to express a big thanks to the gang working on Samba!

In case you missed it, the press release touting the rev is here.

All hail samba…

Related link: http://www.theatlantic.com/unbound/flashbks/computer/bushf.htm

I recently got a chance to reread Vannever Bush’s 1945 Atlantic Monthly article, As We May Think. If you haven’t ever read it, it is well worth the read, and still quite thought provoking. You can also check out a biography of Vannever Bush at Wired magazine — Wired calls him the “Godfather” of the Internet.

In his article, Bush describes a machine called the “Memex.” Here’s how he describes it:

A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

Bush then goes on to describe a machine which is remarkably like the World Wide Web, as it exists today. For example, he describes “trails” between documents (think hyperlinks), indexes of trails (think Google), and publication of trails with commentary (think Blogs).

As far as we have come, I think we still fall short of Bush’s original vision. In Bush’s description, the Memex holds all the pieces of everything you have ever read, and lets you easily add notes, and organize those notes into meaningful research. In my own writing and research, the closest thing I have ever found to this is a small shareware application, called
Zoot. Zoot lets you copy and paste web pages, add random notes, and then organize all this into more meaningful folders. You can then do quick searches on all your ideas, and reorganize your folders yet again. It’s a great tool, and I highly recommend it.

We have great tools for surfing and searching the web, but there’s still a great need for information organizers to help make sense of all the information we take in. In other words, we still need a Memex. And, if anyone ever does build a Memex, I would definitely be first in line to buy one.

Have we fulfilled Bush’s vision for the Memex? What qualities would your ideal Memex machine have?

Related link: http://www.nytimes.com/2003/09/25/technology/circuits/25code.html

Michael Everson, globetrotting font geek, is one of the folks behind Unicode 4.0.

Related link: http://www.angrycoder.com/article.aspx?cid=6&y=2003&m=7&d=31

Here’s a little gem from
Matt Heusser, who’s putting together tomorrow’s
Perl Mongers meeting in Grand Rapids.

The attitude of “Just give me the requirements” fails because it has the customer solving the problem; the software developer becomes just a glorified technical writer that knows how to write in the language of a machine.

That doesn’t diminish the importance of that coding skill. It’s tough to write good, solid code, or else we wouldn’t need jobs. What we help provide is the translation between the business needs and the capabilities of the computer.

The other problem with “Just give me the requirements” is that it’s an attempt on the programmer’s part to absolve himself from the responsibility of the project’s success. If the requirements are handed to the programmer, created in a vacuum, the programmer has a ready scapegoat when the project fails. (And that probably is a “when”, not an “if.”)

Matt’s original article is aimed at independent consultants, not those of us who are part of an IT department, but I suggest that we should all think of ourselves as consultants. When that happens, the “plan for having a scapegoat when the project fails” attitude will disappear. No consultant who wanted to make the mortgage would say “Just give me the requirements,” and neither should those of us in IT.

Related link: http://grand-rapids.pm.org/testing.htm

The Grand Rapids (MI) Perl Mongers meeting on Thursday will feature yours truly, talking about automated testing in Perl. The meeting is open to the public, so if you’re in the area, please come. I think the face-to-face time provided by user group meetings are incredibly important, and I want to do what I can to encourage that.

I’ve been inspired so much by Mark-Jason Dominus in the past year or two. He started me on my path of giving talks by accepting my 5-minute lightning talk at YAPC 2002, and he’s a great speaker to watch and learn from. So recently I was looking through his
page of appearances and found this at the bottom:

I will speak to any .pm or other user group for free.

All you need to provide is an invitation, a room, and audience, and my minimal travel expenses.

Send me mail to arrange it.

P.S.: People sometimes think I will not want to visit their town because it is too small, too out-of-the-way, or too boring. They are mistaken. I want to visit everywhere.

I thought “Hey, I could do that, too, but for the midwest,” so I started talking to people, and voila, I wind up talking to someone in Grand Rapids who saw my talk at OSCON 2003 and got his company to foot the bill. What amazes me is that people rarely take Mark up on his offer. I chatted with briefly about it, and he said he does it maybe once a year. What an opportunity for PM groups, to get someone of Mark’s caliber speaking for next to nothing!

I hope to make this a regular (bimonthly or so) happening, where I visit a different PM group within a day’s drive of Chicago. If you’re interested in having me talk to your group, Perl or otherwise, please let me know.

What do you do to support open source besides contributing code?

I’ve been exploring python quite a bit lately in the context of Damage. At Damage we use c++ and python quite a bit, and I’m writing interfaces that work via different protocols (irc, web, etc..) and had an occasion to really explore the python module reloading mechanism.

Python is a fascinating language in a lot of ways and like all new languages brings its share of adjustments to the table. One of the strongest aspects of python is its ability to reload modules on the fly. One of the trickiest parts with this mechanism is how it seems to relate to the scoping rules of python.

For instance…suppose you import a module at one level and want to call a function that reloads the modules that have changed since last checked. You can do that, but if you do it at the called function level, you are actually only changing it within that scope. So if you want to do a base reload of an entire system, you have to have one known good base and have the sub modules reload all of their submodules in turn within the proper scoping context (If I understand this correctly). The outrageously cool thing about this is that you can wrap these importation statements within try/except blocks and thus give your program an extra level of robustness.

The irony here is that repeated reloads (I tested it to around 1m reloads) presents the same amount of memory leakages as redefining the function would (via executing a file within an exec block for instance) , so you end up not making up any sort memory savings, which appear to be additive, but you do get a kind of correctness of scoping.

Anyhow, thought I’d plop that and the following code on the table and see what O’Reilly Networkingans thought of it. Note, indenting is very important in python and ignored mostly for readability sake here:

The code to reload a module would look something like this:

import dispatch

and then later in the code (and indented after a test for the reloading command event)

try:

reload(dispatch)

except:

print ‘-’*60

traceback.print_exc(file=sys.stdout)

print ‘-’*60

The cool, and slightly dangerous and somewhat hard to debug, thing is that if you aren’t really concerned about allowing the old bot_dispatch code to run, if it fails the old code still exists and runs until you replace it with a good module that doesn’t throw an exception. Of course this can lead to some very lazy programming due to its robustness, but its a nice feature, sort of like an emergency backup.

I’d love to hear how other python folks are doing this kind of reloading, and if my solution is the truly pythonic one or wildly off base and indicative of my ignorance with the language.

Module reloading in python.

In today’s New York Times they have an article pointing a finger at the true culprit for spreading movies via the P2P networks, the studios themselves.

Now, I don’t think that if the studios own folks hadn’t been the ones spreading the movies that they wouldn’t have ended up on the nets, but it does give one pause, specifically will they be suing themselves for their piracy? Will heads roll? Will the Piracy Menace(tm) be rooted out?

Will it matter?

No, it won’t. Movies will still end up on p2p networks and those who are wont to download them will get them all the same.

This brings up another pet peeve of mine. Why do people think that shutting down or hobbling P2P networks will reduce copying. People have the taste for it and so long as people have free will and a fast enough connection there will be large scale illicit copying of music and movies (I refuse to call it piracy, mateys. Arrr), whether it is over SMTP, HTTP, on outrageously inexpensive disks, or on cheap optical media.

P2P Copying Begins at Home

Related link: http://dibona.com

For my first entry on the O’Reilly network, I’d like to start off by thanking O’Reilly (Thanks Bruce, Terri and Sara) for hosting this blog, I’ve actually maintained a journnal on my personal site DiBona.com since late 1999, and that is where I will keep an archive of those old posts. I was pretty happy that O’Reilly wanted me to do one here, specifically because I can be quite lazy when it comes to redoing my website.

If you had followed that link, you would have noted my very 1999 looking site. No prettiness, no rss/xml feed or anything. That said, it served me quite well for some time, by moving my Journal to this network it takes away any last excuses I had for not fixing things up. My eventual goal is to have the xml feed from this blog be on that site and I do all the updates here with regards to what I’m up to, my thoughts and the progress we’re making at both the game studio I co-founded, Damage Studios and its sister company, open source consultancy Konstrux.

My cred: Other than Damage and Konstrux, I co-edited the O’Reilly book “Open Sources: Voices from the open source software revolution.”. I was an editor for Slashdot for about a year, posting something like 750 stories and about 300 polls. I’m also the Linux guy for TechTVs The ScreenSavers in case you want to find me on your TV Screen.

I also think I’ve just barely done my part for digital rights through my work with the EFF and others (I testified in the New York DVD cases). I also ran community relations for VA Software for a few years before the Slashdot gig, where a lot of people know me from.

That’s all for now…talk with you more soon.

Hey there, I’m Chris DiBona

Related link: http://www.guardian.co.uk/business/story/0,3604,1039409,00.html

If the London power grid does down because they accidentally used a one amp fuse instead of a five amp fuse — and they never discovered this over a period of two years, despite a testing program — what does this say about the stability of computer programs, where every user action is a one amp fuse waiting to blow?

Is there any hope for complex systems?

Related link: http://www.sco.com/company/openletter/

It’s obvious that Darl McBride has no understanding of the open source community and its motivations and values.

In his open letter to the open source community he states, “it is clear that the Open Source community needs a business model that is sustainable if it is to grow beyond a part-time avocation into an enterprise-trusted development model.”

I can’t imagine Mr. McBride is not aware of the fact that virtually every businesses everywhere currently depends heavily on open source technology. Why he would state otherwise is ridiculous.

Whether using the Apache web server, Linux, gcc, make, Apache Tomcat, PHP, or one of the many, many other open source packages, virtually every company today is heavily dependent upon open source. Virtually everyone.

The ‘collaborative’ process by which all these projects get created is based on a new business model. It’s a business model defined by the *users* of the technology - not by vendors. Individuals in companies all around the world are simply working together to create the technologies collaboratively - and releasing them under open source licenses that ensure everyone can freely use and contribute to their enhancement.

Darl continues, “Rather than fight for the right for free software, it’s far more valuable to design a new business model that enhances the stability and trustworthiness of the Open Source community in the eyes of enterprise customers.”

Why he believes our efforts are “far more valuable” if incorporated into a “business model” is unimaginable. The immense economic impact of open source on the productivity of the American and the World economies is undeniable — and accomplished without this “business model”.

Open Source doesn’t need to “design a new business model” - SCO does. Mr. McBride has simply not come to grips with the fact that his business model is dying. The open source community doesn’t need his services or his intellectual property. We can innovate faster, work cheaper, achieve higher quality and impact productivity in this country (and throughout the world) much better than SCO — by far.

Open Source is about being ‘free’ - as in ‘freedom’. Open Source developers give away their work so that we will all be enriched - and so that they can get the benefit of the work that others have given away as well.

We don’t need or want his help and we won’t be working with him. SCO simply wants to control and own the ideas of the open source community and the fruits of our labors.

Open your eyes Darl McBride. The new business model that the open source community has already developed without your help is about to flatten you. This new business model is based on companies and individuals increasing productivity and reducing costs by collaborating on ’shared’ IP that no one owns and is free.

The giant sucking sound you’re hearing, Darl, is the sound of your customers all contributing little pieces to the open source picture - so they can get better value and get rid of you.

Related link: http://www.nonesuch.org/p2prevolution.pdf

Many people have predicted that peer-to-peer file-sharing will change
the face of media, but this paper by Scott Jensen is the most
thoroughgoing research I’ve seen into the commercial and artistic
effects that peer-to-peer can potentially lead to. The paper is bold
and futuristic, which means there are plenty of places the path it
lays out could be sidetracked, but I think it’s important reading. It
makes me a little afraid to work for an old-style publication company,
even one as innovative as O’Reilly & Associates.

Will it be?

Signal-to-noise ratios are awful in the world around us.
Consider the following:

• A car alarm goes off in a parking lot. You don’t look to see where it’s coming from, much less call the police.

• Your office has an overhead paging system. A couple of times an hour, a voice breaks in: “Bob Smith, dial 847. Bob Smith, dial 847.” Sometimes, if it’s actually important, the voice begins “may I have your attention, please.”

• You call a customer support line. “Please listen carefully, as our options have changed.” Since when? Since the last time you called six months ago and it had the exact same message?

In each case, something has tried to get our attention, but we’ve learned through repetition that we don’t need to pay any mind to it. It’s become mental clutter.

Now, consider some computer-side cases where we’ve done the same things with how we do our work:

• You’re grepping through log files and need to dump results to a temporary file, like ~/foo.
  But before you execute the command, you notice there’s already a file called foo. It’s got some list of filenames in it. What are they? Are they important? OK, you’ll dump to ~/foo2, but there’s foos 2 through 7 already. Well, time to make foo8.

• You’re doing an emergency rebuild of a server, and you need the CD-ROMs for some RPMs. You open the desk drawer to find at least a dozen discs, three of them labeled “RH 7″. Which is the one you want? Does it matter?

• A process crashes, and tells you that it’s dumped a core file in /tmp. There are 100+ files in /tmp.

• You’ve been getting joe-jobbed with fake spam bounces recently. You set up a rule to throw away anything that comes from MAILER-DAEMON@.

• You’ve set up an hourly smokebot to run automated tests on your project and report failures. One problem is a pain to deal with, so you ignore it for now. Each morning, you delete the dozen failures from the night before, not knowing that another bug has been introduced, too, since you’re ignoring the reports.
  

Our lives are filled with helpful warnings, and we ignore many of them. Worse, our lives are filled with clutter that we let accumulate so that we don’t notice when something is going wrong.

So what to do?

Deal with every problem. Don’t brush the problem aside by making a mental note to deal with it later. That method doesn’t scale. In fact, NOTHING that relies on a single person scales. Make the computer do its work.

If you have to deal with it later, then put it somewhere, like a ticket in your RT ticketing system. The point is to reset the automatic idiot light so that you never think “Oh, I know about that problem.” If your computer is telling you something is wrong, then it better mean something.

The job of the computer is to do the repetitive, mindless work. Your job is to think. If you have to waste brain cycles on whether a given warning is actually a problem, then you’re not using your computer to its fullest.

The job of de-cluttering falls on you, and only you.
If you don’t clean up your crap, who will? Do you expect the crap-cleaning fairy to come and take care of it? If you want to set up your own automatic crap-cleaning fairy that doesn’t gloss over problems, that’s great, but set up something.

If your system is constantly crying “Wolf!”, then it’s doing you a disservice. Whip it into shape and make sure you only get alerted to problems that are real.

Any suggestions on decluttering your computing life?

I had to setup a repository for the class I’m TAing this year, and I bumped into this great Chrooted SSH CVS Server HOWTO (courtesy of the Google cache).

The niftiest part of the HOWTO is not the chroot jail (there are plenty of resources on how to set one up and even some interesting ones on how to possibly break out of them) — but rather the idea of using smrsh as the default shell of a user to allow them to only use CVS and not log in.

smrsh is the restricted shell that Sendmail uses. When you want to execute an arbitrary command from your aliases file, Sendmail, by default, executes the command through smrsh who will only use a command if it has it in a directory it owns. Read this as a poor man’s attempt at preventing arbitrary code from being run. What that document suggested, and I used, was to set the default shell of users to smrsh and then setup smrsh to only be allowed to execute a statically linked CVS binary. Voila! People can then SSH tunnel CVS, but they cannot actually log into the machine.

Anyway, I thought it was cool.

Like the rest of the world, the Internet is getting scarier and
scarier. In fact, it’s becoming a three-ring circus where the
elephants commandeer the trapeze and the acrobats poison the peanuts.
And recently those responsible for the Internet’s creeping anarchy,
even in their greed and vindictiveness, have been showing incredibly
poor judgment:

• Sending out a virus that instructs Windows systems to launch a
  denial-of-service against the Windows update site, which obviously
  very few people use in the first place.

• Bringing down the sales site of SCO, a company that makes its money
  off of legal maneuvers rather than sales.

• Sending out a virus that “phones home” to the instigator’s web site,
  even after the RIAA has shown how easy it is to trace Internet users
  with nothing more than a dynamic IP address and a timestamp.

• Citing privacy as a reason to deny giving the identities of
  individuals to the RIAA–individuals who have put themselves in
  public view by offering up hundreds of files for sharing.

I should say a bit more about file-sharing to minimize the chance of
misunderstanding. Is the RIAA going stark raving mad in suing 275
music fans every week? Yes. Is file-sharing the cusp of a
revolutionary movement opening up the media? Possibly. Does the chance to get free pop songs rank as a legitimate reason for breaking laws, comparable to promoting rights for African-Americans or even offering marijuana to relieve AIDS sufferers? Not in my opinion.

On a related note, a friend of mine denies that flash mobs (such as
stunts where a hundred people show up at one downtown intersection and
do a five-minute dance) are an expression of direct democracy and a
disregard for hierarchy. “They’re the ultimate in subordination and
hierarchy,” she says. “A bunch of people mindlessly giving themselves
over to the whims of one person.” That’s a persuasive argument.
History teaches us not to expect much smart behavior from large
groups–but we aren’t getting it from self-appointed vigilantes
either.

Are we seeing evolution or entropy at work?