O'Reilly Digital Media

My last weblog post explained that anyone can podcast on a budget. Other than my laptop I managed to start with no initial hardware costs. Recently I upgraded my equipment to improve the quality of my show. Like an elite runner I couldn’t finish a marathon in my best time wearing flip-flops (we call them thongs in Australia, and I probably couldn’t complete one wearing a pair of those either). I’d also still look like an idiot with stockings wrapped around my noggin.

After being inspired to listen to a podcast by Michael Goeghagen, who explored his transition from cheap to a more professional rig, I set about finding the equipment I needed for a professional sound.

My focus was on improving the microphone. You’ve seen the pitiful one I originally used, and I wanted to upgrade to one that would produce a better sound, and that I also had more control over. The USB input of the Logitech meant that I could only boost the audio level within the PowerBook. That has led to plenty of frustrations with the sound level on my end.

I’d already read some reviews from other podcasters about the Shure SM58 microphone, and they suggested it is a good choice for voice. There are plenty of others that would no doubt have made a clearer reproduction, but my experience with home audio – like my home theatre – proved that once you reach a point in quality you pay handsomely for very little gain. Podcasts are compressed to small MP3 files – there goes the fantastic quality – so I wasn’t going to stress about spending hundreds on a condenser microphone (which I’m told will produce a better quality sound).

Next up was a pre-amp. This was a necessity for my configuration. The PowerBook’s line-in isn’t powered, so any input needs amplification. I knew that I could buy a cheap pre-amp that did a simple job, but I figured for only a few extra bucks I could get a simple mixer. In hindsight this was a great decision, because I can warm up the sound a little with the tri-band equalizer and add extra gain to increase the volume before it hits the Apple – great for when using Skype and the other party can’t hear me that well.

I grabbed the Behringer Eurorack UB802 because almost everyone in the podosphere says they have one. No point in being someone else’s guinea pig when I can just follow everyone else. I also paid the extra over and above the UB502, because I figured an extra microphone input may come in handy if I ever to interviews with someone in the same room.

The next purchase was completely unnecessary. I sprang for a K&M microphone boom arm that clamps to my desk. It’s mostly because I wanted to show off and look professional, but I used the excuse that our house’s floorboards would carry audio all the way up a freestanding mic boom. The popshield is an added extra, I wanted one that looked better than my last, and I swear these cotton disks are a must to cut out unnecessary plosives. Well worth the extra dollars.

For those scrutinizing the photo, my cans (the professionals term for headphones while I’m still posing) are an old pair of Sony noise cancelling headphones, with the cancelling turned off. I also invested in a Griffin iCurve stand and Apple keyboard and mouse, but that’s purely to help the neck ache I’ve developed since starting the book.

So, how’ve I found the quality? It’s impressive. If you want to compare the difference then check The Gadget Show episode 20 and compare it to 21. Other than the new introduction the changes have only been in the hardware.

So, although I mention in the last post that it doesn’t cost much to start podcasting, if you decide to pursue it as a hobby or as a serious venture, it’s worth investing in a good microphone. At least I don’t look like a lunatic with a stocking flapping around my lugs.

Any advice on podcasting, please chip in.

So there I was, sitting comfortably in my chair reading slashdot. The emails start coming in.. it seems nobody can use subversion, emacs mail was failing, and sendmail was spazzing when it tried to deliver to a file (i.e. not via procmail).

The exact error was: “no record locks available.” Google lead me to believe that everyone in the world has solved this problem with a file server reboot, or by restarting lockd. I rebooted many times.

The next day my boss basically said that I need to stay all night until the problem is resolved. I logged in and ran ‘lockfs -fa;sync;sync;sync;reboot.’ Clearly rebooting this Sun server wasn’t the solution.

To make a very long story short, I finally discovered how to get info from the kernel about record locks. Note: this means locking part of a file (from an offset) as opposed to just a standard lock, those were working fine. It turns out that lockd only has 20 threads to deal with locks, and we were being DoS’d with a lock storm. Before that, I had ran lockd in debug mode and discovered the exact error being returned was: NLM4_DENIED_GRACE_PERIOD (see the RFC), which is very misleading.

So it’s possible to have a lockd storm. Running ’snoop rpc nlockmgr’ showed a tad bit of traffic from a web server that mounts home directories from the file server. A student’s perl cgi was calling lockf() explicitly, and this became a very popular site. MSNbot was crawling it at the time, and the processes never died. They never ate up many resources on the web server, so nobody noticed.

Isn’t that just the strangest thing? :)
My gratitude goes to Dan Leger of Sun Networking Support for helping me, even though he wasn’t required to, by providing “contract only” documentation which led straight to the resolution.

Anyone ever seen that?

For years now I have thought that there is a certain set of immutable rules in the universe that specifically govern the circumstances of a network admin’s life.

One of them has proven itself over and over. It’s a lot like Murphy’s Law, but much more specific:

If you’re leaving town, all the possible bad things that can happen to your [network | servers | personal computers | other geek devices], will happen in the [days | minutes | hours | seconds] before you leave.

I’m getting ready to go to OSCON next week, and so far my Network Admin’s Law is proving itself out yet again. In the past week:

• One of the main circuit breakers in our building overloaded, resulting in a power outage where our generators did not kick in. The actual breaker did not pop, so we didn’t know there was a problem. Before we found the problem and reset the breaker, one UPS unit in our NOC ran out of battery power.
• We lost one crappy Windows server data disk as a result of the power outage. Fortunately this was a good test of our backups as we were able to slap new disks in the machine and restore from tape with no data lost. That burnt about 6 hours post power-outage.
• The weekend after the power outage, yet another Windows server decided to lose two disks simultaneously on it’s RAID 5 system where all of the server data is stored. 4 more hours burnt restoring that system.
• Prior to the power outage, two other UPS units simultaneously decided that their batteries were bad and needed replacing, further reducing our runtime on batteries during the outage.
• Returning home on the weekend after finishing the #2 restore job above, I sat down to work on the second edition of Wireless Hacks (link to the first edition). Removing my PowerBook from my laptop backpack, it slipped out of my fingers and took a double-header on my nice new wood floor. Result? Nice ding in the floor, and a PowerBook with a display so dim you can barely see it at all.
• Of course, even though AppleCare is a great warranty for PowerBooks, it doesn’t cover accidents. It will cost my employer over $1000 to fix the damn laptop. It’s a good thing my boss is understanding.
• I figured that my trusty old Dell C600 notebook would have to be my laptop during OSCON while the PowerBook is off getting repaired. However, as soon as I started using it regularly, in Linux or Windows, I found that the trackpad has gone completely loopy. At totally random times the cursor goes nuts - sometimes even if I’m not touching the trackpad at all. I can’t tell if it’s a pressure issue on the area around the trackpad, or if the pad itself is bad, but there’s certainly no time to fix it before I leave.

Fortunately we haven’t had any additional network, power or system issues (yet). I’m not looking forward to this weekend, though.

And if you see a depressed guy with little hair in a black t-shirt wandering around OSCON with an old G3 iBook instead of his G4 PowerBook, take pity on me and buy me a beer.

Got a better name for my law? Suggestions welcome.

Talk about timing—I was sitting in the tenth row at Apple’s Worldwide Developer Conference when Steve Jobs announced that Macs were switching to Intel chips. As the collective gasp from thousands of developers sucked the air out of the auditorium, I thought numbly of the new G5 FedExing its way to my home.

To put this in perspective, I squeeze a lot of use out of my computers, so I wanted to make sure I’d made the right choice. The last time I’d bought a Mac was in 1998, when Monika Lewinsky was frolicking around the White House and people still called MP3s “MPEGs.” I’d upgraded almost every possible component on that old beige beast (referring to the computer here, not the concubine), but it was definitely holding me back.

However, one thing it still did that no modern Mac can do is run a Korg OASYS PCI card. That way-overengineered board (original price, $2,300) is a combination synthesizer, multieffects processor, and 24-bit audio interface with terrific sound. The free 2.0 software upgrade gave it the samples from a Korg Triton as well.

So after I calmed down and realized I’d get plenty of great stuff out of my new computer, I started thinking about how to integrate it with my old one and a few choice pieces of MIDI gear. That’s when I discovered that Apple had pre-installed a slick diagramming program called OmniGraffle on my hard drive.

OmniGraffle lets you draw boxes and connect them with lines that snap into place “magnetically.” As you drag items around the screen, guidelines pop up to let you know when the items are aligned. It’s really easy and fast.

I plan to use it to design Website maps from now on, especially because each box can have an automatic drop shadow—something I’ve been adding manually to the site maps I produce for client proposals. There’s something about that little shadow that makes a boring “org chart” look so elegant that clients feel comfortable with it right away. I think drop shadow is the graphic artist’s version of digital reverb.

Speaking of music, here’s my very first OmniGraffle drawing, which helped a lot as I was wiring everything up:

The icons are photos I downloaded with the handy Google Image Search Widget from Anomaly Industries. This clever freebie bypasses Google’s annoying framed interface so that one click on an image takes you right to the full-size one online:

Placing the image in OmniGraffle takes one click as well. You just check the Image box in the Image Inspector window, and the program opens a file browser for you. Hit Enter to select the image you want, drag a slider to scale it, and you’re in business. You can export your drawing in a variety of formats, including vector-based ones. Positively presidential.

If drop shadow is the visual equivalent of reverb, what’s embossing?

I’m a photography newbie, but I’ve been considering a DSLR for some time now. I’ve read reviews of Mintolas, Nikons and Canons, but I’m still a bit confused.

Is there a noticable difference between shutter speeds of 1/4000 and 1/8000? Randall Schwartz wrote a piece a while back with some samples of a fountain in downtown Portland. The pictures were amazing, and just as he said, it seemed the water was frozen in time. Really beautiful shots.

Can I expect similar results from something with “only” 1/4000?

I’ve wanted to mention this in a public forum for a while, and now that my O’Reilly blog is up and running I think it’s the perfect venue. Podcasting can be cheap. I can say that now that I’ve spent several hundred dollars upgrading my podcast rig. When I began with my first podcasting adventures, on The Gadget Show, I started on a budget.

Let’s ignore the cost of a computer and bandwidth. Lets make the assumption that if you’re taking the plunge into making your own podcast that you’ve already got this organized. That’s not to de-emphasize this cost in any way, it is just that the cost of podcasting, Internet access, and social class is a completely different topic.

I’ll also leave hosting costs and software to a later post. All of which are important, but I’d never get to the point if I didn’t.

The total cost of the hardware for me to start was zero dollars. That’s right, nada. I cheated; a mate of mine loaned gave me a USB microphone (a $50 Logitech headset if you need to buy one). That’s all I needed to record a show. These days basic recording software comes with the operating system, so that isn’t even a factor.

I’ll confess though. I did spend $32 on Audio Hijack Pro. I wanted to record interviews over Skype, and on the OS X platform, it was the easiest way. For months though, that’s all I needed.

If you do start on a budget, it’s also worth investing in a coat hanger and pair of stockings. No, not as a method to avoid detection by the secret service, but to make your own Pop Shield. That way when you speak it’ll stop the plosives occurring in the recording from the letter P. O’Reilly’s in-house hardware hacker, Phil Torrone, would be proud.

If you’re keen to start a podcast, don’t let people convince you that an entry-level rig is worth a few hundred dollars. It isn’t necessary to launch your own assault on the world’s eardrums. Wait until you’ve given it a try, and if it’s something that you enjoy you can invest more money later.

Successful podcasts aren’t made by hardware alone.

Yes, I looked like an idiot with stockings wrapped around my head.

Have you got advice for a newbie-caster? Leave your comments here.

I decided that my first ever O’Reilly blog post should be a complaint. I’m quite fond of complaining.

To be fair, I’m talking only about the c3750. Here’s the deal:
The 3750 supports IPv6 in hardware, sort of; more on that later. When we first purchased many 3750s we were blissfully unaware that IPv6 software wasn’t available yet. A few years later (really) the Advanced IP Services image came out with IPv6 support and Cisco wanted another chunk of change for it.
Ok, fine.

Running the new image with our previously configured policy routing along side IPv6 required that we adjust the TCAM allocation. Unfortunately, there was no option in sdm prefer to allocate TCAMs for both policy routing and IPv6. It is a cheap router, and we implemented an L2 solution for the thing policy routes used to do, so this wasn’t that big of a deal. Annoying, sure, but our prayers to the hardware gods went unnoticed, so we still have a limited amount of TCAMs.

router#sh sdm prefer
 The current template is "desktop IPv4 and IPv6 default" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

Eight isn’t really enough.. (you have to be really old to get that joke)

After a reboot and configuring OSPF to speak IPv6, we started bringing up interfaces. Everything worked quite nicely, until we actually started to use it. Linux clients nabbed an IP very quickly, Solaris required starting in.ndpd, an interface plumb, and touch /etc/hostname6.eri0, and Windows required ipv6 install then ipv6 if. Oh, Windows clients. That reminds me: we need ACLs.

The terror begins. You can create IPv6 ACLs until your fingers fall off, but actually applying them to an interface is impossible. It seems the required command is missing! You cannot just “conf t” “int vl1″ “ip access-group” anymore. Buried in the documentation, Cisco shows us:

(config-if)# ipv6 traffic-filter  { in | out }

This yields:

(config-if)#ipv6 t?
% Unrecognized command

The command is actually missing. We were using: c3750-advipservicesk9-mz.12.2-25.SEB1 at the time. There was a slightly newer rev available, so we assumed it had some bug fixes. This is where the next major bug reared its head. After installing the new image on the group of switches, it failed to reboot. Long story short, the switch won’t boot with IPv6 interfaces configured. Remove them, boot, add them back, and everything works again. Except we can’t protect the Windows boxes, since the command to apply ACLs is missing.

I should note that the Juniper routers had no problems with IPv6. I haven’t tried it on any expensive Cisco routers yet, but I assume it should work a bit better. So let’s recap some of the unadvertised features you get on the c3750 (with an expensive IP Services image):

• Policy routing and IPv6 cannot live together
• ACLs exist but you can’t apply them to an interface
• You cannot boot the router with IPv6 interfaces configured
• And finally, I forgot to mention, there’s a memory leak if you disable OSPFv3

The last item wasn’t a problem for me, but I’m sure it will bite someone. Quite a list, isn’t it? We’d like to offer IPv6 natively (non-tunneled) for the University, but certain vendors definitely don’t make it easy.

I’d love to hear other people’s experiences with Cisco+IPv6 on 3750s or other devices.

Related link: http://www.oreillynet.com/pub/a/oreilly/digitalmedia/2005/05/18/ds2.html

I’ve been getting lots of mail about my enthusiastic review of the Olympus DS-2 stereo voice recorder. Several readers wondered what I thought of the WS-200S, a smaller, higher-capacity model that came out shortly after I finished the review.

Well, because I really liked the DS-2 but found it a tad too big for my (admittedly overcrowded) pocket, I took a chance and bought a WS online after returning the DS-2. Here’s what I discovered.

Left to right: Olympus WS-200S, DS-2, and V-90, the ancient model that got me hooked on voice recorders. The WS-200S image is pasted on, but accurately sized.

As you can see in the photo, the WS is significantly smaller than the DS-2, which made it the perfect size for me. What you can’t see is that the silver and black halves slide apart, revealing a USB plug, which was a feature I really wanted in a voice recorder. (The DS-2 requires a mini-USB cable or docking station to initiate computer file transfers.) There have already been several times when the WS saved the day by acting as a pocket flash drive. It also has double the memory of the DS-2, 128MB.

MUSICAL SURPRISE

One thing I hadn’t thought to try with the DS-2 was playing back prerecorded music files. (Its 64MB of memory would hold only an hour of music at a listenable resolution.) But the WS’s extra memory inspired me to try, so I dragged a WMA music file into the “E” folder. It worked! I could listen to the music through the stereo headphone output as well as the WS’s 16mm mono speaker. Unfortunately, the DS-2 and WS don’t play MP3s. Encrypted WMAs and WMAs with very high bitrates didn’t work either. (And as far as I know, there’s currently no way to make a WMA file on a Mac short of running Virtual PC. Let me know.)

Unlike the DS-2, the WS has both an on/off switch and a “hold” switch, which disables the other buttons so you won’t press them accidentally. When the WS is switched completely off, it takes about five seconds to boot up, which is annoying. However, leaving it on and in Hold mode doesn’t seem to drain the batteries, so that’s what I do; the battery gauge hasn’t budged in three weeks. Both models take only a second to boot up when switched on from Hold mode.

I also found the DS-2 slightly easier to navigate, thanks to its larger, better-labeled buttons and larger display.

Another thing you can’t tell from the spec lists is that the WS picks up more handling noise than the DS-2. (O’Reilly blogger Giles Turnbull also noted this shortcoming in his quick review of the WS, though it wasn’t clear if he had the mics in high-sensitivity mode.) I’m now careful not to slide my fingers on the case when recording, and to wait a second after the sound I’m recording has stopped before I click the Stop button. That lets me chop off the inevitable thump later in my audio editor.

Speaking of which, the WS doesn’t come with Olympus’s DSS Player software, so to edit recordings, Mac users will have to convert the Olympus WMA files to another format. (Several WMA editors exist on the Windows side.) The shareware program EasyWMA works well for that. I still had the DSS Player software on my computer and was happy to see that it worked with the WS as well.

For transcription, Mac users can try my free AppleScripts, described in “Turn Your Mac into an Audio Transcriber.” A couple of those scripts started misbehaving in QuickTime 7, so I’m currently rewriting them.

FIRE AWAY!

Can’t have a shootout without bullet points, can we? Here’s a list of the benefits each recorder brings to this showdown. (Read my full DS-2 review for more background.)

WS-200S Benefits

• Significantly smaller
• Integrated USB plug
• Twice the memory
• Speaker is on the front
• Uses single AAA battery (DS-2 requires two)
• Slick-looking dual LEDs
• Less expensive (currently $103 on Amazon vs. $119 for the DS-2)

DS-2 Benefits

• Multisegment recording level meters
• Backlight
• Less handling noise
• Bigger buttons and screen
• File naming
• Folder naming
• Alarm
• Timer recording
• Two noise reduction settings
• More Fast Forward and Slow playback speeds, with better sound quality
• Looping
• Bigger, better-sounding speaker (23mm vs. 16mm)
• Docking station
• Carrying case
• Works as USB speaker and microphone
• Comes with DSS software for editing, converting, and playing back files

You might think that getting all that for just $16 more would make choosing the DS-2 a no-brainer, and for most people, I’d agree. For me, though, the WS’s smaller size and integrated USB plug made it the better choice. Because uploading recordings to a computer is so easy, I haven’t come close to filling up the memory with recordings, though I did max it out when using the WS as a thumb drive.

For around $200, you can get a version of the DS-2 with double the memory; it’s called the DS-20.

I also heard from a reporter for Fortune who opted for an Olympus DM-20, also about $200. This model looks very similar to the DS-20, but with a full metal case and the ability to play MP3s. Although it supports optional external stereo mics, its built-in mic is mono and has a frequency response that tops out at just 8kHz.

Here are some links for further comparison.

DS-2 product page

DS-20 product page (not yet listed at Olympus, so this link goes to RadioShack)

WS-200S product page

DM-20 product page

Interactive comparison table

MY RECOMMENDATIONS

If size and integrated USB plug are important, get the WS-200S. If tons of features and enhanced usability are important, get the DS-2 or DS20. (The backlight and alarm are pretty handy, and so is file naming.) If you want to carry an occasional MP3 and don’t need built-in stereo mics, go with the DM-20.

Happy recording! What gadgets shall I tackle next?

Update, 2007-02-08: I just noticed Olympus has a new line of voice recorders, the DS-30 and 40, with 128kbps WMA recording.

Related link: http://news.bbc.co.uk/1/hi/technology/4683385.stm

“I am predicting 50 years of chaos,” says digital media thinker Clay Shirky. He’s been speaking at the TED (Technology, Entertainment and Design) Global conference in Oxford, UK. Story here.

Sounds about right, though I’d guess it could be 50 years worth of chaos crammed into as little as 10 years. Hey, it’ll be just like the ’90’s again.

Even though there’s a large constituency emerging for the blanket license model as advocated by the EFF, the big record labels, through the RIAA, will fight it for as long as they can. The idea looks efficient and clean on its surface: Let people stream and file-share to their hearts’ content in exchange for a surcharge on their ISP bill (or some other levy, such as on computer audio equipment), and allocate money according by tracking popularity. But the major labels hate it because it would take away their control over collecting revenue and give it to a central bureaucracy similar to a performing rights organization, such as ASCAP, which to them looks like socialism. They also point out that there are other forms of media that would also have to be paid for: TV, movies, print, etc.

Given that the big labels control the greatest part of currently popular content, their resistance will continue to make a big difference until such time as someone can demonstrate the real world success of a better system. Free P2P is way more popular and growing faster than any pay distribution system, but record industry sales seem to have bounced off a bottom and are rising again - the claim that file-sharing doesn’t hurt sales cuts both ways. So, for the foreseeable future, it looks like a complicated, messy struggle in an ecosystem of alternative business models.

Usually, I stick to the PHP / HTML / XML aspects of the web, as I tinker around with my blog. Recently, however, I’ve been thinking quite a bit about wikis. This is most likely because the group mind of the web seems to be fixated upon wikis and podcasts of late, as the two hot technologies to gab about. However, although I’m liking iTunes’ new podcasting support, I’m not the biggest podcast fan, because I read faster and with more focus than I listen.

On the other hand, I see a lot of untapped usefulness in wikis. I recently polled the readers over at TUAW to see what they thought were the best wikis out there for the Mac, but my interest isn’t in running it exclusively as a Mac platform. I really want a powerful community-based wiki for a mixed Mac, Windows, UNIX, and Linux population.

I started searching around. I looked at a few resources, like the Social Software Weblog, and then I started visiting several different wikis that I use regularly and scrolled down to the bottom of their pages to discover what wiki package made them tick. Here’s a list of a few of them that I’ve been looking at so far. Please feel free to contribute your thoughts and impressions to this list in the comments and to correct any mistakes I’ve made in my early assessment of these packages:

• MediaWiki—Wikipedia is undeniably the best known wiki out there. I’m starting to hear Wikipedia mentioned by regular users, just like Google suddenly emerged in the normal user jargon a few years ago, and Wikipedia is often discussed among the Academics with whom I work. Wikipedia runs on MediaWiki, which looks to be the most robust wiki package out there. It has a rather impressive feature list, but while it’s a very well-documented project, it’s also a bit daunting. I’m already test-flying this one, and while running the basics of the wiki are relatively easy, there are lots of features, customization options, and different administrative options that I haven’t even begun to touch. There’s a wiki guide to writing pages with wiki formatting in MediaWiki, which is helpful, but I’m anticipating a rather steep learning curve for supervising, deploying and maintaining a MediaWiki based wiki in an Academic setting. Fun!
• PHPWiki—As a Mac blogger, I keep my eye on interesting Mac-oriented weblogs, and The Tao of Mac has been on my radar for quite some time, both because it has good content and because it is part blog and part wiki. According to the credits page, the Tao of Mac wiki is a highly-customized variant of PhpWiki. PHP is familiar ground for me, so I immediately think that the learning curve will be less than with MediaWiki, and this is a package I need to spend some time with. I also like that it would appear to be a highly scalable package, as it’s being used for a single person website by Tao of Mac and in more collaborative settings by others.
• Wikka / Twiki—one of my good friends, Jay Savage, recently launched EngaTiki, which aims to “be a massive public annotated bibliography in Literature and the Humanities.” I talked to Jay and he said he was using Wikka for EngaTiki, and was pleased with it, but that he also really liked Twiki, which I immediately like for its bidibidibidi shout-out to a bygone TV show. I haven’t played with either of these packages, but I trust Jay’s advice and intend to give them both a thorough test in the near future.

So, what am I missing? Is there a better wiki that I am leaving overlooked? Should I be considering more options? Can you recommend any other wiki packages or documentation for these wiki packages? If this blog post were a wiki, you’d be able to amend and correct this list, but since it’s not, feel free to collaborate with your input in the comments.

What other wikis or wiki-related resources do you recommend?

I rather expected the recent W3C Schema Experience Workshop would be a snow job.
People who make something are not the proper people to
conduct a review of it. It the basic principle of auditing,
and of code reviewing. If I had evangelized my employer and its customers to invest heavily and strategically in a technology I had been pivotal of developing, it is unrealistic to expect me to impartially filter and adjudge user feedback and ultimately come up with anything other than variants of “the teething period is taking longer than expected, don’t panic, no change is required.”

So right from the start, when looking at this workshop,
I think we need to acknowledge that we should not imagine it to be an independent review. In order for that, the ringleaders (Henry Thomson, Noah Mendelson, et al), the divide-and-conquerers, and the rest of us meek, herded sheep from the W3C XML Schema Working Group should have to have been kept out of the loop. The Workshop doesn’t seem to have been a complete festival of arse-covering; but the giddy danger of unshaded posteriors loomed large.

Minutes are online at

http://www.w3.org/2005/06/21-xsd-user-minutes.html
and

http://www.w3.org/2005/06/22-xsd-user-minutes.html.
Presubmitted comments, including one by me (endorsed to some extent by Microsoft see below), are at

http://www.w3.org/2005/05/25-schema/all.zip.
All presentations are at

http://www.w3.org/2005/06/presentations.zip.
Thanks to Mike Champion for the heads-up for these, on
XML-DEV.

So even though I am sceptical that the workshop could result in any adjustment of course for XML Schemas, the input reports are extremely interesting. Here are some quotes: do you see any pattern?

• ACORD: …we have random implementations and understandings of the
  specification and it is usually up to the unwary user at the most
  inopportune moment to find out that a feature is not supported.

• BEA: we believe that the complexity of Schema is, in many scenarios,
  unnecessary, and often actively harmful. More than anything, this is the
  feedback we hear from our customers and end users. We are also concerned
  that Schema’s complexity may be the root cause of so many incomplete
  and/or incorrect implementations.

• BT: In our experience, XML Schema is implemented inconsistently in vendor
  tools, especially those which used schemas to generate mappings into code
  and other forms of data. Working around interoperability issues with
  vendor supplied tools is difficult and sometimes impossible when using XML
  Schemas published by third-parties, standards organisations and
  consortia.

• HL7: HL7’s experiences with designing schemas that work across a broad
  array of tools has been extremely disheartening.

• IBM: Anecdotal feedback from the Web services community has suggested
  that inconsistent support for certain valid XML Schema constructs across
  the various development tool environments has contributed significantly to
  the interoperability challenges faced by Web services developers.

• Microsoft: Rick Jelliffe [RJ05] describes the situation faced by an
  actual customer where incomplete support for XSD in various products has
  “stuffed up the ready interoperability they thought they were buying into
  with XML.” That sums up the problem nicely and mirrors the experience of
  many of our users.

• OAGi: Complex type derivation by restriction simply does not work.
• Rogue Wave: Many customer issues come from schemas that are not valid. In
  almost all cases this is the result of a schema generated by a tool.

• SAP: In a heterogeneous environment, usability, implementation and
  language binding issues impact the choice of features that are supported.
  This led SAP to have different levels of support for XML Schema features
  due to the impedance mismatch between the language constructs and XML
  Schema constructs or the frequency of use of certain XML Schema
  constructs. SAP favors this direction (a profiling mechanism) as a
  catalog of profiles and their constraints could then be published and used
  as basis for interoperability when designing a certain class of language
  bindings and applications as well as business vocabularies. Further, it
  would help reduce interoperability issues for hard to implement and
  understand features of XML Schema.

• Sun: When people run these broken schemas against our schema compiler, we
  reject it as an error, which only make them think that ours is broken.
  This trouble-shooting can get quite complicated if it involves in a type
  hierarchy, substitution groups, and/or wildcards.

• WS-I: …few web services implementers use validating XML Schema
  processors. Many users “validate” SOAP messages using only inherent
  SOAP-processing mechanisms, possible with some uncoordinated help from
  type serializers. This situation often means that XML Schema constructs
  like type facets and PSVI are ignored when web services messages are being
  processed, which in turn discourages the use of such constructs by Schema
  Authors.

So tool-makers blame users for generating non-standard schemas, and users blame the spec for being to difficult to know whether their schemas are standard or not, and spec makers blame tool makers for not implementing the spec properly. Who will free us from this cycle of sin and death?

The W3C response is a

databinding best practices blog and group.
Better than nothing?

Syntactic ambiguity versus semantic ambiguity

Reading the material, there is almost constant agreement that the UPA is a frequent stumbling block. (The Unique Particle Attribution rule is XML Schema’s mechanism to prevent schema ambiguity.) I was disappointed to see that the usual spurious comments still being made about RELAX NG, which doesn’t require UPA: first that it couldn’t be used for data binding and second that annotations would be lost without a UPA equivalent. The approach RELAX-NG takes is to allow ambiguity in the base, and then move UPA checking to be an application-dependent constraint. (The idea of layering XML Schemas is anathema to some people, however.) And for unambiguous annotations, for RELAX NG that there are two branches in the grammar is a syntax issue not a semantics issue–the annotations on syntactically alternate but semantically identical nodes will be identical: a schema validator (a nice clean extra layer) can check this.

What to do?

My own take on all this? Well, I am surprised that the user experience feedback is so frank and so bad. Like I said, I expected a snow job.
XML Schemas needs to be refactored so that it

• uses current syntax and namespace where possible
• allows at its base something more like the RELAX NG static-typing structural model, with its semantic-not-syntactic UPA model
• uses XQuery datatype model: keep XSD datatypes as much as possible
• factors out runtime or dynamic typing (xsi:type), runtime schema discovery model, key/uniqueness, derivation-checking (!), and syntactic UPA checking to other related optional parts
• embedded Schematron for co-constraints

The only way that XML Schemas can be refactored is
with a different core XML Schemas working group.
My current expectation is that a lot of nothing
will happen until XQuery/XSLT2 becomes seen as a more central technology than XML Schemas; the goal will then be how to support XQuery most minimally.

In other words, I don’t see the PSVI and the (XQuery) XML Schemas datatypes/facets going away. But all the rest needs to be refactored and layered: wildcards, type derivation, uniqueness and references, lax/strict/NVDL validation, runtime checking and so on. The car needs a new engine.

Which versus How

But I think there is an architectural difficulty that also needs to be addressed. A SOAP envelope and the SOAP payload belong to different layers, and it is incorrect, in general, to think of a schema for the document or even of one effective schema made from all the schemas for the various namespaces. Each layer in a multi-layer protocol is concerned with the semantics of its control data but only the minimum syntax of its payload. When using XML, control data envelopes the payload, confusing the layers. The result is that people either validate too much or not at all.
Schematron’s phase mechanism and ISO DSDL’s Part 4
>Namespace-based validation dispatching language
(NVDL) get it more right.

What DSDL Part 4 gets right is that selecting which information to validate should be a separate issue from how the information is validated.
The Schema Experience Workshop has some people wanting
better wildcards, and other people wanting fewer: the
superficial conclusion is that these are irreconcilable differences and so they cannot both be satisfied. But many uses cases could be satisfied by taking wildcards out from the schema language, and adding a projection layer, such as NVDL.

I think there is the glimmer of awareness of this:
Henry Thompson has recycled James Clark’s double validation
method (used by James to validate XHTML) for a certain XML Schema issue, and David Orchard and others have been working on schema projection, which is like NVDL but using DBMS concepts.

But I bet you won’t be seeing Ads for XSD tools or systems with “extremely disheartening”, “difficult and sometimes impossible”, and “trouble-shooting can get
quite complicated” :-) or will we get nothing but sunshine, snow and anal modesty.

Related link: http://longtail.typepad.com/the_long_tail/2005/07/prefiltering_vs.html

Most discussions of new digital media business models get around to filters pretty fast. Digital tech takes the cost of production and distribution pretty close to zero. This leads to an infinite explosion of content, so now there’s value in filters that help you find the good stuff.

As Long Tail originator Chris Anderson put it recently:

“…the day when people like me [i.e. magazine editors] decide what makes it to market and what doesn’t is fading. Soon everything will make it to market and the real opportunity will be in sorting it all out.”

Here’s the part I keep wrestling with: What’s the new model filter for mojo? That is, the mystique that makes a large number of people want to buy a piece of pop culture. All the new filters I know of rely on recommendations based on assessments of merit, as in the case of blogs or Weed or garageband.com. But as much as I like and root for some of these filters, I’m not sure any has yet got to what makes people buy in large enough quantities to make it pay a living wage for the artist, not just the filter.

I don’t think most people buy music or magazines or movies because of merit. They buy them as an act of tribal identification and participation. Merit of some sort is almost always necessary, but it isn’t sufficient - good reviews, on their own, usually don’t drive sales. Another way of looking at it is that in pop entertainment, pop culture is the medium: The pop artist modulates pop culture like a trumpet player modulates the air. Modulating pop culture is what trends are about. And a trend traditionally has been jump-started by a big marketing investment.

This may be why the new filters have yet to produce hits. Garageband.com can point to Bo Bice making it to 2nd place in the most recent American Idol contest, as well as Geoff Byrd gaining an independent record deal. But that means that the artists are now the beneficiaries of old-style, celebrity-based marketing. And note that while American Idol is younger than many of the new filters, unlike them it has already produced several hit performers.

So what’s the new filter for mojo? Is there one?

Related link: http://www.creativesynth.com/BLOG/C34648494/E1182853940/index.html

Not only does Darwin Grosse have the perfect name for a James Bond supervillain, he also runs a superb site on electronic music production, CreativeSynth.com.

While browsing the site today, I came across a blog entry called “My (Personal) Top Ten” that got me thinking. In this entertaining list, Darwin—who’s also the CTO of Cycling ’74—recalls some of the groundbreaking music technology that inspired him. His retrospective on the Kurzweil K2000 is especially thought-provoking. It made me remember how after my first synthesizer was stolen, the replacement (the exact same model) never quite gave me the same connected feeling. I think even electronic instruments start to take on the player’s personality after a while, particularly if you create your own sounds.

And that reminded me of a conversation I had with Darwin earlier this year, when he revealed that he came up with a bunch of patch names for one Cycling ’74 product by plundering a superhero encyclopedia. I’m sure the mere act of naming a patch, say, “Eggplant Man” caused people to use it in a certain way.

Anyhow, check out the site. You’ll find a gross of inspiring ideas.

What’s in your Top 10?

Report from FlashForward NYC 2005 - Wed July 6th. Day 1

FlashForward kicked off today. I caught only the tail end of the keynote by Ze Frank, but learned the new term “Authorship Society.”

To quote the blurb: “The transition to an authorship society brings with it massive amounts of content to consume, but it also brings a new relationship between our society and the notion of creativity. Frank will explore the notion of real-time creativity and how it affects the way we think about teaching and learning.”

The well-known ActionScript developer gave a talk on the issues (forces) in deploying Flash for different platforms and devices. I was familiar with most of it but did pick up a few tidbits. Most notably, Flash 7.2 includes a new ActionScript class, Delegate , that lets you execute a script within a certain scope/context. This makes it much easier to write callback handlers that are scoped to another class instead of the callback object itself. See the Flash MX 2004 online help or Macromedia.com’s LiveDocs for details.

Peter Hall showed his Ripple framework for creating HTML-driven Flash pages. Why would you want to do that? Well, if you structure it right, you get two-for-one, both a Flash site and HTML site for not too much extra work. And, it makes it much easier to make your Flash site indexable by Google and other serach engines. And it lets you add HTML hyperlinks to or from any Flash page in your presentation automatically. And it includes a history object, with support for back/forward (and future integration with the browser’s back/forward buttons). In short, if you have clients with one foot still in the HTML world, or you want to create HTML (or XML) driven Flash sites, give Ripple a look.

I spent the next hour learning about Processing, a very nifty little programming tool that exports to Java. Amit Pitaru demonstrated how to create a drawing program with a color picker in just a few lines of code. Very easy to use. I’ll definitely try my hand at programming some ideas I’ve had for a while that aren’t very easy to implement in Director or Flash.

Tom Igoe gave an interesting talk on Physical Computing. He demonstrated using a stuff monkey (monskey) with flex inputs attached to its arem to control two pong paddles. He also covered some basics on the hardware available for creating your own projects with inputs from the physical world. Very cool stuff and prices for hardware have come down considerably. And the software is improving too. He demonstrated stuff with Processing, but you can use other environments as well.

Well, that is my report on Day 1 from the field. Unfortunately, I won’t be able to attend the second and third day, but I’m sure others will blog it. This morning there was a keynote by Kevin Lynch with more excitement to come, undoubtedly.

Related link: http://msn-cnet.com.com/4520-6475_7-6255653-1.html?part=msn-cnet

Video space shifting article.