This particular essay has been sitting as a draft on my laptop for some time. I’ve been intrigued lately by the bigger issues facing the IT market, and especially of the people who work in it. The news of late seems to be ominously similar in tone no matter where you go, that even in the face of softening employment elsewhere companies can’t fill programmers fast enough. I don’t think this is peculiar to this particular moment, but may in fact shadow a significant shift in the relationship of the IT workforce to the companies who employs it. My apologies for the length, but there’s a lot to cover …

An interesting paradox is occurring right now in the IT market, one that I predicted back in 2002 when jobs for programmers were as scarce as new IPOs about then. In a nutshell, even as the general economy is beginning to suffer from significant doldrums, the demand for skilled IT professionals has seldom been higher … and many companies are having to shelve new production because they can’t find the programmers to build their applications … globally. From the United States and Canada to England, Germany, India, China, Japan, Australia … there just don’t seem to be enough skilled programmers to fill more than about fifty percent of all IT-related jobs.

A number of reasons for this have been pushed forward for this, each with a certain degree of merit. Demographics certainly is a factor - the typical image of the programmer - a male in his mid-twenties or early thirties, usually with at least a four year college education, factors into every recruiter’s mind when they are hiring, usually for good reason. Such programmers are typically single (and hence don’t mind working 12 and 14 hour days), don’t necessarily know their real full worth, are fairly easily malleable, and are often most adept with the hottest technology of the day. The older you get, the less BS you’re willing to put up with, the higher your rates, and typically the greater your expectations that you will be involved at a management level, none of which is attractive to a company that’s hoping to get the most for the least.

Yet the workforce is getting older. The boomer generation’s birth-rate peaked in 1955, which puts the echo boomers - their children, hitting their peak back in 1980. The average age of the Echo generation is now 27 years old, or, put another (and rather disturbing way), the demographic trends indicate a fairly significant drop-off of “prime” technical talent from this year on, with the next major bump not likely to occur until nearly 2020 (a considerably smaller bump, as it’s the peak of the smaller GenX population’s children).

The market itself also shoulders a fair amount of blame on this. Stock options are a form of corporate roulette. For those early enough into the game, stock options proved to be incredibly powerful, both because the likelihood of a company increasing in valuation several thousand-fold was very real and because, from the standpoint of many companies, stock-options meant that rather than the company actually bearing the cost of paying for good talent, the company was able to push that cost to the stock market; in essence, the stock speculators became the real owners of that talent. I think that this head the effect of salaries being undervalued significantly in the market for good IT professionals, most of whom either figured or were coerced into believing that their real compensation would come when the company went public.

The tech market collapse in the early part of this decade gave the lie to that. Many inventive people ended up giving up their key marketable products to companies that paid them poorly then dismissed them when the market was no longer there to keep the float going. Now there’s a funny thing about both programmers and engineers: they are not stupid. They may appear a little naive sometimes, somewhat disconnected from the rest of the world, but that has more to do with the fact that they are thinking about considerably more abstract issues than just winning the latest account or who was booted off American Idol or whether this tie goes with that shirt. Programmers don’t like being lied to, and once their trust is lost, its hard to get it back, because the effect of such trust is that they HAVE to deal with the real world, something they generally don’t like to do.

This has had several effects. Many people with solid IT skills went into other areas, returning to school in great numbers during the tech winter to retrain themselves as doctors, lawyers, managers or teachers, among other professions, or they left pure IT to go into biotech. They discouraged young college students from entering the field, meaning that many people that may have trained in IT have instead gone elsewhere. The ones that had made a fair amount of money during the boom became entrepeneurs themselves, often starting tech companies that in turn absorbed the available talent even faster. Others followed the money into financial services or real estate, and while some may be contemplating a return to IT as that market has softened its unlikely that there will be a mass influx of older but wiser programmers.

The problem with IT skills, you see, is that they degrade quickly. If you are out of the marketplace for a year, you face a challenge getting back up to speed in your field. If you are out of the marketplace for more than two years, then the barriers to re-entry are almost insurmountable (a point I’ll come back to in a bit). This means that for many people, once you leave IT for any significant duration, you’re effectively out of it completely, at least as far as programming goes.

Moreover, a fairly significant number of developers who entered the field in the late 1990s came from the legacy field in response to Y2K - in other words, they were already semi-retired from programming to begin with, and were drawn back because their particular languages - COBOL, RPG, etc., were in significant demand. This helped pad retirement accounts, but again with the tech winter, most of these people have moved back out of active programming, this time likely for good.

The upshot of this is that there is a continuing drain upon the IT field due to programmers moving on or moving out, coupled with an impending drop due to raw demographics coupled with an overall negative image being painted of the IT field at the college level that are whittling away all but the most dedicated developers before they even graduate.

Educational funding and priorities have also played a part in this process, unfortunately. University level curricula take three to eight years to implement, typically, which means that they are usually quite adept at teaching what was contemporary half a decade before. Thus, it’s likely that courses in Ruby or AJAX will likely not appear in the typical university catalog until 2009 at the earliest. The community colleges are generally more nimble, though they too have a lag time that they have to fight.

While this can be seen as a necessary brake on “fad languages”, the reality is that from the time that a person enters college, they will be looking at seven to eight years before they are even at a journeyman programmer level, of which nearly half that ends up being borne as “on-the-job” training to companies. Those same companies, of course, see such OTJ training as a drain on their internal resources, and usually end up outsourcing that training to certification companies (often with the programmer picking up the tab for that training). Moreover, such companies also fear (with some justification) that once an employee receives an adequate amount of technical training, they will take those new skills to an employer willing to pay a higher wage.

All this has, of course, occurred against the backdrop of the US government shifting its spending priorities from education and training to fighting wars on multiple fronts. This has meant that in general colleges have been receiving less funding for technical education programs, except in the context of military training, which in turn means that even among a shrinking demographic, the people most likely to get university or even college training are those who can afford increasingly expensive schools with fewer grants and now increasingly expensive loans as mortgage rate increases dry up credit liquidity.

Choosing a career is making an investment, and as the number of tech IPOs decrease relative to the overall stock market, as the amount of open source code reduces the market niches for software companies, as software as a service kicks into high gear, the opportunities for the “big kill” became increasingly rare. This doesn’t mean that it won’t continue to happen, mind you, but what happens is that programming is rapidly becoming yet another one of the “Lottery Professions” - underpaid for the work and time invested in 99.9% of the case, but a multibillion dollar lotto winner in that remaining tenth of one percent.

Such lottery professions aren’t limited to software, of course. Any creative profession is beginning to look like this - if you are already well connected, have the means to be at the right place at the right time, or just happen to accidentally stumble upon the ever shifting public zeitgeist, you do well. Otherwise, its not worth the time. Of course, even this tends to work against employers - given the danger of being too far from the forefront of the tech revolution (where you’re able to buy more lotto tickets, if you will), programmers tend to instinctively shy away from older tech, even if it appears more lucrative at first, because the opportunities for the “big breaks” are fewer and farther between.

What this means to many companies is simple - the compensation and training costs for technical employees is not a drain on the coffers but an investment to keep them with the company. In one company I saw, one lynchpin employee - a senior architect - left the company because they had both cut pay and dramatically cut back on training in a “cost-cutting measure”. His departure led to the departure of nearly a dozen other programmers within the next six weeks (they saw the handwriting on the wall) which in turn meant the company lost several multimillion dollar contracts because they didn’t have the staff to bid for them. Many other companies that decimated their engineering and technical staff during the tech recession (often for no reason other than everyone else was doing it) have since discovered that they can’t get back even to half of their pre-2000 levels in  IT and the people they are hiring are considerably less seasoned.

Of course, there’s always outsourcing … or is there? The same demographic trends that are hitting the US are hitting globally - and indeed, the US is probable safer here. This means that many countries are now competing for the same IT talent that previously was supplying the US with IT workers during the 1990s. Moreover, many of these places have brought back the wealth from those lucrative contracts and reinvested at home in order to build up their own domestic IT infrastructure, because of the demand for it there. Thus, the number of people available to outsource has been dropping pretty dramatically as well. Add into this the fact that the US dollar has fallen in value by anywhere from 10% to 30% in the last decade against most currencies in the world, and the cost benefits to outsourcing have quickly disappeared.

The same trend is occurring here in Canada, of course, though on the other side - its become increasingly difficult for Canadian software shops, production studios or the like to compete, because they can’t hire the talent, the talent they can hire is consequently paid considerably more, and the rising Loonie (up more than 25% in the last three years) reduces the cost advantage of going to Canada in the first place.

I personally believe that the days of cheap IT labor are behind us for several years to come at a minimum. In the short run, this is probably going to be a good time to be in IT if you’re a programmer. In the long run its likely the herald that the “software age” has reached a major transition point. Companies are already beginning to scale back IT projects because they can’t get enough developers in house, and their traditionally suppliers of talent, IT body shops, are ironically in a great deal of trouble.

The IT body shops of the late 90s did quite well in the dot-com rush because they could hire people cheaply and apply a significant markup, most of which was pure profit, largely because those programmers were fairly young, inexperienced, and eager to break into the market. During the dot-com bust, they had more people on their rosters than they could find work for, and while things were lean even for them, they tended to be very dismissive of IT people. Now, while they’re getting significant bounties for finding talent, the number of people they have on the bench, the ones that earn them the bulk of their income, is leaner than its ever been, and most even semi-experienced programmers recognize that they will do better to go through the companies themselves rather than letting the body shops agent them.

While the short term looks good for developers as a consequence, the longer term benefits for the industry overall is considerably more unsettled. Tech jobs in general are quite attractive to most politicians - they pay well (and consequently can be taxed well), they tend to attract a core of intelligent people who often participate disproportionately in the cultural life of their community, tech jobs are generally “green” in that they require comparatively little polluting infrastructure to sustain, and they tend to have a comparatively light footprint in terms of crime, drug abuse, and other societal “ills” compared to other groups.

Yet these jobs are also in danger of disappearing - projects unfulfilled get canceled and new projects can’t find the people to staff them, the growing base of services oriented software (and open source software) tends to fulfill the “good enough” needs of many who need software services, software developers end up working remotely as the technology has improved, meaning that it becomes more difficult to build that “critical mass” of developers that have long defined places like Seattle, Boston or San Francisco. I don’t think that we’ll have another market collapse similar to 2001-2004 (at least not in the tech sector); rather there is going to be a long, slow attrition, which will push IT managers to focus on software that is better able to adapt to business conditions and thus require fewer people to maintain.

As economic conditions improved in 2005, many companies started shifting their resources away from consultants (and service shops) and towards building up in-house staffs. However, the cost of maintaining that staff will continue to climb, and ultimately, especially as the US economy tilts into recession, its likely that such staff positions will be phased out in favor of near-source vendors for all but the most core maintenance functions. I don’t see this necessarily impacting salaries or job availability significantly for the developer, it just shifts which company they work for (and likely increases dramatically the ability for more senior developers to run their own companies or consultancies).

In other words, the long term trend for programmers is towards becoming consultants or joining small (up to maybe twenty five or so) specialty shops, and for those programmers to consequently shift away from being generalists to being specialists in a particular domain (in other words, their work relationships begins to look a lot more like that of doctors or lawyers than it does of tradesmen and women). Even in large software development companies (such as Microsoft, Sun, Google, or IBM) this shift is reshaping the way that such developers interact with companies - moving towards small, self-contained units that have a high degree of autonomy and where the individual members of those units are as likely to be working remotely as commuting in daily to the corporate headquarters (put another way, the campus model is dying, to be replaced by satellite campuses and telecommutes).

Gridlock and explosive housing prices factor into this as well - you may be able to make $150K a year in San Francisco as a developer, but kick in taxes, high gas and food prices, absurd housing prices and time lost due to commuting (not to mention the stress on yourself, your family, and your car), and the amount at the end of the month doesn’t look anywhere near as attractive as it might have otherwise.

Offshore development shops (where offshore is defined as being where the labor is cheapest), so indispensable to the corporate bottom line in the 1990s, are now becoming far less attractive  as the cost of managing those resources rise and the competitive pressures on these same outsource agents force them to raise their own rates. At the same time, most developers in the IT field have adapted to working remotely, and it is the rare programmer who doesn’t have projects coming from outside their driving range. This is not as advantageous to business, but after nearly three decades of being in a position where they could dominate the labor/business relationship, demographics and technology are now both shifting that power to give programmers much more say in how they work.

This is the direction of the future. By 2010, qualifications for a project manager will include the ability to coordinate programmers from all over the world and the compensation model will evolve into a fixed-fee plus royalty basis (perhaps with a stake in the project as a side bet) rather than a per-hour basis, since the latter ultimately requires someone with a clock being able to track individual workers, difficult enough in a closed environment and impossible in an open, distributed one. The expectations on the developer will be that the they will become responsible for the purchase of their own equipment (most prefer that anyway, as a contract gone sour isn’t going to leave them without a computer when they most need it), pension plans, health care and so forth. There’s a business there for someone who can supply at least some of this, but that business needs to recognize that programmers are getting tired of the agent’s 15-30% take on their salaries for the “benefit” of providing some tax bookkeeping.

While it is unlikely that programmers will ever formally join “unions”, already networks are emerging that give those programmers nearly the same thing - information. A developer’s stock in trade is information, and as those networks (such as LinkedIn) strengthen, programmers will be able to tell whether a company is negotiating in good faith, will be able to determine what constitutes an appropriate compensation package, and will be able to warn away others from bad situations. Unions were competitive in the 1950s and 60s because companies had to negotiate with all of the members through their proxies, with failure to do so resulting in a strike or work-slowdown. I see the more informal networks of today as serving much the same purpose, but doing so without empowering yet another hierarchy at the expense of its members - having one’s company criticized among the very circle of developers that you are hoping to hire can prove a powerful incentive to play fairly.

Finally, a disclaimer - this is all a guess, based upon what I’m seeing and hearing from others in the industry. The IT field has long had the distinction of being a profession where the normal rules don’t apply, but demographics and technology both are trending towards what was discussed here, and in the absence of some ground-breaking revolution, I see developers moving into the catbird seat for some time to come.

Kurt Cagle is an author, consultant, and information architect specializing in XML and web technologies, and is the webmaster for XForms.org. He lives in Victoria, British Columbia, and can be reached at kurt.cagle@gmail.com.