The Missing Open Source Projects, from an Enterprise POV
 | ||
 | Email weblog link | |
 | Discuss | |
 | Blog this |
Tim O'Reilly
Jul. 11, 2003 12:53 PM
Permalink
Finally this morning, I had a chance to sit and enjoy some of the talks. And as I settled in to the last session but one, I thought I would do my bit to contribute to the conference blogosphere by taking notes on Robert Lefkowitz (r0ml)'s talk on "The Missing Open Source Projects". Robert works on Wall Street (though he's not speaking for his employers). R0ml gave us some real insight into why proprietary software companies aren't going to go away for a long time, and just how much investment goes into software in the enterprise space. What's more, he emphasized just how many of the dollars go into operations, not software per se, and how the opportunity is for software to help manage some of those operations costs, and other business needs.
For each of seven application areas, r0ml tried to characterize the problem and make it vivid for open source developers. He started, though, with a big picture view. (Much of what follows is a transcription of his comments or slides, but some of it is my summary, and I don't have clear differentiation between the two. Please excuse any errors. And as r0ml accelerated the pace, the notes get sketchier. Hopefully, we'll eventually get the slides themselves up online.)
Enterprise Applications are characterized by many people, many computers, and many years (more than 10,000 people/machines, and more than 10 years). What's more, all the people and the computers need to be interchangeable.
He then talked about the top ten software vendors, and showed the kinds of business areas.
Then he went on to what he believes are the top six projects that are missing from an enterprise point of view. He actually listed seven, saying that two of them were only half-value (but he didn't say which ones.) He's not going to talk about things like an Exchange replacement, since he just recently saw Opengroupware.org
1. Distributed cron
What if I had 80,000 jobs running on 10,000 computers? Cross platform, with dependencies, and it's was somebody's job to contact someone else if something went wrong. In financial services, if jobs don't run, or run twice, money gets lost, and occasionally, people go to jail.
What people use now: Computer Associates Autosys. But google and you'll find others.
Here's what you have to be able to do: Keep a database of job state and run information in a single database. Commit to manageing all the batch jobs for a dozen open source porjects, schools and non-profits, and any conference attendees who agree. Including any non-Linux machines. If people object because of security, you have to fix that too.
Grid is starting to look at problems in this space. But they haven't started to look at the problem of who gets sued if you get it wrong.
This isn't just about writing code, it's about managing the process.
2. Asset Management
What if I had 3000 different software products running on 50,000 computers, with hundreds of paid data sources used by thousands of people? And support models based on number of users/number of installations/number of uses? And there are financial penalties for getting it wrong.
Audit requirements give financial and legal exposure. Inability to track right now drives companies to sign "enterprise deals, which create long-term lock in.
An effective open source asset management tool would help highlight how much money could be saved by using open source software. Software is currently about 15% of the corporate IT budget.
What goes into asset management: tracking (agents running everywhere); authorization workflow, including auto-authorization; asset redistribution. "Knowing what yu use and how is always a better idea than ignorance."
Google "asset tracking" to see who does this now. Tivoli, Asset Insight, others.
At this point, r0ml told a long story about the first time someone threatened to sue him and his company for many millions of dollars for automating a formerly manual process, thereby allowing users to switch what licensed data screens they were looking at. Automation can cut costs significantly, but vendors are often opposed to it. Open source could potentially help.
3. Single Sign-on
The average person needs many passwords to get their job doen. And many are required to change every 30-90 days. And they all have different policies around allowed values. This is so bad in a corporate setting that he never even tries to remember passwords, but starts right in with the "I forgot my password link" and just types in something random for the new password.
More than 50% of the calls to help desks are for password resets.
Merrill Lynch now spends 2 billion a year on IT. 20% of that is user support, and if half of that is password resets, are they really spending $200 million a year?
The missing project is to modify every database, browswer , email client, so that it uses LDAP and other open standards... (I lost the end of the thread here as r0ml accelerated the pace.)
4. Messaging
Why isn't email the answer for reliable messaging? In the interests of time, r0ml skipped over this one...
5. Change Management
Deciding what to change. Tracking who changed what. Making the change. Backing out the change. Keeping track of current state.
There are between 200 and 300 tracked changes a day. There is one change-related outage every day, with a 99.5% success rate for changes. And change-related outages are a large expense.
This can be as small as puliing out a cable at the wrong time to move a piece of equipment. In a financial services context, this can cost a lot of money.
Tokyo sys admins have it worst, since they are the start of the world trading day. It's like facing the tsunami every Monday morning, as they deal with the problems caused by changes in New York on the preceding Friday.
Some vendor products: Rational Merant, visible Razor. Change management is not just a software package, though, but a methodology and a process. Microsofts MSF/MOF, SEI's CMM.
He described a book on software change management process that starts with buying a spiral bound notebook, how to label it, what to put in it. A whole process around operating a spiral bound notebook in order to build software!
In the open source world, there's a lot of folklore, but there's no how-to on process.
To make this concrete in an OSS context: What would it take to rev Debian stable on a weekly basis? How can I tie a CVS check-in to a bug report. Both ways. If I undo an upgrade, I need to automatically notify the maintainer responsible. I have to undo across multiple hosts.
6. Relationship Management
Keep track of your customers. This generates increased satisfaction and lowers sales costs.
What percentage of bug ruports to open source projects are submitted by employees of financial services firms? Which industries have the highest patch submission to running copies ratio. How are you going to write software that is useful for people if you don't know if or how they're using your features? Like BitKeeper, should your software automatically collect data about how people are using it?
Ideas from Doc Searls: Markets are not just about money (free as in ride), or about ideas (visible or transparent source helps), but about relationships.
Don't throw the software over the (fire)wall -- demand a relationship.
Companies spend more on marketing than on development because the relationships are more important than the code.
7. Source Terminator
It's easier to combine proprietary software than open source software. "In the proprietary world, we're always mushing stuff together because the big fish eat the little fish."
In open source, we should aspire to create definitive literature for particular problem domains.
There is some movement in this direction: parrot, oscom, mysql and SAPdb are all doing some combining of ideas and code from other projects.
In conclusion, if they want to get into the enterprise, open source developers need to think about operations, the financial business case, help desk automation, methodology, customer focus, and marketing. All the things that businesses think about. There's a lot of opportunity here, and a lot of money.
Tim O'Reilly is the founder and CEO of O'Reilly Media, Inc., thought by many to be the best computer book publisher in the world. In addition to Foo Camps ("Friends of O'Reilly" Camps, which gave rise to the "un-conference" movement), O'Reilly Media also hosts conferences on technology topics, including the Web 2.0 Summit, the Web 2.0 Expo, the O'Reilly Open Source Convention, the Gov 2.0 Summit, and the Gov 2.0 Expo. Tim's blog, the O'Reilly Radar, "watches the alpha geeks" to determine emerging technology trends, and serves as a platform for advocacy about issues of importance to the technical community. Tim's long-term vision for his company is to change the world by spreading the knowledge of innovators. In addition to O'Reilly Media, Tim is a founder of Safari Books Online, a pioneering subscription service for accessing books online, and O'Reilly AlphaTech Ventures, an early-stage venture firm.
Showing messages 1 through 13 of 13.
-
Missing projects
2003-07-16 06:33:46 anonymous2 [Reply | View]
We definitely need OpenMQ, for want of a better name.
But what we also need are some other lower-level components to build these large scale systems on. And we don't really have these yet.
We definitely need a good, stable, scalable, easy to configure and manage RDBMS (NOT MySQL - that's a cardfile - PostGresQL? perhaps)
We need a development environment with REALLY good debugging support, including distributed debugging - and a good UI - there's nothing out there now. Plenty of powerful languages but all have very weak debugging support, particularly if you have client/server or n-tier components (i.e I want to put a breakpoint in a remote process and single-step it etc. as if I were locally connected to it). I don't care what language it is, Perl, Java, Python,C#,... just give me an enterprise-level development environment for it!.
Finally, we need to be able to use Mozilla as a reusable component. This is similar to how you would embed Internet Explorer as a COM control in, say, a VB program and have IE do all the user interface handling. Currently, because there's no inter-platform component service like COM in the open source arena, that's just not possible, even if the Mozilla control project wasn't so immature.
Alternatively, can we have Display PostScript like Apple and have a common rendering language for display and printing? with tools to support its creation.
Until we have these things its silly to wonder why large-scale Open Source projects don't exist. The plumbing is not there - apart from operating system low-level services the support stuff just doesn't exist.
-
Open Asset Standard
2003-07-15 11:41:42 anonymous2 [Reply | View]
We need Open Standard for Asset Management more than Open Software. Each system vendor has it's own propriety tool. MS has things like WMI. All report different things and supposedly interface with SMTP, with config nightmares I can't even begin to describe.
So here's a thought. An Open standard for Asset Management. Standard data interfaces (dare I say which ones). Gather information on any hardware, software, driver, blah, blah, blah… that you want. Drill into from everything from CPU speed to Vendor S/N. An open standard would allow central reporting. Easy accessibility to information. And more importantly, removal or propriety methods and systems.
-
Distributed CRON and Messaging
2003-07-15 11:40:53 patrickdlogan [Reply | View]
I would tie these two topics together. One solid open source foundation has been very well proven: Erlang OTLP.
The Erlang system seems to me to be well more than 80 percent of an industrial strength CRON.
Other open source language providers should look to Erlang OLTP for the ingredients that simplify scalable, monitored, distributed systems.
-Patrick -
Distributed CRON and Messaging
2003-07-19 13:43:42 r0ml [Reply | View]
I agree that Erlang has a number of capabilities that simplify building distributed systems. However, it is a language and a platform -- one would still need to build a distributed cron application. And as you point out on your blog, it would be difficult to build a sustaining community around such a project developed in Erlang.
One could then argue that the facilities that Erlang provides could be incorporated into other more popular development tools / languages. That suggests the "Source Terminator" project -- combining good ideas from multiple projects. Unfortunately for distributed cron, the work involved in combining Erlang and Mono, or Erlang and Python probably exceeds the effort required to just go ahead and build distributed cron using C the old fashioned way.
It is a shame that it is so difficult to build sustaining projects out of brilliant but unknown languages (check out Aplus ). Likewise a shame that we don't know how to incorporate these ideas into the mainstream more effectively.
r0ml
-
Distributed CRON and Messaging
2003-07-23 21:06:23 mvw [Reply | View]
Erlang is one of a few languages that are designed from the ground up for distributed programming, in this case from the functional programmers camp.
An imperative example is CxC (a C/C++ ish language), which seems to be more aimed for number crunching while Erlang has a focus on networking and reliable operation.
Erlang is the only functional language, where I have not the feeling that it is just an academic toy. This is because it is used in heavy real world apps in the telecommunications area.
It comes with lots of cool ideas and apps, among it a distributed database.
I wish the documentation would be more suited to newcomers, but that is just a matter of time.
I disagree that it would be hard to find Erlang coders.
Ever looked at the Zope sources and then claiming that Python is easy?
I would rather stick the effort into learning the Erlang base language. :)
So please mail me your cron requirements.
And lets see, how far we get.
Regards,
Marc
-
Missing framework
2003-07-15 07:24:52 anonymous2 [Reply | View]
Having the missing pieces would be nice. Having the existing ones talk to each other would be better. Dashboard, trending, ticket system, gnat. Sourceforge shows 2200+ projects for system administration, installation,distribution,logging. I doubt any of them talk to each other.
-
SSO
2003-07-14 13:37:31 anonymous2 [Reply | View]
Well, we have an Open Source network directory management system at http://www.arlut.utexas.edu/gash2/ which people are more than welcome to take and use. It's got its flaws and limitations, but it's also got .25 mil lines of Java code and is designed to be heavily customizable and integrable with almost anything.
-
On reliable messaging
2003-07-14 13:04:14 anonymous2 [Reply | View]
On item (4): he was probably referring to messaging middleware, which provides transaction support. All investment banks use this heavily (typically IBM MQSeries), and thre is no Open Source version. -
On reliable messaging
2003-07-14 16:30:09 anonymous2 [Reply | View]
yes, he explicitly referred to MQ series.
-
SSO is huge
2003-07-11 18:09:48 anonymous2 [Reply | View]
I hear the SSO requirement regularly. Probably the archetype for a strong enterprise architecture has a firewall outside and a directory of identities inside that passes authentication to Kerberos. Businesses with directory servers are trying to consolidate, but Microsoft has succeeded in making that an ugly problem unless it's all AD. Regardless, Gartner had a study 3 years ago with sage advice: the idea that there will be one single directory per organization is fallacious.
| Showing messages 1 through 13 of 13. |
Return to weblogs.oreilly.com.
Weblog authors are solely responsible for the content and accuracy of their weblogs, including opinions they express, and O'Reilly Media, Inc., disclaims any and all liabililty for that content, its accuracy, and opinions it may contain.
This work is licensed under a
Creative Commons License.
And there is a very subversive nature to having asset management as open source.
Once you know what you have, then you can begin to make experiments to replace proprietary components with open source alternatives...
http://www.nexb.com