I believe that Python should be dominating in the LAMP arena. But, frankly, it is lagging behind terribly. I’m not saying that it’s lagging behind in its usefulness and usability, but in the breadth of its adoption.
There are some excellent tools (such as SQLObject) as well as some fantastic web frameworks (such as CherryPy) which make developing web applications faster, easier, and more fun. Python as a language has every conceivable facility available to it to empower web development. The syntax is extremely clean and nimble. It is OO, but not demandingly so. It has a wealth of re-usable code in the standard library specifically geared to facilitate web development.
Ian Bicking wrote a really great article/blog on this topic I guess a few months back. Ian could not have been any more correct with his premise of fixing Python’s web programming problems being the most important thing we can do to market Python. In that article, he stated
We also need input from people trying to do commodity Python hosting, and we need to pay attention to what they say.
In that vain, let me recount my most recent web development experience. (Let me first of all say that I’m not a web developer by trade. I piddle and I hack, but it’s not my day job.) I am in the process of creating a website for my wife. Initially, it was going to be just plain HTML. Then, it was going to need some file-upload capability. Now, it’s going to need user management, a shopping cart, a product catalog, etc., etc.
I initially started just creating plain HTML when that’s all it needed. There was no need to over-engineer, right? Next, when she wanted a file-upload section, I reached for Python CGI. That should do the job pretty simply. Then, as we faced further scope creep, my heart reached for TurboGears, but I knew that wasn’t an option. Not on my hosting plan, anyway. So, I reached for PHP. (And, by the way, I’m not altogether dissatisfied with it, either. But I’ll save that for another day.)
Why did I reach for PHP rather than something Python-based? I can’t believe that I’m so different from everyone else. I believe that this experience (and the answer to the question which I just posed but haven’t answered yet) may shed some light on the current Python situation. I didn’t try Python any further than Python CGI because Python CGI is the most ubiquitous Python offering in the web hosting space; any other options are very sparse.
Let me outline the 3 general options for depoying Python web applications. 1) Standalone processes. Turbogears which sits on top of CherryPy require a standalone server in order to run. They don’t integrate nicely with Apache like PHP does. Actually, that’s not entirely true, but I’ll get to that in a second.
2) CGI. This is by far the most available option for commodity web hosting. But I don’t believe it supports sessions out of the box. You could probably contrive something to stuff a “?PYTHONCGISESSIONID=12ASD83234JALJSDF879S87DF98SA7DF987″ string at the end of every URL and store session data in a database, but that just feels nasty.
3) mod_python. This is currently, IMHO, possibly the best option for Python web development using commodity hosting services. I’ve even read that you can deploy CherryPy and TurboGears in it. But the availability is just not there yet. Google sometime for “mod_python web hosting” vs “php hosting” and you’ll see what I mean.
To sum things up, after failing to find more than a handful of hosting services who would support Python hosting in a configuration suitable to me, I chose a more commonly available technology: PHP. The hosting is cheaper and if I run into a problem with a hosting service, I’m not stuck with a bad host. I suspect that the majority of Python web application development takes place in one of two settings: 1) large deployments where the developer/customer have full access to the boxes they are deploying to, or 2) small deployments for either in-home use or on an intranet where the developer is using a spare Linux box and has full access to it. That’s just a guess, though.
I really don’t have a solution, merely my experiences and an observation of the current state of things. I don’t know how Python will gain a ubiquitous commodity-ready hosting solution overnight. Or, how it will take what appears to be a commodity-ready hosting solution (such as mod_python) and make it ubiquitous. I can’t help feeling that I’m missing something obvious, here, though.
No session management?
I'm not much of a web developer, but I don't think it's unreasonable to require that the user has cookies turned on, and python supports those. Take a peek at http://docs.python.org/lib/module-Cookie.html if you don't believe me ;)
It's education
Python has some nice tools for doing web programming, but very few people are talking about them and outlining how to use them. Take TurboGears. I played with it a bit, but its major shortcoming was the documentation. There was a nice getting started tutorial, but once I started to venture outside of that, the only thing I could find was the spit out of all the classes in alphabetical order. Actual documentation was sparse (mostly just method signatures), and the classes weren't organized in any way meaningful to new users. (These auto-documentation generaters are actually a bit of a pet peeve of mine because they make devs think they are 'documenting' their tools; they're not, they're just spitting out method/class names.)
The "CatWalk" GUI looked like it might be very helpful, but again, the old version didn't have instructions on how to make it work with 0.8, it just said to get TG 0.9 from SVN which has it built in. But the instructions on the TG site didn't make it clear exactly how I should install and start with the SVN branch. So I just backed off, thinking "I'll wait until this is more polished."
Perl and PHP have had years and years to build up a reputation for being solid solutions to web programming problems. To make a dent in that, Python has to have solutions that are both easy to understand and easy to implement. Python is an excellent tool, but there aren't many people outlining/detailing solutions to use it for web programming. (And those that do just detail rudimentary CGI that can be done just as easily with Perl/PHP.)
IMHO, promoters are just as important as programmers in terms of getting good technologies adopted. Unfortunately, they're a lot harder to find in open source. ;-/
No session management?
I don't think I did a great job getting across what I meant here. The problem isn't whether Python supports cookies. It does. CherryPy has supported cookies for quite a while now. The problem is how to do session management using CGI where the Python process is executed every time a page it hit. If the process is constantly starting and stopping, it makes it hard for the web server to shove information into memory for each specific session. Hope this clears up what I was talking about.
More hosting companies than you might expect
CGI is not a good option unless your site is very low traffic and your code is simple. The cost of doing a fork for each request is just too high.
This leaves mod_python or one of the long-running-process solutions (FastCGI, SCGI or standalone server with mod_proxy/rewrite). mod_python is not ubiquitous as you mention.
But, there *are* hosting companies (inexpensive ones, too) that will let you run TurboGears in a long-running config. TurboGears.org will have a list of these soon (they include TextDrive, Dreamhost and Python-Hosting.com which actually offers TurboGears hosting explicitly).
There are also quite a few companies providing, for a little more money, virtual private servers on which you can run anything.
When TurboGears was first released, it didn't make a lot of sense to try to list hosting companies... now it certainly does, and I'll be sure to get that list up soon.
It's education
Beyond the Getting Started Guide, the main docs for TurboGears, at present, are the docs available at the individual project sites. There are links to the current docs for CherryPy, Kid, MochiKit and SQLObject on the docs page of TurboGears.
The project has grown quite a bit in a short period of time, and we're starting to get documentation contributors as well as code contributors, which helps a great deal.
It's also worth noting that at least one TurboGears book project is approaching a deal with a publisher.
So, "polish" is definitely needed and it's getting there. One thing I will say, though, is that the mailing list has *many* useful people on it who can help answer questions that are not covered as well in the docs at present.
Been there. Done that.
I initially wrote a simple, CGI based web store for my wife too. We moved on to a commercial, open source like PHP based shopping cart/storefront as the business grew. What a PITA that PHP crap is. *sigh*
Now I'm looking at a dedicated machine so I'll probably write some new apps in Subway and/or TurboGears real soon now, but I don't plan on ditching the PHP cart any time soon. It's unpleasant, but it also works fine.
Best of luck.
What?!!
Let me first of all preface this with saying that I haven't batted about with PHP 5, so it's been a while. But...
As far as web development goes, Python is far and away superior. The biggest reasons for this are that it makes network programming (as my friend put it) like falling off a log. I have looked at Twisted, but haven't implemented anything yet. The biggest reason is, I DON'T HAVE TO. I've got a 12 line CGI server running with the PYthon base CGIHTTPServer module, and I get to use all of the Python goodies to design the entire web application. I confess that I don't know how well Python handles session management (but as I recall, PHP was no picnic there either). And as for SQL, BAH, who needs it?! Aside from being perhaps, marginally faster and smaller, SQL doesn't hold a candle to XML as a data storage format. My CGI Server (or collection of them, running automatically on self-discovered ports on my Darwin machine requires no httpd.conf fishing, no database configuration, all oogledie-googledie Python goodness. I design classes to handle my own server-side includes to make the painful terdium of HTML form development modular and reusable. And the whole operation is portable. Zip server and data up and throw them anywhere and they land on their feet with zero configuration necessary.
I'll grant that Python probably doesn't play the Apache, IIS, etc second fiddle role to well. Also, my web apps are reltively small, so I can't speak to how well they might scale. But as a top to bottom solution, server, data, everything, it pretty much rocks!
What?!!
I actually agree with you that Python is superior to PHP as a language. I'm not willing to wholly discount PHP because it does have some redeeming features. I keep meaning to blog about those.
Twisted is cool. I haven't built a project in it. Go get Abe Fettig's Twisted Book (http://www.oreilly.com/catalog/twistedadn/) . It's worth a read.
Regarding PHP session management, it's really a snap. All you have to do to stuff into session memory is assign to the $_SESSION array. I think CherryPy does something similar...can't remember.
I'm a really big fan of XML, but I think you're unwarrantedly undervaluing RDBMSs and overvaluing XML for general use. XML may meet your needs better in this case, but RDBMSs have their place as well. I would rather use SQL and an RDBMS when having to cross-reference data in different tables. Doing it in XML is potentially resolving a solved problem.
I do like your deployment mechanism. I wish more apps were like that.
Check out django
http://www.djangoproject.com/
PHP is easy to learn and easy to deploy
PHP is easy to learn, anyone who knows HTML can just start to copy and paste bits of PHP into their website to make it work. mod_python, although growing is still somewhat of a beast to work with and VERY poorly documented and hard to get running, it breaks all the time for no reason because it's too complicated and tries to expose too much low level HTTP. I don't like it.
CherryPy and even Twisted are some people's favourites but the truth is people go with what's cost effective and when it comes to $7/mo hosting for 3 domains with all the whistles you've GOT to build something that can be served by Apache. Mass hosting operations don't generaly let you run your own processes.
I love Python as a language, I hate the state of it's web development options (as of December 2005)
Long running vs short running
Long vs short is something of a silly debate, if your website gets 5 hits a second for 8 hours a day and say 1 for the other 16 that's going to be 75,600 hits a day, does your site get that many hits? No... I see.... Well You can handle that kind of load on a pretty crummy server. 5 web hits a second each with 1 normal select from a RDBMS such as PostgreSQL and some mild templating can be handed by a 1.5Ghz 512MB machine with Apache and Python as CGI not even FAST CGI. It's not like most web servers are 133Mhz Pentiums anymore... Have you actually benchmarked your server? You know what makes it slow? YOUR bad code. Write better SQL statments and you'll be fine.
So you do get more than 75,000 hits a day? Well then I'm sure you can afford to have your own servers and run whatever you like so it's a moot point. And if you can't why the heck aren't you using Google Ads?
Why isn't Python doing any better?... period.
I recently raised the issue of Python's lack of general acceptance as a programming language. The response from Python developers was... there is no problem.
There doesn't seem to be much interest in gaining wide acceptance. Some even wish for less acceptance. Instead the focus is on being "right".
So is it any wonder that Python is lagging in LAMP? I think it's not just an issue with LAMP.
John
One solution
I agree that Python Web hosting is severely lacking. With that said, though, there are a fair number of Django-friendly Web hosts:
http://code.djangoproject.com/wiki/DjangoFriendlyWebHosts
And it runs perfectly with Apache/mod_python:
http://www.djangoproject.com/documentation/modpython/
It's education
We try to improve the Django documentation (djangoproject.com/documentation) every time somebody asks a question that the docs don't answer. I think you'll find the Django docs that exist are fantastic. I (a core Django developer) am a professional journalist and care passionately about good writing.
Of course, there are plenty of features that are not yet documented, but, one by one, we'll get there.
wtf
You chose PHP because you had more cheap-ass underpowered hosting plans to choose from? Hello? You can get a VPS for $15/m these days from any number of providers. Please stop blogging now if you do so little programming that it's not worth even that much.
wtf
Yes. That is exactly the reason I chose PHP over a Python-based solution. And I believe that is a perfectly reasonable basis for deciding on the technology to use for developing a website. And I believe that unless Python folk don't address that reality, they will continue to lose their potential share of the web development market.
Your line of reasoning is reminiscent of Linux folks back in 1998 proclaiming Linux a superior desktop to Windows. Which it was...if you were a geek who was committed to Linux. That wasn't the majority of desktop users back then (or now). And the majority of web developers aren't currently committed to Python. So, to tell web developers that they should pay more for hosting so that they can code in Python is not going to fly, especially when there's no obvious payback for doing so.
Or maybe you're just telling me that. And if you're just telling me that, that's not going to fly, either. There is a set of requirements around the site that I'm building. One is that hosting for the technology chosen to build the site must be reasonably available. I really don't want to be limited in my options, so this is perfectly reasonable to me.
Another requirement is that the hosting must be cheap. We're starting a business on a shoestring. "Shoestring" is a relative term. It can mean one thing for folks with VC. It will definitely mean another thing for those of us who aren't even interested in VC. We're starting this business in our spare time with our spare change. And on top of that, we're extremely frugal people, so even $15 a month is way out of reason for us. I even cringe at the thought of $10 a month.
Speaking of relative terms, "underpowered" is an interesting relative term you threw out. If the hosting I can get for $5 a month does what I need it to do, does it matter if it may be underpowered by your estimation? We're expecting very low traffic, so "cheap" and "underpowered" suit us fine at this moment.
I'm still trying to figure out exactly what you meant when you said, "Please stop blogging now if you do so little programming that it's not worth even that much." So, are you saying that my perspective on web development is invalid because I am an amateur (which I freely admitted in the blog entry you responded to) rather than a professional web developer? I think Paul Graham (http://www.itconversations.com/audio/download/ITConversations-657.mp3) would disagree with you. I guess I just don't get what you mean.
Thanks for the post and thanks for the candor.
Things are changing
I've been in similar situations as you have, and so can understand completely why PHP was an attractive option. For quite a while, if you wanted to do Python webdev, you either used under-powered cgi or you owned (and administered) the box (or vps...).
The main strength of PHP is low barrier to entry - it's relatively easy to learn, has buckets of support libraries, and it's deployed *everywhere*.
That said - it appears things are changing, driven to good extent by the ruckus surrounding Rails. Thanks to Rails-buzz, large commodity hosts (Dreamhost) and with smaller more focused hosting companies (TextDrive, GeekISP, Python-Hosting, etc.) support FastCGI/SCGI.
My main point is that cheap hosting for Django/TurboGears is available now, and there's enough competition in that market so that a web developer can feel comfortable they won't be stuck with a bad host. Thanks in large part to Rails, other commodity hosting companies will be forced to follow suit, which makes simply increases deployment options for us python-heads.
The main obstacle now is the ease with which an average web developer can start and deploy a web app using fcgi. There are cookbooks in blogs & such, but it still takes a fair amount of mucking about to make it happen. I expect that to get easier as TurboGears/Django/etc. mature and attract more developers.
Thanks for starting the conversation!
mod_python
mod_python has the most potential right now, but IME it's an absolut b*tch to set up. With PHP, all you do is get mod_php running and you're mostly done, sans file extension binding. With mod_python, I copy&pasted whole parts of htaccess/httpd.conf and it _still_ didn't work, or worked different than expected. Quite discouraging; it's definitely not Pythonic (as in don't-get-in-your-user's-way) ATM.
Why isn't Python doing any better?... period.
I'm not certain that the core Python developers need to concern themselves with Python acceptance. Maybe Guido doesn't even need to do so. (I think he should be concerned with it, but that's just me.) But I believe that some Python folks need to have a marketing/acceptance perspective. It seems that Python has gotten as far as it has on excellence alone. I think if it wants to gain more than the niche that it has, it needs to be concerned with acceptance and do things to promote it. I don't know who should do it nor what they should do, but something needs to happen if Python is going to be able to compete at an enterprise level. I believe that it mostly can as far as a technology, but in order to be taken seriously and even considered for use in the enterprise by C level executives (CIO, CTO, etc.), some serious marketing needs to take place.
One solution
Thanks for pointing these out. I'm probably going to use Dreamhost soon.
It's education
I applaud your efforts. I'm sad to say that I haven't even dug into Django yet. Maybe I'll give it a spin sometime soon. I've got to say that CherryPy fit my brain well as I've used it off and on for the past couple of years. Lately, TurboGears has done that same, only more so.
I keep intending to contribute to the TurboGears project if by no other means than improving the documentation. Maybe I'll get around to it in all my spare time :-)
Things are changing
Thanks for posting this. Your points on growing TurboGears-friendly hosting as a result of Ruby on Rails is a huge reason why I've ported this project to TurboGears. I'm planning on blogging about it soon, but this is my first mention of such.
Thanks for the post!
mod_python
I agree with your estimation of mod_python having the most potential for now. I've gotten code to run under mod_python, but haven't tried to deploy a CherryPy or TG app under it. I'll take your word for the PITA estimation :-)
No session management?
Session management using cookies and Python CGI is fairly easy though.
It's the approach I use in logintools - http://www.voidspace.org.uk/python/logintools.html (CGI authentication and account management for Python CGI)
It uses a dictionary like object to store state in-between accesses.
TurboGears hosting options
TurboGears hosting options (not a definitive list by any means):
http://trac.turbogears.org/turbogears/wiki/hosting