As Derrick Story points out, iPhoto 2 is Mostly Good News — lots of great new features — but if you read the follow-up threads you’ll notice that customers don’t feel that performance has improved. Built-in CD archiving is implemented excellently and iPhoto librarian is a life-saver, but it’s still a pain to have to maintain multiple libraries manually if your collection is larger than 1,000 or so images (i.e. amateur size).
Likewise, iMovie 3 is a great leap forward in many respects, but has a couple of media database problems of its own. It’s able to bring my photo database into the Photos view almost instantaneously, but only sees the default iPhoto database, not the one I’m currently working on (fortunately, it is possible to drag and drop from iPhoto into iMovie). The real bottleneck comes when I click the Audio button to see my iTunes database. I clocked this: 35 seconds from clicking the button to registering a 15,560-song library. That’s 35 seconds with no application feedback whatsoever - no beachball, no thermometer. The app appears totally hung as I wait and wonder whether to terminate the process. I get a similar effect when clicking the Slideshow button in iPhoto, since it has to present a track selector before it can proceed. Meanwhile, iTunes itself still struggles to keep up when deleting tracks, selecting items, bringing up the ID3 editor, etc.
To me, all of these database issues point to a similar need — find a more efficient backing store for the iApps. The more I ask around, the more it seems that XML is the smoking gun on iLife performance drags - it’s a great format for interoperability, but horribly inefficient and resource consumptive. Maybe, just maybe, it’s time to reconsider using XML for the iApps. Maybe, just maybe, Apple should consider using some of the highly efficient open source database code out there — MySQL would do nicely I’m sure.
And since the iLife apps are all so wonderfully integrated now, why not place all of my media in a single, integrated media database? Whether such a database would store media objects themselves (allowing full export to original formats of course) or just references to them (with iTunes-style non-breaking inode references) is unimportant to me. With modern Mac hardware, I should be getting modern media database performance where it counts the most — when using my Mac as the digital lifestyle hub it’s touted as.
It’s important to keep in mind that a future of media collection lies in front of us all. Most of us have been collecting digital video, MP3s and photos for a year or two. What’s your media collection going to look like in five years, or ten, or twenty? Our Macs will ship with terabyte drives at some point in the future, no doubt. How will your iApps deal with that much media? The time to plan for the inevitability of truly massive personal media collections is now. We’re already feeling the pinch.
I like to steer clear of the “armchair CEO” syndrome, where users get in the habit of telling the vendor what it “should” do, so let’s just say this is one of my top wishlist items for the future of Mac OS. Interestingly, when I mentioned iApps, performance, and XML to an Apple employee at the last MacWorld, he rolled his eyes skyward and made some vague reference to “Apple always working on interesting stuff.” Hmmm…
Bottom Line
You want things to be faster. This is symptomatic of bigger problems. Namely Apples hardware, speed it up an order of magnitude and I bet people would have an order of magnitude less complaints about how this or that is sluggish. Apple has already made key infrastructure choices that they can't just give up on without messing with their customers again. Perhaps the 970's will make a difference.
Bottom Line
I disagree with the brute force approach. This is more about resource consumption than CPU cycles anyway. We know that a five-year-old Mac can serve up queries from a Filemaker database with a million records and be perfectly fast about it. My 867 is plenty fast -- it just shouldn't have to slog through inefficient storage formats. Throwing more CPU at the problem isn't going to make the root cause go away.
XML databases?
Are any of the XML databases ready for prime time? Would that be the best of both worlds?
Bottom Line
CS101: however much you speed up your processor, an order n-squared algorithm will never be as fast as an order n algorithm.
XML database
Either switch to using MySQL or i guess Apple (or 3rd parties) could look at http://xml.apache.org/xindice/
My only concern would be, can apple maintain an 'Open-ness' about them. Can 3rd parties easily hook into it? Its easy, with one hand, to ask for a faster database, and then on the other hand, condemn them for not allowing 3rd parties to compete ...
iApps are much slower than the hardware would suggest
Every Apple program (e.g. iPhoto, Mail, Address Book (*pathetically* slow), iTunes) has this problem. On the same hardware, a real database like MySQL is faster - not just a little but by several orders of magnitude - hundredths of a second versus multiple seconds. I can search a MySQL database with 15 million records on the same machine and get results in a few hundredths of a second (less than .01 seconds if it's an indexed key) - I do not think that the best answer is waiting until they can get a .5THz processor.
There's nothing quite like using a bad database (whatever Apple uses, Filemaker, etc.) to kill an application's utility. It becomes a chore, people use it only when they have to and all of the little niche uses never become practical. A counter example is the combination of simplicity and speed which is the Be filesystem - people found all sorts of interesting ways to use it because it didn't artificially restrict your imagination.
Several notes
Use of XML for storage is a mistake in any application which deals with more than trivial amounts of data. XML is wonderful as an interchange format - the role it was designed for - and almost completely the inverse of a format designed for speed.
Fortunately, MySQL 4 is not only blazingly fast but it also has the ability to be embedded. Even if Apple didn't want to make MySQL a system service (I think the pros outweigh the cons there but there are some decent arguments against this) they could embed the database engine inside an app such as iPhoto or iTunes, which would probably improve performance by at least an order of magnitude. Of course, going the system service route would also make a lot of interaction with that database trivial, such as allowing devices like the SliMP3 to update play counts or efficiently producing a web-view for your iPhoto catalog.
Unfortunately, the data storage format may in fact be the least of Apple's worries. I think slow GUI code is the heart of the iApp sluggishness - those painfully slow redraws are a sign of either inefficient code or unnecessary calls. Profiling iPhoto 1 showed almost all of the execution time being spent in the redraw logic, which is a pretty good indication that they had a naive implementation of the scroll views (I'm guessing that they repaint the entire view instead of the visible portion).
Apple should consider something like SQLite or BDB
embedded, inline, standalone, opensource, SQL compliant, blazingly fast.
what more could one want.
Relational DBs suck at multimedia
The only viable option would be an object database. Which is not so far of from a file system wit xml-meatadata. Which is what we have now.
Just wait for some optimalization to slip through.
Total agreement
It's long past time for Apple to include a standard database engine in OS X. I agree that BDB is an excellent choice; open source, high performance, extensible, with relational, procedural, and scriptable APIs.
In fact, it's already included in the DBM package for Apache, Apple only has to make it part of the base distribution.
"Unified Media Database", ugh.
Is it time for iTunes and iPhoto to scale better with large libraries? Yes. Is it time for these apps to share a "Unified Media Database"? Absolutely not!
The file formats used with iTunes and iPhoto, mp3 and JPEG, can already store metadata alongside the sound or graphic data with ID3 or Exif, respectively. What is needed is a good way of caching all this metadata so the application may scale better and respond more quickly.
With the current use of the basic filesystem with standard file formats, I feel confident that I will be able to maintain and grow my collection as I migrate to newer computers, to newer versions of software, and probably to different platforms. It is difficult enough to maintain backups of a basic filesystem, and I'm not convinced that allowing export to standard file formats would make archiving a "Unified Media Database" practical.
A "Unified Media Database" only gives me visions of vendor lock-in, future incompatibilities, and losing my music and pictures.
One Database to Rule them All
Please, please, please, NEVER suggest that everything ought to be in *one* database. Never ever ever. Sure, it sounds nice, once you have all your media catalogued and place just where you want it to be, but let's face it: humans are not the born organizers that many computer systems expect us to be!
Trust my programmer's intuition on this one: once you have a central repository, you *will* bump into situations where you will be unnecessarily constrained by the design and layout of the database. It *will* happen. Much, much better would be the creation of a file system object that acts like a folder but is more like a "group" or a
"pile" of documents. Then, at the programmer's level, you would have a unified way of accessing "related" documents and no more mucking around with all these incompatible file formats for databases.
15k files is NOT alot! http://www.apple.com/feedback/itunes.html
i started buying CD's in '87. That's 16 years of music purchasing. i bought a ton of CD's in highshcool and even more when i got to college. I still buy a bunch of CD's. Why is it unreasonable to think that iTunes should be able to manage all of my music, as the description on apples software webpage describes. "iTunes 3. Manage your music, burn CDs and sync to iPod" I have a 20GB iPod that i can fit 1/5 of my collection on. This seems reasonable to me. I want to "Rip, Mix, Burn" dammit. i still think it's ass that i can't use my iPod on both my work and my home computer's iTunes out of the box. damm DRM.
I suggest people use the feedback at apple's iTunes page to tell apple to speed it up!
http://www.apple.com/feedback/itunes.html
Not a lot!
Surely the whole point of having computers is that they do big things for us. I don't want my computer telling me that 5000 photos is 'a lot' as though it had to sit down and paste them into albums by hand.
There are no professional tools which are affordable and easy to use - that is the whole point of iLife or am I missing something