Article:		Graphical Composition in Avalon
Subject:		macos x already works like this
Date:		2004-03-14 10:23:25
From:		mweiher
Response to: macos x already works like this
I think it would be easier if you separated defending your article (which rightfully addresses the needs of Windows developers) from trying to refute the statement "macos x already works like this". The fact is that Mac OS X already *does* work mostly like this, except for retained vectors within the graphics API, and wether this is a good thing or not is at the very least debatable (I don't agree it is). I don't think you needed to dwell on Quartz, but a short reference to existing state of the art is almost always appopriate, if just to provide context. With that out of the way, I do believe you made a number of statements in your replies that need to be addressed. 1. High DPI displays Quartz will be quite capable of adapting to high DPI displays, just not using the mechanism you envisage. In fact, Quartz uses the PDF/Postscript imaging model, which as you may recall, is device independent and theoretically arbitrarily scalable, and practically scales at least to imagesetter and film-printers, with around 2500 dpi, and has done so for a couple of decades. The mechanism is quite simple and has been in uses for several decades: apply a device-transform with a higher scale factor before any other rendering is done. Voilá, you get a bitmap rendered at a higher resolution. Your proposed mechanism of applying a compositing transform to the completed Window is unlikely to work well even in the Windows world, because it assumes that *all* drawing by *all* applications will use the retained vector API. This assumption seems unlikely unless non-retained APIs are removed, and completely hopeless once you consider legacy applications. 2. "You have to do your own redrawing" This is false. Or more precisely, this really depends on who you mean with "you". If you mean "some entity outside the low graphics library", then it is technically true. However, if you mean "you, the developer", this is simply false (and since you put this under "it makes life easier for the developer:", that seems the only reasonable definition). Most modern app development on Mac OS X is done using the Cocoa frameworks, and these will handle the refresh logic, calling your redraw methods when necessary (and appropriately clipped) without you being aware of it. You might reasonably counter that this goes outside the scope of graphics APIs, and the answer is that yes it does. However, just because something isn't handled by a particular API doesn't mean it isn't handled, and just possibly a retained-mode graphics API isn't the best way of handling this situation. [I would argue that it is not] 3. "big limitation: there is no vector-level retention" You assert that this is a big limitation. Unlike the other two points, which were factual errors, this is more of a value judgement that I can't factually refute quite as easily. However, I do disagree, and point out that your primary arguments supporting your assertion [1+2 above] have been shown to be factually incorrect. 3a. newness The retained vector model is not new in any sense of the word, it is actually one of the oldest in town, with GKS and similar systems going back to the 70ies and before (anyone remember Tektronix storage tube displays? :-) ) Now that doesn't mean it is bad, just like claiming it is new doesn't make it good. However, it should be noted that retained-mode graphics systems were the norm a while ago, and then were rejected in favor of immediate drawing, at least for 2D graphics. Attempts to reintroduce retained mode drawing such as QuickDraw GX did not meet with success. 3b. drawing using the GPU vs. unified imaging model You seem to assume that drawing via the GPU is inherently "better" or "more advanced", without apparently even considering the possibility that *not* drawing via the GPU may be a conscious design decision. However, this was precisely the decision taken for DisplayPostscript and I am almost certain also for Quartz. The reason was/is that the Postscript/PDF imaging model is precisely defined (in a device independent manner, see above), and most GPUs simply don't draw precisely way, and even if some do, there is no guarantee of this. You don't need to be convinced that this is a good idea, but at the very least it is an alternative view and means that GPU-based drawing is not necessarily "better" or "more advanced", or "leap-frogging". 3c. (vector level) retention automatically fast(er) Once you are not drawing retained vectors with the GPU, there is no particular reason why it should be faster than immediate mode drawing, and experience with DPS, for example, showed that it almost never was. The reason for this is that you are doing extra work: building up the retained graphics representation from the application-level representation, and then drawing that. Drawing directly from the application-level representation eliminates one step. 3d. (vector level) retention makes for easier APIs As I already showed above, it is not true that vector-level retention automatically makes for less work by the programmer. In fact, retained mode APIs tend to make things more complicated, not easier. The reason is that almost all applications will already *have* a retained-mode representation of whatever it is they want to draw, in application-level terms. This will be true in every case where the application-level model is not exactly the same as the retained-mode graphics model that the API offers. I can't imagine an application where this will *not* be the case, except maybe for a trivial "Avalon" viewer. So you end up simultaneously maintaining *two* models, instead of just one, and trying to propagate changes between them. Doing this via "diffs", which is the advantage you are proposing, is very complex and error prone. So complex that I am not going to go into too much detail here. All this extra complexity for what? Supposedly faster 2D drawing. Hmm....I just profiled TextEdit during live resize, and actual drawing was a negligible percentage of total CPU usage. So what are we gaining here? Optimizing a small fraction of an operation that is already fast enough, with no guarantee that the "optimization" will actually be an improvement. That isn't just premature optimziation, that is completely superluous optimization. Classical "Mount Everest Syndrome". 4. Quartz Extreme and Window Level compositing Another factual error: window-level compositing is not a feature of Quartz Extreme. It was always in Quartz, Extreme just made things faster by putting the compositor on the graphics board. I find it interesting that you think that hardware support and feature support are intrinsically linked, because that is exactly the way Avalon seems to be going...

Showing messages 1 through 18 of 18.

• macos x already works like this
  2004-03-15 05:51:48  Ian Griffiths | [View]
  
  
  "The fact is that Mac OS X already *does* work mostly like this, except for retained vectors"
  
  But my point is that the retained vectors are crucial.
  
  But other than this fundamentally important feature, yes, very roughly speaking, Mac OS X more or less works this way. I say "roughly speaking" because there is still the issue that the compositor and the 2D drawing APIs are two seperate entities. (At least that's what Apple's documentation shows in all the pictures of Quartz's architecture.) In Avalon, composition and drawing are tightly linked.
  
  To answer your points:
  
  (1) This PDF/Postscript thing keeps coming up. I think I already said this, but just to be clear, that's entirely internal to the Quartz 2D API, and has absolutely no bearing on the Quartz Compositor which is, as far as I can tell, resolutely bitmap based.
  
  Try taking a screen grab using the Shift-Command-3 shortcut. This produces a PDF. When I first saw that, I thought, "Aha, this must be because Quartz uses the PDF/Postscript imaging model." But look at the PDF - it contains a big bitmap. If you zoom in on it, everything pixellates. Why? Presumably it's because displaying a window involves composition, and the Quartz Compositor is *not* in fact PDF/Postscript based, it's bitmap based. So by the time you get to composition, the PDF/Poscript nature of the Quartz 2D API has been lost, and it's just so many pixels.
  
  So while PDF files are theoretically able to scale to typesetting resolutions, the ones the come out of the Quartz compositor don't. Try to print one of these things out on a high resolution device, and voilá, you get a low quality pixellated image.
  
  Perhaps there is some different way of getting PDFs of the display that I don't know about. If there is, please let me know. But Apple's documentation suggests that I shouldn't be surprised that these things are all pixel-based, because that's how they claim their compositor works. And unsurprisingly, the evidence backs them up.
  
  The fact that these things pixellate doesn't bode well for the likely quality of support for high-DPI displays. And it's hardly surprising, given that the compositor offers a range of effects that are not, as far as I know, supported in PDF. (E.g., someone else mentioned non-linear transforms such as the Genie effect. Can you do that in a PDF without dropping down to bitmaps?)
  
  By the way, I think Apple could mostly fix this in a future version if they want to. (It would require a significant change to Quartz's design, but I can't think of any fundamental reasons they couldn't do it if they really wanted. The only truly problematic thing that springs to mind is the way gradient fills require a callback function - makes it kind of hard to push that onto the pixel shaders...) I hope they do fix these issues in the future. I'm just saying how it appears to work at the moment - after all, that is the subject of the thread. But Apple need to start making moves in this direction now if they want to be ready for high DPI displays, which brings me onto the next point:
  
  There seems to be a lack of leadership here from Apple - all of their graphical widgets appear to be bitmaps, not vector images. This means that even if the PDF-ness of Quartz 2D did manage to permeate all the way through to the composition level, what you'd see instead of one big bitmap is a PDF containing lots of individual bitmaps. This won't scale any better than one big bitmap.
  
  Once again, I'd better point out that I'm not a Quartz expert, I'm just reporting my findings. If someone can tell me what I'm doing wrong, and show me evidence of resolution independence I'll happily admit to error. But so far, all the experiments I've tried point strongly towards the direction of purely bitmap-oriented composition. (As do all of Apple's architecture diagrams showing how the compositor relates to the other parts of Quartz.)
  
  I have a couple of Macs, so just tell me what I should be doing to see evidence of this supposed resolution independence, and you'll make me happy. But for the meantime, I can only go on the basis of what Apple's documentation claims, and what I've seen with my own eyes.
  
  
  "Your proposed mechanism of applying a compositing transform to the completed Window is unlikely to work well even in the Windows world, because it assumes that *all* drawing by *all* applications will use the retained vector API. This assumption seems unlikely unless non-retained APIs are removed, and completely hopeless once you consider legacy applications."
  
  You are right that legacy applications are indeed a problem. The quality of display for those will be inferior to those of native Avalon applications. But for new applications, Microsoft are providing a strong and clear lead in telling everyone that vector-based UIs are the way to go. They're also providing tool support to make this easy in practice. I see no such leadership in this direction from Apple, which leads me to conclude that Apple don't regard this as important.
  
  So I would agree that for today's Windows applications, high DPI will be less than completely satisfactory, and the same will be true for today's OS X applications. Both OSs will have to resort to the same tricks, resulting in less than ideal image quality. But for the future, Microsoft have made their strategy clear, and have made it equally clear what high DPI support will require from developers. I don't believe Apple have made their strategy for high DPI clear yet. I hope they do soon. (And I really hope they realise that the PDF/Postscript level drawing isn't the whole story - what worries me is the possibility that they don't even know they have a problem yet. Windows' GDI+ API uses a model which is almost identical, and it isn't a complete solution to the high DPI issue either, for the same reasons.)
  
  (Of course, both Mac OS X and Windows could introduce partial solutions to these problems - they could apply bitmap scaling to those parts of the display relying on bitmaps, and high resolution versions of everything else. But this is a difficult area to get right - even when drawing with a resolution-independent API such as GDI+ or Quartz 2D, it's surprisingly easy to write code that only works properly at one resolution, and has some visual anomalies at any other resolution. That said, I think Quartz 2D will probably have fewer problems here than Win32.)
  
  
  (2)
  
  "Most modern app development on Mac OS X is done using the Cocoa frameworks, and these will handle the refresh logic, calling your redraw methods when necessary"
  
  That would be an example of doing your own redrawing. The fact that OS X is calling your redraw methods is a clear indication that it has not retained your UI - if it were retaining the UI, why would it need to send you a message saying "Please redraw your UI?"
  
  
  3. Yes, I continue to assert that the lack of vector-level retention is a key issue. I believe it is the cause of the problem mentioned above in point (1) - the pixel-based nature of the compositor's output. It's also the reason behind (2).
  
  I'm well aware that vector retention is an old model. I considered writing a retrospective intro to the article on our industry's tendancy to revisit old solutions, but I was already over the requested length by a large factor...
  
  The integration with the GPU is an important feature simply because it's the only way you're going to get adequate performance for anything other than static images or very simple animations. The frame rates achievable with OpenGL are far better than with Quartz 2D. And likewise on Windows, DirectX fares better than GDI+. So in both cases, the accelerated pipeline works far faster than the unaccelerated postscript-style API.
  
  (The reason acceleration is faster, by the way, has nothing to do with how fast the drawing can be performed, by the way. It's all to do with minimizing the amount of data that needs to be sent across the bus to the GPU. The fact that the CPU may be able to render something as fast as the GPU would have is of no consolation if you now have to transfer 200K of data into the graphics subsystem, rather than a few hundred bytes.)
  
  I understand your point that Apple may have painted themselves into a corner here by requiring precise rendering semantics that think cannot be achieved with acceleration, although I'm not sure that's still true with the latest generation of graphics cards. GPUs have moved away from the inflexible fixed pipeline architectures popular a few years ago, and now have a much more reconfigurable set of acceleration building blocks.
  
  Microsoft made exactly the same decision with GDI+ as Apple did with Quartz 2D - these two APIs are more or less contemporaries, and both use software rendering because of the limitations of the hardware of the late 90s and early 2000s. These hardware restrictions no longer apply, so that decision looks a little outmoded now, which is why I think it's reasonable to say that accelerating the 2D pipeline is more advanced. An unaccelerated pipeline will become increasingly anachronistic.
  
  More importantly though, software rendering becomes a performance liability. Now that these drawing operations can be accelerated, it makes little sense not to accelerate them, because it's very easy to max out the CPU and/or PCI bus (mostly the latter) with Quartz 2D, limiting use of animation. You can happily use animations for the other parts of Quartz - QuickTime, OpenGL and the compositor itself all enable animation just fine, because not much data needs to be sent to the graphics card for each change. But you just use animation in Quartz 2D except in very limited ways. (So it's OK for the highlighted button to pulse gently because that's a tiny screen area and doesn't require a very high update rate to get that effect. But anything more ambitious is likely to peg the PCI bus or CPU.)
  
  You might argue that this is a good thing - too much animation is distracting and pointless. And poor use of animation is just that. But I regularly watch subtle but extremely effective use of animation to present information on television. But unless/until Quartz 2D gets acceleration, it's not going to have the performance to offer the quality of visualization that is commonplace on television.
  
  (Also, remember that 'animation' effectively includes interactive stuff like the user picking something up and moving it around. The speed at which such drawing can be updated has a significant impact on the feel of the operation.)
  
  
  3c. - your point is predicated on the assumption that 2D drawing with the quality offered by Quartz 2D and GDI+ can never be accelerated. As I said above, I believe this is no longer true, so I reject your conclusion on this point for that reason.
  
  Perhaps Apple is betting the farm on the prediction that the advantage offered by current GPUs will be transient, and that they will win in the long run, because doing everything in the CPU is more flexible. And then you can just invoke Moore's law as the solution to any performance problems. Except I think that's a flawed strategy because it ignores one very important fact: the speed at which computers can move data from one piece of memory to another has not been keeping up with Moore's law. The trend for a long time now has been for memory speed to drop further and further behind the speed at which CPUs can execute instructions.
  
  This is an aspect that worries me about Quartz 2d's current approach: it requires large bodies of data (the bitmaps generated by the Quartz 2D code on the CPU) to be shifted to an external memory system (the graphics card) any time anything changes. This is likely to be a bottleneck already, and will become a worse one as time goes one - Moore's law is the enemy of this graphics architecture, not its friend. (A different physical architecture for the display and memory systems might mitigate this, but can Apple really afford to stop using off-the-shelf GPUs and commission the design of custom graphics chips?)
  
  
  3d "As I already showed above, it is not true that vector-level retention automatically makes for less work by the programmer."
  
  I don't think you did show it, you just claimed it. You made two points. With the first, under your label '2', you missed something crucial - having the OS send your code a message saying "you must redraw now" is not the same as not having to redraw. It's pretty much the opposite in fact. And the second thing you mentioned was QuickDraw GX, which I have to admit I have no familiarity with.
  
  Have you tried writing any code that targets Avalon? I'm guessing not from what you've written, in which case, what are you basing your claims on? It sounds like neither of us has enough common ground to agree on here - I suspect you've not used Avalon, and I've not used QuickDraw GX. All I can say is that so far, my experience with Avalon has been positive.
  
  
  "you end up simultaneously maintaining *two* models"
  
  I agree that this issue is a potential cause of trouble, but on the other hand, there are plenty of precedents for this kind of thing working just fine: there are controls today that retain information (but not the actual drawing), so this is hardly new. (Text controls and list controls being the two obvious examples. Even though the table views go through the data source object, you still need to perform work to keep the view and the source synchronized.)
  
  The notion of maintaining seperate objects for your view and your model isn't a new one. I've written code for both this approach, and the imperative style used by Quartz 2D and GDI+, and so far, I prefer the retained model.
  
  But programming styles are a very personal thing - you might well simply prefer the imperative style. Fortunately, Avalon lets you use that and still get most of the performance benefit of retention: the OS will retain any drawing you give it either declaratively or imperatively and not ask you to repaint until you send it a messages saying "this piece of the UI tree is out of date, I'd like to replace it with something else." In this style of programming, you still have a repaint method just like you do in Quartz, meaning that you're no longer obliged to sync up your model with the OS's (in both cases you just say "Junk what I gave you before - I want to redraw from scratch"). It's just that you only get the repaint requests when you tell the OS that something has changed. (So it actually feels more like the relationship between the NSTableView and its data source in Cocoa.) So you can have the same programming model that OS X and GDI+ use today if you really want it without paying too much of a performance hit. (There is some perf cost - rebuilding part of the UI tree is likely to be more expensive than adjusting it. But it's still better than having to recreate it every time the OS decides it needs to refresh the display!)
  
  
  "Hmm....I just profiled TextEdit during live resize, and actual drawing was a negligible percentage of total CPU usage. So what are we gaining here? [...] That isn't just premature optimziation, that is completely superluous optimization."
  
  What kind of Mac do you have? On my 1GHz G4 PowerBook, and also on my 866Mhz iMac, resizing of windows is pretty sluggish. It seems to do resizing at a much slower rate than, say, scrolling - it behaves as though the resize rate has actually been throttled back. It's certainly not resizing at an update rate that would be fast enough on either of my Macs to do smooth transitions. (It's noticably much less smooth than scrolling, dock animations and the like.) So is that actually a realistic way of measuring performance? You're measuring something which (on both my Macs at least) is visibly obviously not going fast enough.
  
  (Of course it could be that resizing is a difficult operation for the Quartz Extreme engine - because Quartz uses a bitmap-based composition model, it will have to rejuggle all of its composition buffers when things change sizes. So it's possible that in this particular scenario, the CPU load is low because everything is stalled waiting for the compositor; you may simply have pegged the GPU rather than the CPU. Remember that a maxed out composition engine won't max out the CPU on Quartz Extreme because with QE, composition is done by the graphics card. That's obviously speculation of course - I've no idea why resizing is so much slower than everything else in Quartz, but I'm definitely not the only person to have noticed the problem if the web is anything to go by.)
• macos x already works like this
  2004-04-06 06:17:25  musnat [View]
  
  
  I don't understand why you suddenly get personal, most probably because you are a mac idiot. Instead of trying to be to the point, you seem to be more agressive, despite the fact that the author is really trying to discuss the real issues.
  
  By doing this, you void your whole posts, because it seems to me that they are either made-up or that you simply don't know enough about APIs and Quartz.
  
  You already stopped discussing the real issues, you are trying to discussing your own stuff. You never attempted to understand what the author says, instead you seem to try to save Apple's ass here, thinking that if you somehow make the point that quartz is great and actually far more better, than you would be done.
  
  Here is how things went.
  
  A mac idiot claimed that Quartz is actually working the same way as Avalon, because he reads too much slashdot or mac magazines.
  
  The author, as someone who knows what he is talking about refuted the idea.
  
  Another mac idiot came in and said bunch of stuff which didn't say much with respect to the fact that quartz is not a revolutionary api after all. He sort of continued to attack the author, but suprisingly (to me) the author was very patient and still to the point.
  
  Finally another mac idiot come in, who in this case seem to know more than other mac idiots, and again said bunch of stuff.
  
  If you ask whether quartz is something like avalon or not, it is clear that it is not and that quartz is not something revolutionary and that what you have in quartz is also in windows xp.
  
  With all the mac idiots coming in, trying to attack the author personally, all we got is a long thread without much substance from the mac idiots. I don't get this, why are you guys so stupid?
• macos x already works like this
  2004-03-17 04:33:50  Ian Griffiths | [View]
  
  
  Oh one more thing...
  
  (For those interested in this discussion, please note that O'Reilly's forums seem to be chopping the conversation off once the replies get more than 13 levels deep. If you click on the "View" link under mweiher's post of 2004-03-15 13:58:01 you'll see that I replied, even though that reply does not show up on the main page. And then I replied again when the first reply seemed not to have appeared...oops!)
  
  We seem to have moved onto arguing over the relative merits of vector vs bitmap retention, and the right place of the compositor in relation to the drawing mechanisms.
  
  Does this mean that we can all at least agree that macos x does not in fact work like this today? (Despite what the subject line says.)
• macos x already works like this
  2004-03-15 13:58:01  mweiher [View]
  
  
  1. "But my point is that the retained vectors are crucial."
  
  Yes, it is obvious that you think so. But this is just your claim. Alas, you seem to not even have attempted to understand the points I have raised, so I am not sure wether continuing this discussion is going to be useful. I am going to give it one more shot
  
  2. "This PDF/Postscript thing keeps coming up."
  
  Yes, it will continue to come up until you begin to understand it. Quartz has a device independent imaging model, implemented by the Quartz 2D renderer and based on the Postscript/PDF imaging model. It is not tied to screen resolution.
  
  First of all, let's be clear about one thing: Quartz is not DPS, it is not "DisplayPDF", it is rendering engine closely tied to the PS/PDF *imaging model*, in such a way that calls to CoreGraphics API correspond closely to PDF 'operators'. That way, PDF can be used as the 'storage' for a stream of CoreGraphics commands.
  
  Second, despite your fixation on "Quartz compositor", the part that actually does the rendering is "Quartz 2D". The compositor just handles "windowing services", meaning mostly the interaction between different application's windows, such as the various bits of transparency between windows (translucent titles, menu-bars, drop-shadows).
  
  I am not sure why you are so fixated with the compositor, or what led you to believe that this is the part to look at. Maybe it is because Avalon pushed its rendering back to the last step, or because Windows uses multitudes of composed Windows for a lot of its drawing. Whatever the reason, the Quartz compositor is (essentially) just for handling what you would refer to as "top level" windows and how they interact.
  
  
  This has *nothing* to do with an applications drawing objects into its window(s), wether with transparency or not. That is all handled by the Quartz 2D rasterizer, the drawing engine if you will. Once again, increasing the resolution of the output of Quartz 2D simply means concatenating a device-(scale)-matrix before you do the rest of your drawing. I do this all the time, and you get beautiful high-resolution bitmaps. In fact, this is how Quartz rasterizes for bitmap printers.
  
  I have to admit I am somewhat flabbergasted by your screen-grab example and your mention of PDFs coming out of the compositor. I can only assume you were hurried and didn't think your response through.
  
  PDFs go *into* Quarz 2D (the rasterizer part), they don't come *out* of it. Also, they go into Quartz 2D (again, the rasterizer) not the compositor. Once they have been rasterized, they are, of course, rasters. What did you think a rasterizer does?
  
  Anyway, I have to agree that Apple does make things confusing by saving the (bitmap) screenshots in PDF format, because they are really just bitmaps. However, and repeating redundantly just to make sure, that is *after* rasterization.
  
  The way to handle other resolutions is, of course, not *after* rasterization, because then you will obviously be scaling rasters, but *during* rasterization. And, repeating myself, the process is simple: just supply a different initial transformation matrix (or device matrix in PS/PDF terms). And yes, this works today, though of course not when going to screen, where you'd just get a bigger bitmap, not a higher-resolution one...
  
  So once again, you are looking at the wrong thing. The compositor is not the part that handles what you want to do, Quartz 2D is. And it handles it just fine. And yes, Quartz 2D also handles transparency (and any compositing it may have to do internally to achieve transparency effects).
  
  I am getting the feeling that you have been misled to believe that all transparency handling (all 'compositing') is handled by the Quartz compositor. This is not the case (or at least not visible). Once again, Quartz 2D handles this just fine.
  
  I have written a Postscript interpreter that can be hooked up to Quartz to do its rendering, resulting in a little live PS viewer with live output.
  
  The Postscript code shown below results in black text shining through the translucent red rectangle painted on top of it:
  
  
0 setgray
/Times-Italic 50 selectfont
50 150 moveto (This is an example of Quartz transparency!) show
1 0 0 setrgbcolor
.5 setalpha
100 100 200 200 rectfill

  
  The result can be seen here:
  
  quartz2d_transparency.tiff
  
  [Note: this is actually different from the way DisplayPostscript handled transparency, where painting the rectangle would have obscured the text below using the standard painter algorithm. The transparency would only have mattered if the resulting bitmap were later composited with another bitmap. I am guessing that this is actually close to the way you are thinking about it, but that is NOT the way Qartz works. Quartz applies the transperency immediately]
  
  This will render at any resolution I want it to, not buts about it and nor jaggies from bitmap scaling. Of course, drawing this results in some sort of compositing operation occuring internally, but how this is handled is up to Quartz 2D.
  
  In fact, the compositor is for the most part not even a publically accessible API. I am guessing that you got confused because Avalon seems to mix these functions up into one.
  
  So to sum up: Quartz 2D can handle higher resolutions just fine, including rendering transparency, which may or may not include a hidden "compositing" step, without scaling bitmaps. This compositing step has nothing to do with the Quartz compositor (unless possible internally) and preserves full resolution.
  
  3. "doing your own redrawing"
  
  My comments already contained a reply to your reply, but maybe it was a bit terse, so I will try to expand.
  
  Once again, you put this point under "it makes life easier for the developer:". So the question is *not* "does OS X retain graphics", it does not. The question is wether your assumption that not having graphics that are retained by the OS means extra developer burden is correct.
  
  Your assumption that this means extra burden is not correct. With Cocoa, the *developer* does not have to do anything extra.
  
  Let's examine this step by step.
  
  At some point, the developer's code has to somehow draw/paint/define the display. Agreed?
  
  With Cocoa, this is done once, in the View's -drawRect: method. That is all there is.
  
  With a retained drawing model, your application also has to do this (at least) once. So at this point were are even. Agreed?
  
  The difference you want to see is when a redraw happens. Agreed?
  
  So let's see what happens in the retained drawing case. Part of the system software (the graphics subsystem) notices it needs to regenerate part of the display. It consults the model it has been passed (a display list) to regenerate that part of the display. Easy, no new code for the programmer to write.
  
  So now let's see what happens in Cocoa's case. Part of the system software (the Cocoa framework) notices it needs to regenerate part of the display. It consults the display model the programmer (already) defined (the NSView hierarchy) to regenerate the part of the display. Also no new code for the programmer to write because the framework just uses the code from above.
  
  From the *developer's* point of view the two are absolutely equivalent: no extra code to handle the update event.
  
  And it was *developer* ease of use that you claimed was the advantage.
  
  Of course, there *is* a difference in that it is actually the application's code that is being exercised in the redraw in the Cocoa case, but that is a question of how system-software is partitioned, not a question of developer ease of use.
  
  To sum up:
  
  4. Lack of retained graphics is a core issue
  
  I hope I have managed to communicate more effectively why the problems that you think exist ( "high dpi" , "more programmer work") actually do not.
  
  5. GDI+ vs. Quartz
  
  You keep claiming that GDI+ is somehow up to the level of Quartz. This is simply not the case. GDI+ is, from an imaging model point of view, much more like DisplayPostscript (around 1989).
  
  6. Speed
  
  There are actually a number of smaller factual errors that I am not going to go into now, this post is already too long. However, I seem to have not managed to get the point of my TextEdit rescaling example across at all. I will have to do better.
  
  You say: "Of course it could be that resizing is a difficult operation for the Quartz Extreme engine"
  
  This isn't even close.
  
  What the profile showed that most of the time was *not* spent rendering graphics. It was spent on the *application* side, not in the graphics subsystem. It was spend recalculating the line-layout, doing typography, not rendering it.
  
  So the graphics subsystem (Quartz 2D, not Quartz Extreme, which is just a marketing name for the hardware accelerated compositor that isn't really much involved here at all, see above) is not the limiting part of this. If we could drop in Avalon and Avalon were infinitely fast, it would hardly speed this operation up at all.
  
  Is it at all clear what I am saying here? If you have an operation (a live resize), with 90% of the time being taken up by recalculating your line-layouts, and 10% by redraws, then accelerating the 10% part is not going to have a significant effect on performance.
  
  And of course, if you actually do junk the entire retained model each time you update the application model, then you are not going to gain any performance, because you are actually doing *more* work by building up this model that you then promptly destroy. Trust me, this has been hashed out before...
  
  To sum up: graphics subsystem performance is not the limiting factor of the TextEdit live-redraw, making redraw faster would have negligible effect on performance.
  
  7. Other issues
  
  There are actually a whole number of other issues that need to be addressed, but these are a bit more subtle and complex than the very simple issues here. So I don't think there is much of a point of starting on those before the much more simple (mostly basic factual errors) here have been dealt with.
  
  Summary:
  
  - The immediate-drawing model of Quartz 2D does not have the "problems" you think you identified (inherently incapable of handling higher res displays, more work for the developer)
  - Quartz 2D renders vector graphics with correct transparency effects today, at any resolution desired (subject to implementation limits). You do not have to invoke the Quartz compositor to do this, the Quartz compositor is used for different tasks. (The implementation probably uses compositing operators to achieve these effects, this is hidden).
  - Lack of retained graphics is not a "core issue", it is an implementation choice.
  - Drawing performance is not the limiting factor for "normal" applications, even with during highly interactive and graphics-intensive operations.
  
• macos x already works like this
  2004-04-06 06:16:16  musnat [View]
  
  
  I don't understand why you suddenly get personal, most probably because you are a mac idiot. Instead of trying to be to the point, you seem to be more agressive, despite the fact that the author is really trying to discuss the real issues.
  
  By doing this, you void your whole posts, because it seems to me that they are either made-up or that you simply don't know enough about APIs and Quartz.
  
  You already stopped discussing the real issues, you are trying to discussing your own stuff. You never attempted to understand what the author says, instead you seem to try to save Apple's ass here, thinking that if you somehow make the point that quartz is great and actually far more better, than you would be done.
  
  Here is how things went.
  
  A mac idiot claimed that Quartz is actually working the same way as Avalon, because he reads too much slashdot or mac magazines.
  
  The author, as someone who knows what he is talking about refuted the idea.
  
  Another mac idiot came in and said bunch of stuff which didn't say much with respect to the fact that quartz is not a revolutionary api after all. He sort of continued to attack the author, but suprisingly (to me) the author was very patient and still to the point.
  
  Finally another mac idiot come in, who in this case seem to know more than other mac idiots, and again said bunch of stuff.
  
  If you ask whether quartz is something like avalon or not, it is clear that it is not and that quartz is not something revolutionary and that what you have in quartz is also in windows xp.
  
  With all the mac idiots coming in, trying to attack the author personally, all we got is a long thread without much substance from the mac idiots. I don't get this, why are you guys so stupid?
• macos x already works like this
  2004-03-17 04:29:36  Ian Griffiths | [View]
  
  
  For some reason my previous post didn't get through, possibly because it had embedded URLs, so here it is again, but without the URLs...
  
  
  "Alas, you seem to not even have attempted to understand the points I have raised"
  
  Since I replied to every point that you raised, and explained which parts I was finding hard to understand or impossible to resolve with experimental evidence, that seems like a harsh comment. Particularly as I have been trying out code samples on both Mac OS X and the Longhorn preview to try and make sure that all my statements were based on real observations, rather than conjecture.
  
  
  "Quartz has a device independent imaging model, implemented by the Quartz 2D renderer and based on the Postscript/PDF imaging model."
  
  I've read this whole section of yours again, and nothing you say contradicts my current understanding. You've gone into a detailed explanation of a bunch of stuff I already knew... I don't disagree with anything you said about the relationship between Quartz 2D, PDF/poscript drawing primitives and the like. Neither does it have any bearing on what I'm trying to explain.
  
  So I'm evidently not making my point clearly enough! But it occurs to me that I could express my view of what I think the problem is in a different way, because then it might be more clear what I'm on about. Here, in a nutshell, is the problem:
  
  Just because your drawing API is resolution independent doesn't mean your GUIs are.
  
  Why on earth would I say that? Mainly because of the extensive experience I've had over the last 15 years with three different drawing APIs that were all resolution independent, two of which were modelled closely on the drawing facilities of Postscript, and none of which yielded resolution-independent APIs.
  
  The APIs in question are:
  
  GDI32, the original Win32 drawing API
  GDI+, the new Win32 drawing API
  DrawingModule, the vector drawing section of the long-since-defunct RISC OS
  
  GDI32 is and always has been resolution independent - not only does it support a variety of drawing units, it also supports arbitrary transformations - all drawing can be put through a world transform matrix that allows all the normal affine transforms. The only APIs that deal in actual pixels are the bitmap APIs.
  
  GDI+ is the same. The main differences are that is supports a wider range of drawing primitives, and it also supports anti-aliasing and alpha blending.
  
  The (rather obscure) RISC OS vector drawing facilities are pretty similar again - a set of vector drawing facilities providing the same basic set of primitives as Postscript, but because it's much older (introduced in 1987) it doesn't have any of the new stuff that GDI+ has like alpha blending.
  
  All of these can perform the trick you showed with your postscript sample - you can stream out a series of drawing commands into a file, and then play them back at any resolution with no loss of precision. (In Win32 you do this with EMF files, GDI+ uses EMF+ files, and RISC OS had, if memory serves, what were called !Draw files.)
  
  And yet none of these could scale the entire UI. They could all scale individual fragments of drawing, just like you showed, but that's the easy bit. The hard part is scaling the entire UI.
  
  Why is that hard? Well, the key to this was revealed when you said this:
  
  "Whatever the reason, the Quartz compositor is (essentially) just for handling what you would refer to as "top level" windows and how they interact.
  
  "This has *nothing* to do with an applications drawing objects into its window(s), wether with transparency or not. That is all handled by the Quartz 2D rasterizer, the drawing engine if you will."
  
  But you can put an OpenGL surface in a window. You can put a QuickTime surface in a window. These can both coexist in the same window as Quartz 2D stuff, and yet neither of these things are drawn by the Quartz 2D rasterizer are they?
  
  So what you're saying doesn't appear to be consistent with reality - how can the window be built entirely by Quartz 2D when it contains non-Quartz-2D elements?
  
  It is clear that a window in Mac OS X can be composed from multiple elements, not all of which are Quartz 2D elements. So what you say here cannot possibly be true, can it? Something must be composing the 2D stuff with the OpenGL stuff and the QuickTime stuff. If it's not the Quartz Compositor, then what is it? (And according to all of Apple's diagrams, it is the Quartz compositor. But you're saying this is not so. What do you think does this composition?)
  
  
  If the inside of every window was a purely Quartz 2D world (which appears to be what you're saying, even though firing up the QuickTime player will disprove it pretty quickly) then yes, we would have instant scalability. And in the same way, if the inside of a window in a Windows application is a pure GDI32 world or a GDI+ world, then again you have full scalability.
  
  But certainly in Windows, the problem is that you don't have just one GDI+ surface or one GDI32 context for a whole window. What you have is a bunch of little drawing areas composed together. And where Windows and RISC OS all fell short is that the technology that composed these windows together wasn't resolution independent.
  
  You're telling me that Quartz doesn't suffer from this because you believe the compositor is irrelevant. So here's the thing I don't understand: if the compositor is irrelevant, how do QuickTime and OpenGL fit into all this? Are you saying that if I put down an NSMovieView in a window, it's not actually QuickTime that renders it, but the Quartz 2D engine? I'm guessing you won't claim that, in which case, how can these things end up side by side in one window without getting the compositor involved?
  
  
  "I have to admit I am somewhat flabbergasted by your screen-grab example and your mention of PDFs coming out of the compositor. I can only assume you were hurried and didn't think your response through."
  
  I was trying to see if I could find some concrete evidence to prove your claim that the UI is scalable to be correct. (Actually I first tried this experiment ages ago, back when I believed that the UI was scalable.)
  
  Here's the thinking behind the experiment. There doesn't appear to be any way of scaling windows up on screen built in to OS X. (And the Expose stuff that scales things down *definitely* does bitmap scaling, not vector scaling.) So I was looking for some way of obtaining the contents of a window in scalable form.
  
  PDFs are, as you say, normally scalable to very high resolutions, so given that the built-in screen grab shortcut generates a PDF, then surely if the whole UI is scalable, there's no good reason for that PDF not to be a scalable one. It should be possible for OS X to generate a PDF that contains a scalable version of what's in the window, if the whole UI really is scalable - it should be trivial in fact.
  
  But as it happens, that's not what you get - it's just a bitmap. (I don't really see why they put it in a PDF if all they're going to produce is a bitmap...)
  
  While that's disappointing, I'm aware that it's also inconclusive. If the PDF had been a proper vector PDF, then that would have proved your argument conclusively - it would have demonstrated the scalability of the UI by showing a scalable representation of the UI. However, the fact that it just produced a bitmap is odd, but inconclusive. (Particularly when it does this even for individual windows. I could see how a full screen grab would be problematic, as it has to show the drop shadows, but why would it need to use a bitmap for a single window? Unless perhaps it uses bitmap composition within windows too...)
  
  The reason I described what I had done despite the inconclusive results was to illustrate the kind of thing I was hoping to see - a complete scalable representation of a window. I was hoping you'd show me something similar, but which actually works.
  
  I understand that Quartz 2D isn't using PDF from end to end, and is instead using a repertoire of drawing primitives very similar to what PDF offers. (I've been using drawing APIs that work that way for 15 years now, so I've had a while to get used to the idea!) But that's not a reason for it *not* to generate resolution independent output from a screen grab, if, as you're claiming, the UI really is scalable. I know that the vector handling is not as simple as the screen being one big PDF, but nonetheless, it could just convert from its internal representation back to PDF at the last minute, as long as its internal representation really is completely vector-based. (After all, it's perfectly capable of letting you print to PDF. So the fact that it can't do the same with a window is suspicious.)
  
  
  So the screen grab experiement is a failure. I mentioned it because I was hoping you were going to show me some technique for getting a scalable representation of a window.
  
  Instead you showed me a little bit of postscript.
  
  In other words you showed me the easy bit - scaling a bit of drawing, rather than scaling the whole UI. The three other drawing technologies I mentioned can all do that too. Even Win16 could do that with the WMF file format. (10 years ago I wrote a Windows program that used WMFs for all of its internal icons precisely because I wanted to be able to scale the drawings. And it works just fine. But I couldn't scale the rest of the UI.) The support for individual snippets of fully scalable drawing either held in internal data structures or serialized out to some stream has been in Windows for a decade and a half, and yet Windows does not have a scalable UI.
  
  This is why I was trying to see if there was some way of getting a scalable representation of a whole window - that's the tricky part, and is the acid test of genuine UI resolution independence; rendering snippets proves nothing. The whole UI is tricky because it might not just contain a single Quartz 2D surface. It could easily contain other surfaces. This is precisely where Windows falls over. It was precisely where RISC OS fell over. So it seems likely that Mac OS X will fall over here too, since it really isn't significantly different from GDI+. (Despite what you appear to think GDI+ is. But I'll come back to that later.)
  
  As I said before, just because your drawing API is scalable doesn't mean your UI will be, unless your UI uses nothing but your scalable drawing API. And Quartz supports multiple drawing APIs within a single window - that's the problem. (And because it has to support this, the support impinges on the design even when you're not using the functionality.)
  
  I challenge you to show me an example demonstrating a whole *window* in a scalable form, toolbars title bars and all. I suspect it cannot be done. If you can show me an example of this, then great! I'm happy to believe evidence, and it will reverse the disappointment I expressed earlier int his thread. All the evidence I've seen from the experiments I've tried seem to indicate that it cannot. (And I've tried lots of things to see if it can be done - I initially just assumed that it could be done when I first got my Mac, because my understanding of how Quartz worked was, well, pretty much what you've described. I've since discovered that my understanding seems to have been a little oversimplified, and that it apparently can't do all the things I thought and you still do think it could do.)
  
  
  
  "doing your own redrawing"
  
  "Once again, you put this point under "it makes life easier for the developer:". "
  
  That's one of the points, although not the only advantage. The other big advantage is perf - if the application is required to respond to redraw requests, it forces everything the app redraws to go across a bus to the graphics card in some form, and that bus is a major bottleneck.
  
  But I did also make claims on easy of programming. As an example of how retained UI makes things easier, do you think that HTML pages would be as easy to author if you had to handle repaint messages, rather than simply declaring what you want on screen? I know loads of non-technical people who are quite capable of creating web pages. I believe there is a gulf between this level of ease of use and the ease of use of handling repaint requests from the OS. Do you really disagree with that?
  
  
  "At some point, the developer's code has to somehow draw/paint/define the display. Agreed?"
  
  Actually no - read my earlier article in this series on XAML for details. Or indeed learn about HTML.
  
  (This is a complete new subject area by the way, and I know that it's kind of irritating when someone starts talking about something else in the middle of an argument, but you made that statement, and ask me if I agreed. I disagree strongly, and now feel compelled to explain why, so I apologise in advance for talking about a new topic. You did ask.)
  
  A declarative UI model mandates a retained model, although a retained model does not require a declarative UI. So just to be clear, you can use either declarative+retained or imperative+retained in Avalon.
  
  If you employ a declarative UI model, there is absolutely no need for developers to get involved with redrawing. They can just create the display with so much markup.
  
  You shouldn't really be surprised at that, because that's not so very different from how you build UIs in OS X. The Interface Builder uses drag and drop while today, Microsoft don't have an interactive Avalon GUI builder ready, but that will come. And while Avalon uses markup, Interface Builder generates serialized objects, if I understand correctly. But it's still a case of telling the system "One of these here, one of those there" and letting it handle the detail without having to write any code to manage the UI. Where Mac OS X falls short is that you can only assemble existing components this way - it doesn't let you handle drawing primitives in this way, forcing you instead to write code to generate those primitives.
  
  In my opinion, that's bad for ease of use.
  
  It's also ridiculously easy for code to make use of stuff defined in the markup, so don't fall into the trap of thinking that this is only any use for static UIs. It's not so different from the HTML DOM (except you're not obliged to use a scripting language - any .NET-capable language will do) - everything you define in markup is an object, so you can write code to go and twiddle with it at runtime.
  
  I wrote an example showing how to draw 'rubber band' outline selections without having to write a line of drawing code that illustrates both how you can use markup instead of code to create a UI, and also illustrates how that UI can still be dynamic. It's in the second half of this blog entry [URL removed; Google for: ian griffiths rubber band). I suggest you take a look - given the blanket statement you made, it sounds like you've not even considered this style of programming before. It might help you understand where I'm coming from. (Again I'd skip over the first part - that's mostly for the benefit of Windows developers, and will be old hat for anyone familiar with Quartz 2D.)
  
  
  So no, I strongly disagree with your thesis that drawing is necessarily the programmer's job. (And since your thesis was the underpinning of everything that followed that section of your post, there's not a whole lot else I can say in reply. Your axioms are wrong, so the rest of the argument isn't useful.)
  
  
  "I hope I have managed to communicate more effectively why the problems that you think exist ( "high dpi" , "more programmer work") actually do not."
  
  Well... you've successfully explained how I used to think Quartz 2D works. But try as I might I can't find any way of proving that a window can necessarily be scaled, and I hope I've presented some issues that show that some kind of compositor that is not part of Quartz 2D is necessarily involved. So I now hope you can refute this with a simple example. I'm afraid that I'm the kind of person who prefers proof, so if you can't show me an example that demonstrates the scalability of an entire window (rather than just an isolated piece of graphics) you're going to have a hard time convincing me.
  
  
  GDI+ vs. Quartz
  
  "You keep claiming that GDI+ is somehow up to the level of Quartz. This is simply not the case. GDI+ is, from an imaging model point of view, much more like DisplayPostscript (around 1989)."
  
  Could you be specific there? As far as I know there's nothing Quartz 2D can do that GDI+ can't. And GDI+ is way ahead of circa 1989 DPS - there's plenty GDI+ can do that display postscript can't. (You gave the example of transparency handling and how display postscript uses the painters algorithm. That's what GDI32 uses too. But it's not what GDI+ uses - you can specify alpha for any drawing operation in GDI+, just like you can in Quartz 2D.)
  
  Moreover, until Jaguar came along, GDI+ was able to do things Quartz 2D couldn't. Quartz 2D didn't original support gradient fills, for example, but these have been in GDI+ from the start.
  
  Are you sure you're not confusing GDI+ with GDI32? Your description characterises GDI32 perfectly, but it's an inaccurate description of GDI+.
  
  
  On profiling
  
  "What the profile showed that most of the time was *not* spent rendering graphics. It was spent on the *application* side, not in the graphics subsystem. It was spend recalculating the line-layout, doing typography, not rendering it"
  
  You don't consider line layout and typography to be graphics operations?
  
  To be honest, you've not given enough detail for me to be able to draw any conclusions from your profiling. Any chance you could post what you actually found somewhere? This forum doesn't really make it easy to provide enough detail. For example, it now seems like you're saying the complete opposite of what you said last time about the profiling results. (I'm sure that's not what you thought you said, but that's honestly how it came across to me.)
  
  "If we could drop in Avalon and Avalon were infinitely fast, it would hardly speed this operation up at all."
  
  Layout and typography support is all part of what Avalon supplies, so that's not in fact correct. If Avalon were infinitely fast, it would make this operation infinitely fast.
  
  Of course that has nothing to do with acceleration, since it's pretty unlikely that Avalon will hardware accelerate layout. I'm pretty amazed that it really is spending this much time in layout code, which is why I'm intrigued to see your actual profiling results. From what you've told me so far, the most likely explanation would seem to be that the layout code is a bit crappy - maybe you've just found a weak spot of Quartz 2D. However, I don't think that's the whole story - resize performance is pretty ropey on *all* windows on all Macs I've ever used (all the ones using OS X at least). Even on a little program that draws nothing but a single ellipse in a resizable view, and I'm pretty sure that program isn't spending all its time laying out text because there was no text...
  
  So if you can share it I'd love to see your profiling setup and the data you generated.
  
  
  
  "And of course, if you actually do junk the entire retained model each time you update the application model, then you are not going to gain any performance, because you are actually doing *more* work by building up this model that you then promptly destroy. Trust me, this has been hashed out before..."
  
  But that's exactly the model you've been saying is so fine in Quartz 2D! There you absolutely have to reconstruct the entire model every time you get sent a redraw request.
  
  Also, you're ignoring what I said about the problems of a software 2D engine (regardless of whether drawings are retained) - the gap between bus transfer speeds and CPU speeds gets ever higher, so any model that rasterizes on the CPU and then transfers the results across a bus to the graphics card is going to find that transfer becoming a more and more glaring bottleneck.
  
  It has indeed been hashed out before - the games industry has been hashing this out for years now, the introduction of GPU-side vertex handling indicates that they've all settled on a retained model, and they long since settled on moving rasterization into the GPU. This is the prevailing approaches because it's the only one that scales to the complexity and speed they need. So it's been hashed out before, the end-to-end-vectors model won long ago, and the consensus amongst graphics card manufacturers seems to be that vector retention model is also the way to go!
  
  
  "Summary:"
  
  "- The immediate-drawing model of Quartz 2D does not have the "problems" you think you identified (inherently incapable of handling higher res displays, more work for the developer)"
  
  Until you produce convincing evidence, I have to believe what the Apple docs and my experiments tell me. Rendering snippets is not convincing, because even 16-bit Windows can do that in a resolution independent way. Show me a bitmap containing a whole window at double the resolution (or a PDF, or a postscript file - whatever is easiest), and I'll believe you. My challenge to you is that I don't believe this can be done without bitmap scaling.
  
  "- Quartz 2D renders vector graphics with correct transparency effects today, at any resolution desired (subject to implementation limits)."
  
  Also true of GDI+. None of which alters the fact that GDI+ hasn't solved Windows' lack of resolution independence.
  
  
  "You do not have to invoke the Quartz compositor to do this, the Quartz compositor is used for different tasks. (The implementation probably uses compositing operators to achieve these effects, this is hidden)."
  
  Of course it uses compositing operators to achieve this - otherwise you'd only ever see the last drawing primitive you drew! But of course it's a different code path from the Quartz Compositor, because Quartz 2D is all software.
  
  "- Drawing performance is not the limiting factor for "normal" applications, even with during highly interactive and graphics-intensive operations."
  
  The two most highly graphics-intensive kinds of applications around today are video editing and 3D work. Neither of these use the Quartz 2D pipeline.
  
  It's true that for static form-like displays, Quartz 2D performance is adequate. But it doesn't take long to hit its limits if you try to do anything ambitious. If you have access to a box running Longhorn (and make sure it's one with a supported graphics card), try running this extremely simple drawing demo (URL removed; google for:ian griffiths zoom xaml). Compare the smoothness of response with Adobe Illustrator on a Mac. (Illustrator's pretty slow even with very simple drawings.) Particularly notice how you can zoom in and out smoothly in realtime with a drag operation. Compare this to how Illustrator only works in chunks and takes about half a second to change zoom factor.
  
  (Just to reiterate, you will need a PC with a suitable graphics card. Because the Longhorn Developer Preview is a very early release, it supports only a small set of graphics cards. If it doesn't know your graphics card and therefore drops down to software rendering, the results will be rather disappointing. Although it's a pretty good example of why you don't want software rasterization...)
  
  I believe this demonstrates that a faster drawing engine enables styles of user interaction that simply weren't possible before. You say that drawing performance is not a limiting factor. I think that its an ubiquitous constraint we all learnt to live with so long ago that we work within its limits without even thinking about it. Liberation from these restrictions gives us the opportunity to enhance usability no end. It's not about making existing things faster, it's about enabling things that were previously impossible.
  
• macos x already works like this
  2004-03-16 04:43:14  Ian Griffiths | [View]
  
  
  "Alas, you seem to not even have attempted to understand the points I have raised"
  
  Since I replied to every point that you raised, and explained which parts I was finding hard to understand or impossible to resolve with experimental evidence, that seems like a harsh comment. Particularly as I have been trying out code samples on both Mac OS X and the Longhorn preview to try and make sure that all my statements were based on real observations, rather than conjecture.
  
  
  "Quartz has a device independent imaging model, implemented by the Quartz 2D renderer and based on the Postscript/PDF imaging model."
  
  I've read this whole section of yours again, and nothing you say contradicts my current understanding. You've gone into a detailed explanation of a bunch of stuff I already knew... I don't disagree with anything you said about the relationship between Quartz 2D, PDF/poscript drawing primitives and the like. Neither does it have any bearing on what I'm trying to explain.
  
  So I'm evidently not making my point clearly enough! But it occurs to me that I could express my view of what I think the problem is in a different way, because then it might be more clear what I'm on about. Here, in a nutshell, is the problem:
  
  Just because your drawing API is resolution independent doesn't mean your GUIs are.
  
  Why on earth would I say that? Mainly because of the extensive experience I've had over the last 15 years with three different drawing APIs that were all resolution independent, two of which were modelled closely on the drawing facilities of Postscript, and none of which yielded resolution-independent APIs.
  
  The APIs in question are:
  
  GDI32, the original Win32 drawing API
  GDI+, the new Win32 drawing API
  DrawingModule, the vector drawing section of the long-since-defunct RISC OS
  
  GDI32 is and always has been resolution independent - not only does it support a variety of drawing units, it also supports arbitrary transformations - all drawing can be put through a world transform matrix that allows all the normal affine transforms. The only APIs that deal in actual pixels are the bitmap APIs.
  
  GDI+ is the same. The main differences are that is supports a wider range of drawing primitives, and it also supports anti-aliasing and alpha blending.
  
  The (rather obscure) RISC OS vector drawing facilities are pretty similar again - a set of vector drawing facilities providing the same basic set of primitives as Postscript, but because it's much older (introduced in 1987) it doesn't have any of the new stuff that GDI+ has like alpha blending.
  
  All of these can perform the trick you showed with your postscript sample - you can stream out a series of drawing commands into a file, and then play them back at any resolution with no loss of precision. (In Win32 you do this with EMF files, GDI+ uses EMF+ files, and RISC OS had, if memory serves, what were called !Draw files.)
  
  And yet none of these could scale the entire UI. They could all scale individual fragments of drawing, just like you showed, but that's the easy bit. The hard part is scaling the entire UI.
  
  Why is that hard? Well, the key to this was revealed when you said this:
  
  "Whatever the reason, the Quartz compositor is (essentially) just for handling what you would refer to as "top level" windows and how they interact.
  
  "This has *nothing* to do with an applications drawing objects into its window(s), wether with transparency or not. That is all handled by the Quartz 2D rasterizer, the drawing engine if you will."
  
  But you can put an OpenGL surface in a window. You can put a QuickTime surface in a window. These can both coexist in the same window as Quartz 2D stuff, and yet neither of these things are drawn by the Quartz 2D rasterizer are they?
  
  So what you're saying doesn't appear to be consistent with reality - how can the window be built entirely by Quartz 2D when it contains non-Quartz-2D elements?
  
  It is clear that a window in Mac OS X can be composed from multiple elements, not all of which are Quartz 2D elements. So what you say here cannot possibly be true, can it? Something must be composing the 2D stuff with the OpenGL stuff and the QuickTime stuff. If it's not the Quartz Compositor, then what is it? (And according to all of Apple's diagrams, it is the Quartz compositor. But you're saying this is not so. What do you think does this composition?)
  
  
  If the inside of every window was a purely Quartz 2D world (which appears to be what you're saying, even though firing up the QuickTime player will disprove it pretty quickly) then yes, we would have instant scalability. And in the same way, if the inside of a window in a Windows application is a pure GDI32 world or a GDI+ world, then again you have full scalability.
  
  But certainly in Windows, the problem is that you don't have just one GDI+ surface or one GDI32 context for a whole window. What you have is a bunch of little drawing areas composed together. And where Windows and RISC OS all fell short is that the technology that composed these windows together wasn't resolution independent.
  
  You're telling me that Quartz doesn't suffer from this because you believe the compositor is irrelevant. So here's the thing I don't understand: if the compositor is irrelevant, how do QuickTime and OpenGL fit into all this? Are you saying that if I put down an NSMovieView in a window, it's not actually QuickTime that renders it, but the Quartz 2D engine? I'm guessing you won't claim that, in which case, how can these things end up side by side in one window without getting the compositor involved?
  
  
  "I have to admit I am somewhat flabbergasted by your screen-grab example and your mention of PDFs coming out of the compositor. I can only assume you were hurried and didn't think your response through."
  
  I was trying to see if I could find some concrete evidence to prove your claim that the UI is scalable to be correct. (Actually I first tried this experiment ages ago, back when I believed that the UI was scalable.)
  
  Here's the thinking behind the experiment. There doesn't appear to be any way of scaling windows up on screen built in to OS X. (And the Expose stuff that scales things down *definitely* does bitmap scaling, not vector scaling.) So I was looking for some way of obtaining the contents of a window in scalable form.
  
  PDFs are, as you say, normally scalable to very high resolutions, so given that the built-in screen grab shortcut generates a PDF, then surely if the whole UI is scalable, there's no good reason for that PDF not to be a scalable one. It should be possible for OS X to generate a PDF that contains a scalable version of what's in the window, if the whole UI really is scalable - it should be trivial in fact.
  
  But as it happens, that's not what you get - it's just a bitmap. (I don't really see why they put it in a PDF if all they're going to produce is a bitmap...)
  
  While that's disappointing, I'm aware that it's also inconclusive. If the PDF had been a proper vector PDF, then that would have proved your argument conclusively - it would have demonstrated the scalability of the UI by showing a scalable representation of the UI. However, the fact that it just produced a bitmap is odd, but inconclusive. (Particularly when it does this even for individual windows. I could see how a full screen grab would be problematic, as it has to show the drop shadows, but why would it need to use a bitmap for a single window? Unless perhaps it uses bitmap composition within windows too...)
  
  The reason I described what I had done despite the inconclusive results was to illustrate the kind of thing I was hoping to see - a complete scalable representation of a window. I was hoping you'd show me something similar, but which actually works.
  
  I understand that Quartz 2D isn't using PDF from end to end, and is instead using a repertoire of drawing primitives very similar to what PDF offers. (I've been using drawing APIs that work that way for 15 years now, so I've had a while to get used to the idea!) But that's not a reason for it *not* to generate resolution independent output from a screen grab, if, as you're claiming, the UI really is scalable. I know that the vector handling is not as simple as the screen being one big PDF, but nonetheless, it could just convert from its internal representation back to PDF at the last minute, as long as its internal representation really is completely vector-based. (After all, it's perfectly capable of letting you print to PDF. So the fact that it can't do the same with a window is suspicious.)
  
  
  So the screen grab experiement is a failure. I mentioned it because I was hoping you were going to show me some technique for getting a scalable representation of a window.
  
  Instead you showed me a little bit of postscript.
  
  In other words you showed me the easy bit - scaling a bit of drawing, rather than scaling the whole UI. The three other drawing technologies I mentioned can all do that too. Even Win16 could do that with the WMF file format. (10 years ago I wrote a Windows program that used WMFs for all of its internal icons precisely because I wanted to be able to scale the drawings. And it works just fine. But I couldn't scale the rest of the UI.) The support for individual snippets of fully scalable drawing either held in internal data structures or serialized out to some stream has been in Windows for a decade and a half, and yet Windows does not have a scalable UI.
  
  This is why I was trying to see if there was some way of getting a scalable representation of a whole window - that's the tricky part, and is the acid test of genuine UI resolution independence; rendering snippets proves nothing. The whole UI is tricky because it might not just contain a single Quartz 2D surface. It could easily contain other surfaces. This is precisely where Windows falls over. It was precisely where RISC OS fell over. So it seems likely that Mac OS X will fall over here too, since it really isn't significantly different from GDI+. (Despite what you appear to think GDI+ is. But I'll come back to that later.)
  
  As I said before, just because your drawing API is scalable doesn't mean your UI will be, unless your UI uses nothing but your scalable drawing API. And Quartz supports multiple drawing APIs within a single window - that's the problem. (And because it has to support this, the support impinges on the design even when you're not using the functionality.)
  
  I challenge you to show me an example demonstrating a whole *window* in a scalable form, toolbars title bars and all. I suspect it cannot be done. If you can show me an example of this, then great! I'm happy to believe evidence, and it will reverse the disappointment I expressed earlier int his thread. All the evidence I've seen from the experiments I've tried seem to indicate that it cannot. (And I've tried lots of things to see if it can be done - I initially just assumed that it could be done when I first got my Mac, because my understanding of how Quartz worked was, well, pretty much what you've described. I've since discovered that my understanding seems to have been a little oversimplified, and that it apparently can't do all the things I thought and you still do think it could do.)
  
  
  
  "doing your own redrawing"
  
  "Once again, you put this point under "it makes life easier for the developer:". "
  
  That's one of the points, although not the only advantage. The other big advantage is perf - if the application is required to respond to redraw requests, it forces everything the app redraws to go across a bus to the graphics card in some form, and that bus is a major bottleneck.
  
  But I did also make claims on easy of programming. As an example of how retained UI makes things easier, do you think that HTML pages would be as easy to author if you had to handle repaint messages, rather than simply declaring what you want on screen? I know loads of non-technical people who are quite capable of creating web pages. I believe there is a gulf between this level of ease of use and the ease of use of handling repaint requests from the OS. Do you really disagree with that?
  
  
  "At some point, the developer's code has to somehow draw/paint/define the display. Agreed?"
  
  Actually no - read my earlier article in this series on XAML for details. Or indeed learn about HTML.
  
  (This is a complete new subject area by the way, and I know that it's kind of irritating when someone starts talking about something else in the middle of an argument, but you made that statement, and ask me if I agreed. I disagree strongly, and now feel compelled to explain why, so I apologise in advance for talking about a new topic. You did ask.)
  
  A declarative UI model mandates a retained model, although a retained model does not require a declarative UI. So just to be clear, you can use either declarative+retained or imperative+retained in Avalon.
  
  If you employ a declarative UI model, there is absolutely no need for developers to get involved with redrawing. They can just create the display with so much markup.
  
  You shouldn't really be surprised at that, because that's not so very different from how you build UIs in OS X. The Interface Builder uses drag and drop while today, Microsoft don't have an interactive Avalon GUI builder ready, but that will come. And while Avalon uses markup, Interface Builder generates serialized objects, if I understand correctly. But it's still a case of telling the system "One of these here, one of those there" and letting it handle the detail without having to write any code to manage the UI. Where Mac OS X falls short is that you can only assemble existing components this way - it doesn't let you handle drawing primitives in this way, forcing you instead to write code to generate those primitives.
  
  In my opinion, that's bad for ease of use.
  
  It's also ridiculously easy for code to make use of stuff defined in the markup, so don't fall into the trap of thinking that this is only any use for static UIs. It's not so different from the HTML DOM (except you're not obliged to use a scripting language - any .NET-capable language will do) - everything you define in markup is an object, so you can write code to go and twiddle with it at runtime.
  
  I wrote an example showing how to draw 'rubber band' outline selections without having to write a line of drawing code that illustrates both how you can use markup instead of code to create a UI, and also illustrates how that UI can still be dynamic. It's in the second half of this blog entry. I suggest you take a look - given the blanket statement you made, it sounds like you've not even considered this style of programming before. It might help you understand where I'm coming from. (Again I'd skip over the first part - that's mostly for the benefit of Windows developers, and will be old hat for anyone familiar with Quartz 2D.)
  
  
  So no, I strongly disagree with your thesis that drawing is necessarily the programmer's job. (And since your thesis was the underpinning of everything that followed that section of your post, there's not a whole lot else I can say in reply. Your axioms are wrong, so the rest of the argument isn't useful.)
  
  
  "I hope I have managed to communicate more effectively why the problems that you think exist ( "high dpi" , "more programmer work") actually do not."
  
  Well... you've successfully explained how I used to think Quartz 2D works. But try as I might I can't find any way of proving that a window can necessarily be scaled, and I hope I've presented some issues that show that some kind of compositor that is not part of Quartz 2D is necessarily involved. So I now hope you can refute this with a simple example. I'm afraid that I'm the kind of person who prefers proof, so if you can't show me an example that demonstrates the scalability of an entire window (rather than just an isolated piece of graphics) you're going to have a hard time convincing me.
  
  
  GDI+ vs. Quartz
  
  "You keep claiming that GDI+ is somehow up to the level of Quartz. This is simply not the case. GDI+ is, from an imaging model point of view, much more like DisplayPostscript (around 1989)."
  
  Could you be specific there? As far as I know there's nothing Quartz 2D can do that GDI+ can't. And GDI+ is way ahead of circa 1989 DPS - there's plenty GDI+ can do that display postscript can't. (You gave the example of transparency handling and how display postscript uses the painters algorithm. That's what GDI32 uses too. But it's not what GDI+ uses - you can specify alpha for any drawing operation in GDI+, just like you can in Quartz 2D.)
  
  Moreover, until Jaguar came along, GDI+ was able to do things Quartz 2D couldn't. Quartz 2D didn't original support gradient fills, for example, but these have been in GDI+ from the start.
  
  Are you sure you're not confusing GDI+ with GDI32? Your description characterises GDI32 perfectly, but it's an inaccurate description of GDI+.
  
  
  On profiling
  
  "What the profile showed that most of the time was *not* spent rendering graphics. It was spent on the *application* side, not in the graphics subsystem. It was spend recalculating the line-layout, doing typography, not rendering it"
  
  You don't consider line layout and typography to be graphics operations?
  
  To be honest, you've not given enough detail for me to be able to draw any conclusions from your profiling. Any chance you could post what you actually found somewhere? This forum doesn't really make it easy to provide enough detail. For example, it now seems like you're saying the complete opposite of what you said last time about the profiling results. (I'm sure that's not what you thought you said, but that's honestly how it came across to me.)
  
  "If we could drop in Avalon and Avalon were infinitely fast, it would hardly speed this operation up at all."
  
  Layout and typography support is all part of what Avalon supplies, so that's not in fact correct. If Avalon were infinitely fast, it would make this operation infinitely fast.
  
  Of course that has nothing to do with acceleration, since it's pretty unlikely that Avalon will hardware accelerate layout. I'm pretty amazed that it really is spending this much time in layout code, which is why I'm intrigued to see your actual profiling results. From what you've told me so far, the most likely explanation would seem to be that the layout code is a bit crappy - maybe you've just found a weak spot of Quartz 2D. However, I don't think that's the whole story - resize performance is pretty ropey on *all* windows on all Macs I've ever used (all the ones using OS X at least). Even on a little program that draws nothing but a single ellipse in a resizable view, and I'm pretty sure that program isn't spending all its time laying out text because there was no text...
  
  So if you can share it I'd love to see your profiling setup and the data you generated.
  
  
  
  "And of course, if you actually do junk the entire retained model each time you update the application model, then you are not going to gain any performance, because you are actually doing *more* work by building up this model that you then promptly destroy. Trust me, this has been hashed out before..."
  
  But that's exactly the model you've been saying is so fine in Quartz 2D! There you absolutely have to reconstruct the entire model every time you get sent a redraw request.
  
  Also, you're ignoring what I said about the problems of a software 2D engine (regardless of whether drawings are retained) - the gap between bus transfer speeds and CPU speeds gets ever higher, so any model that rasterizes on the CPU and then transfers the results across a bus to the graphics card is going to find that transfer becoming a more and more glaring bottleneck.
  
  It has indeed been hashed out before - the games industry has been hashing this out for years now, the introduction of GPU-side vertex handling indicates that they've all settled on a retained model, and they long since settled on moving rasterization into the GPU. This is the prevailing approaches because it's the only one that scales to the complexity and speed they need. So it's been hashed out before, the end-to-end-vectors model won long ago, and the consensus amongst graphics card manufacturers seems to be that vector retention model is also the way to go!
  
  
  "Summary:"
  
  "- The immediate-drawing model of Quartz 2D does not have the "problems" you think you identified (inherently incapable of handling higher res displays, more work for the developer)"
  
  Until you produce convincing evidence, I have to believe what the Apple docs and my experiments tell me. Rendering snippets is not convincing, because even 16-bit Windows can do that in a resolution independent way. Show me a bitmap containing a whole window at double the resolution (or a PDF, or a postscript file - whatever is easiest), and I'll believe you. My challenge to you is that I don't believe this can be done without bitmap scaling.
  
  "- Quartz 2D renders vector graphics with correct transparency effects today, at any resolution desired (subject to implementation limits)."
  
  Also true of GDI+. None of which alters the fact that GDI+ hasn't solved Windows' lack of resolution independence.
  
  
  "You do not have to invoke the Quartz compositor to do this, the Quartz compositor is used for different tasks. (The implementation probably uses compositing operators to achieve these effects, this is hidden)."
  
  Of course it uses compositing operators to achieve this - otherwise you'd only ever see the last drawing primitive you drew! But of course it's a different code path from the Quartz Compositor, because Quartz 2D is all software.
  
  "- Drawing performance is not the limiting factor for "normal" applications, even with during highly interactive and graphics-intensive operations."
  
  The two most highly graphics-intensive kinds of applications around today are video editing and 3D work. Neither of these use the Quartz 2D pipeline.
  
  It's true that for static form-like displays, Quartz 2D performance is adequate. But it doesn't take long to hit its limits if you try to do anything ambitious. If you have access to a box running Longhorn (and make sure it's one with a supported graphics card), try running this extremely simple drawing demo. Compare the smoothness of response with Adobe Illustrator on a Mac. (Illustrator's pretty slow even with very simple drawings.) Particularly notice how you can zoom in and out smoothly in realtime with a drag operation. Compare this to how Illustrator only works in chunks and takes about half a second to change zoom factor.
  
  (Just to reiterate, you will need a PC with a suitable graphics card. Because the Longhorn Developer Preview is a very early release, it supports only a small set of graphics cards. If it doesn't know your graphics card and therefore drops down to software rendering, the results will be rather disappointing. Although it's a pretty good example of why you don't want software rasterization...)
  
  I believe this demonstrates that a faster drawing engine enables styles of user interaction that simply weren't possible before. You say that drawing performance is not a limiting factor. I think that its an ubiquitous constraint we all learnt to live with so long ago that we work within its limits without even thinking about it. Liberation from these restrictions gives us the opportunity to enhance usability no end. It's not about making existing things faster, it's about enabling things that were previously impossible.
• macos x already works like this
  2004-03-19 17:33:59  mweiher [View]
  
  
  "I challenge you to show me an example demonstrating a whole *window* in a scalable form, toolbars title bars and all"
  
  Easy:
  
  Open Xcode.
  Project -> New -> New Cocoa Project
  Build + Run
  Print -> Save as PDF
  Voilá
  
  You will notice that the text on the title bar as well as the gradients used in the title bar are all smoothly scalable.
  
  As well as anything scalable you place inside the window, if you decide to do so.
  
  You will also notice that the traffic-light buttons do not scale as well. Of course, they still scale, Quartz 2D is device independent after all, but because they are drawn as bitmaps they scale as bitmaps and do not look as nice as the other elements.
  
  As I have explained before, this has nothing to do with the drawing or windowing API used, but is simply a matter of application choice. Having a retained representation wouldn't change this by one bit.
  
  EmptyProjectPrint.pdf
  
  
• macos x already works like this
  2004-03-21 11:57:47  Ian Griffiths | [View]
  
  
  That title bar is not in vector format - it's three bitmaps! Your reason for thinking that it is in vector form is superficially plausible, but it turns out to be fallacious:
  
  "the gradients used in the title bar are all smoothly scalable."
  
  I'm assuming that you used the OS X Preview application to look at it - that applies some kind of interpolation (it looks like bilinear interpolation) to any bitmaps it finds in a PDF. Try saving out a whole screenshot using the Shift-Command-3 key press. That generates a bitmap of the screen stored in a PDF. Zoom in on that, and you'll see exactly the same effect - the gradient remains perfectly smooth as you zoom in, despite the fact that these screenshot PDFs just contain one big bitmap.
  
  This gradient smoothness, which you have mistaken for evidence of vector imagery, is just the result of the interpolating scaling algorithm being used to scale the bitmap - linear gradient fills in bitmaps happen to come out looking just great if you use either bilinear or bicubic interpolation to expand them.
  
  If you open your PDF up in Adobe Illustrator, you'll be able to see clearly that the title bar is just three bitmaps - the main area and the two end caps. (And the traffic lights are another partially transparent bitmap drawn on top of the main title bar bitmap.)
  
  Alternatively, if you don't have Illustrator, open it up in a PDF viewer that uses nearest neighbour scaling rather than a smoothing interpolation. The current PC version of Adobe Acrobat Reader does this. When you zoom in with such a view, it becomes perfectly clear that the title bar is just three composed bitmaps.
  
  So this remains unconvincing - you have produced a PDF containing a bunch of bitmaps. This simply appears to add further support to my two hypotheses (1: that OS X cannot produce scalable window content, and 2: that this is because once you get outside of a Quartz 2D surface, Quartz works by composing bitmaps).
  
  I tried a variation on this experiment. I wrote a Cocoa application with an NSImageView on it. I drew a scalable image in OmniGraffle and made that image the NSImageView's image, and also set the NSImageView's autosizing to make it resize with the application.
  
  When resizing the window, the contents of the NSImageView get redrawn every time. This of course gets Quartz 2D to redraw the contents from scratch every time. So on each redraw Quartz 2D rerasterizes the image to the approprate size and then hands it over to the compositor. So the imagery appears to be scalable. But look what happens when I print to PDF:
  
  ScalingFailure.pdf
  
  (I apologise for the exceptionally poor drawing skills, but Dammit Jim, I'm a progammer, not a graphic designer!)
  
  Even though you mistook the bitmaps in your image for vector imagery, surely you cannot fail to see that this PDF is clearly made up of bitmaps - the image view that I gave scalable image data to is quite clearly a bitmap in this PDF. (And I used exactly the same technique to generate this bitmap as you described by the way. I just added the NSImageView. And a button, to see what that would look like. I notice that scales poorly too, although that's presumably because it's done as a bitmap just like the traffic lights.)
  
  This leaves just the text in vector form. But even Win32 can do that. In fact it can do slightly better (even if you use the old GDI32 API) - if you set up an Enhanced Metafile device context and then pass it using a WM_PRINT message to a window, it will write a scalable representation of the window into the enhanced metafile. (This is the closest equivalent I could get to of printing a window using Save as PDF... under Windows, because Windows doesn't have built-in PDF support.)
  
  If generate a scalable EMF file of a window in this way on Windows XP, all the window decorations suffer from the same problem as your OS X example exhibits - the decorations all come out as bitmaps. Any text will come out in text form rather than bitmaps, as with your example. But unlike your example, in Win32, the resulting enhanced metafile will also put any vector drawing primitives (like ellipses, bezier curves etc) in vector format too. (Contrast this with what happened in my NSImageView example on the Mac.)
  
  So it's possible to do a slightly better job with Windows than you have done here. I therefore remain unconvinced. (But I'm still retaining an open mind. If anyone can show me a PDF of a UI that really is scalable, rather than simply being a bunch of bitmaps, then please do!)
• macos x already works like this
  2004-03-22 12:32:11  mweiher [View]
  
  
  1. "That title bar is not in vector format - it's three bitmaps"
  
  This turns out not to be the case. While you are correct that I did look at it in Preview, I also went to the trouble of actually dumping out the PDF drawing stream.
  
  While there are some bitmaps used, the main part of the title bar is a rectangle (vectors) that is filled with a pattern. It is *not* a bitmap.
  
  This is the dump of the actual page content stream:
  
  
  "q Q q 181 466.5 12 22 re W n 0 sc /Gs1 gs 181 466.5 12 22 re f /Perceptual
  ri /Gs2 gs q 24 0 0 22 181 466.5 cm /Im1 Do Q Q q 649 466.5 12 22 re
  W n 0 sc /Gs1 gs 649 466.5 12 22 re f /Perceptual ri /Gs2 gs q 24 0
  0 22 637 466.5 cm /Im2 Do Q Q q 193 466.5 456 22 re W n /Pattern cs
  /P1 scn 193 466.5 456 22 re f Q q 394 469.5 54 17 re W n 1 1 1 0 k
  q 1 0 0 -1 394 486.5 cm BT 13 0 0 -13 2 14 Tm /F1.0 1 Tf (Window) Tj
  ET Q Q q 181 106.5 480 382 re W n /Pattern cs /P2 scn 181 106.5 480
  360 re f Q q 189 468.5 14 16 re W n /Perceptual ri q 14 0 0 16 189
  468.5 cm /Im3 Do Q Q q 231 468.5 14 16 re W n /Perceptual ri q 14 0
  0 16 231 468.5 cm /Im4 Do Q Q q 210 468.5 14 16 re W n /Perceptual
  ri q 14 0 0 16 210 468.5 cm /Im5 Do Q Q q 181 106.5 480 382 re W n
  /Perceptual ri q 15 0 0 15 646 106.5 cm /Im6 Do Q Q"
  
  
  The crucial part is the following bit:
  
  
  /Pattern cs /P1 scn 193 466.5 456 22 re f
  
  
  This sets up a pattern color-space, then draws a rectangle that is filled with this pattern.
  
  The pattern that is used is defined as follows:
  
  
  q Q q /Perceptual ri /Im7 Do Q
  
  
  So here we get the "bitmap" that Illustrator seems to pick up. But this is not a bitmap that is drawn, it is a bitmap pattern that is used as a fill-color for a rectangle. This bitmap is a 24 x 22 vertical gradient from 0xfe to 0x9d.
  
  So why are we seeing a shape filled with a pattern that is a bitmap? Simple: a "bitmap" of gradually changing values is actually the recommended method for drawing a linear gradient in PDF! Or more precisely, was until version 1.3 of the spec. introduced shading dictionaries.
  
  Because we are using a well defined and device-independent imaging model, we know that drawing an image with the 256 gray values 0-255 is exactly the same as drawing 256 rectangles with gray values ranging from 0.0 - 1.0 (if we set up the color-spaces correctly). This can then be scaled to the required dimensions.
  
  So what you are seeing is a gradient implemented in PDF, at 24 step "resolution". Which, incidentally, is completely equivalent to drawing 24 (vector) rectangles, not just practically but also theoretically! Now they could have used more steps to define the gradient, but that would have been pretty pointless given the fact that this is a window title bar. They could also have used a shading dictionary, but that is only available in newer PDF versions, and my guess is that it just wasn't needed.
  
  2. "Scaling failure" with NSImageView
  
  You have made a well-known beginner's mistake in using NSImage that dates back to NextStep days. If you look up the NSImge documentation, you will find that it creates and caches a device-specific rendered version of any resource it uses. And it will then throw away the original, unless it is told otherwise. This behavior is appropriate for the NSImage's intended use: drawing icons and other screen ornaments.
  
  I am guessing you set up your NSImageView completely within Interface Builder, including referencing the pdf "image" by name from within the nib. That is how you get the behavior you describe, and I have duplicated this behavior in the following PDF example:
  
  NonScalable.pdf
  
  As you have noted, even that example will "scale" with the window as you change size, because the NSImage notices that its cached representation is not at the appropriate resolution for the current scale and will thus generate a new representation that does have the appropriate resolution for that particular scale.
  
  If you don't want that behavior, just use - (void)setDataRetained:(BOOL)flag to tell it to retain the original, which it will automatically do if initialized with a file path (instead of an image name, for exmaple). So, to get perfectly scalable output, just make the NSImageView "Editable" (there's a check box in Interface Builder) and drop a pdf in at runtime. I actually took the exact same PDF that I was referencing from within the nib and dropped it in the image-view. Voilá, perfectly scalable output, because the original scalable representation is retained:
  
  Scalable.pdf
  
  In fact, you can use the same pdf both time. Incidentally, the reason that the output did appear scalable while you were resizing is that NSImage will automagically notice that the scale it is drawing at to screen has changed and go back to the original source to re-render a cached representation for that scale. So as you can see, all the machinery is already in place, and WORKING, for automatic adjustment to different resolutions.
  
  Of course, invoking -setDataRetained: is timing-sensitive, there is no use doing this after the NSImage has already discarded the original.
  
  So to sum up: both of your "proofs" have evaporated, as usual, and just show that you didn't know what you're doing on Mac OS X. Of course, not knowing what you're doing on the Mac is OK, you're a Windows programmer after all. However, it is usually better to avoid making bold assertions by drawing wrong conclusions from an incomplete and flawed understanding of the facts.
  
• macos x already works like this
  2004-07-01 06:13:41  Ian Griffiths | [View]
  
  
  "But this is not a bitmap that is drawn, it is a bitmap pattern that is used as a fill-color for a rectangle."
  
  You're splitting hairs here. It's only "not a bitmap" if you use a narrow definition of "bitmap" that exludeds "bitmap patterns". But this is a dicussion about scalability, and bitmap patterns aren't any more scalable than non-bitmap patterns, so there's no significant difference between the two - 'bitmaps' and 'bitmap patterns' are just two ways of painting with bitmap imagery.
  
  In other words, it's still a bitmap, even if it's in pattern form. Indeed it's a 24x22 bitmap, according to your analysis, which is not scalable - the pixellation is quite obvious when you enlarge it, and that is my point. You previously claimed that it was perfectly smooth and scalable, but now you're agreeing that it is essentially bitmap data. (And yes, it's a bitmap pattern - it's still a bitmap-like image source; the fact that it's being used as a fill pattern rather than a standalone image doesn't change its intrinsic non-scalability.)
  
  So all I was saying is that it doesn't make your argument look very credible when you produce what you have now admitted is a 24x22 bitmap pattern, but claim that it is "smoothly scalable". This is emphatically not compelling evidence for the allegedly scalable nature you are claiming Quartz 2D has.
  
  Bitmaps may be the standard way of hacking up graduated fills in PDF, but they're just that - a hack. If you use a 16 bit per channel colour space, then 256 rectangles is not a proper representation of a graduated fill. (Also, drawing 256 rectangles is emphatically *not* the same things as a bitmap with 256 pixels - you have a widely-held but wrong-headed view of what a bitmap represents. This is, as you put it, a rookie mistake. I won't reproduce the argument here, because you can find an excellent explanation of why you're wrong if you google for "A pixel is not a little square") And in any case your argument doesn't even stand up on its own: you've not got 256 levels here - only 24. Which is not enough, and here's that word again, to make it *scalable* - it looks wrong when you enlarge it. The pixellation is clearly visible.
  
  
  
  "You have made a well-known beginner's mistake in using NSImage"
  
  No doubt - I am after all not experienced with OS X. But why on earth would it behave this way? This just shows that by default, OS X applications are not necessarily resolution independent, even if they use resolution independent image sources.
  
  If this is, as you say, a well-known mistake, doubtless that means loads of people have made it. (Particularly since this mistake causes absolutely no problems on screen that I can see - it behaves absolutely correctly when the application is running at the native screen resolution. In fact for on-screen use, the term 'mistake' doesn't seem applicable - there's nothing wrong with the application's behaviour. Problems only start to occur when we attempt to get a scalable version of the display.) Does that mean that every application that makes this mistake is going to look crummy if Apple ever get around to supporting high resolution displays?
  
  This all add weight to my argument that OS X is, by default, not scalable. If a perfectly straightforward way of putting vector imagery into A UI turns out not to be resolution independent, then OS X is not intrinsically resolution independent, end of story.
  
  It remains *possible* to write resolution-independent apps. But you have to set out to do that. It won't happen by accident. The same has been true in Windows for a decade now.
  
  
  
  "So to sum up: both of your "proofs" have evaporated,"
  
  What proofs? I was just pointing out that the image you supplied was not what you claimed it to be. And apparently I was right - you have now admitted that the fill is in fact a bitmap pattern. (That may be the standard way of doing grad fills, but it doesn't change the fact that it is, according to your analysis, a 22x24 bitmap pattern, which is not a scalable way of creating an image.)
  
  My assertion is, and has always been, that OS X is just like Windows today - it's a platform you can build resolution-independent UIs with if you make the effort, but in which UIs will typically not be resolution by default, because it requires effort to achieve resolution independence.
  
  I may have blundered into these proofs, but that doesn't change the basic facts. I, as a beginner, wrote an application that works absolutely fine in the native resolution, but turns out not to be resolution indepent. This proves point: resolution independence is not automatic in OS X.
  
  That's the point I've been putting to you all along, and you've been denying all along. All it took was a rookie mistake to proove the point... :-)
  
  The fact that I blundered into this existence proof doesn't reduce its worth. You seem to be attempting to dismiss me because I don't know a lot about OS X. That's a falacious argument, and I think you know it is. I can tell that, say, the elements of the button in the PDF I produced don't line up properly at all resolutions without having to be an expert Mac developer. Remember, you started generating PDFs as a way of trying to prove your point - I'm just using my non-technology-specific ability to recognize cruddy imagery when I see it to point out that these PDFs seem to be demonstrating the opposite of what you claim.
  
  (Unless of course creating PDFs like this was just the wrong way to go about things. Maybe that's a facility that doesn't really work right anyway. In which case, all these sample PDFs don't really proove anything at all.)
  
  
  
  Meanwhile, I see you've quietly ignored the other problem I illustrated in my example - that the button looks terrible. In fact it looks wrong even at the natural resolution, and just looks worse as you scale it.
  
  Actually I'll go into a little more detail on the button, because I didn't mention the most interesting thing about it. Try a few different magnifications. Notice how sometimes, the middle section lines up with the caps, and sometimes it doesn't. This is typical of the problems you will see if you try to scale the output of a program that has never been tested at any resolution other than its native resolution.
  
  This is one of the two main reasons why Apple can't simply apply a scale factor of 1.5 to applications running on a 150dpi display - lots of little glitches like that will come crawling out of the woodwork. In other words, it's pretty much exactly the same reason Microsoft don't do the very same thing on Windows. (The second reason is the widespread reliance on bitmaps in most UIs.)
  
  In fact it wouldn't surprise me if this problem - the fact that a scalable drawing API turns out not to be as useful as you might thing for making scalable UIs - is the whole reason they have decided to 'solve' the problem by arbtrarily declaring recently that 100dpi is the perfect resolution. This to me looks like a positive spin on an admission of defeat that they can't solve the scalability problem with the technology they have today. (Surely nobody actually *believes* that 100dpi is the ideal resolution. Are Apple going to recommend we throw out our 600dpi laser printers and dig out those old dot matrix machines?)
  
  Or did I make a rookie mistake with the button too?
• macos x already works like this
  2004-03-18 16:28:15  mweiher [View]
  
  
  Where to start? Phew!
  
  Easy bits first:
  
  1. Illustrator vs. your Longhorn example
  
  1a. Illustrator does not use Quartz. It uses Bravo, Adobe's own drawing engine. So you aren't even comparing to Quartz at all. [and yes, it uses some API to bring the result to screen, this could be Quartz but I hope we agree that that's not the point!]
  
  1b. I was doing live redraw with NeXTStep on 100 MHz 486 boxes more than a decade ago. Both in a shipping product (eXTRASLIDE) and in internal demos. Actually, the internal demos did live text-rescale on 25 MHz NeXTStations.
  
  1c. And yes, the Postscript interpreter hooked up to Quartz was also used in various live-redrawing apps on OS X (not publicly available), with the Postscript code intelligently adapting to the size changes to maintain certain corporate-design guidelines. I even had it hooked up to OpenGL textures, with each resize of the 3D element triggering a ( Postscript -> Quartz 2D -> OpenGL texture ) redraw.
  
  Smoothly.
  
  2. Your screen grab example
  
  I am glad you realize that at the very least this is "inconclusive". In fact, the only thing it could show is wether there is a retained vector level representation or not at the window level. However, there is no disagreement on this point: OS X does not have a retained vector representation at that level.
  
  The dispute is not (a) wether OS X has such a mechanism, it doesn't, but (b) wether or not such a retained vector-level representation is required for higher DPI displays.
  
  Instead of showing (b), you keep showing (a) and apparently somehow thinking you're done.
  
  3. Drawing engine vs. UI resolution independence.
  
  You rightly observe that having one doesn't imply the other. Then why on earth do you keep saying the opposite, that a particular type of API (retained vectors within the graphics API) is required?
  
  Having such an API wouldn't change the issues one bit.
  
  3a. Agreement on Quartz 2D scalability
  
  It seems you understand (and have understood) that Quartz 2D has no problem whatsoever producing higher resolution output and will therefore have no problem driving higher resolution displays. Glad that is settled.
  
  3b. Composited elements: OpenGL and QuickTime
  
  With Quartz 2D out of the way, let's look at what's left that might prevent the UI from scaling. You mentioned QuickTime and OpenGL surfaces that are integrated into the display by the Quartz compositor.
  
  Ignoring the vector parts of QuickTime for a moment, QuickTime is used for displaying animations of bitmaps. So you are going to doing bitmap scaling anyway, aren't you? How would a "retained vector representation" help here? Wether this scaling is done by QuickTime or the Quartz compositor is probably not that important.
  
  For OpenGL, you again have 2 options: either tell the OpenGL engine that it has to render to higher resolution, or scale up using the compositor, with the former being the higher quality option, one that seems pretty achievable considering how OpenGL works. Again, I don't see how a retained 2D vector representation for the rest of the display system will help here, or even make any difference whatsoever.
  
  So both OpenGL and QuickTime (a) do not represent a problem and (b) would not benefit from a "retained vector representation". Furthermore, both OpenGL and QuickTime are really just relevant for special media applications.
  
  3c. UI bitmaps
  
  You forgot a third one: hand-painted bitmaps used for the display but rendered through Quartz 2D. Again, how would those benefit from a retained representation? They wouldn't, the retained drawing engine would need to scale bitmaps just as before. In fact, this isn't an API issue at all, it is a question of higher resolution resources provided by application programs and OS level GUI libraries. So Apple and Third Parties will have to provide higher resolution bitmaps,
  
  However, it turns out that the primary bitmap that all third parties provide is the app icon. And this is already provided at a whopping 128x128 pixels! Much, much larger than you ever need except at insane zooms on a 72 dpi display. Want to venture a guess why?
  
  Anyway, I think I've now covered all the elements that go to screen.
  
  - bitmaps \ OpenGL \
  - vectors ---> Quartz 2D ------> screen
  - text / QuickTime /
  
  
  
  
  3d. Scaling DPS
  
  I actually did "scalable" screen-grabs from DPS by listening in on the WindowServer / app connection. Had to edit the output because the code referred to stored state ( execuserobject ... ), but otherwise it worked. Did wonders for text legibility.
  
  4. Performance intensive apps
  
  While we're on the subject, I am glad you brought up the fact that the apps with the highest performance requirements are video editing and 3D, and that both of these don't use Quartz 2D. I was going to mention this myself, but the post was already too long.
  
  Anyway: if both of these do not use Quartz 2D, how would speeding up Quartz 2D help them? Answer: it wouldn't.
  
  Apple seems to have made a deliberate choice of providing special (industry standard, cross platform) APIs for specific media types (movies, 3D), while providing an extremely high quality drawing engine for just about everything else. As you yourself have pointed out, both speed and quality requirements are different for dynamic and static media, so making this split seems like a wise idea.
  
  I am going to reply to the "developer ease of use" issue another time.
  
  
• macos x already works like this
  2004-03-21 14:39:30  Ian Griffiths | [View]
  
  
  "Illustrator does not use Quartz."
  
  Fair enough - my mistake.
  
  Can you show me an application that illustrates zooming in on vector imagery in OS X that does use Quartz then? I'd like to be able to see a proper side by side comparison. What about Omni Graffle? Is that using Quartz 2D? That only seems to manage about 4 redraws a second if I hammer on the mouse button using the zoom tool. It's not keeping up with the mouse clicks for a relatively simple drawing. (The terrible horse drawing I posted earlier.) This suggests that it isn't going to be good enough for a smooth (as in full field rate video playback smooth) zooming.
  
  
  "The dispute is not (a) wether OS X has such a mechanism, it doesn't, but (b) wether or not such a retained vector-level representation is required for higher DPI displays."
  
  *cough* Actually originally it was whether "macos x already works like this" - hence the subject line of the thread.
  
  So do you agree that Mac OS X does not already work like this?
  
  But yes, the debate certainly seems to have shifted to whether its desirable to have a retained vector architecture. One of the subtopics is whether it's necessary for high-DPI.
  
  
  "Instead of showing (b), you keep showing (a) and apparently somehow thinking you're done."
  
  I'm trying to show both. If you'll agree that (a) is true (Mac OS X doesn't work like this), I'll stop banging on about that and concentrate on (b).
  
  
  "3. Drawing engine vs. UI resolution independence.
  
  "You rightly observe that having one doesn't imply the other. Then why on earth do you keep saying the opposite, that a particular type of API (retained vectors within the graphics API) is required?"
  
  Well, what I said was this: "Just because your drawing API is resolution independent doesn't mean your GUIs are."
  
  The corollary of that would be "If your drawing API is not resolution independent, this means your GUI won't be."
  
  I think both of those statements are true - do you disagree?
  
  The problem I'm perceiving with Quartz is that only part of it is resolution independent. (And as I've always said, this is just based on my own experiments, and what the documentation says.) Quartz 2D meets the minimum requirement - it is resolution independent. But it plugs into something which is not - the Quartz Compositor.
  
  I am hypothesizing that this is a problem, because it will cause UIs to be non-scalable in practice. Your earlier post where you showed a window in a PDF is consistent with my theory (but doesn't prove it) because it was just a bunch of bitmaps in a file.
  
  My thinking behind this theory is that unless *everything* that must be done to render a window is resolution independent, you don't have a completely resolution independent system. And given that (a) you can mix Quartz 2D, Open GL and QuickTime into a window, and (b) the only documented part of Apple's architecture capable of combining the output of these three things is the Quartz Compositor, and (c) the Quartz Compositor is pixel based, it seems like there's a problem. (And the PDFs we have both managed to produce so far seem to exhibit the problems you would expect to see if I'm right. As do on-screen whole-window scaling effects like Expose. Both of these seem to be using bitmap-based scaling in practice.)
  
  
  Do I think that retained vectors are absolutely the only solution for high-DPI? I will concede that it may not the only possible solution. I therefore recant my claim that "you absolutely need this for high-dpi displays to be supported automatically for all applications." Looking back at that, this was a blanket statement. I probably can't prove that other techniques do not exist. (Although I will maintain that *something* has to retain a non-bitmap representation somewhere. But I don't think that's controversial.)
  
  This doesn't alter the fact that I stand by my original claim:
  
  "as far as I know, Quartz doesn't have a good way of dealing with high-dpi displays"
  
  I have yet to see any concrete evidence that Quartz has solved this problem - I merely know that it has some of the necessary ingredients. If a resolution-independent drawing API was all you needed for resolution-independence, Windows would have had everything it needed for resolution-independence on the day that Windows NT 3.1 shipped - GDI32 is a resolution-independent API.
  
  And yet Windows has problems with resolution independence. The pixel-based composition is one of the causes of problems in Windows. So this is why I am worried about Quartz. It might have the ingredients for success, but it also appears to have the ingredients for failure.
  
  So when you said: "It seems you understand (and have understood) that Quartz 2D has no problem whatsoever producing higher resolution output and will therefore have no problem driving higher resolution displays."
  
  I don't really go along with everything after the "and will therefore".
  
  Looked at in isolation, Quartz 2D has everything it needs to drive a high resolution display. But then looked at in isolation Win32's GDI32 drawing API has everything it needs to drive a high resolution display.
  
  The problem in Windows was that GDI32 did not sit in isolation. Quartz 2D doesn't seem to either, according to any of Apple's architecture diagrams and that's what worries me.
  
  One of the specific problem in Windows was that a given window was formed by composing lots of individual windows together. It was this composition step that was an issue, because the compositor was not resolution independent.
  
  There a number of other problems, most problematic of which is that applications that didn't test at different resolutions would work just fine at their default resolution, but you'd get a bunch of off-by-one errors when changing resolution, even though they were theoretically doing the right thing. You can actually see exactly this kind of effect in the PDF I produced on OS X earlier - the middle of the button isn't quite the same height as the rest of the button. It looks OK at some sizes, but at others, it looks clearly wrong. The net result is that even when you do get things to scale, everything suddenly ends up looking really rough round the edges. This is a real problem - it means that even if you're hypothetically resolution-independent, things probably won't look right unless you've tested at multiple resolutions. Given that OS X does not, as far as I know, provide a means of changing the notional pixel resolution of the screen, presumably nobody is testing for this today. So the point at which we find out whether everything works in practice is if/when such a feature eventually gets added.
  
  If there's some resolution independence testing tool I've not come across, then I'll happily recant on this point too. Is there some test harness I can run my apps in to see how they'll get rendered on a high DPI display? I'm not happy trusting to faith that things will just work. (In my experience, this isn't true, even when you've tried to do everything in the right way. I've seen too much code end up drawing almost but not quite the right thing at scale factors other than the default.)
  
  Apparently lots of people don't even know that it's possible to change the notional pixel resolution settings on current versions of Windows, because nobody tests. You can do it, but so much software breaks it's barely worth it.
  
  
  But back to composition...
  
  If I understand you correctly, you're saying that each window in OS X is really one big Quartz 2D surface. If that really is the case, then the only remaining problem (other than the lack of a test harness to view your windows at non-default scales) is a cultural one: Apple set a lead by using bitmaps for all of their widgets, and most application developers have followed suit. But if a window really is one big 2D surface, that's a problem that has a technical solution, so it merely needs Apple to provide some leadership here.
  
  What I'm still having a hard time grasping is your explanation of how this 'one big Quartz 2D surface per window' thing squares with integrating other imagery sources. You say that QuickTime can be instructed to render to the appropriate resolution, but I'm still not quite clear on how the results get composited. Somehow the rasters generated by the QuickTime engine have to appear on the same screen as the rasters generated by the Quartz 2D engine. What's responsible for doing this? According to Apple's documentation, it's the compositor. Are you saying it's not?
  
  If it is the compositor, how is the QuickTime engine (or the OpenGL engine) supposed to find out about what scaling is supposed to be applied? Because the transformations are all part of Quartz 2D, they're not something the compositor understands, so there must presumably be some out-of-band mechanism to keep the scaling of the various different drawing engines you are using in sync. How is that done? Is this the responsibilty of applications? Also, the compositor appears to work in whole pixels. (Run Expose in slow mode - Shift-f9 or Shift-f10. Watch how it is unable to deal with windows which, according to their current scaling, should really have edges at sub-pixel locations. Put a window near the top left, and watch how its top position steps up one pixel at a time rather than moving smoothly like the rest of the screen.) So what happens when your transform model says that QuickTime should be rendering into something that's 24.3 pixels wide, and located at 100.2,80.8? It's going to have to bump it to some whole number of pixels location and size. Or can the QuickTime renderer deal with this subpixel stuff too?
  
  Also, out of interest, can the QuickTime renderer deal with other transformations like rotation and shearing? (And if not, what happens if you're using these in Quartz 2D in the same UI that you're using QuickTime in?)
  
  
  
  What my current thinking boils down to is this: I see a system with a combination of elements I have known to cause problems with scalability in the past, and this is why I suspect there will be problems, particularly as you can't test a whole application for resolution independence today. I cannot prove to you that there is definitely a problem. Indeed it is possible, if windows really are one great big Quartz 2D surface, that high-DPI really will just work. However, the fact that you can't test for high-DPI correctness today makes me suspect that a lot of applications will look a bit wonky in practice - I've found that in general, if you don't test there's a high probability of failure. I still maintain that vector retention is a better solution to the problem because it has better performance characteristics. (Remember, doubling the resolution will quadruple the number of pixels that need to be pushed. In Quartz 2D's case, those pixels need to be pushed across the bus from main memory into the GPU.)
  
  
  "You forgot a third one: hand-painted bitmaps used for the display but rendered through Quartz 2D. Again, how would those benefit from a retained representation?"
  
  I already brought this topic up actually. Earlier in the conversation I complained that "There seems to be a lack of leadership here from Apple - all of their graphical widgets appear to be bitmaps, not vector images." You've brought up the very issue I was referring to.
  
  Obviously hand-painted UI bitmaps are anathema to device independence. This is why Microsoft are pushing a very clear message that you should no longer do this, and it's why they are moving all their own widgets over to a vector form.
  
  It's very clear what The Right Thing to do here is. Microsoft have made it very clear that high-DPI is considered an important feature, and that vector imagery is the way to go. (This was repeated many times over at the conference where Avalon was launched. Microsoft really want people to be aware of this.)
  
  This is not a technical issue of course. You can use vector imagery today in OS X. It's just a leadership issue. (The 128x128 pixel standard for icons is OK so long as you think displays will never go above their current resolutions. But 200dpi displays have been around for a while now - a 300dpi display is a resonable possibility, and a desirable one at that - after all, it's still possible to make out the individual pixels on a 200dpi display. At 300 dpi, your 128x128 pixel image is now less than half an inch across. This constrains your design - I happen to rather like the cartoon-like vastness of the images I get today when Alt-Tab switching in OS X, but I'm not going to be able to have those on a 300dpi display without loss of image quality.)
  
  
  
  "...the apps with the highest performance requirements are video editing and 3D, and that both of these don't use Quartz 2D [...] Anyway: if both of these do not use Quartz 2D, how would speeding up Quartz 2D help them? Answer: it wouldn't."
  
  Am I right in saying that you're making the following unstated argument: only the two most demanding application areas on the Mac get high performance display piplines, so it must therefore be pointless speeding up anything other than the two applications with the highest demands.
  
  (I'm trying to understand why you said what you said. The only way I've been able to make sense of it so far is if you are taking that argument as being implicit.)
  
  If I've understood your argument correctly, my response would be that the one reason you don't see any graphically demanding applications using Quartz 2D is that Quartz 2D's architecture makes it impossible to write graphically demanding applications.
  
  You can't point to the dearth of graphically intensive applications available that use Quartz 2D and use that as evidence that there is no requirement for better performance. Couldn't it be that Quartz 2D's architecture is stifling innovation in this area?
  
  
  "As you yourself have pointed out, both speed and quality requirements are different for dynamic and static media"
  
  Actually I don't think I did say that. What I said is that the applications that require high quality don't use the Quartz 2D pipeline. So I don't regard what you said there are an accurate characterization of what I said.
  
  The point I was making was simply that the Quartz 2D pipeline is no good for high performance work. This is very different from saying that there is no need for a high performance 2D pipeline. I was just saying that Quartz doesn't have a high performance 2D pipeline.
  
  
  
  By the way, you've not replied to my comments on the relative abilities of Quartz 2D and GDI+. I asked if you could name anything Quartz 2D could do that GDI+ could not. (You claimed that GDI+ was not up to the level of Quartz, and I was asking you to substantiate that claim.) Did you forget? Or are you changing your position on this point?
• macos x already works like this
  2004-05-23 22:06:41  glenlow [View]
  
  
  "Can you show me an application that illustrates zooming in on vector imagery in OS X that does use Quartz then? I'd like to be able to see a proper side by side comparison. What about Omni Graffle? Is that using Quartz 2D? That only seems to manage about 4 redraws a second if I hammer on the mouse button using the zoom tool. It's not keeping up with the mouse clicks for a relatively simple drawing. (The terrible horse drawing I posted earlier.) This suggests that it isn't going to be good enough for a smooth (as in full field rate video playback smooth) zooming."
  
  Take a look at my Mac OS X port of Graphviz: http://www.pixelglow.com/graphviz/, a released and production-used app. In particular it uses pure PDF (via Quartz 2D) to draw the graphs, and specifically disables cached bitmaps. As a result rendering is smooth and fast enough to work with a dynamic, live zoom (a slider control).
  
  Try opening graphs with a lot of nodes e.g. awilliams.dot. You can fiddle around with the default colors to get alpha transparency happening and in 10.3 a drop shadow delineates the graph boundary, all within the Quartz 2D/Cocoa API.
  
  The graph PDF is a retained vector representation, of course. As you can see, the Quartz 2D pipeline is pretty good for high performance work!
  
  My main irritation with Quartz PDF is that it's hard to do selective edits of a PDF, since the Quartz model is more a "lay down paint" model like PostScript rather than a full object model.
  
  Cheers,
  Glen Low, Pixelglow Software
  www.pixelglow.com
• macos x already works like this
  2004-03-22 15:32:48  mweiher [View]
  
  
  
  

  1. Application that provides smooth zoom

  
  
  

  Just modify your NSImageView a tiny little bit. Add a class BetterNSImageView with an instance variable id pdfImageRep.

  
  
  

  To this add the following two methods:

  
  

  
  
  -(void)drawRect:(NSRect)r
  
  {
  
          if ( pdfImageRep ) {
  
                  [pdfImageRep drawInRect:r];
  
          } else {
  
                  [super drawRect:r];
  
          }
  
  }
  
  
  
  -(void)setImage:(NSImage*)anImage
  
  {
  
          [anImage setDataRetained:YES];
  
          pdfImageRep=[[[anImage representations]
  
              lastObject] retain];
  
          [super setImage:anImage];
  
  }
  
  

  
  
  

  (This code will leak the image-rep, but that's OK for us here)

  
  
  

  The clou here is drawing the PDF directly, instead of recaching an NSImage each time around. That makes things significantly faster. I haven't put in the code that takes care of keeping the aspect ratio the same, this doesn't affect the speed and is left as an exercise for the reader.

  
  
  

  This redraws interactively on my dual 1 GHz G4 and my 867 MHz 12" PowerBook, though it is a bit more sluggish on the PowerBook. Or, to put numbers on that, add the following trivial "benchmark" code:

  
  
  

  
  
  -(void)doScaleBench:sender
  
  {
  
  	int i;
  
  	NSWindow* win=[self window];
  
  	NSRect r=[win frame];
  
  	
  
  	NSLog(@"start %@",[NSDate date]);
  
  	for (i=0;i<10 ;i++) {
  
  		r.size.width++;
  
  		r.size.height++;
  
  		[win setFrame:r display:YES];
  
  	}
  
  	NSLog(@"stop %@",[NSDate date]);
  
  }
  
  

  
  
  

  Hook that up to a menu item in Interface Builder.

  
  
  

  On my 867MHz 12" PowerBook, that yields between 7.41 redraws/s for a complex PDF with a full page of text and 17.2 redraws a second for a simple PDF with a couple of shapes and a small amount of text. A full page of text being rescaled is very rough, because text is expensive to render and you aren't getting much use out of the text cache.

  
  
  

  On the 1 GHz G4, we get 12.5 and 31 redraws / second, respectively. A current model G5 should be more than twice as fast, and I wonder what will be available by the time Longhorn/Avalon ships. But it's always surprised me how much faster Safari is on my 12" PowerBook than the 1.4 GHz Dell I have at work. Apart from looking incomparably better...

  
  
  
  

  2. "wether mac os x has a [retained vector] mechanism like this"

  
  
  

  *double cough* This is from the start of my first post to this thread:

  
  
  

  "The fact is that Mac OS X already *does* work mostly like this, except for retained vectors"

  
  
  

  To which you replied:

  
  
  

  "But my point is that the retained vectors are crucial."

  
  
  

  'nuff said.

  
  
  

  3. Quartz compositor hypotheses

  
  
  

  Just complete bollocks. Completely unrelated. Apart from being untrue, this is still bitmaps being drawn with Quartz 2D, not stuff that is turning into bitmaps as a result of the Quartz compositor, which has nothing to do with that at all.

  
  
  

  4. OpenGL + QuickTime + Quartz

  
  
  

  5. Diffuse worries

  
  
  

  OK, so you had bad experiences with Windows drawing APIs that just didn't work as advertised. Yes, that is also what I heard. Especially the transition from the initial GDI to the one that supported a "Postscript-like" imaging model, with arbitrary affine transformations. I remember thinking to myself "hey, this does all I need", but then hearing that, in fact, it doesn't actually work. However, I never tried it myself, but you seem to corroborate this.

  
  
  

  This is also where I get confused as to which Windows API is which. I remember that the original GDI did not support arbitrary (affine) transformations, or bezier-based paths. Did it support path-based clipping? The API that followed was very closely modelled after the Postscript imaging model, but that was the one that didn't really work as advertised, at least initially. Was that GDI+? I seem to be getting conflicting messages on that. You seem to say that GDI+ came with XP, but that would mean there must have been an intermediate API. Was there some renaming in the meantime?

  
  
  

  Anyway, whatever bad experience you had with Windows APIS, the good news is: Quartz Just Works(tm), as did DPS. Your worries, as applicable as they are in the Windows world, simply do not apply here. I have lots of ideas why this is the case, but these aren't really that important. What is important is that it does just work.

  
  
  

  [Possible reasons: well-defined drawing model, with precisely specified rendering semantics; history of implementations that have always been and continue to be device-independent stretching back more than two decades; surrounding APIs that have traditionally always worked in conjunction with a fully device-independent imaging model and count on that working correctly; developers that have worked with a device-indepndent imaging model for more than a decade etc.]

  
  
  

  6. Concrete compositing worries

  
  
  

  Anyway, you do give one concrete reason why there were problems with Windows, which is that windows are composed of lots of smaller windows. This isn't the case with normal Quartz windows. It is only true (to a much lesser degree) when the Quartz compositor is used to combine other media-types.

  
  
  

  However, virtually all the concrete worries you have expressed about (sub-)pixel boundaries, arbitrary scaling and so forth already apply today in exactly the same way! So either everything is as broken as you think it will be in the future today, or it won't be broken in the future either.

  
  
  

  [Exposé as 'proof' of compositor limitations: how many more #$%!@# lame examples are you going to come up with?! Exposé is intended to give you a rough overview over your windows, not provide precise rendering. Also, if anything it is an example of limitations of GPU-based drawing...]

  
  
  

  7. One big Quartz 2D surface

  
  
  

  For most windows, that is exactly what it is. The only exceptions are those that explicitly include dynamic media such as QuickTime movies and OpenGL-rendered 3D graphics

  .
  
  

  And yes, of course it is the compositor that is placing those QuickTime streams inside Quartz windows. What else? I really understand where you're getting all this from. You seem to have a profound misunderstanding of how all this works somewhere, but I am at a bit of a loss where it is.

  
  
  
  

  8. "At 300 dpi, your 128x128 pixel image is now less than half an inch across."

  
  
  

  This makes me wonder wether you understand what device-independent graphics means at all. Probably not. The same image will be precisely the same size as it was before. It will also look (almost) exactly the same. It just won't improve as much as a vector drawing will. However, most of the time those 128 x 128 icons are drawn much smaller than 1:1, and at those sizes a higer DPI display will display those icons with much finer details than possible today.

  
  
  

  Incidentally, that applies to all the other bitmaps in the UI, for example the traffic-light close/minimize buttons. They will render just fine even at the higher resolution, they just won't benefit much from it. That is what device-independent rendering is all about. Of course, it will be trivial for Apple to provide higher-resolution bitmaps for those higher resolution displays, or vectorized versions if that is appropriate/desirable.

  
  
  
  

  9. Static vs. dynamic graphics pipelines (and retained vectors)

  
  
  

  '"As you yourself have pointed out, both speed and quality requirements are different for dynamic and static media" Actually I don't think I did say that.'

  
  
  

  Er, yes you did. Earlier. On bitmap-stretching used by, for example, the genie effect:

  
  
  

  "Quartz gets away with this for high-speed animations like minimization because they're over before you have a chance to see the imperfections. But it would not stand up to close sustained scrutiny. And a crummy image would completely defeat the purpose of a high-dpi display!"

  
  
  

  So, at high speeds, you can get a way with imperfections. At low speeds, or a static display, you can't.

  
  
  

  So once again, for 3D and video work, you have the fast 3D and video pipelines. For mostly static displays, you have the very high quality Quartz 2D pipeline (and you can mix in the other pipelines when necessary). Quartz 2D is plenty fast for 2D non-media, non-movie work, and offers precision/quality that other APIs can't match.

  
  
  

  Also, you still haven't explained how 2D retained vectors would improve movie playback or OpenGL based 3D work. Are we going to vectorize movies on the fly? Is OpenGL going to render to a 2D vector representation? I don't think so.

  
  
  

  10. "retained vectors" ... "may not the only possible solution" for "High DPI"

  
  
  

  Good, we are making progress, then. However, it turns out that wether vectors are retained or not is completely irrelevant to the High DPI "problem".

  
  
  

  Unless you are going to make the absolutely absurd claim that QuickTime movies are going to vectorized on the fly and OpenGL will render down to 2D vectors instead of bitmaps, you are still going to have to composite at least some bitmaps together when you join up these different pipelines, at some point, wether you have a retained vector representation or not. It only happens at a different point in the pipeline that is all, but the actual issues of compositing the bitmaps together remains exactly the same.

  
  
  

  The other issue is bitmaps shoved into the graphics pipeline at the application end. Again, this has nothing to do with wether you have the ability to retain (vector) representations inside your graphics subsystem (or on the graphics card for that matter) at all. It is *solely* a matter of the applications providing vector or bitmap data. So once again, the API makes no difference whatsoever

  .
  
  
• macos x already works like this
  2004-03-19 20:59:37  rosyna [View]
  
  
  I cannot seem to respond to Ian's reply that spawned this reply (server error). It might have been too long since the last reply was made on this thread so no one may ever see this post.
  
  Ian, your argument now seems to be that OS X windows aren't scalable (say to 2000%) without loss of quality. This is true of course because OS X uses bitmap images for GUIs. For the Close, Minimize, Zoom buttons, toolbar buttons (tic-tac), background patterns, and menu backgrounds bitmaps are used. This doesn't mean that Quartz 2d isn't scalable it means that OS X doesn't use PDF operations for widgets. There are many reasons for it. First, it'd be REALLY slow on any graphics card or CPU right now to have 400 vector objects on the screen at any given time. It'd also be much, much slower for machines with 233Mhz G3 processors that support OS X.
  
  You can't seriously be suggesting that Longhorn will be completely scalable with no lose. It'd mean that everything would be vectors. Widgets, title bar stuff, window backgrounds.
  
  Let's look at two different windows. We have this one from longhorn. Look at the metal lookalike. One very smooth image with no imperfections "patterns". Now let us look at this clock thing here. Notice the wallpaper. Not scalable. It is a bitmap. Look at the image on the clock window. Also not scalable. Now, the Finder. You cannot tell too much from the screenshot but that metal window has imperfections. It *looks* like metal. It is not flat. It cannot be represented as a vector image without 1000s (or more) of paths. No way a GPU is ever going to handle it. So basically as the amount of vectorness goes up, the amount of uniqueness decreases while the amount of "flatness" increases. This is how OS X would look if it was all vector.
  
  And for the TextEdit example. You seriously expect a GPU to handle typographic layout? You want them to look at the kerning of the fonts, ligatures, baseline spacing, direction of type, and word wrap? That's what you seem to be saying. It is Not and operation for video cards to do. Ever.
• macos x already works like this
  2004-03-21 12:50:03  Ian Griffiths | [View]
  
  
  "Ian, your argument now seems to be that OS X windows aren't scalable (say to 2000%) without loss of quality. This is true of course because OS X uses bitmap images for GUIs."
  
  That's not the whole problem. Even if you do provide scalable image data, Quartz 2D still rasterizes to the native display resolution, leaving the compositor to do bitmap scaling. This is evident in Expose - this is just scaling bitmaps. It is evident when you use the accessibility features of OS X to enlarge the display - again, it just scales up bitmaps. And if you look at the PDF I posted in response to mweiher's PDF, it appears that this remains true even when generating PDFs of a window. Only text remains scalable - everything else comes out in bitmap form even if the source data is in vector form.
  
  
  
  "It'd also be much, much slower for machines with 233Mhz G3 processors that support OS X.
  
  "You can't seriously be suggesting that Longhorn will be completely scalable with no lose. It'd mean that everything would be vectors. Widgets, title bar stuff, window backgrounds."
  
  That is indeed the plan. (Or at least it's what Microsoft is currently telling everyone is their plan. If you don't like that, tell them - the whole reason for the Longhorn preview was so people could see where Microsoft were going early enough to offer feedback before everything is set in stone.) Microsoft aren't planning to convert everyone over to Longhorn - its minimum system requirements will include a graphics spec which today counts as high end. (By the time Longhorn ships, such specs will be mainstream.) So yes, it would be absolutely unusable on old machines, which is why Microsoft have no intention of telling people they have to upgrade - they know it isn't going to work on old hardware. (Although they are apparently intending to support 3 levels of quality of user interface, so it will be able to support older hardware to some extent. But the design means that there will inevitably be limits to how old your machine can be and still have a hope of being useable.)
  
  This does mean of course that applications using Avalon won't run on old machines. This is by design - the whole point of this architecture is that it is intended to last as long as the previous graphics architecture. This inevitably means that old hardware will be left behind. The same thing happened with OS X remember - Apple eventually had to give refunds to people who had attempted to install OS X on systems that were too old. As Apple already discovered, if you want to move forward technically, you have to let go of the distant past. Microsoft already have a transitional API that will let you write for both, and which makes it relatively easy to make applications that can take advantage of Avalon if it's there, but still run if it isn't.
  
  Is it too soon to make such a radical change so dependent on high-performance GPUs? Well I guess that depends on (a) when you think Longhorn will ship, and (b) how soon you think developers will actually start moving over to Avalon. I think the answer to both is "Not any time soon". This stuff really is an awfully long way out - it probably won't become mainstream for another 5 years yet.
  
  (The only reason Microsoft are showing it this early is because their customers have asked them again and again over the years to provide more information on their long term plans.)
  
  
  "First, it'd be REALLY slow on any graphics card or CPU right now to have 400 vector objects on the screen at any given time."
  
  That flies in the face of experience with 3D software. Modern 3D games have far more than 400 vector objects on screen at any given time, and they routinely achieve frame rates of 60Hz. (For example, Quake 3 would have up to 6000 polygons of scenery actually getting rendered in any given frame. And that's after all the culling to make sure that it doesn't even bother trying to draw obscured polygons. So even if you make a conservative assumption that your drawing primitives are actually going to need be broken up into 10 individual operations on the actual GPU, you're still not getting all that ambitious by today's standards.)
  
  With modern GPUs, it turns out that processing large numbers of vectors is not only easily achievable, but likely to be faster than rasterizing in software and then copying the results over. (And also faster than just using bitmaps for everything.) The principal reasons for this are:
  
  

  
  
  • Drawing commands are usually smaller than bitmaps containing the results of these commands, and can therefore be moved around faster
  
  
  • GPUs have incredibly fast fill rates, so it's going to be substantially faster to get a GPU to do the drawing than it is to draw it all on the CPU and then copy the drawing over - regardless of whether you've got PCI express, AGPx4 or AGPx8, an interconnecting bus will never be able to approach the data transfer rates that the graphics subsystem achieves internally.
  
  

  
  
  The Avalon team has a target for supporting displays showing around 10,000 primitives. Given that they already have caching techniques (described in earlier comments in this thread) in place for retaining bitmaps on the graphics card for complex bits of the UI that don't change much, this should be easily achievable - it means that if a given piece of UI gets so complex that it really is faster to move over to bitmaps, they can do that dynamically. (And in practice, they won't have to redraw 10,000 primitives every time anything changes. But even if they did, this doesn't take all that long on a modern GPU.)
  
  
  "Look at the image on the clock window. Also not scalable."
  
  It's a photograph, so you wouldn't expect it to be. That's a piece of UI designed for showing photographs...
  
  But the clock itself IS scalable. It's one of the most immediately visible demonstrations in the Longhorn preview of the advantages of a vector-based UI. That clock can remain visible at all times on the sidebar, and if you resize the tile that contains it, the clock resizes with it. And because it's all vector-based, it looks perfect at any size. And it redraws at the full framerate of the monitor (at least it does on my system) so it looks gorgeously smooth as it does so. Moreover, it does so without significant CPU usage.
  
  The clock must have at least 130 vector elements - each minute has at least 2 (a mark and a shadow), and there are a bunch more for doing the hands, the bezel, and the reflection on the face. And remember, this is a developer preview of the OS - they've got an awful lot of work to do on performance, and even so it all works fast enough.
  
  That clock illustrates the kind of thing you can do with vector-based UI.
  
  (The other stuff you pointed to is all XP imagery. The Longhorn preview given out at the PDC didn't have the new shell - it had a slightly modified version of the old XP shell. This is why all of these icons are bitmap based.)
  
  
  "You seriously expect a GPU to handle typographic layout?"
  
  No, that's why I said "it's pretty unlikely that Avalon will hardware accelerate layout." You pretty much reversed what I actually said there.
• macos x already works like this
  2004-03-22 12:20:26  rosyna [View]
  
  
  You seem to be confusing Direct3d/OpenGL with vector image drawing. For OpenGL and the like (what a GPU accelerates) it is a polygon with no AA (until recently and even then FSAA is crap) with a bitmap texture in it. Have you ever seen any applications that use the video card to render text? It ain't pretty and isn't scalable at all.
  
  And I also produced a widow image (with no code) with an image well. SmileyWindow.pdf. I made the smiley image with Sketch then chose Save a Copy as From the file menu. Format was PDF. Made a new application nib in Interface Builder, added an image well (NSImageView) to the nib, dragged the PDF into the well, pressed command R (to run the nib) and printed the window. I'm not sure how you exported OmniGraffle's image though. Sketch is in /Developer/Examples/AppKit/Sketch/.
  
  " Quartz 2D still rasterizes to the native display resolution, leaving the compositor to do bitmap scaling. This is evident in Expose - this is just scaling bitmaps."
  
  Exposé is *only* doing scaling and repositioning, nothing else. It is not possible to do DPI-level scaling because applications that use QuickDraw would look hideous (they don't support DPI independence) and text may look really messed up as well (when using ATSUI, Text using QD will look screwed up.
  
  However, it is true there is a bug in OS X that causes the window server to not properly interpolate the scaled window image even if it is requested, works fine for subviews though. Also Exposé does it the way it does it because it is faster. OS X doesn't have to redraw a thing neither does the GPU (except it resizes a texture to a smaller size). If it was all vector (and retained), something would have to redraw everything.
  
  "That is indeed the plan. (Or at least it's what Microsoft is currently telling everyone is their plan. If you don't like that, tell them - the whole reason for the Longhorn preview was so people could see where Microsoft were going early enough to offer feedback before everything is set in stone.)"
  
  Then it will fail miserably unless common GPUs get massive 2d acceleration abilities by the time longhorn is released. And with it getting pushed back as much as it is, that might be possible.
  
  "That flies in the face of experience with 3D software. Modern 3D games have far more than 400 vector objects on screen at any given time, and they routinely achieve frame rates of 60Hz."
  
  We aren't talking about 3D operations but 2D operations. Even on the best video card, 2D objects in 3D games look like crap. You can clearly see it in FFXI (a game I sadly cannot put down). 2D and 3D operations are a different beast both logically and by implementation. Because of these differences, I've never gotten into OpenGL. However, I would gladly eat crow if you could show me a 2D OpenGL/Direct3D application that is both pretty, scalable, and fast.
  
  It's a photograph, so you wouldn't expect it to be. That's a piece of UI designed for showing photographs...
  
  And anything that uses bitmap images will look horrible when the DPI of the display goes above the DPI of the image. Especially if that app uses vector art everywhere else.
  
  No, that's why I said "it's pretty unlikely that Avalon will hardware accelerate layout." You pretty much reversed what I actually said there.
  
  I was referring to the comment "You don't consider typography a graphics operation?"
  
  Which I don't... cause it isn't. It requires a lot of math and a lot of cached file access. And I don't think it is wise to make the GPU the main processor of an OS which seems to be what people are ultimately suggesting as of late.
  
  As for the QuickTime conundrum, Quartz does create surfaces for different QT formats, OpenGL, and other things that can be accelerated if it can just composite them together. You can easily notice this if you rotate a window, the surface rotates out of sync with the window and due to a bug in Panther if you take a windowed screenshot of such a window it does not compile the surfaces so they all disappear.
  
  Yes, it is possible to rotate a bunch of windows in OS X at the same time with little to no CPU usage (if you don't code the rotate in a tight loop like I did, that is) if QE is enable. Turn it off and the computer becomes unusable.
  
  And can you make a screenshot of longhorn (any part) that is scalable?