Women in Technology

Since I replied to every point that you raised, and explained which parts I was finding hard to understand or impossible to resolve with experimental evidence, that seems like a harsh comment. Particularly as I have been trying out code samples on both Mac OS X and the Longhorn preview to try and make sure that all my statements were based on real observations, rather than conjecture.

"Quartz has a device independent imaging model, implemented by the Quartz 2D renderer and based on the Postscript/PDF imaging model."

I've read this whole section of yours again, and nothing you say contradicts my current understanding. You've gone into a detailed explanation of a bunch of stuff I already knew... I don't disagree with anything you said about the relationship between Quartz 2D, PDF/poscript drawing primitives and the like. Neither does it have any bearing on what I'm trying to explain.

So I'm evidently not making my point clearly enough! But it occurs to me that I could express my view of what I think the problem is in a different way, because then it might be more clear what I'm on about. Here, in a nutshell, is the problem:

Just because your drawing API is resolution independent doesn't mean your GUIs are.

Why on earth would I say that? Mainly because of the extensive experience I've had over the last 15 years with three different drawing APIs that were all resolution independent, two of which were modelled closely on the drawing facilities of Postscript, and none of which yielded resolution-independent APIs.

The APIs in question are:

GDI32, the original Win32 drawing API
GDI+, the new Win32 drawing API
DrawingModule, the vector drawing section of the long-since-defunct RISC OS

GDI32 is and always has been resolution independent - not only does it support a variety of drawing units, it also supports arbitrary transformations - all drawing can be put through a world transform matrix that allows all the normal affine transforms. The only APIs that deal in actual pixels are the bitmap APIs.

GDI+ is the same. The main differences are that is supports a wider range of drawing primitives, and it also supports anti-aliasing and alpha blending.

The (rather obscure) RISC OS vector drawing facilities are pretty similar again - a set of vector drawing facilities providing the same basic set of primitives as Postscript, but because it's much older (introduced in 1987) it doesn't have any of the new stuff that GDI+ has like alpha blending.

All of these can perform the trick you showed with your postscript sample - you can stream out a series of drawing commands into a file, and then play them back at any resolution with no loss of precision. (In Win32 you do this with EMF files, GDI+ uses EMF+ files, and RISC OS had, if memory serves, what were called !Draw files.)

And yet none of these could scale the entire UI. They could all scale individual fragments of drawing, just like you showed, but that's the easy bit. The hard part is scaling the entire UI.

Why is that hard? Well, the key to this was revealed when you said this:

"Whatever the reason, the Quartz compositor is (essentially) just for handling what you would refer to as "top level" windows and how they interact.

"This has *nothing* to do with an applications drawing objects into its window(s), wether with transparency or not. That is all handled by the Quartz 2D rasterizer, the drawing engine if you will."

But you can put an OpenGL surface in a window. You can put a QuickTime surface in a window. These can both coexist in the same window as Quartz 2D stuff, and yet neither of these things are drawn by the Quartz 2D rasterizer are they?

So what you're saying doesn't appear to be consistent with reality - how can the window be built entirely by Quartz 2D when it contains non-Quartz-2D elements?

It is clear that a window in Mac OS X can be composed from multiple elements, not all of which are Quartz 2D elements. So what you say here cannot possibly be true, can it? Something must be composing the 2D stuff with the OpenGL stuff and the QuickTime stuff. If it's not the Quartz Compositor, then what is it? (And according to all of Apple's diagrams, it is the Quartz compositor. But you're saying this is not so. What do you think does this composition?)

If the inside of every window was a purely Quartz 2D world (which appears to be what you're saying, even though firing up the QuickTime player will disprove it pretty quickly) then yes, we would have instant scalability. And in the same way, if the inside of a window in a Windows application is a pure GDI32 world or a GDI+ world, then again you have full scalability.

But certainly in Windows, the problem is that you don't have just one GDI+ surface or one GDI32 context for a whole window. What you have is a bunch of little drawing areas composed together. And where Windows and RISC OS all fell short is that the technology that composed these windows together wasn't resolution independent.

You're telling me that Quartz doesn't suffer from this because you believe the compositor is irrelevant. So here's the thing I don't understand: if the compositor is irrelevant, how do QuickTime and OpenGL fit into all this? Are you saying that if I put down an NSMovieView in a window, it's not actually QuickTime that renders it, but the Quartz 2D engine? I'm guessing you won't claim that, in which case, how can these things end up side by side in one window without getting the compositor involved?

"I have to admit I am somewhat flabbergasted by your screen-grab example and your mention of PDFs coming out of the compositor. I can only assume you were hurried and didn't think your response through."

I was trying to see if I could find some concrete evidence to prove your claim that the UI is scalable to be correct. (Actually I first tried this experiment ages ago, back when I believed that the UI was scalable.)

Here's the thinking behind the experiment. There doesn't appear to be any way of scaling windows up on screen built in to OS X. (And the Expose stuff that scales things down *definitely* does bitmap scaling, not vector scaling.) So I was looking for some way of obtaining the contents of a window in scalable form.

PDFs are, as you say, normally scalable to very high resolutions, so given that the built-in screen grab shortcut generates a PDF, then surely if the whole UI is scalable, there's no good reason for that PDF not to be a scalable one. It should be possible for OS X to generate a PDF that contains a scalable version of what's in the window, if the whole UI really is scalable - it should be trivial in fact.

But as it happens, that's not what you get - it's just a bitmap. (I don't really see why they put it in a PDF if all they're going to produce is a bitmap...)

While that's disappointing, I'm aware that it's also inconclusive. If the PDF had been a proper vector PDF, then that would have proved your argument conclusively - it would have demonstrated the scalability of the UI by showing a scalable representation of the UI. However, the fact that it just produced a bitmap is odd, but inconclusive. (Particularly when it does this even for individual windows. I could see how a full screen grab would be problematic, as it has to show the drop shadows, but why would it need to use a bitmap for a single window? Unless perhaps it uses bitmap composition within windows too...)

The reason I described what I had done despite the inconclusive results was to illustrate the kind of thing I was hoping to see - a complete scalable representation of a window. I was hoping you'd show me something similar, but which actually works.

I understand that Quartz 2D isn't using PDF from end to end, and is instead using a repertoire of drawing primitives very similar to what PDF offers. (I've been using drawing APIs that work that way for 15 years now, so I've had a while to get used to the idea!) But that's not a reason for it *not* to generate resolution independent output from a screen grab, if, as you're claiming, the UI really is scalable. I know that the vector handling is not as simple as the screen being one big PDF, but nonetheless, it could just convert from its internal representation back to PDF at the last minute, as long as its internal representation really is completely vector-based. (After all, it's perfectly capable of letting you print to PDF. So the fact that it can't do the same with a window is suspicious.)

So the screen grab experiement is a failure. I mentioned it because I was hoping you were going to show me some technique for getting a scalable representation of a window.

Instead you showed me a little bit of postscript.

In other words you showed me the easy bit - scaling a bit of drawing, rather than scaling the whole UI. The three other drawing technologies I mentioned can all do that too. Even Win16 could do that with the WMF file format. (10 years ago I wrote a Windows program that used WMFs for all of its internal icons precisely because I wanted to be able to scale the drawings. And it works just fine. But I couldn't scale the rest of the UI.) The support for individual snippets of fully scalable drawing either held in internal data structures or serialized out to some stream has been in Windows for a decade and a half, and yet Windows does not have a scalable UI.

This is why I was trying to see if there was some way of getting a scalable representation of a whole window - that's the tricky part, and is the acid test of genuine UI resolution independence; rendering snippets proves nothing. The whole UI is tricky because it might not just contain a single Quartz 2D surface. It could easily contain other surfaces. This is precisely where Windows falls over. It was precisely where RISC OS fell over. So it seems likely that Mac OS X will fall over here too, since it really isn't significantly different from GDI+. (Despite what you appear to think GDI+ is. But I'll come back to that later.)

As I said before, just because your drawing API is scalable doesn't mean your UI will be, unless your UI uses nothing but your scalable drawing API. And Quartz supports multiple drawing APIs within a single window - that's the problem. (And because it has to support this, the support impinges on the design even when you're not using the functionality.)

I challenge you to show me an example demonstrating a whole *window* in a scalable form, toolbars title bars and all. I suspect it cannot be done. If you can show me an example of this, then great! I'm happy to believe evidence, and it will reverse the disappointment I expressed earlier int his thread. All the evidence I've seen from the experiments I've tried seem to indicate that it cannot. (And I've tried lots of things to see if it can be done - I initially just assumed that it could be done when I first got my Mac, because my understanding of how Quartz worked was, well, pretty much what you've described. I've since discovered that my understanding seems to have been a little oversimplified, and that it apparently can't do all the things I thought and you still do think it could do.)

"doing your own redrawing"

"Once again, you put this point under "it makes life easier for the developer:". "

That's one of the points, although not the only advantage. The other big advantage is perf - if the application is required to respond to redraw requests, it forces everything the app redraws to go across a bus to the graphics card in some form, and that bus is a major bottleneck.

But I did also make claims on easy of programming. As an example of how retained UI makes things easier, do you think that HTML pages would be as easy to author if you had to handle repaint messages, rather than simply declaring what you want on screen? I know loads of non-technical people who are quite capable of creating web pages. I believe there is a gulf between this level of ease of use and the ease of use of handling repaint requests from the OS. Do you really disagree with that?

"At some point, the developer's code has to somehow draw/paint/define the display. Agreed?"

Actually no - read my earlier article in this series on XAML for details. Or indeed learn about HTML.

(This is a complete new subject area by the way, and I know that it's kind of irritating when someone starts talking about something else in the middle of an argument, but you made that statement, and ask me if I agreed. I disagree strongly, and now feel compelled to explain why, so I apologise in advance for talking about a new topic. You did ask.)

A declarative UI model mandates a retained model, although a retained model does not require a declarative UI. So just to be clear, you can use either declarative+retained or imperative+retained in Avalon.

If you employ a declarative UI model, there is absolutely no need for developers to get involved with redrawing. They can just create the display with so much markup.

You shouldn't really be surprised at that, because that's not so very different from how you build UIs in OS X. The Interface Builder uses drag and drop while today, Microsoft don't have an interactive Avalon GUI builder ready, but that will come. And while Avalon uses markup, Interface Builder generates serialized objects, if I understand correctly. But it's still a case of telling the system "One of these here, one of those there" and letting it handle the detail without having to write any code to manage the UI. Where Mac OS X falls short is that you can only assemble existing components this way - it doesn't let you handle drawing primitives in this way, forcing you instead to write code to generate those primitives.

In my opinion, that's bad for ease of use.

It's also ridiculously easy for code to make use of stuff defined in the markup, so don't fall into the trap of thinking that this is only any use for static UIs. It's not so different from the HTML DOM (except you're not obliged to use a scripting language - any .NET-capable language will do) - everything you define in markup is an object, so you can write code to go and twiddle with it at runtime.

I wrote an example showing how to draw 'rubber band' outline selections without having to write a line of drawing code that illustrates both how you can use markup instead of code to create a UI, and also illustrates how that UI can still be dynamic. It's in the second half of this blog entry. I suggest you take a look - given the blanket statement you made, it sounds like you've not even considered this style of programming before. It might help you understand where I'm coming from. (Again I'd skip over the first part - that's mostly for the benefit of Windows developers, and will be old hat for anyone familiar with Quartz 2D.)

So no, I strongly disagree with your thesis that drawing is necessarily the programmer's job. (And since your thesis was the underpinning of everything that followed that section of your post, there's not a whole lot else I can say in reply. Your axioms are wrong, so the rest of the argument isn't useful.)

"I hope I have managed to communicate more effectively why the problems that you think exist ( "high dpi" , "more programmer work") actually do not."

Well... you've successfully explained how I used to think Quartz 2D works. But try as I might I can't find any way of proving that a window can necessarily be scaled, and I hope I've presented some issues that show that some kind of compositor that is not part of Quartz 2D is necessarily involved. So I now hope you can refute this with a simple example. I'm afraid that I'm the kind of person who prefers proof, so if you can't show me an example that demonstrates the scalability of an entire window (rather than just an isolated piece of graphics) you're going to have a hard time convincing me.

GDI+ vs. Quartz

"You keep claiming that GDI+ is somehow up to the level of Quartz. This is simply not the case. GDI+ is, from an imaging model point of view, much more like DisplayPostscript (around 1989)."

Could you be specific there? As far as I know there's nothing Quartz 2D can do that GDI+ can't. And GDI+ is way ahead of circa 1989 DPS - there's plenty GDI+ can do that display postscript can't. (You gave the example of transparency handling and how display postscript uses the painters algorithm. That's what GDI32 uses too. But it's not what GDI+ uses - you can specify alpha for any drawing operation in GDI+, just like you can in Quartz 2D.)

Moreover, until Jaguar came along, GDI+ was able to do things Quartz 2D couldn't. Quartz 2D didn't original support gradient fills, for example, but these have been in GDI+ from the start.

Are you sure you're not confusing GDI+ with GDI32? Your description characterises GDI32 perfectly, but it's an inaccurate description of GDI+.

On profiling

"What the profile showed that most of the time was *not* spent rendering graphics. It was spent on the *application* side, not in the graphics subsystem. It was spend recalculating the line-layout, doing typography, not rendering it"

You don't consider line layout and typography to be graphics operations?

To be honest, you've not given enough detail for me to be able to draw any conclusions from your profiling. Any chance you could post what you actually found somewhere? This forum doesn't really make it easy to provide enough detail. For example, it now seems like you're saying the complete opposite of what you said last time about the profiling results. (I'm sure that's not what you thought you said, but that's honestly how it came across to me.)

"If we could drop in Avalon and Avalon were infinitely fast, it would hardly speed this operation up at all."

Layout and typography support is all part of what Avalon supplies, so that's not in fact correct. If Avalon were infinitely fast, it would make this operation infinitely fast.

Of course that has nothing to do with acceleration, since it's pretty unlikely that Avalon will hardware accelerate layout. I'm pretty amazed that it really is spending this much time in layout code, which is why I'm intrigued to see your actual profiling results. From what you've told me so far, the most likely explanation would seem to be that the layout code is a bit crappy - maybe you've just found a weak spot of Quartz 2D. However, I don't think that's the whole story - resize performance is pretty ropey on *all* windows on all Macs I've ever used (all the ones using OS X at least). Even on a little program that draws nothing but a single ellipse in a resizable view, and I'm pretty sure that program isn't spending all its time laying out text because there was no text...

So if you can share it I'd love to see your profiling setup and the data you generated.

"And of course, if you actually do junk the entire retained model each time you update the application model, then you are not going to gain any performance, because you are actually doing *more* work by building up this model that you then promptly destroy. Trust me, this has been hashed out before..."

But that's exactly the model you've been saying is so fine in Quartz 2D! There you absolutely have to reconstruct the entire model every time you get sent a redraw request.

Also, you're ignoring what I said about the problems of a software 2D engine (regardless of whether drawings are retained) - the gap between bus transfer speeds and CPU speeds gets ever higher, so any model that rasterizes on the CPU and then transfers the results across a bus to the graphics card is going to find that transfer becoming a more and more glaring bottleneck.

It has indeed been hashed out before - the games industry has been hashing this out for years now, the introduction of GPU-side vertex handling indicates that they've all settled on a retained model, and they long since settled on moving rasterization into the GPU. This is the prevailing approaches because it's the only one that scales to the complexity and speed they need. So it's been hashed out before, the end-to-end-vectors model won long ago, and the consensus amongst graphics card manufacturers seems to be that vector retention model is also the way to go!

"Summary:"

"- The immediate-drawing model of Quartz 2D does not have the "problems" you think you identified (inherently incapable of handling higher res displays, more work for the developer)"

Until you produce convincing evidence, I have to believe what the Apple docs and my experiments tell me. Rendering snippets is not convincing, because even 16-bit Windows can do that in a resolution independent way. Show me a bitmap containing a whole window at double the resolution (or a PDF, or a postscript file - whatever is easiest), and I'll believe you. My challenge to you is that I don't believe this can be done without bitmap scaling.

"- Quartz 2D renders vector graphics with correct transparency effects today, at any resolution desired (subject to implementation limits)."

Also true of GDI+. None of which alters the fact that GDI+ hasn't solved Windows' lack of resolution independence.

"You do not have to invoke the Quartz compositor to do this, the Quartz compositor is used for different tasks. (The implementation probably uses compositing operators to achieve these effects, this is hidden)."

Of course it uses compositing operators to achieve this - otherwise you'd only ever see the last drawing primitive you drew! But of course it's a different code path from the Quartz Compositor, because Quartz 2D is all software.

"- Drawing performance is not the limiting factor for "normal" applications, even with during highly interactive and graphics-intensive operations."

The two most highly graphics-intensive kinds of applications around today are video editing and 3D work. Neither of these use the Quartz 2D pipeline.

It's true that for static form-like displays, Quartz 2D performance is adequate. But it doesn't take long to hit its limits if you try to do anything ambitious. If you have access to a box running Longhorn (and make sure it's one with a supported graphics card), try running this extremely simple drawing demo. Compare the smoothness of response with Adobe Illustrator on a Mac. (Illustrator's pretty slow even with very simple drawings.) Particularly notice how you can zoom in and out smoothly in realtime with a drag operation. Compare this to how Illustrator only works in chunks and takes about half a second to change zoom factor.

(Just to reiterate, you will need a PC with a suitable graphics card. Because the Longhorn Developer Preview is a very early release, it supports only a small set of graphics cards. If it doesn't know your graphics card and therefore drops down to software rendering, the results will be rather disappointing. Although it's a pretty good example of why you don't want software rasterization...)

I believe this demonstrates that a faster drawing engine enables styles of user interaction that simply weren't possible before. You say that drawing performance is not a limiting factor. I think that its an ubiquitous constraint we all learnt to live with so long ago that we work within its limits without even thinking about it. Liberation from these restrictions gives us the opportunity to enhance usability no end. It's not about making existing things faster, it's about enabling things that were previously impossible.