Pixel abuse: the wonders of 3D graphics

Pixel abuse: the wonders of 3D graphics

It's all over the press, and a battle will soon be joined:

Desktops need 3D.

But what kind of 3D? The term is vague, would it be isometric 3D, Z-buffered vector 3D used to display 2D elements - as we can see in games -, or something hopefully more profound?

Now, frankly - it was time for a little innovation in the computer display area, and 3D may just be it. Starting with the Xerox Alto (and subsequent Apple Lisa) interface, computer users left the single-dimension of the command-line to attain 2D: mosaics, windows, icons, taskbars, and such interface pieces that we now take for granted. That was in 1973, and reached the public in 1983 (darn, Wikipedia can really make someone feel smart!).

To give a better idea of the lack of progress in this area, consider that the X window system (which just celebrated the first minor revision of its 7th major version) is now 22 years old. Of course, pseudo-3D has been creeping in for the past few years - stacked virtual desktops, 3D desktop switchers, motion capturing input devices...

Desktops themselves have seen little breakthroughs: the un-innovative Microsoft Windows 95 (and subsequent version) looked so much like 1986's GEM environment I wonder how they didn't get sued over it, Gnome is still mind-numbingly bare, KDE looks more like an overturned toolbox than a desktop, Enlightenment is getting less innovative (I preferred e16 to e17), and MacOS is a nice piece of wrapped candy. Yes, I tried to spare no one.

What do these interfaces have in common? I'll tell you what: you still need to open a terminal and a Command-Line Interface to run complex tasks efficiently in all these environments. You still only have a keyboard and a mouse to interface with your machine, and it is still using a matrix-based output.

On one side, we have Microsoft touting a brand-new OS with a 3D interface requiring state-of-the-art graphics capabilities to work transluscent window borders, stacked windows in the focus switcher, and transparent windows when you can figure out the keyboard shortcut. This is also what Windows users are supposed to be stuck with for the next five years at least, with no improvements to come - looks more like isometric 3D (it gives an idea of depth, but you can't make use of it).

On the other side, you have composite display managers: Xgl, Xegl, and all. It works more or less well depending on your hardware's driver capabilities, it blends more or less with existing desktop managers, adds this or that effect depending on whatever crazy idea crossed the mind of its developer and/or developer... That's better: the cube-shaped virtual desktop is really starting to look like 3D, mosaic effects give a live overlook on all your running applications while they run, and transparent windows add real depth to your desktop - this is like Z-buffered vector 3D (it allows some 3D use, provided you only have one finger).

Then comes the outsider:

For those of you to have seen the Movie "Minority Report", how many of you found him(or her)self drooling over Tom Cruise's workspace? You know, the thingy where you could grab windows, stack them, turn them with a wave of the hand, zoom on a video by pulling on it, put it to the side like swatting a fly, reach for a detail inside a picture, and all that, faster than a puppeteer on a caffeine high?

Now, maybe that's a bit much - we're no closer to manufacturing holographic screens than we are to mastering site to site teleports. Simple motion capture, though...

Why all this complete raving, you may ask? Just a looong introduction towards this little question:

If the keyboard was king of single-dimension interfaces, the mouse queen of 2D, why are we getting 3D without a tool suited for it?

The answer may very well be that until now, no one (maybe Nintendo did - the Wii looks like a step in that direction at least) has even considered to link new display methods with new input methods...

If the Battle for 3D is indeed on, free software may win it not by working better or providing more useful (or useless) effects, but it may be by offering the first large scale development of a complete 3D desktop solution: a 3D display, with 3D object, and a 3D-accessed interface...



R0nin's picture
Submitted by R0nin on

> Desktops need 3D.

Ok, I'll play devils advocate on that one... Do they really? From what I've seen, everyone would like things to run faster, but most users (excluding gamers) Don't need, or even want much more than 2D. One might point out that whatever wizardry you do, you are really dealing with a 2D screen surface, so you can only create the 'illusion' of 3D. When we get that true holographic environment, then you can talk about 3D. Until then, I'll take my 'overturned toolbox' as preferable to whatever else I've seen.

I do like your comment on interface devices. There are days when I think that Bell Labs stuck us with the absolute worst solution. That's certainly an area that could use a rethink, and any attempt at future 3D will require it.

It would seem to be Gamers that are driving this thing, and while it may be important to them, 3D emulation currently adds little to most tasks or apps on a computer. While I agree that Desktop design is pretty bad, I just don't see that 3D is the answer to that bad design.

Mauro Bieg's picture
Submitted by Mauro Bieg on

You're absolutely right! Free Software could win battles over new technologies by simply being first.
As a matter of fact, I was lately thinking the same idea, just that I thought of innovative touchscreen-GUIs instead of 3D.

but it may be by offering the first large scale development of a complete 3D desktop solution: a 3D display, with 3D object, and a 3D-accessed interface...
Sounds great, but what is a 3D display if not a holographic screen about which you said that we're no closer to manufacturing holographic screens than we are to mastering site to site teleports. However, if we could provide a 3D input-device (as the Nintendo Wii's one) and a intuitive 3D-accessed interface in which you could navigate easily, this would be great. Somehow I've the feeling, that it makes only (or at least much more) sense to navigate with a 3D input-device in a 3D virtual world, not in a 2D virtual world. But the trouble with 3D objects (especially on 2D-screens) is always that the human eyes, seeing (nearly) only 2D, can't see the object from all sides at the same time. The switch from the command-line to the GUI made better use of the same 2D-screen and navigation got easier. But the switch from 2D to 3D could make things more difficult, considering that we're only seeing 2D. Therefore I think that a 3D interface for simple tasks is overkill.
Why a 3D environment on a 2D-screen if you just want to write a text, hear some music, or edit an image? I think for such things, touchscreens could improve the user experience in a much greater way. Pointing on an icon on the screen is much more intuitive than moving the mouse until the cursor is on the right position and than double-click with a mouse button. Maybe we have to wait until touchscreens that allow multiple touchs on the screen at the same time come to market. This video looks quite promising: Multi-Touch Interaction Research
But sooner or later there will come a good touchscreen GUI, not like current 'solutions' as the 'Microsoft Windows XP Tablet PC Edition' which are simply the old familiar mouse-GUI from the 1980s with tabs and windows, displayed on a touchscreen. I hope that this first innovative GUI will be Free Software and run (exclusively) on GNU/Linux.

However, I completely agree that there could be a battle over 3D. On more complex tasks as heavy multi-tasking, 3D-animation (Blender) and naturally games, a 3D input-device combined with a good GUI might work very well and could be a breakthrough for Free Software and especially GNU/Linux. But i think we would have to pay attention and spend some money for good advertising. I mean, Apple was also first with the GUI, but Microsoft simply copied it and combined with an evil business strategy began to dominate the PC-market. It would be a pity if the same thing happened to the 3D interface and GNU/Linux. Also considering Terry Hancock's blog post Is Microsoft bracketed by GNU/Linux?, an idea might be to run GNU/Linux with this (to be invented) 3D user-interface on the Nintendo Wii or the PS3, both featuring movement-sensitive controllers.

Let's see what the future brings us! :-) And let's hope for the best for Free Software.

Terry Hancock's picture

The existing 2D "desktop" metaphor is not likey to be improved much by 3D -- certainly not 3D graphics alone.

But there is probably a more interesting metaphor to be had. That won't really happen without a lot of experimenation, and it'll probably be "gamers" who will have enough interest in the subject to do the experimentation.

Without major improvements in input devices, though, it's likely that it won't get too far. Nevertheless the mouse, for all its flaws, is cheap. In order to become a ubiquitous successful platform, a prospective 3D environment is going to need a better metaphor and a better interaction model, before it takes over. Graphics is not enough.

nachokb's picture
Submitted by nachokb on

Please don't ruin current efforts to *NOT* convert the desktop metaphor to 3D.
Since a little more than 10 yrs ago people have been trying to "make it 3D" (by making your desktop a room... something like Quake, using badly implemented extensions to Win32's shells, X DEs, etc...). Perhaps the first ones were awfully implemented (in those years, there weren't any hw accel). Then, slowly, some people started thinking about using 3D video cards (which are much better accelerated than their 2D counterparts) to just give a touch to a 2D desktop (a long needed touch). Many projects were ruined when some other people read "desktop" and "3D" in the same phrase. So, the first one who got to deploy a 3D accelerated 2D desktop (alright, just composited, but still) without being disturbed by the "3D desktop" crowd was Apple, circa 1999 (I think). That's to their credit. Others followed, some understood (XGL, Vista sorta) some didn't (Project Looking Glass... oh god why on Earth would I want windows to rotate on a Z axis???).
I admit I liked the XGL rotating-cube desktop-switcher (though I really dislike the idea of multiple desktops from a usability standpoint). But that's as far as I can go. In summary: N O 3 D P L E A S E ! ! ! ! (or, please start your own 3D-desktop project without disturbing current efforts to not convert the desktop to 3D).



Terry Hancock's picture

With free software projects, these things are done by people who are interested in them. So just because some of us want 'true 3D metaphors' doesn't mean the rest of us have to convert to that. Only if something really compelling comes along will that happen, and so there's no reason to oppose someone trying. It's not a like a company that provides only one interface, and you have to learn it.

There are actually dozens of desktop interfaces for GNU/Linux (KDE and Gnome are just the more conventional front-runners -- there are plenty of bare window manager environments, and some things like project looking-glass (which intrigues me enough that I'm going to have to try it out on a spare machine some time).

As for multiple desktops -- I have 20 right now, and I'm P.O.'d that KDE won't let me have more than that! Big, big, *big* timesaver for me. I use them to sort incoming information when doing web research, as well as to work on multiple projects simultaneously (which can save a lot of time when they involve long builds or downloads, for example).

So clearly, people's needs differ. That's why it's good that there doesn't have to be just one way to do it.

jackbnymbl's picture
Submitted by jackbnymbl on

Displays that produce true 3d, although not exactly like the hologram you described, are not far away, and some already exist. But what about an even simpler (if maybe not cost effective) way. How about a bunch of Liquid Crystal Dispays that can turn each pixel transparent or opaque individually in a row, and then a projector projects color onto each one. Whats more, unlike the laptop mentioned in the popsci article, if you move, the the image changes as if it were real. Not that any of these, except the laptop (which I've heard is not the best thing ever, it can make your eyes hurt, and only works if your in exactly the right position) are even close to being available to the public at a reasonable price, but they are interesting ideas, and, while not holograms, except for one, they may be able to serve the purpose of one. I would like to see a 3d interface, but one that makes use of a computer faster and more efficient (for instance, the program tactile 3d, while a neat idea, doesnt nessicerely work any better that the standard 2d equivelant).

Now that are problem with screens is solved, its time to start working on teleporters, which unfortunately for us, because of the heisenberg uncertainty principle, may be impossible, or at least very difficult, to create.

nachokb's picture
Submitted by nachokb on

Problem is, over the years, many projects (Free and otherwise) tried to bring 3D to the desktop. But it wasn't until one company (developing a *proprietary* product, damn it!) stood firmly behind a "no 3D" sign that using 3D hw to render the desktop became a reality. And today, we are all going toward that direction (nobody knows what happened to 3ddesktop -- at least that's the deb package's name -- or so may others even I tried on Windows in the mid to late 90s). Don't get me started with Project Looking Glass (not even they know what it will be... 2D on 3D hardware or 3D desktop, if it ever gets released).
Of course there are many DEs out there, which got started just because people wanted to start them. But my point is: in the last 10 yrs every single 3D desktop project have failed (PLG seems to be the next one). A few (before Apple) had the idea of using 3D hardware for drawing plain-old, boring 2D. And this is where I take issue, they never got anywhere because of all the noise generated when people read "3D hardware" and immediately thought "3D desktop", not realising that that "2" in "2D desktop" wasn't a typo.
In the end, if you want to start a 3D desktop project, do it. If you want to fork XGL, AIGLX or whatever to create your 3D desktop project, by all means, do it. But don't derail those projects, please.

About virtual desktops, I didn't explain myself: I find them useful. I use it. Yet they are an awful hack, usability-wise. I gues the principle behind it is: hierarchy. A linux user has to deal with virtual terminals, each having virtual desktops, each having multiple windows, each having tabs (which, in the case of Eclipse, Azureus, and in general SWT apps it seems, contains more tabs inside -- with each level looking and behaving in slightly different ways). I can't imagine my grandma building a mental tree in her head!! We must flatten things for users who don't want to spend their time even knowing what a virtual desktop is. That's (one of the reasons) why tags are better than folders for organizing people's stuff. I remember someone complaining about tabs in Safari because they didn't appear in Exposé (the same could be said with Windows's taskbar).

We tend to think that people are ok with tree-like structuring of data (after all, that's really close to our mental model of many things). But people don't even realise that a text document should strictly follow a tree-like structure (they still use Microsoft Word, a fashionable typewriter... indeed OOo Writer is the same thing, and I don't really know of a word-proceesor-like app which just lets you type your content, and then think of style, without any "Bold" button around -- I tried things surrounding TeX, like LyX, LaTeX, but they are way too geeky).


Anonymous visitor's picture
Submitted by Anonymous visitor (not verified) on

To come across with the conception of how this works and or how it really works would be so helpful in the beginning stages of Invention. Some people could not even grasp the idea but if explanation of how it would work, what it would take, beginning with a hologram, say a large cube or sphere with drawers, spinning. The spinning cause by the ability to interact with the hologram by spinning the item with your fingers, slowing or stopping the item, object, starting or speeding up the object
understand a hyper sphere from the fourth dimension as a sequence of spheres, from this statement in understanding we may graduate to the third degree, so if you understand can you simply explain in lay mans terms so an inventor may begin one of the most loved discoveries of all times.

"Interactive Holographic Wedlock".

Quote from above on this page:-)(-:Then comes the outsider:

For those of you to have seen the Movie "Minority Report", how many of you found him(or her)self drooling over Tom Cruise's work space? You know, the thing where you could grab windows, stack them, turn them with a wave of the hand, zoom on a video by pulling on it, put it to the side like swatting a fly, reach for a detail inside a picture, and all that, faster than a puppeteer on a caffeine high?

Now, maybe that's a bit much - we're no closer to manufacturing holographic screens than we are to mastering site to site teleports. Simple motion capture, though...

Say some one begins with " simple motion capture' after the capture we set free the item or object and help it along to new found freedoms and the the adventure begins.

So does someone want to begin, simple explanation? by doing so, you may create a vaccumm of dreamers, inventors, explorers, freelancing their way, our way into discovery holographic style.

"the hologram with physical interaction and dialog or bust!"

Anonymous visitor's picture
Submitted by Anonymous visitor (not verified) on

As I watched minority report, the coolness of the technology seemed more visually stunning than useful. Movnig windows and the sheer managability of such a workspace seems to mimic more the typical desktop analogus of the real world than be prone towards the real requirements of such a time. That system, as depicted in the film, seems rather obtuse and malformed. Unfortunately, we are far from such a system, let alone a system that could potentially be, useful, without relyin on real world "analogies".

Mitch Meyran's picture

Well, train people into thinking deeper than 2D, wider than 3D, ascend them through 4D thinking... Then maybe you'll get interfaces more complex than real world analogies.

However, the human beast is still terribly set in the real world. Have a go at Ghost in the Shell-styled neural interfaces, maybe.
A computer is like air conditioning: it becomes useless when you open windows.

Author information

Mitch Meyran's picture


Have you ever fixed a computer with a hammer, glue and a soldering iron? Why not? It's fun!