Linux performance: is Linux becoming just too slow and bloated?

Linux performance: is Linux becoming just too slow and bloated?


This is an aspect of FOSS that is regaining some measure of interest: for years, it was considered that writing production-ready FOSS meant lean and mean software. However, recent events have shown that, in the case of the Linux kernel, this is no longer exactly true: performance is dropping slowly yet steadily.

How come?

Bloat

The Linux kernel is monolithic: it means that every hardware driver runs in kernel memory space. Every time you add a driver, you add stuff to the kernel. Moreover, for performance reasons, several user-space elements make their way into the kernel and increase its bloat even further.

Moreover, Linux has historically been developed for x86 platforms, and is tightly linked to the architecture, going as far as using hardware interrupts in code (essentially, mixing x86 assembly in the much more generic and portable C code).

On top of that, since everything hardware related runs in kernel space, it is unstable by design: a bug in a hardware driver means that a network card can crash the system. Linux is literally the opposite of a micro kernel.

Now, some of you may remember several years ago when Andy Tanenbaum and Linus Torvalds had a 'shouting match' over the Intarwebz on the merits of a micro kernel. As I won't pretend to be something I'm not (a Ph.D-level teacher, writer and software designer), I'll let you discover for yourself what the heck this was about: first on Andrew S. Tanenbaum's page on MINIX and Linux, then in an interview made for FSM. For the sake of brevity, I'll sum up what is, and the advantages of a micro kernel.

A micro kernel: MINIX

A micro kernel is, by definition, a kernel that has NO integrated hardware driver -- none! Not even an interrupt scheduler nor a memory controller. It is nothing more than a messaging system, and everything that is not the kernel will run in user space. It is lean: a micro kernel might not need more than a few thousands lines of C code. It is stable: if a hardware driver fails, it won't bring down the kernel. It can therefore crash, and be reloaded good as new. Moreover, such a design makes updates very easy. As the kernel itself is so simple it is relatively easy to make it bug free; if a driver is updated, it can be unloaded and a new version loaded in its place. No more reboots.

Now, MINIX was made as a teaching tool: it means that it should be as hardware-independent as possible. It will, thus, be made entirely out of portable code (like, say, C – with no assembly whatsoever).

MINIX is now in its third version. It can handle a windowing system and pseudo terminals. It can work as a server, or as a desktop system (but very few desktop tools have been ported). It is POSIX compliant (which doesn't actually mean much).

So why isn't it replacing Linux?

First, for historical reasons. MINIX is first and foremost a teaching tool. It is the most basic functional UNIX clone currently in existence, and it must, for that reason, remain very simple. Features added must not be integrated as patches, but as extra drivers – so as not to be used until needed. This, coupled with as little code duplication as can be achieved, means that improving MINIX is a very delicate process.

Second, for performance reasons: micro kernels use messaging to communicate between the kernel and drivers. This process uses up CPU cycles that direct communication in kernel space avoids. The very fact that the memory controller is decoupled from the kernel entails a 20% performance loss alone. Thus, in most cases, MINIX will be at least 20% slower than Linux – or BSD.

Another performance reason would be that since Linux has assembly-optimized code for x86, even without the micro kernel's design, it would still be faster than MINIX, with its C-only code, on x86 systems (performance would depend on compiler efficiency, but it would be there nonetheless). However, we need to nuance this.

(Editor's note: Minix was also non-free up until much later in its life. This was one of the main points Linus had while talking to Tanenbaum...)

A strange monolithic kernel: Linux

One of the biggest improvements made to Linux ever since its inception has been its ability to support modules: originally, you had to compile a whole kernel every time you changed a piece of hardware, or you had to compile a huge kernel with all drivers loaded even though you would then end up with wasted RAM for unused drivers. Modules allow you to split your kernel up into several pieces that can then be loaded and unloaded pretty much at will. This means that you can compile a smallish basic kernel and then load needed modules at run-time. You can even unload a module, compile a replacement, and load the updated version – just like in micro kernels. But, for those who didn't pay attention, this is not where a micro kernel design is theoretically better than a monolithic one. That occurs on the interface between the driver and the kernel.

A case in point: if you load a buggy graphics driver module it can crash the kernel, freeze the computer, etc.

Now, there are several mitigating factors that make the picture less problematic than it actually is: the module interface in Linux gets tighter all the time, so it is becoming harder for a driver bug to bring down the whole kernel. Moreover, the open-source nature of the kernel means that it is quite unlikely for a piece of code to contain bugs that can lead to crashes – and those that do exist get squashed rapidly. In short, while MINIX is stable because it doesn't permit bugs to bring it down, Linux is stable because it doesn't have too many bugs to bring it down – and usage difference is sufficiently large enough that there is a difference between the two FOSS projects.

Moreover, while Torvalds (and the Linux developer community in general) disagrees with Tanenbaum on the matter of micro kernels, he does acknowledge that many of his points are valid. That's why, with time, more and more code is added to the kernel in order to isolate subsystems from each other. This kind of coding adds complexity to the functions of the kernel, thus using up CPU cycles and so reducing performance...

That's point one.

Since Linux is now in use on more than one platform (x86), many parts of it had to be rewritten: ARM, PPC and Itanium are but a few of the platforms on which the kernel runs. Most replicate the same functionalities that exist on x86, but in many cases it is less advanced or refined – so while Linux is probably one of the best performing kernels on x86, it is not the case on any other platform. Since this is indeed a concern, most work occurring on x86-specific code is now designed to be as compartmentalized as possible:

  • code that would have no functional equivalent on other platforms (at a reasonable performance cost, or code such as an x87 FPU emulator or a specific memory controller) is kept inside x86 but must be hack- and shim-less proof,

  • code that would be functionally equivalent on other platforms (at reasonable performance cost) is programmed in plain C and taken out of x86 to find its way in the 'kernel' section.

You'll notice that, along the way, several 'speed hacks' (as this is valid for both re-factored and new code) could have gone missing. At the same time, thanks to code re-use, the 'central' part of the kernel on all platforms (let's say, the scheduler, the memory controller and the rest of the kernel's internals) may be getting slimmer for core functions. Slimmer, more elegant code doesn't mean faster though.

That's point two.

Performance is also in the eye of the beholder: sometimes it can go down due to a bug or regression, but at other times it may be the result of a balance shift. We'll take for example the memory allocator and the ext4 driver.

That's point three, and this one deserves a more detailed analysis.

Feature and safety at a performance cost

While code modularization has a generic performance cost that is often negligible, considering the gains in development time and overall code quality, there are features that have performance costs with no easily measurable benefit – but benefit there is. We'll take two of the most noticed “regressions that aren't”.

Do keep in mind though that regressions do happen, and are usually corrected a few kernel releases later.

The memory allocator dilemma

The Linux kernel has, currently, three different allocators: SLOB, which is historical and rather dedicated, SLAB, which used to be the preference, and SLUB, which is newer and generally slower.

According to kernel documentation, SLUB has been written to replace SLAB on multicore systems: SLAB is faster than SLUB, no question about it, but on multicore system it is also heavier on the RAM. In very extreme cases, SLAB can end up using more than a gigabyte of RAM, which negates its performance advantages by causing paging and data bus contention. SLUB solves this problem by being heavier on the CPU but using much less RAM – in these extreme cases, SLUB performs better 'live' than SLAB. Since it also isn't THAT costlier CPU-wise and machines get more cores all the time (an Intel quad-core with Hyper-threading will count as an 8-core, and waste around 17 Mb of RAM – say goodbye to your CPU cache), it's become the default.

But a database benchmarks will still find that a SLUB-using kernel will perform worse than a SLAB-using one.

The scheduler stress

Initially, Linux was made to work on server-oriented tasks: one task that had to be accomplished as fast as possible. Linux is also a co-operative multitasking kernel: it will give a process more resources when it asks for it if:

  • the process' priority (its 'nice' level) is higher than other processes requiring resources,

  • other running processes have notified the kernel that they're done for now.

There is more to say about this, but you get the basic idea.

Typically, on server loads, you have a handful of foreground processes with high priorities that aim at completing their tasks as fast as possible before releasing their resources: in those cases, delaying one task so that another finishes is the most efficient system. Less cache misses on the CPU, less wild head movement on hard disk drives... This kind of scheduling is good for servers.

But, on a desktop system, it means that, say, mouse movements will freeze if a process gets very heavy on the CPU. Which is Bad (tm).

So, the Completely Fair Scheduler (CFS) was written, and the kernel redesigned somewhat. It takes into account stuff like how much resource a process has already asked for. It allows pre-emption of a process on more execution points and it polls processes for additional execution points at more regular times.

But a database benchmarks will still find that a CFS-using kernel will perform worse than an older one.

Mitigating factors

Now, it seems that the kernel slowing down isn't such a bad thing if it is the price for a more versatile and safer system. However, there are still debates raging on various points that may seem to contradict all that's being said. I'll tackle a few of these here.

Should file system drivers be found in user space, or in kernel?

This one is very sensitive, and for more than one reason: since, on UNIX, everything is a file, one file system at the very least has to be very close to the kernel. After all, even processes are found on a file system. Look at /proc! or even devices, in /dev! Back to our micro kernel vs. monolithic kernel argument. Performance costs for all file systems being found in user space would be high. So, let's put all file system drivers in memory! But, there are thousands of them and many are more buggy than an ant hill! Do you really want that?

I'll take a very controversial example, which shows how difficult it is to answer this question – until you put it into perspective: let's take NTFS, which supports POSIX, DOS and NT file spaces, alternate data streams, metadata journalling, file-level compression and encryption, built-in defragmentation and out of order block allocation.

Version 3, starting with Windows 2000 is the version currently in use. Not all its features are used or even supported by Microsoft's OSes, meaning that it could trash data found on otherwise perfectly valid and sane NTFS volumes. You have been warned.

NTFS support has been very long in coming; first, because there was no real use for it until 2003, and second because it is very complex and not very well documented. Still, there was support in the kernel since 2001. But, it was for an older version of NTFS (NT4's) which was not compatible with 3.x – leading to massive corruption. Around 2004, a rewritten NTFS driver appeared, that allowed read and then eventually partial write access to NTFS partitions. A third-generation driver, however, made its debuts in 2006, and soon attracted interest: NTFS-3G runs in user space and provides full read/write access to NTFS partitions. Performance was not its primary goal, and it started becoming a primary concern only very recently, although there were drastic improvements along the way.

Its lack of performance was taken as reason why file system drivers should be kept in kernel space. From my own semi-formal tests, that's completely bogus. What I could observe was that:

  • Windows NTFS driver has a lousy block allocator: it can split a big file into more than 3000 pieces on a used partition, while NTFS-3G will only do 30 or so (test done on a 60% full 80Gb partition);

  • NTFS-3G will eat up a lot of CPU time on writes (latest version 20091015 mitigates that a lot): many small files or a few big ones are not much of a difference any more, but on lower -end systems (say, netbooks) transfer rate is essentially limited by CPU clock;

  • Windows, NTFS-3G and Linux NTFS have very similar read speed and CPU use on reads;

  • depending on whether you go through the SCSI interface (that includes IDE and SATA in later kernels) or USB, data rates can vary wildly on Linux. The same is true if you use the full FUSE package or the reduced, streamlined version provided by NTFS-3G.

This mean that NTFS-3G cannot be used as an example of why in-kernel drivers are better than user space ones: not only is it as fast as the in-kernel driver where they share functionality (and a good chunk of code) with similar CPU use, there is a higher performance impact coming from bus controllers! Moreover, considering that most of the time on reads and writes is spent waiting for the drive to answer, optimizations as small as direct kernel access and/or assembly code are useless – period. Optimizations on read order, data sorting and allocation are far more important.

Now, for in-memory file systems, it may be a different matter. Still, there is no reason for a file system driver to be ported in-kernel if it works well in user space, nor is there real reason to take a file system outside the kernel if it's already complete and well tested. Actually, one project under way is to modularize block allocation and error reporting in kernel for all file system drivers: while this is not, stricto sensus, the best place for that, it is more practical in a 'code repository' way but it requires time to work well.

Outside of x86, Linux actually sucks... Or does it?

Not actually wrong, but not true either: as a matter of fact, Linux on PPC works rather well, if for no other reason that Linus Torvalds' main rig has for a long time been a dual CPU Apple Mac G5; so, if Linux-x86 is good, Linux-PPC is a close second. Also, the x86-64 architecture has recently achieved a place of its own in the kernel, and is no longer a bastardized x86 branch. Alpha and Itanium, well, they're getting quite moldy – I mean their architectures. There are several embedded systems that see support, or not – these are rather short lived.

That leaves ARM essentially as an architecture. And it is a fact that Linux on ARM isn't very good. But here, we've got some input from Debian. The kernel may not be top-notch but most of the ARM performance loss rests on GNU's glibc library – which is why Debian decided to replace glibc with eglibc, which is essentially a heavily patched glibc. According to the fork's authors, GNU developers aren't too quick on the uptake and take ages to integrate ARM patches – prompting the fork. What's nice about eglibc is that, although it is patched for ARM, it remains compatible with glibc – and thus, runs on x86 (both of them), PPC etc.

So, it does look like Linux isn't so hard to port to a different platform any more, and a failed port depends on much more than just the kernel. But it also requires time to be done right.

Conclusion

Linux is becoming bloated, and it's also getting slower. You can't however, correlate the two directly, as the kernel gets bigger every time a driver is added, but you're not forced to compile or load all drivers. The fact is that, for a similar set of drivers, Linux isn't really getting bigger.

So, using existing metrics, Linux's performance is, indeed, declining steadily; those metrics being task-oriented and those tasks being typical for a server, one must take this with a grain of salt.

If indeed, regressions do find their way inside the kernel they are usually corrected quite rapidly. “False” regressions are usually caused by security and/or stability features being added, along with a good helping of user-oriented optimizations. What's lacking is the tools to catch regression and the time so that added features don't bloat the kernel.

In that light, projects like Phoronix's test suite and tracker, which test both network and user oriented tasks and allow variations to become automatically available, are tools that have been common for a long time under Windows and those results are invaluable.

Sources and further reading

Phoronix website

The Register: Linus calls Linux bloated and huge

Slashdot: Debian Switching From Glibc To Eglibc

Category: 

Comments

Terry Hancock's picture

As more people use Linux for "desktop" and "workstation" applications, more of them are going to care about "features and safety" than about "reliability and efficiency".

I use GNU/Linux primarily as a desktop (or a workstation -- the distinction is kind of subtle). Most of the time, I don't really care if I have to reboot once in awhile. I know we razz Windows for this problem, but is it really that much trouble?

It's not if I'm the only one using it. And if ferreting out whatever has actually gone wrong is going to take more time than rebooting, I'll sometimes just hit the power button and wait through all of the obligatory filesystem repair work that has to go on in the next boot. So what? You never lose files that weren't open anyway (well, hardly ever, anyway).

On the other hand, if I can't get my graphics card or sound card to work right, I'm totally screwed. My worries are over things like finding an OpenGL-capable 3D-accelerated graphics card that will work adequately with free-licensed drivers. That's not just because I'm ideologically opposed to closed source drivers (though I am), it's also because they're just so much of a pain in the neck to install -- and some of them, like NVidia's, are notably buggy and cause my system to crash frequently enough to be really annoying.

Of course, there will continue to be GNU/Linux distributions optimized for servers, and there are always the various versions of BSD Unix to choose instead.

So, I'm not entirely sure this is a bad trend. GNU/Linux is starting to look a little more like Windows and Mac, because it's trying to do the same things. And like them, GNU/Linux can stand some bloat for these kinds of applications.

Mitch Meyran's picture

I also use GNU/Linux full time: workstation, entertainment, and gaming (I use Wine). And it can, indeed, happen that the system hangs due to driver errors.

Most of there errors can be tracked back to the only non-Free driver I use: AMD's fglrx. It's been a long time since any ALSA driver got wonky (I did say ALSA, not PulseAudio, which is another matter), and I won't mention the network driver: ever since Nvidia decided to directly support forcedeth, I've been network-related problems free, be it on AMD, Intel or Nvidia hardware. USB problems also disappeared since the last USB stack rewrite. I have yet to test the 1394 driver, but I've heard nothing but good stuff about it.

When it doesn't downright crash, I find it faster and safer to switch to init 1 (single user mode, all services stopped) and rmmod fglrx (it worked when I had nvidia, too) then go back to init 5.

No, what's left are, as you said, graphics drivers. And I can't wait and test Gallium3D drivers for Nvidia and AMD!

---
A computer is like air conditioning: it becomes useless when you open windows.

Terry Hancock's picture

Actually, I find most graphics hangups will get fixed by just resetting the X server (which has "Ctrl-Alt-Backspace" as a hot key combination on my Debian system).

The cold boot approach is only for when that doesn't work, which does occasionally happen.

Mitch Meyran's picture

This is true for FOSS drivers and proprietary ones alike: sometimes, the kernel module gets its bits in a knot, and needs unloading (something that ctrl-alt-backspace, or a Xorg restart) doesn't cause. And it sometimes complains that it is still in use, so I may need to switch to single user mode and unload most processes in order to unload it.

---
A computer is like air conditioning: it becomes useless when you open windows.

Mitch Meyran's picture

It seems a comment/addendum I tried to do yesterday got scrapped. So I'll repeat it.

'nice' defines how high the priority of a Linux process: the lower 'nice' is, the higher the process' priority. 'nice' means how nice a process is to others. I didn't make that distinction explicit in the post, so here it is.

---
A computer is like air conditioning: it becomes useless when you open windows.

Ryan Cartwright's picture

As an established user your comments should go through withotu approval so the scrapping wasn't done by a human. is it possible you clicked Preview and then navigated away (thus not submitting the comment)? I've done that a few times.

Ryan
--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

Mitch Meyran's picture

Just as you said, I probably merely hit 'preview'. So it got scrapped - by me. Sigh.

---
A computer is like air conditioning: it becomes useless when you open windows.

Mitch Meyran's picture

Why is it that I mentioned ext4, and ended up mentioning NTFS-3G instead? Simply because ext4 wasn't the most counterproductive example I could find, and I tested its latest build while I wrote that piece - and forgot to come back and correct it.

Still, what I intended to say is, recently ext4 got a bad performance regression - one that related to a problem that is also susceptible to happen on ext3, and which may cause corruption at inopportune times. Barriers.

On Linux, barriers are used to prevent any data from being sent to the disk drive while it is flushing - as in, committing every operation it has been sent. The reason why is because, with more drives using NCQ, there is a chance that the journal could be written to disk, data would be too, and the journal closed (data committed) - but if the drive reordered its operation, putting the actual data's commit AFTER the journal has been closed, and the disk crashed before that was actually the case, then the journal wouldn't match what is actually on the drive! for more information on barriers, read this rather old, but very relevant, piece: http://lwn.net/Articles/283161/

ext3's default behavior is to mount with barriers disabled; on ext4, due to it very recent, that behaviour was changed recently so that it defaulted to enabled. Resulting on a huge performance drop on intensive, small writes requiring many sync() - such as databases (PostGreSQL got a 82% performance drop!).

Right now, developers work at reducing that huge performance loss (by making barrier calls less necessary), but this is indeed an illustration as to how reduced performance may actually come from improved reliability.
---
A computer is like air conditioning: it becomes useless when you open windows.

Mitch Meyran's picture

Phoronix just ran a series on benchmarks on all kernel versions since 2.6.24 to 2.6.33 (meaning, on 2 1/2 years of kernel releases).

It appears that several regressions were fixed in 2.6.33, and performance actually got BETTER across the board - or didn't change, except in database-based scenarii. Pure CPU-based benches either stayed stable or improved some - or greatly.

It would seem that barriers were enabled one ext3/4 file systems, resulting in very slow commits. However, caching seems vastly improved.

Linux might be getting bloated, but it seems performance isn't a problem yet.


A computer is like air conditioning: it becomes useless when you open windows.

Pantherman's picture
Submitted by Pantherman on

Being a Debian user myself, I've not encountered that many problems with the OS itself. I still think that it is more stable than XP. (I was getting the BSOD on a regular basis in the end).

I'm willing to put up with some "slowness" of the system as long as it remains a stable system. I'm just a "typical" end user, I do no programming or anything special with Linux aside from internet use and e-mail and the like. I tried re-compiling the kernel a few times to see if I could make it faster and just reverted back to the normal kernel mostly because I didn't really see any improvement, and I lost some functionality in one instance.

I however, can't attest to the longevity of my computer's up-time. Per the family's request all computers are shut off at night when not in use.

Terry Hancock's picture

I can.

My desktop system has run for as much as several months without rebooting before, under moderate use.

Now I have had to reset the X-server a few times during that time, and there have been some dicey times with particular pieces of equipment or software that caused me to have to reboot multiple times in a few days, but under normal use, uptimes of multiple months are common.

Frankly my electricity is less reliable than my operating system. ;-)

So when we're talking about decreased reliability, we're talking about "compared to server Linux or Unix systems" not "other desktop operating systems like Windows or Mac". In other words, we're really spoiled. :-)

sam_tuke's picture
Submitted by sam_tuke on

My understanding of the kernel is limited, however it seems to me that the kernel must remain independent of the needs of specific implementations of it in order to continue to fulfill its myriad of roles. That is to say that the kernel is a critical part of any GNU/Linux OS and so its important that it does not reflect the needs of one particular type of implementation, such as workstations rather than servers or embedded devices, but rather remains agnostic to any given type of implementation. As I understand it, the legendary scalability of GNU/Linux was not built upon technical decisions made to reflect one particular deployment environment.

If the trend is currently that the kernel is moving to reflect desktop needs rather than server or embedded needs, then it does seem to me that we have a serious problem on our hands that is likely to develop into a time bomb (as this doesn't seem like the kind of trend that will easily reverse itself, particularly as GNU/Linux is currently seeing most growth in desktop/workstation markets).

So to the experts I put the question: is the Linux kernel becoming less lean and efficient because of a trend towards desktop hardware compatibility over server stability and efficiency, as this article seems to suggest? I don't see that the recent phronix testing sheds any light on this issue as it is purely comparative and does not isolate kernel specific performance.

Terry Hancock's picture

I think we're mostly talking about the effect of various collections of modules. Since these generally aren't used in server or embedded applications they don't have much impact on those uses.

IOW, I think we're talking about the characteristics of the delivered builds of Linux as included in distributions, not Linux itself.

Pantherman's picture
Submitted by Pantherman on

Really? because I too got the impression that the kernel itself was what was being called "bloated", rather than the distributions.

Mitch Meyran's picture

Modules are part of the kernel, inasmuch as they ship with it. Distros may add some modules not included in 'vanilla', or include 'staging' drivers, but a module is part of the kernel in any case. And distributions will, by default, ship with most modules compiled and ready for loading.

When you do a kernel build yourself, you can disable most modules and options you don't need, and get a much smaller kernel.

---
A computer is like air conditioning: it becomes useless when you open windows.

Author information

Mitch Meyran's picture

Biography

Have you ever fixed a computer with a hammer, glue and a soldering iron? Why not? It's fun!