Software and hardware annotations, q2 2005

title           Xen Fedora 5
kernel          (hd1,5)/boot/xen.gz dom0mem=120000
module          (hd1,5)/boot/vmlinuz-2.6.12-1.1400_FC5xen0 ro \
                  vga=ext reboot=warm pci=biosirq \
                  elevator=anticipatory root=/dev/hdb6
module          (hd1,5)/boot/initrd-2.6.12-1.1400_FC5xen0.img

• It is not possible to access the X font system, only the evil Fontconfig/Xft2 one (this probably can be fixed only by recompiling).
• The bitmaps fonts are only available in the X font system, not the the Fontconfig/Xft2 one (this can be fixed by adding a few lines to /etc/fonts/local.conf.

title		hda1
root		(hd0,0)
kernel		(hd0,0)/boot/bzimage root=/dev/hda1

title		C:
rootnoverify	(hd0,2)
chainloader	(hd0,2)+1

mkisofs -no-emul-boot \
  -boot-info-table -c boot/boot.catalog \
  -boot-load-size 32 -b boot/grub/iso9660_stage1_5 \
  -r -J -l -o /tmp/grubboot.iso /tmp/grubboot/

$ rpm -qlp libtermcap-2.0.8-39.src.rpm
libtermcap-2.0.8-ia64.patch
libtermcap-aaargh.patch
libtermcap.spec
termcap-116934.patch
termcap-2.0.8-bufsize.patch
termcap-2.0.8-colon.patch
termcap-2.0.8-compat21.patch
termcap-2.0.8-fix-tc.patch
termcap-2.0.8-glibc22.patch
termcap-2.0.8-ignore-p.patch
termcap-2.0.8-instnoroot.patch
termcap-2.0.8-setuid.patch
termcap-2.0.8-shared.patch
termcap-2.0.8-xref.patch
termcap-2.0.8.tar.bz2
termcap-buffer.patch

$ rpm -qlp ncurses-5.4-17.src.rpm
ncurses-5.4-20041218.patch
ncurses-5.4-20041225.patch
ncurses-5.4-20050101.patch
ncurses-5.4-20050108.patch
ncurses-5.4-20050115.patch
ncurses-5.4-20050122.patch
ncurses-5.4-xterm-kbs.patch
ncurses-5.4.tar.bz2
ncurses-linux
ncurses-linux-m
ncurses-resetall.sh
ncurses.spec
patch-5.4-20041211.sh

Red Hat will create the Fedora Foundation with the intent of moving Fedora project development work and copyright ownership of contributed code to the Foundation.

• Making fine prints in your digital darkroom Monitor calibration and gamma.
• Yet another gamma correction page.
• "Brightness" and "Contrast" controls.
• The Monitor calibration and Gamma assessment page. I particularly like the target gamma visual tests in this page, which are provided at three different intensity levels. My LG L1710B LCD monitor was tuned fairly easily. I can get a final gamma of 1.8 with X gamma set to 1.2 and contrast set to 90/100 and brightness to 75/100. Image quality is rather good. I had set it almost to those values by judgement with brightness and contrast set a tiny bit too low. I suspect that lots of people set their monitor to excessively low brightness levels, which are the default.

ORN: A lot of companies have been using OpenSSH in their products (Sun Microsystems, Cisco, Apple, GNU/Linux vendors, etc.). Did they give anything back, like donations or hardware?
Henning Brauer: Nobody ever gave us anything back. A plethora of vendors ship OpenSSH --commercial Unix vendors (basically all of them), all of the Linux distributors, and lots of hardware vendors (like HP in their switches)-- but none of them seem to care; none of them ever gave us anything back. All of them should very well know that quality software doesn't "just happen," but needs some funding. Yet, they don't help at all.

ORN: This is the first release that includes X.Org. Why did you choose to import it instead of XFree86 4.5.0?
Matthieu Herrb: The primary reason is that the new revision 1.1 of the XFree86 license is less free than the old MIT license that had been used for years by XFree86. OpenBSD already avoided shipping the final XFree86 4.4 release that also uses the new license in 3.6. Then, as many other projects moved away from XFree86 because of the license, it became obvious that most new developments in the X window system now take place in X.Org. Having said that, projects like OpenBSD have to stay vigilant that X.Org doesn't turn into a Linux-only project (that would slowly slip to a GNU General Public License).

sub read_vint {
        my $b = ord CORE::getc($_[0]->[0]);
        my $i = $b & 0x7F;
        for (my $s = 7 ; ($b & 0x80) != 0 ; $s += 7) {
                $b = ord CORE::getc $_[0]->[0];
                $i |= ($b & 0x7F) << $s;
        }
        return $i;
}

sub doc {
        my ($self, $n) = @_;
        $self->{index}->seek($n * 8, 0);
        my $pos = $self->{index}->read_long;
        $self->{fields}->seek($pos, 0);
        my $doc = Plucene::Document->new();
        for (1 .. $self->{fields}->read_vint) {
                my $fi = $self->{field_infos}->{bynumber}->[ $self->{fields}->read_vint ];
                my $bits = $self->{fields}->read_byte;
                $doc->add(
                        bless {
                                name         => $fi->name,
                                string       => $self->{fields}->read_string,
                                is_stored    => 1,
                                is_indexed   => $fi->is_indexed,
                                is_tokenized => (($bits & 1) != 0)              # No, really
                                } => 'Plucene::Document::Field'
                );
        }
        return $doc;
}

Laptops...beware?
Ext3 has a stellar reputation for being a rock-solid filesystem, so I was surprised to learn that quite a few laptop users were having filesystem corruption problems when they switched to ext3. [ ... ] had nothing to do with ext3 itself, but were being caused by certain laptop hard drives.
The write cache
[ ... ] Unfortunately, certain laptop hard drives now on the market have the dubious feature of ignoring any official ATA request to flush their write cache to disk. This isn't a wonderful design feature, although it has been allowed by the ATA spec up until recently [ ... ]
However, it gets worse. Some modern laptop hard drives have an even nastier habit of throwing away their write cache whenever the system is rebooted or suspended. Obviously, if a hard drive has both of these problems, it's going to regularly corrupt data, and there's nothing that Linux can do to prevent it from doing so.

During the past year, more than 4,100 patches from Red Hat employees were integrated into the upstream 2.6 kernel. In contrast other companies boast that their offering contains the most patches on top of the community kernel.

Upstream - doing all our development in an open community manner. We don't sit on our technology for competitive advantage, only to spring it on the world as late as possible.

• The issue is not that other distributions do not publish their patches, because they do, and in a timely manner, thanks to the GPL, but that they do not actively submit them upstream, and since upstream is busy enough, upstream people do not go out of their way to scout for patches, but only deal with those that are actively submitted.
  I think this is a very important observation on the actual process of free software development, which is totally dominated by scratch my itch logic.
• My perception is that RedHat used to be as reluctant to actively contribute the results of their QA to upstream as the others. It is very welcome that their official position is now different.
  However, I think that the kernels in their products are still very different from upstream kernels because of the number of feature patches they use to differentiate it from upstream. Yes, it is good that differentiation is not based on the number of semi-hoarded fix patches, but they are still semi-forking the kernel.
• I think there is little competitive advantage to be had from being reluctant to contribute QA to the upstream kernel for Linux companies; competitors, unlike upstream, can always proactively fetch those patches, so not actively contributing them is rather futile.
  I reckon that semi-hoarding patches instead benefits the kernel maintainers by enhancing their job security. After all a policy of having a kernel that is kept quite different from the upstream kernel implies the need for a team to maintain it instead of relying on upstream and community maintenance.
  Most of RedHat's competitors have already done rounds of downsizing, in the middle of a big IT recession, and probably their remaining employees are very keen to make themselves as indispensable as possible (and think of Mandrake who have just acquired Conectiva, that is a team of distribution developers that cost one third per year and are as good).
  Job security through obscurity (whether of source or of process) is a winner in many cases, and every little helps, unfortunately :-(.

The plan is to reduce the amount of memory that Gnome applications consume. Gnome is barely usable on a machine with 128 MB of RAM; contrast this with Windows XP, which is very snappy on such a configuration.

Why do you want to reduce memory consumption?

Looking at the top 25 contributors to the Linux kernel today, you'll discover that more than 90% of them are on the corporate payroll full-time for companies such as HP (HPQ), IBM, Intel (INTC), Novell (NOVL), Oracle, Red Hat (RHAT) and Veritas (VRTS), among many others.

• I spent the last month halving the memory used by these applications
• I spent the last year adding to this application these snazzy transparency effects and ten more cool features as you can see in this demo

Benchmarks and lack of visibility

Most popular benchmarks are pure speed benchmarks and can be run on systems with enough memory that the whole application being benchmarked is memory resident. Also, most popular benchmarks are application benchmarks, and not system benchmarks, and that's another reason why the whole application can be memory resident.
There are also some intrinsic problems with (mostly nonexistent) sysem and memory benchmarks:

• A memory benchmark for a system is intrisically less repeatable and thus impressive a fact than a speed benchmark for a single application.
• Creating a realistic system and memory benchmark and running it takes a lot more effort than a synthetic or single application speed benchmark.

Large pages

I have seen very persuasive research evidence that for unoptimized programs the optimal page size is 256 bytes, and with pages sizes above 1024 bytes the number of pages in the working set is roughly constant; in other words a 4096 byte page size means that other things being equal the working set of a program is four times larger in bytes than with a 1024 byte page size. Larger page sizes amplify considerably the impact of the other issues.

Use of shared libraries

Shared libraries mean that any use of a function on a page brings in the whole page. Bad news if functions in a shared library are not clustered with respect to expected dynamic usage. Usually functions are clustered in random order or in alphabetic order...

Use of many shared libraries with lots of impure data

Shared libraries contain a significant amount of inpure (writable) pages typically containing either external (interlibrary) links or internal (intralibrary) links. Internal links contain mostly relocation entries.
When any internal or external link on a library is fixated the entire page on which it resides it must be duplicated for the process in which it is. As this page says about a GNOME Hello world! program:

That simple program uses 73 shared libraries that allocate a total of 13Mb of non-shared static data.

This can be alleviated in the following ways:

• Better library and language design.
• Prerelocating libraries to a nominal address range, and prelinking them (KDE does a bit of this).
• Forking processes that use the same set of libraries from a master one that has them already prerelocated and linked in (KDE does this too by default).
• Generating PIE (position independent code) which eliminates most of the intra library links, at the price of higher register usage usually.
• Merging libraries together.

Bad memory allocators

Many common memory allocators are designed to maximize memory usage and resident sets, for example one or more of:

• Use of buddy-system logic, which tends to overallocate memory by about 30%. But it is fast, so benchmarks on infinite memory reference PCs are flattered.
• Extensive list walks to find or free allocated blocks, so they touch a lot of pages.
• No attempt to cluster block allocations.
• Inability to coalesce freed blocks and to release back large chunks of memory to the operating system, which does not directly impact resident sets, but indirectly via larger virtual memory mapping tables.

Careless program designs

Many programs embody deep and thin flow trees, where going from event to event resolution can involve dozens of nested procedures, each of which is often on a different page from the next.
A legendary case if the reference X server implementation: most of the time the X server just receives bitmaps to BLIT on the screen, so only one or two pages should be touched; but to get from reading the BLIT request to the BLIT code that satisfies it may involve a very deep and dispersed call sequence, and so very many pages are kept constantly referenced.
Some programs also involve maximization of data, not just code, memory resources. For example client side fonts imply that each X client needs to have a list and cache of rendered glyphs, which it constantly references, instead of having them in the graphics card's framebuffer memory, as it happens with server side fonts.
Also, a lot of cool graphics tricks require a lot of memory, for example to handle screen damage events (for example background pictures or transparency or non rectangular shapes).

April 08, 2005
Pete Zaitcev: Thomas in a cage with Firewire
http://thomas.apestaart.org/log/index.php?p=291
Yay. Thomas is about to find why Firewire is unsupported even on Fedora (let alone on RHEL).
But If someone asked me what Firewire needed, I would answer, "only a hacker with a brain". He may be able to pull it off, though I'm not too optimistic. Firewire is about as complex as USB and we all know how well that goes despite a sustained effort by Greg K-H, David-B, Stern and myself. My approach was to hide whenever Firewire came too close and wait for it to die in the marketplace (which, I suspect, is inevitable at this point).

• Architecturally, client side fonts don't fit with the X architecture: X is designed so that a single host may support multiple X servers, a single X server may have multiple screens with completely different characteristics, a process may be connected to multiple X servers on multiple hosts, as well as a single X server being used by many processes from different hosts. X is designed to support this flexibility this not quite as transparently as it should, but at least with some display independence, where font rendering is delegated to the server, which is what drives directly the specific target screens.
  A process connected to multiple X displays or screens of the same X display need to have multiple font rasterizer contexts if it uses client side fonts. With server side fonts all the process has to do is to send strings and they will be rendered optimally for the specific target display, no matter its characteristics, which host it is connected to, and so on.
• From a performance point of view client side fonts are bad now and will be even worse in the future. Their only performance advantage is in some modest win as to protocol latency: since the rendering of glyphs happens on the client side, it knows exactly the geometry of the rendered text, and the X server does not need to send back font metrics; so, it is possible to contrive a demo in which gives a small overall advantage in a very special case.

• Rasterization and caching of the rasters has to be performed in every single application connected to the same X server; if an application is connected to multiple X screens, it has to rasterize and cache the same glyphs multiple times. Also, almost always the majority of applications connected to an X server use the same fonts.
  Of course if memory and CPU time are in effect infinite this does not matter; and therefore it is almost unnoticeable for single processes, but unfortunately I have often several dozen processes running connected to one or two X servers with one or two screns with different characterics each.
• Bandwidth is consumed by the sending of text rasters. This applies to multiple levels of the communication chain: even if the client and the X server are on the same host, with client side fonts he X server must rasterize the glyps every time to the display card, thus consuming AGP or PCI bus bandwidth.
  With server side fonts the X server can use all of the framebuffer that is not used for the screen as a glyph cache, making the rasterization of text almost entirely an onboard operation, saving on AGP and PCI bus bandwidth and latency.
• Latency increases because each application has to first rasterize all the glyphs in a text strings, and then send the raster as a whole, and the X server can only start blitting it when it is fully received.
  With client side font rendering not only the X can cache the glyphs in the frame buffer, but the text of the string to render is received quickly, and then the X server can render that text in way of principle incrementally, glyph by glyph.
• Sending pixmaps to blit instead of character strings to render currently is way less efficient than sending text strings in relative terms, but not that much in absolute terms because most current screens have rather low DPI values, so the bitmaps aren't that big.
  Unfortunately now that the world has switched to LCD screens higher DPIs are becoming more common, and higher DPI raise dramatically the size of the pixmaps to be sent, especially if they are antialiased, as this both increases the size of the pixmap and reduces the effectiveness of compression, if any; but then antialiasing is fortunately less necessary on high DPI displays.
  In any case rasterizing text over the wire on a 1280x1024 pixel display is not quite the same as on a 4000x3000 display.
  The growing size of client side text pixmaps impact performance severely in several ways:
  • Each client has to spend that much more CPU time to create bigger glyph rasters, and that much more RAM to cache them.
  • The bandwidth needed to send the text rasters increases considerably, both at the network level or at the AGP/PCI/... bus level.
  • The latency also increases.

• A considerable lack of documentation. For example the Fontconfig font specification is not that clearly defined anywhere, and there are several obscurities in the specification of the Fontconfig configuration files; and the Fontconfig configuration files are a classic example of moronic abuse of text markup, but then this is a much wider issue, admittedly.
• A number of gross misdesigns have been made in the implementation of Fontconfig; version 1 for example did not even cache font descriptions, and version 2 does, but in a rather clumsy way. After quite a wait now Fontconfig maps font files instead of copying them into memory, but of course this only works for filesystems that support mmap efficiently.
• Each client process must depends on a number of additional shared libraries, and since different clients will be executing different parts of the shared library at any one time, the memory residence footprint will be way large than if those shared libraries were in use just by the X server.
• Each client process has a private glyph cache, and I am not aware of any user level mechanism to influence the caching policy, including the minimum and maximum size of the cache; perhaps this is possible, but I am pretty sure most people do not use it. Instead cache policy settings are obvious in server side embedded and standalone font servers.
• It seems impossible to specify which precise font one wants to select, in particular I have been unable to ensure that the PCF version of Helvetica is selected instead of the Type 1 version. This may be due to lack of documentation or my misunderstanding, but I suspect not.

• ESIX Unix System V dev system multi user US$1499, std system 2 users US$619.
• UNIX System V TCP/IP US$359.
• UNIX System V Windowing runtime US$219, dev sys US$599.