Software and hardware annotations, q2 2005

This document contains only my personal opinions and calls of judgement, and where any comment is made as to the quality of anybody's work, the comment is an opinion, in my judgement.

June 2005

050630
My ALSA latency notes recommend setting the PCI latency timer to a uniform lowish values like 32 cycles, and I have been told by many that this often fixes sound quality issues when there are PCI/AGP cards, like many video cards, that hog the bus by default.
But I have just been surprised to discover that in particular setting the host bridge latency timer to any non zero value reduces the effective transfer rate of the IDE disk subsystem (as detected with hdparm -t), for example from about 55MB/s to about 33MB/s for a value of 32 cycles.
This is somewhat surprising, because the latency timer is an upper limit, and if the device does not need to hog the bus for that long it is supposed not to. But it looks instead as if at least the host bridge in my chipset uses it as a lower limit.
050629
I have been trying out the Xen hypervisor for virtual partitions under Fedora 4 and I have noticed that some people have had difficulty figuring out how to boot it under GRUB, which is indeed not totally obvious, and here is an example (with the bleeding edge experimental kernels for Fedora 5):
title           Xen Fedora 5
kernel          (hd1,5)/boot/xen.gz dom0mem=120000
module          (hd1,5)/boot/vmlinuz-2.6.12-1.1400_FC5xen0 ro \
                  vga=ext reboot=warm pci=biosirq \
                  elevator=anticipatory root=/dev/hdb6
module          (hd1,5)/boot/initrd-2.6.12-1.1400_FC5xen0.img
So far Xen works pretty well and is quite fast, even if I have had some lockups. I suspect the bleeding edge Fedora kernel; also the Fedora firewall seems to have some issues. It seems quite practical to just run always under Xen, to enjoy for example fast save and restore to disk.
050628
The dumbness of sysfs and hotplug can only get worse: the zd1201 driver requires that the firmware for the peripheral it supports be loaded, and what now happens is that it creates on loading a sysfs entry to enable the loading, and after a short timeout this entry is removed. This means that just about the only way to load the firmware is to have hotplug enabled.
Note that there is no problem with just leaving the firmware loading entries around permanently until the firmware is loaded, for example manually. Just more job security via complexity for the maintainers of hotplug and sysfs.
050627
Just noticed that the ZyDAS 1201 driver (zd1201) is incorporated in the mainline kernel as of release 2.6.12 so that the patch mentioned previously is no longer necessary.
050623
More client side font pain as I discovered while having a look at Fedora 4 that under KDE (and GNOME too I guess):
  • It is not possible to access the X font system, only the evil Fontconfig/Xft2 one (this probably can be fixed only by recompiling).
  • The bitmaps fonts are only available in the X font system, not the the Fontconfig/Xft2 one (this can be fixed by adding a few lines to /etc/fonts/local.conf.
050618
Got a pointer to an interview with Theo de Raadt comparing Linux with OpenBSD as to code quality. I agree mostly that OpenBSD code quality is broadly higher, both in the kernel and in many userspace utilities. But the point with Linux are that it is GPL'ed and has lots more drivers.
Linus seems well aware that large portions of Linux are messy but that's just one of the issue, however painful it is to me.
050617
I have been looking into making bootable CDs or DVDs with Linux, and it is now possible with recent versions of GNU GRUB to use it to boot directly, instead of using a floppy image or SYSLINUX (more precisely ISOLINUX). A simple example of a GRUB menu for booting from a hard disk partition:
title		hda1
root		(hd0,0)
kernel		(hd0,0)/boot/bzimage root=/dev/hda1

title		C:
rootnoverify	(hd0,2)
chainloader	(hd0,2)+1
and a mkisofs line to create a filesystem image for booting GRUB:
mkisofs -no-emul-boot \
  -boot-info-table -c boot/boot.catalog \
  -boot-load-size 32 -b boot/grub/iso9660_stage1_5 \
  -r -J -l -o /tmp/grubboot.iso /tmp/grubboot/
If the bootloader one uses does not support boot choices/menus it is possible to construct a CD image with multiple boot image choices using the -eltorito-alt-boot options to mkisofs but this require multiple boot choice handling in the BIOS and this is not implemented in a significant number of cases.
I have also found some quite interesting interesting Microsoft Windows based utility for those interested in such practices.
050616
Just spotted an amusing if slightly confused article which suggests that splitting a 512MB swap partition into four improved swapping performance, as long as they have the same priority.
This is highly surprising, because four swap partitions on different discs should give better performance, but not on the same disc.
But a note says that the swap space handler in the Linux kernel at least in 2.4.x is poorly implemented, and uses a linear scan to find unused swap space, and splitting the swap partition is a workaround to that.
Or perhaps having four partitions with the same priority gets the kernel to issue four times as many IO requests for swapping, allowing better disc scheduling.
050615
Well, Fedora 4 has been released and I had a look, because I am still looking at something that does not have Debian's awesomely irritating packaging suckages. Unfortunately it turns out that dumb n00b packaging ideas are contagious: Debian's comical practice of calling a binary package with a base name differently from its source package, by adding a leading lib in front of the name if it is a runtime library package, has appeared in Fedora too, after having been dumbly adopted by Mandriva.
For example the source for the termcap library is called termcap-2.0.8.tar.bz2 and is packaged as libtermcap-2.0.8-39.src.rpm. Looking inside the source package one also finds some silly inconsistency in naming, with patches have different base names:
$ rpm -qlp libtermcap-2.0.8-39.src.rpm
libtermcap-2.0.8-ia64.patch
libtermcap-aaargh.patch
libtermcap.spec
termcap-116934.patch
termcap-2.0.8-bufsize.patch
termcap-2.0.8-colon.patch
termcap-2.0.8-compat21.patch
termcap-2.0.8-fix-tc.patch
termcap-2.0.8-glibc22.patch
termcap-2.0.8-ignore-p.patch
termcap-2.0.8-instnoroot.patch
termcap-2.0.8-setuid.patch
termcap-2.0.8-shared.patch
termcap-2.0.8-xref.patch
termcap-2.0.8.tar.bz2
termcap-buffer.patch

This is not the sole example, but of course for the sake of confusion some packages that contain mostly libraries are packaged without the annoying spurious lib prefix, for example ncurses-5.4.tar.bz2 is packaged quite properly:
$ rpm -qlp ncurses-5.4-17.src.rpm
ncurses-5.4-20041218.patch
ncurses-5.4-20041225.patch
ncurses-5.4-20050101.patch
ncurses-5.4-20050108.patch
ncurses-5.4-20050115.patch
ncurses-5.4-20050122.patch
ncurses-5.4-xterm-kbs.patch
ncurses-5.4.tar.bz2
ncurses-linux
ncurses-linux-m
ncurses-resetall.sh
ncurses.spec
patch-5.4-20041211.sh
still note the marvelous idea of having one of the patches called patch.
This silly, contagious mispackaging idea has two bad consequences; one is that it breaks simple, direct mappings from original package names to RPM package names, and most importantly it breaks sorting consistency. The most important one is that since it is both gratuitous, in the sense of having no obvious advantages, and it does have significant disadvantages, and it is also plain ugly, demonstrates a lack of what Linus Torvalds calls good taste.
Now good taste in packaging or programming is very important because it guides people to do things that suck less, and that matters greatly in the complicated issues that arise from packaging. In other words it puts the judgement of a packaging team into serious question; if they get simple things like issues of good taste wrong, one wonders what else is wrong.
For example once upon a time RedHat switched from putting the internal components of INN2 in the various traditional package specific directories, as for example in /usr/lib/news/bin/procbatch, to merging all those internal components into the public base directories, like /usr/bin/procbatch. I then submitted a bug report, and was told to lump it. Cool, and roughly on the same day I stopped using RedHat, because if they had people who could gleefully do that and persist obviously things were going downhill.
Overall my impression is that of the major distributions the least badly packaged is still SUSE, but it has always been politically incorrect (playing the 99.9% free but with proprietary bits as limpet mines game that RedHat have later finessed to a truly clever degree with proprietary _trademarked_ names logos and icons). Then Mandriva at one point switched to the stupid lib prefix naming convention, and at that demonstration of loss of good taste I just switched to Debian. After all if one is prepared to tolerate dumb bad taste, let's get it direct from the source, which at least is politically correct.
Unfortunately Debian policy mandates other dumb examples of bad taste, like the use of DPKG (who ever though that ar archives containing tar.gz archives was a good package format? Never mind the other huge problems), the starting of daemons on package upgrade even if they are disabled in the init runlevel config, and the social problems that cause very infrequent releases, and so on.
I have been attracted to Fedora at this point because for those wanting to try the latest and greatest and are prepared to cope with the resulting issues it seems pretty OK. Wait a moment, now that I think of it, does Fedora still have the same tradermarked names and icons issues as RedHat? It looks like it has less of them, but the resulting FUD effect is still rather unpleasant. Hopefully the move by RedHat to endow a separate entity, the Fedora Foundation, the endowment is described as:
Red Hat will create the Fedora Foundation with the intent of moving Fedora project development work and copyright ownership of contributed code to the Foundation.
with no mention of trademarks, a beautiful example of corporate cleverness. Copyrights matter a lot less because those are GPL'ed anyhow, but there is no GPL for trademarks or GPL equivalent permissions for RedHat's trademarks.
050612
More memory horrors... Konqueror has grown from around 50MB to around 250MB (about 70-80% memory resident) so I tried to quit it and restart it. However engagingly it did not terminate, because I had specified in the misnamed Performance preferences to preload a copy in the background. My naive expectation was that on quitting it would because of that restart it. Fat chance.
Also, how a Konqueror process where all pages have been closed can still have almost 200MB resident when on creation it has only got around 35MB baffles me. I suspect grave mistakes in the caching logic. The curious thing is that even when running Konqueror under gc that memory is not collected, so it is not quite a leak.
050611
Returning from a meeting of the pretty good Greater London Linux User Group meeting where there were four quite intesting talks. Of the four talks, I was quite personally interested in the one on the SVK source control system by Chia Liang Kao and the one on the Xen hypervisor for virtual partitions because I often have to deal with document/source archives and I'd like to run different Linuxes in different virtual partitions.
The SVK presentation made a convincing case for looking seriously at it, as it is based on the reuse of existing known working code and ideas from other projects, and was designed to be quite compatible with some popular existing version control systems, and to avoid some of their most hateful limitations.
Among the more interesting aspect of SVK is that as of now it is purely a client side system, which relies for now, and is therefore compatible with, the subversion server for shared version storage.
The Xen talk was particularly enjoyable because of the quality of the delivery by Dr. Ian Pratt.
Xen is a very interesting take on the virtualization idea. It is a kind of microkernel that creates a set of virtual machines whose architecture is not identical to the underlying hardware architecture, but is much simpler so it can be simulated much more efficiently. As such it is a paravirtualization system like UML or CoLinux but with a radically different architecture strongly reminiscent (and probably the most useful subset) of MERT (aka UNIX/RT) by Heinz Lycklama (see also this reference for the place of MERT in UNIX history). but considerably modernized and extended in one crucial way: the Xen hypervisor by defining a simple abstract software virtual machine can snapshot it incrementally, and either save the snapshots or transfer them to another machine running Xen hypervisor, thus achieving dynamic process migration, which was the original motivation for the development of Xen in the Xenoserver project. Another operating system kernel famously based on an hypervisor archirecture was Mach from CMU which eventually evolved into OSF/1 and the GNU Hurd. Even the Microsoft Windows NT HAL is a sort of hypervisor for the various operating system personalities (WIN32, OS/2, POSIX) that it supports.
The Xen architecture is very different from that of UML, and is based on booting a miminal hypervisor kernel which creates one or more virtual machines, into which a Linux (or FreeBSD, or other) kernel, ported to the Xen virtual machine architecture, is then booted.
The hypervisor itself has little or no drivers because of an astute ruse: one (or more) of the virtual machines (usually number 0) is given more or less direct access to the peripherals, and this acts as a server for the other virtual machines. So for example the disc driver runs in virtual machine 0, and all other virtual machines access the disc by sending requests to virtual machine 0. This architecture reminds me fairly somewhat of IBM's MVS or VM clusters using real (MVS) or virtual (VM) CTCAs to communicate among individual systems.
050608
While helping someone with getting a good intensity calibration on his monitor(s) I (re)discovered that many people don't quite get the subtle issues related to color and intensity. Some very nive people have helped by writing interesting papers with illustrations, for example:
050607
There is an intensely ironic part of an interview about the recent OpenBSD 3.7 release which is:
ORN: A lot of companies have been using OpenSSH in their products (Sun Microsystems, Cisco, Apple, GNU/Linux vendors, etc.). Did they give anything back, like donations or hardware?
Henning Brauer: Nobody ever gave us anything back. A plethora of vendors ship OpenSSH --commercial Unix vendors (basically all of them), all of the Linux distributors, and lots of hardware vendors (like HP in their switches)-- but none of them seem to care; none of them ever gave us anything back. All of them should very well know that quality software doesn't "just happen," but needs some funding. Yet, they don't help at all.
It is so ironic because the OpenBSD project is committed to the BSD license, which does not require anything other than credit or, in its new edition, nothing at all, in the way of contributions from adopters. At least the GPL requires vendors to contribute back their improvements, and this has worked really well in a number of cases.
I have been particularly amused by this other remark further down:
ORN: This is the first release that includes X.Org. Why did you choose to import it instead of XFree86 4.5.0?
Matthieu Herrb: The primary reason is that the new revision 1.1 of the XFree86 license is less free than the old MIT license that had been used for years by XFree86. OpenBSD already avoided shipping the final XFree86 4.4 release that also uses the new license in 3.6. Then, as many other projects moved away from XFree86 because of the license, it became obvious that most new developments in the X window system now take place in X.Org. Having said that, projects like OpenBSD have to stay vigilant that X.Org doesn't turn into a Linux-only project (that would slowly slip to a GNU General Public License).
These people are worried by a slow switch to the GPL; so ironic too. Especially as I was once following a discussion on the Xorg IRC channels and one of the authors said he had quite a bit of code to do an improvement someone was requesting, but since the server was not GPL licensed, the code was proprietary and could not be shared. It used to be that the development of the X reference server code was mostly funded by major corporates, so they chose the licence that best served their embrace and extend competitive advantage stategies.

May 2005

050531
I have recently published sabishape and dokde.
050527
Still looking at memory usage horrors including the bizarre discovery than under Debian unstable, using the unofficial 3.4.0 packages, the KDE Konqueror browser grows to inordinate size (like more than 200MB) and under SUSE 9.2 the KDE 3.3.2 Konqueror stays at around 50MB.
So I decided to use the latest BD conservative garbage collector compiled in malloc-override mode (with build option --enable-redirect-malloc) and then to set export GC_PRINT_STATS=1, and then to preload it with export LD_PRELOAD=/usr/local/lib/libgc.so.1.0.2) and it is such good fun watching what it snitches on something like Konqueror.
050525
Some to-remain-nameless culprit working on the Koha open source library automation program has been asking me how to improve the speed of their catalog searching program, which uses MySQL tables as inversion indexes, which is inappropriate in itself. Even worse, I have had a look at the schema and queries and code that does that and I have found it an appallingly awful farrago, and then found that MySQL has what looks to me a poorly written query planner. So I have supplied first some decently written SQL that works around MySQL's query planner limitations, and then suggested looking at proper text indexing systems, like Lucene.
Koha however is mostly based on Perl, not Java like Lucene, so that someone had a look at Plucene a translation of Lucene to Perl. Unfortunately this translation seems slow. How slow? Well, on 2GHz CPU it takes over one hour to index 170,000 record totaling around 7MB, and the worst part of it is that it is solidly CPU bound.
Now this is quite extraordinary: over one hour of CPU time to index that amount of data requires skills previously unsuspected, as it is usually expected that indexing and searching be IO bound operations.
So in order to learn from such marvels I have had a look at the lower levels of the system, as I fervently hope that at least the overall hash table index strategy is not unusual.
By looking at the lower level datum and aggregate code I have learned so much! For example, this piece of low level code to write a variable length binary integer:
sub read_vint {
        my $b = ord CORE::getc($_[0]->[0]);
        my $i = $b & 0x7F;
        for (my $s = 7 ; ($b & 0x80) != 0 ; $s += 7) {
                $b = ord CORE::getc $_[0]->[0];
                $i |= ($b & 0x7F) << $s;
        }
        return $i;
}
brought me tears and screams because of its depth and daring, and never mind this other splendid example of Perl programming, whose style seems to be representative of much other code in Plucene:
sub doc {
        my ($self, $n) = @_;
        $self->{index}->seek($n * 8, 0);
        my $pos = $self->{index}->read_long;
        $self->{fields}->seek($pos, 0);
        my $doc = Plucene::Document->new();
        for (1 .. $self->{fields}->read_vint) {
                my $fi = $self->{field_infos}->{bynumber}->[ $self->{fields}->read_vint ];
                my $bits = $self->{fields}->read_byte;
                $doc->add(
                        bless {
                                name         => $fi->name,
                                string       => $self->{fields}->read_string,
                                is_stored    => 1,
                                is_indexed   => $fi->is_indexed,
                                is_tokenized => (($bits & 1) != 0)              # No, really
                                } => 'Plucene::Document::Field'
                );
        }
        return $doc;
}
Watch and learn! And these apparently are the literal translations into Perl from Java code of equivalent magnificence.
050523
I have been reading several articles about filesystems alternative to Ext3, in particular JFS and XFS. In a related article about Ext3 I have noticed this scary story:
Laptops...beware?
Ext3 has a stellar reputation for being a rock-solid filesystem, so I was surprised to learn that quite a few laptop users were having filesystem corruption problems when they switched to ext3. [ ... ] had nothing to do with ext3 itself, but were being caused by certain laptop hard drives.
The write cache
[ ... ] Unfortunately, certain laptop hard drives now on the market have the dubious feature of ignoring any official ATA request to flush their write cache to disk. This isn't a wonderful design feature, although it has been allowed by the ATA spec up until recently [ ... ]
However, it gets worse. Some modern laptop hard drives have an even nastier habit of throwing away their write cache whenever the system is rebooted or suspended. Obviously, if a hard drive has both of these problems, it's going to regularly corrupt data, and there's nothing that Linux can do to prevent it from doing so.
The same article has interesting details about the various modes of journaling that Ext3 offers, and in particular that data=journal can be very fast in some special case (probably reading from the journal as it is writing to it). Also, about making it flush the buffer cache more frequently prevents huge write storms.
050518
While reading an article in Linux Journal about the new RedHat EL 4 release process I was greatly amused by some points about their kernel release manager, as to how committed RedHat are to share their QA process with the kernel mainline, unlike other vendors:
During the past year, more than 4,100 patches from Red Hat employees were integrated into the upstream 2.6 kernel. In contrast other companies boast that their offering contains the most patches on top of the community kernel.
and
Upstream - doing all our development in an open community manner. We don't sit on our technology for competitive advantage, only to spring it on the world as late as possible.
These statements are commendable, and reflect some of my own thoughts but one can make some points:
  • The issue is not that other distributions do not publish their patches, because they do, and in a timely manner, thanks to the GPL, but that they do not actively submit them upstream, and since upstream is busy enough, upstream people do not go out of their way to scout for patches, but only deal with those that are actively submitted.
    I think this is a very important observation on the actual process of free software development, which is totally dominated by scratch my itch logic.
  • My perception is that RedHat used to be as reluctant to actively contribute the results of their QA to upstream as the others. It is very welcome that their official position is now different.
    However, I think that the kernels in their products are still very different from upstream kernels because of the number of feature patches they use to differentiate it from upstream. Yes, it is good that differentiation is not based on the number of semi-hoarded fix patches, but they are still semi-forking the kernel.
  • I think there is little competitive advantage to be had from being reluctant to contribute QA to the upstream kernel for Linux companies; competitors, unlike upstream, can always proactively fetch those patches, so not actively contributing them is rather futile.
    I reckon that semi-hoarding patches instead benefits the kernel maintainers by enhancing their job security. After all a policy of having a kernel that is kept quite different from the upstream kernel implies the need for a team to maintain it instead of relying on upstream and community maintenance.
    Most of RedHat's competitors have already done rounds of downsizing, in the middle of a big IT recession, and probably their remaining employees are very keen to make themselves as indispensable as possible (and think of Mandrake who have just acquired Conectiva, that is a team of distribution developers that cost one third per year and are as good).
    Job security through obscurity (whether of source or of process) is a winner in many cases, and every little helps, unfortunately :-(.
050513
As an ulterior proof of the frequently useless sensationalism of SlashDot there has been a big discussion about this apparent discovery that it is hard to guarantee something has actually been written to disc. This has been well known for years at least to those reading the comp.arch newsgroup. Things also are much subtler and more complex than apparent from this late discovery.
To me the most interesting question is why if disk write caching is disabled then sequential write performance goes down typically by a factor of as much as ten (e.g. from 40MB/s to 4MB/s).
050512
After an article about the evolution of GNOME I have had a look at the GNOME memory reduction project which has comments like:
The plan is to reduce the amount of memory that Gnome applications consume. Gnome is barely usable on a machine with 128 MB of RAM; contrast this with Windows XP, which is very snappy on such a configuration.
Even more amusing is the question that comes next:
Why do you want to reduce memory consumption?
to which some answers are given. Unfortunately none seems convincing to me; I reckon that there is a vanishingly small and largely powerless constituency for fixing the many and horrifying memory wastages in most GNU/Linux applications, so the answers are merely pious hopes.
GNU/Linux projects are based on the volunteer side on the scratch my itch principle, and many volunteers by now just have very large PCs, with at least 1GB of memory.
Also many volunteers are now richly paid employees of big corporates instead; for example this article on BusinessWeek.com points out that:
Looking at the top 25 contributors to the Linux kernel today, you'll discover that more than 90% of them are on the corporate payroll full-time for companies such as HP (HPQ), IBM, Intel (INTC), Novell (NOVL), Oracle, Red Hat (RHAT) and Veritas (VRTS), among many others.
and obviously these big corporations are not limiting their highly paid employees with PCs having only 128MB RAM, and I would imagine that most of them have no interest whatsoever in wasting their expensive time minimizing memory consumption in the kernel or in applications. The employees themselves are now paid large enough salaries that buying more memory even for their personal system is simply no longer an issue for them.
To them high memory usage is not an itch that has to be scratched other than by buying more memory; adding transparent windows and other cool effects seem to be high priority itches, also perhaps because they help feature/demo driven career progression: try to imagine them wowing the manager who decides their raises and bonuses saying one of:
  • I spent the last month halving the memory used by these applications
  • I spent the last year adding to this application these snazzy transparency effects and ten more cool features as you can see in this demo
As I have remarked before some virtual console applications have memory footprints of 5 megabytes (optimistically: my current Konsole processes have resident set sizes of over 30 megabytes, of which only 5-8 are shared), that is more than one thousand (4KiB) pages, when all they do 99% of the time is to receive a string of a few characters and send a request to BLIT them on the screen.
Also, the hideous memory usage is a system problem; most applications, on their own, fit into 128MB systems, even virtual terminals that have 30MB memory footprints. It is when several are used together that laughably large memory footprints have an impact. But the author of each one has an easy defense of it is fine by itself, and in any case what really matters is the demo, not actual use in a loaded system.
As to the merits of what is causing enormous aggregate memory footprints, here are some of my guesses, corroborated by quite a bit of observation:
Benchmarks and lack of visibility
Most popular benchmarks are pure speed benchmarks and can be run on systems with enough memory that the whole application being benchmarked is memory resident. Also, most popular benchmarks are application benchmarks, and not system benchmarks, and that's another reason why the whole application can be memory resident.
There are also some intrinsic problems with (mostly nonexistent) sysem and memory benchmarks:
  • A memory benchmark for a system is intrisically less repeatable and thus impressive a fact than a speed benchmark for a single application.
  • Creating a realistic system and memory benchmark and running it takes a lot more effort than a synthetic or single application speed benchmark.
Large pages
I have seen very persuasive research evidence that for unoptimized programs the optimal page size is 256 bytes, and with pages sizes above 1024 bytes the number of pages in the working set is roughly constant; in other words a 4096 byte page size means that other things being equal the working set of a program is four times larger in bytes than with a 1024 byte page size. Larger page sizes amplify considerably the impact of the other issues.
Use of shared libraries
Shared libraries mean that any use of a function on a page brings in the whole page. Bad news if functions in a shared library are not clustered with respect to expected dynamic usage. Usually functions are clustered in random order or in alphabetic order...
Use of many shared libraries with lots of impure data
Shared libraries contain a significant amount of inpure (writable) pages typically containing either external (interlibrary) links or internal (intralibrary) links. Internal links contain mostly relocation entries.
When any internal or external link on a library is fixated the entire page on which it resides it must be duplicated for the process in which it is. As this page says about a GNOME Hello world! program:
That simple program uses 73 shared libraries that allocate a total of 13Mb of non-shared static data.
This can be alleviated in the following ways:
  • Better library and language design.
  • Prerelocating libraries to a nominal address range, and prelinking them (KDE does a bit of this).
  • Forking processes that use the same set of libraries from a master one that has them already prerelocated and linked in (KDE does this too by default).
  • Generating PIE (position independent code) which eliminates most of the intra library links, at the price of higher register usage usually.
  • Merging libraries together.
Bad memory allocators
Many common memory allocators are designed to maximize memory usage and resident sets, for example one or more of:
  • Use of buddy-system logic, which tends to overallocate memory by about 30%. But it is fast, so benchmarks on infinite memory reference PCs are flattered.
  • Extensive list walks to find or free allocated blocks, so they touch a lot of pages.
  • No attempt to cluster block allocations.
  • Inability to coalesce freed blocks and to release back large chunks of memory to the operating system, which does not directly impact resident sets, but indirectly via larger virtual memory mapping tables.
Careless program designs
Many programs embody deep and thin flow trees, where going from event to event resolution can involve dozens of nested procedures, each of which is often on a different page from the next.
A legendary case if the reference X server implementation: most of the time the X server just receives bitmaps to BLIT on the screen, so only one or two pages should be touched; but to get from reading the BLIT request to the BLIT code that satisfies it may involve a very deep and dispersed call sequence, and so very many pages are kept constantly referenced.
Some programs also involve maximization of data, not just code, memory resources. For example client side fonts imply that each X client needs to have a list and cache of rendered glyphs, which it constantly references, instead of having them in the graphics card's framebuffer memory, as it happens with server side fonts.
Also, a lot of cool graphics tricks require a lot of memory, for example to handle screen damage events (for example background pictures or transparency or non rectangular shapes).
050510
Interesting discovery on some details of font systems. The X11 bitmap font module (for PCF format bitmap fonts) is loaded by default and this is unfortunate because it has a misdesign in which it forces the DPI of fonts to be either 75 or 100 only. Omitting it from the list of modules to load is no good.
However I found that if the freetype font module (for PCF, TrueType and Type1 fonts and a few other types too) is specified before bitmap, it registers itself for PCF fonts and seems to preempt the bitmap module, which seems an acceptable workaround.
050503
I have discovered that some nice people have produced two fairly high quality repackagings in Type1 of the Computer Modern typefaces with fairly standard (not TeX) encodings (1, 2).
The on screen quality seems fairly good (particularly for Latin Modern), and probably better than that of the Bitstream Vera typefaces, which really require antialiasing to mask the lack of hinting.
The above only applies if the rasterization is left to the X11 freetype font module, and the type1 font module is not loaded. The freetype font module uses the FreeType library which seems to have a fairly good Type 1 rasterizer, one that does some decent autohinting (a side effect I guess of FreeType having to have an autohinter for TrueType fonts), while the original type1 module does not, and produces fairly crude low DPI bitmaps.
The font packages are the Latin Modern (there is a nice PDF paper describing them) and Computer Modern Unicode sets.

April 2005

050419
I have been a bit wondering why my external Firewire backup hard disk is so awkward to use, as in requiring such great care in the exact sequence of actions that don't result in OOPSes or hangs, and here are the not so good news from a kernel developer:
April 08, 2005
Pete Zaitcev: Thomas in a cage with Firewire
http://thomas.apestaart.org/log/index.php?p=291
Yay. Thomas is about to find why Firewire is unsupported even on Fedora (let alone on RHEL).
But If someone asked me what Firewire needed, I would answer, "only a hacker with a brain". He may be able to pull it off, though I'm not too optimistic. Firewire is about as complex as USB and we all know how well that goes despite a sustained effort by Greg K-H, David-B, Stern and myself. My approach was to hide whenever Firewire came too close and wait for it to die in the marketplace (which, I suspect, is inevitable at this point).
Now, my external hard drive box also supports USB2, so I tried that. Bad news! It does not work at all with the usb-storage driver, presumably because it uses a chipset (ALi) that is designed to the MS USB driver interface, not the official one. It works, very slowly, with the slow device ub driver, but this anyhow only supports up to 138GB drives, and my drive has a 160GB capacity.
050418
Cleaning up things in my host and user configurations, I have been as usual feeling very disappointed with client side fonts particularly as implemented in Fontconfig/Xft2 (and Xft1 is best forgotten).
Client side fonts are not a good idea in general for X:
  • Architecturally, client side fonts don't fit with the X architecture: X is designed so that a single host may support multiple X servers, a single X server may have multiple screens with completely different characteristics, a process may be connected to multiple X servers on multiple hosts, as well as a single X server being used by many processes from different hosts. X is designed to support this flexibility this not quite as transparently as it should, but at least with some display independence, where font rendering is delegated to the server, which is what drives directly the specific target screens.
    A process connected to multiple X displays or screens of the same X display need to have multiple font rasterizer contexts if it uses client side fonts. With server side fonts all the process has to do is to send strings and they will be rendered optimally for the specific target display, no matter its characteristics, which host it is connected to, and so on.
  • From a performance point of view client side fonts are bad now and will be even worse in the future. Their only performance advantage is in some modest win as to protocol latency: since the rendering of glyphs happens on the client side, it knows exactly the geometry of the rendered text, and the X server does not need to send back font metrics; so, it is possible to contrive a demo in which gives a small overall advantage in a very special case.
As to the client side performance disadvantages, present and future:
  • Rasterization and caching of the rasters has to be performed in every single application connected to the same X server; if an application is connected to multiple X screens, it has to rasterize and cache the same glyphs multiple times. Also, almost always the majority of applications connected to an X server use the same fonts.
    Of course if memory and CPU time are in effect infinite this does not matter; and therefore it is almost unnoticeable for single processes, but unfortunately I have often several dozen processes running connected to one or two X servers with one or two screns with different characterics each.
  • Bandwidth is consumed by the sending of text rasters. This applies to multiple levels of the communication chain: even if the client and the X server are on the same host, with client side fonts he X server must rasterize the glyps every time to the display card, thus consuming AGP or PCI bus bandwidth.
    With server side fonts the X server can use all of the framebuffer that is not used for the screen as a glyph cache, making the rasterization of text almost entirely an onboard operation, saving on AGP and PCI bus bandwidth and latency.
  • Latency increases because each application has to first rasterize all the glyphs in a text strings, and then send the raster as a whole, and the X server can only start blitting it when it is fully received.
    With client side font rendering not only the X can cache the glyphs in the frame buffer, but the text of the string to render is received quickly, and then the X server can render that text in way of principle incrementally, glyph by glyph.
  • Sending pixmaps to blit instead of character strings to render currently is way less efficient than sending text strings in relative terms, but not that much in absolute terms because most current screens have rather low DPI values, so the bitmaps aren't that big.
    Unfortunately now that the world has switched to LCD screens higher DPIs are becoming more common, and higher DPI raise dramatically the size of the pixmaps to be sent, especially if they are antialiased, as this both increases the size of the pixmap and reduces the effectiveness of compression, if any; but then antialiasing is fortunately less necessary on high DPI displays.
    In any case rasterizing text over the wire on a 1280x1024 pixel display is not quite the same as on a 4000x3000 display.
    The growing size of client side text pixmaps impact performance severely in several ways:
    • Each client has to spend that much more CPU time to create bigger glyph rasters, and that much more RAM to cache them.
    • The bandwidth needed to send the text rasters increases considerably, both at the network level or at the AGP/PCI/... bus level.
    • The latency also increases.
These issues apply to any strictly client side scheme, but Fontconfig/Xft2 have their own specific problems, for example:
  • A considerable lack of documentation. For example the Fontconfig font specification is not that clearly defined anywhere, and there are several obscurities in the specification of the Fontconfig configuration files; and the Fontconfig configuration files are a classic example of moronic abuse of text markup, but then this is a much wider issue, admittedly.
  • A number of gross misdesigns have been made in the implementation of Fontconfig; version 1 for example did not even cache font descriptions, and version 2 does, but in a rather clumsy way. After quite a wait now Fontconfig maps font files instead of copying them into memory, but of course this only works for filesystems that support mmap efficiently.
  • Each client process must depends on a number of additional shared libraries, and since different clients will be executing different parts of the shared library at any one time, the memory residence footprint will be way large than if those shared libraries were in use just by the X server.
  • Each client process has a private glyph cache, and I am not aware of any user level mechanism to influence the caching policy, including the minimum and maximum size of the cache; perhaps this is possible, but I am pretty sure most people do not use it. Instead cache policy settings are obvious in server side embedded and standalone font servers.
  • It seems impossible to specify which precise font one wants to select, in particular I have been unable to ensure that the PCF version of Helvetica is selected instead of the Type 1 version. This may be due to lack of documentation or my misunderstanding, but I suspect not.
050409
While sorting out my enormous archive of articles and papers I found some old (around 1990) UNIX for PC software pricelists, with several entertaining items:
  • ESIX Unix System V dev system multi user US$1499, std system 2 users US$619.
  • UNIX System V TCP/IP US$359.
  • UNIX System V Windowing runtime US$219, dev sys US$599.
and so on (some scans, including some modem prices: 1, 2, 3). Over the years I have bought, and I still have the installation media, Microport System V.2 for the 286, then ESIX System V.3 (ESIX was a brand from Everex), and finally Dell System V.4, before switching to GNU Linux.
050408
Just found a fairly nice page on the more important design patterns.
It looks like to me the usual confusion between design, patterns, algorithms, methods, recipes and pious homilies.
So I am reminded that they may be considered a buzzword bandied by some to make their employers or customers believe they possess some kind of secret sauce that makes their work tastier and less fattening at the same time.
Even more usefully, like most previous software fads, their underlying promise is that they give managers more powers over their minions by deskilling them; that is enables hiring low skilled labourers telling them to follow a set of simple principles or rules obediently and dumbly, as long as they they are the secret sauce ones.
As to the those of these patterns that seem actually design oriented they tend to be trivializations of simple databased design rules.
There is of course a benefit, small but significant, to the talk about patterns: whatever they are, they name recurrent and topical practices, whether related to design or not, and this might sometimes improve communication with and between otherwise unskilled people; which may be of benefit given the industry tendency towards employing unskilled workers.
050407
I have been pleasantly surprised with KIAX an IAX2 protocol based software phone. It feels much better designed and reliable than either Linphone or KPhone, as decribed more extensively in my (updated) more specific comments on them.
050405
I have been experimenting with Asterisk which is an IP and PSTN telephony multiprotocol switch program, and to thsi effect I have been experimenting with both some GNU/Linux based software phones and the Gradwell VoIP services.
As to the Gradwell VoIP services, they have a somewhat attractive IAX2 based service, by which those who run an Asterisk instance can just register with an Asterisk server at Gradwell's usingn it to switch calls from and to the PSTN.
The service charge is just a £1+VAT registration fee plus (low) call charges. However, this is not enough to use their PSTN gateway service, because to use in the outgoing direction one must supply an authenticated caller ID number, which must be either a manually authenticated PSTN number already assigned to the user, and this manual authentication costs £35+VAT, or one already supplied to the same customer by Gradwell themselves. Gradwell can register a block of 10 numbers for £3+VAT per month (minimum term 3 months) or a single number with a SIP account for £4+VAT per month, first three months free (mininum term 3 months).
So in practice one has to register for the IAX2 services, and then either pay a once only but fairly high fee, or register for a monthly-fee service. Since I wanted to experiment with SIP telephony anyhow I have registered for the SIP account.
Readily available to me for Debian are two SIP software phones, Linphone and KPhone.
They are both half-finished, mostly working, mostly undocumented; they are sort of useful but shoddy.
Of the two it is easier to get Linphone to work, but KPhone seems more reliable and lightweight; I have written up some specific comments.
050404
Another day, another great Debian script annoyance: I have installed zeroconf a nice little utility which is like a DHCP client, but without the server, in that it can configure a network interface automagically a bit like for IPv6, but (usually) in the IPv4 169.254.0.0/16 range reserved for that.
So far it is good and nice, but the Debian zeroconf package installs also /etc/network/if-up.d/zeroconf-up which is a script that runs the application every time an interface is activated, whether or not I want it. Allegedly autoconfiguration should always work, but there are cases where I simply want to bring up an interface in a fully passive way.
However, I have had a look at the various /etc/network/if-X.d/zeroconf-up directories and the scripts therein, and as usual the Debian Way is to try to do things automagically and in ways to that me feel rather shoddy.
Having something like zeroconf (or IPv4LL is definitely a nice idea, having it started by default is not nice. But this pales compared the the horror I feel when updating a package containing a daemon starts the daemon even if I have disabled its activation in the runlevel configuration.