Computing notes 2017 part two

This document contains only my personal opinions and calls of judgement, and where any comment is made as to the quality of anybody's work, the comment is an opinion, in my judgement.

[file this blog page at: digg del.icio.us Technorati]

2017 July

170724 Mon: Different types of M.2 slots

As an another example that there are no generic products I was looking at a PC card to hold an M.2 SSD device and interface it to the host bus PCIe and the description carried a warning:

Note, this adapter is designed only for 'M' key M.2 PCIe x4 SSD's such as the Samsung XP941 or SM951. It will not work with a 'B' key M.2 PCIe x2 SSD or the 'B' key M.2 SATA SSD.

There are indeed several types of M slots and with different widths and speeds, supporting different protocols, and this is one of the most recent and faster variants. Indeed among the user reviews there is also a comment as to the speed achievable by an M.2 flash SSD attached to it:

I purchased LyCOM DT-120 to overcome the limit of my motherboard's M.2. slot. Installation was breeze. The SSD is immediately visible to the system, no drivers required. Now I enjoy 2500 MB/s reading and 1500 MB/s writing sequential speed. Be careful to install the device on a PCI x4 slot at least, or you will still be hindered.

Those are pretty remarkable speeds and much higher (peak sequential tranfer) than those for a memristor SSD.

170720 Thu: The O_PONIES summary and too much durability

So last night I was discussing the O_PONIES controversy and was asked to summarise it, which I did as follows:

There is the additional problem that available memory has grown at a much faster rate than IO speed, at least that of hard disks, and this has meant that users and application writers have been happy to let very large amounts of unwritten data accumulate in the Linux page cache, which then takes a long time to be written to persistent storage.

The comments I got on this story were entirely expected, and I was a bit disappointed, but in particular one line of comment offers me the opportunity to explain a particularly popular and delusional point of view:

the behavior I want should be indistinguishable from copying the filesystem into an infinitely large RAM, and atomically snapshotting that RAM copy back onto disk. Once that's done, we can talk about making fsync() do a subset of the snapshot faster or out of order.

right. I'm arguing that the cases where you really care are actually very rare, but there have been some "design choices" that have resulted in fsync() additions being argued for applications..and then things like dpkg get really f'n slow, which makes me want to kill somebody when all I'm doing is debootstrapping some test container

in the "echo hi > a.new; mv a.new a; echo bye > b.new; mv b.new a" case, writing a.new is only necessary if the mv b.new a doesn't complete. A filesystem could opportunistically start writing a.new if it's not already busy elsewhere. In no circumstances should the mv a.new operation block, unless the system is out of dirty buffers

you have way too much emphasis on durability

The latter point is the key and, as I understand them, the implicit arguments are:

In particular the requirement that metadata operations (on directories) should be individually durable as prescribed by POSIX, and file writes be made durable with fsync are both unnecessary.

As to that I pointed out that any decent implementation of O_PONIES indeed is based on the true and obvious point that it is pointless and wasteful to make a sequence of metadata and data updates persistent until just before a loss of non-persistent storage content and that therefore all that was needed were two systems internal functions returning the time interval to the next loss of non-persistent storage content, and to the end of the current semantically-coherent sequence of operations.

Note: the same reasoning of course applies to backups of persistent storage content: it is pointless and wasteful to make them until just before a loss of that persistent storage.

Given those O_PONIES functions, it would never be necessary to explicitly fsync, or implicitly fsync them in almost all cases, a sequence of operations like:

echo hi > a.new; mv a.new a; echo bye > b.new; mv b.new a

Because they would be implicitly made persistent only once it were known that a loss of non-persistent storage content would happen before that sequence would complete.

As simple as that!

Unfortunately until someone sends patches to Linus Torvalds implementing those two simple functions there should be way too much emphasis on durability because:

170716c Sun: Hardware specification of "converged" mobile phone

In the same issue of Computer Shopper as the review of a memristor SSD there is also a review of a mobile phone also from Samsung, the model Galaxy S8+ which has an 8 CPU 2.3GHz chip, 4GiB of memory, and 64GiB of flash SSD builtin. That is the configuration of a pretty powerful desktop or a small server.

Most notably for me it has a 6.2in 2960×1440 AMOLED display, which is both physically large and with a higher pixel size than most desktop or laptop displays. There have been mobile phones with 2560×1440 displays for a few years which is amazing in itself, but this is a further step. There are currently almost no desktop or laptop AMOLED displays, and very few laptops have pixel sizes larger than 1366×768, or even decent IPS displays. Some do have 1920×1080 IPS LCD displays, only a very few even have 3200×1800 pixel sizes.

The other notable characteristic of the S8+ is that also given its processing power is huge it has an optional docking station that allows it to use an external monitor, keyboard and mouse (most likely the keyboard and mouse can be used anyhow as long as they are Bluetooth ones).

This is particularly interesting as desktop-mobile convergence (1, 2, 3) was the primary goal of the Ubuntu strategy of Canonical:

Our strategic priority for Ubuntu is making the best converged operating system for phones, tablets, desktops and more.

Earlier this year that strategy was abandoned and I now suspect that a significant motivation for that was that Samsung was introducing convergence themselves with Dex, and for Android, and on a scale and sophistication that Canonical could not match, not being a mobile phone manufacturer itself.

170716b Sun: Hardware specification of smart watch and related

In the same Computer Shopper UK, issue 255 with a review of a memristor SSD there is also a review of a series of fitness-oriented smartwatches, and some of them like the Samsung Gear S3 have 4GB of flash storage, 768MiB of RAM and a two CPU 1GHz chip. That can run a server for a significant workload.

170716 Sun: Miracles do happen: memristor product exists

Apparently memristor storage has arrived as Computer Shopper UK, issue 255 has a practical review of Intel Intel 32GB Optane Memory in an M.2 form factor. That is based on the mythical 3D XPoint memristor memory brand. The price and specifications have some interesting aspects:

The specifications don't say it, but it is not a mass storage device, with a SATA or SAS protocol, it is a sort of memory technology device as far as the Linux kernel is concerned, and for optimal access it requires special handling.

Overall this first memristor product is underwhelming: it is more expensive and slower than equivalent M.2 flash SSDs, even if the read and write access time are much better.

170712 Wed: Special case indentation for PSGML

To edit HTML and XML files like this one I use EMACS with the PSGML library, as it is driven by the relevant DTD and this drives validation (which is fairly complete), and indentation. As to the latter some HTML elements should not be indented further, because they indicate types of speech, rather than parts of the document, and some do not need to be indented at all, as the indicate preformatted text.

Having looked at PSGML there is for the latter case a variable in psgml-html.el that seems relevant:

	sgml-inhibit-indent-tags     '("pre")

but is is otherwise unimplemented. So I have come up with a more complete scheme:

(defvar sgml-inhibit-indent-tags nil
  "*An alist of tags that should not be indented at all")

(defvar sgml-same-indent-tags nil
  "*An alist of tags that should not be indented further")

(defun sgml-mode-customise ()

  (defun sgml-indent-according-to-level (element)
    (let ((name (symbol-name (sgml-element-name element))))
      (cond
	((member name sgml-inhibit-indent-tags) 0)
	((member name sgml-same-indent-tags)
	  (* sgml-indent-step
	    (- (sgml-element-level element) 1)))
	(t
	  (* sgml-indent-step
	    (sgml-element-level element)))
      )
    )
  )
)

(if-true (and (fboundp 'sgml-mode) (not noninteractive))
  (if (fboundp 'eval-after-load)
     (eval-after-load "psgml" '(sgml-mode-customise))
     (sgml-mode-customise)
  )
)

(defun html-sgml-mode ()
  (interactive)
  "Simplified activation of HTML as an application of SGML mode."

  (sgml-mode)
  (html-mode)

  (make-local-variable			'sgml-default-doctype-name)
  (make-local-variable			'sgml-inhibit-indent-tags)
  (make-local-variable			'sgml-same-indent-tags)

  (setq
    sgml-default-doctype-name		"html"
    ; tags must be listed in upper case
    sgml-inhibit-indent-tags		'("PRE")
    sgml-same-indent-tags		'("EM" "TT" "I" "B" "CITE" "VAR" "CODE" "DFN" "STRONG")
  )
)

It works well enough, except that I would prefer the elements with tags listed as sgml-inhibit-indent-tags to have the start and end also not-indented, not just the content; but PSGML indents those as content of the enclosing element, so to achieve that would require more invasive modications of the indentation code.

170704 Tue: Fast route lookup in Linux with large numbers of routes

Fascinating report with graphs on how route lookup has improved in the linux kernel, and the very low lookup times reached:

Two scenarios are tested:

  • 500,000 routes extracted from an Internet router (half of them are /24), and
  • 500,000 host routes (/32) tightly packed in 4 distinct subnets.

Since kernel version 3.6.11 the routing lookup cost was 140s and 40ns, since 4.1.42 it is 35ns and 25ns. Dedicated "enterprise" routers with hardware routing probably equivalent. In a previous post amount of memory used is given: With only 256 MiB, about 2 million routes can be stored!.

As previously mentioned, once upon a time IP routing was much more expensive than Ethernet forwarding and therefore there was a speed case for both Ethernet forwarding across multiple hubs or switches, and for routing based on prefixes; despite the big problems that arise from Ethernet forwarding across multiple switches and the limitations that are consequent to subnet routing.

But it has been a long time since subnet routing has been as cheap as Ethernet forwarding, and it is now pretty obvious that even per-address host routing is cheap enough at least on the datacentre and very likely on the campus level (500,000 addresses is huge!).

Therefore so-called host routing can result in a considerable change of design decisions, but that depends on realizing that host routing is improper terminology and the difference between IP adddresses and Ethernet addresses:

Note: the XNS internetorking protocols used addresses formed of a 32 bit prefix as network identifier (possibly subnetted) and 48 bit Ethernet identifier which to me seems a very good combination.

The popularity of Ethernet and in particular of VLAN broadcast domains spanning multiple switches depends critically on Ethernet addresses being actually identifiers: an interface can be attached to any network access point on any switch at any time and be reachable without further formality.

Now so-called host routes turn IP addresses into (within The Internet) endpoint identifiers, because the endpoint can change location, from one interface to another, or one host to another, and still identify the service it represents.

170701 Sat: Fixing an Ubuntu phone after a power loss crash

So I have this Aquaris E4.5 with Ubuntu Touch, where the battery is a bit fatigued, so it sometimes runs out before I could recharge it. Recently when it restarted the installed Ubuntu system seemed damaged: various things had bizarre or no behaviours.

For example when I clicked on System Settings>Wi-Fi I see nothing listed except that Previous networks lists a number of them, some 2-3 times, but when I click on any of them, System Settings ends; no SIM card is recognized (they work in my spare identical E4.5), and no non-local contacts appear.

Also in System Settings when clicking About, Phone, Security & Privacy, Reset it just crashes.

Previous "battery exhausted" crashes had no bad outcomes, as expected, as battery power usually runs out when the system is idle.

After some time looking into it figured out that part of the issue was that:

$HOME/.config/com.ubuntu.address-book/AddressBookApp.conf.lock
$HOME/.config/connectivity-service/config.ini.lock

were blocking the startups of the relevant services, so removing them allowed them to proceed.

As to System Settings crashing, it was doing SEGV, so strace let me figure out that was for running out of memory just after accessing something under $HOME/.cache/, so I just emptied that directory, and then all worked. Some cached setting had been perhaps corrupted. I suspect that the cache needs occasional cleaning out.

Note: another small detail: /usr/bin/qdbus invokes /usr/lib/arm-linux-gnueabihf/qt5/bin/qdbus which is missing, but /usr/lib/arm-linux-gnueabihf/qt4/bin/qdbus works.