Compile static copy of Midnight Commander

I work with quite a few clusters, and in trying to get batch jobs running, I tend to generate many tiny files which are usually almost identical and almost useless.  The rare exception is the lone non-identical file, which is invariably critical in solving a problem.  Run a test job ten or fifty times, and a directory can fill up rapidly.  ls and rm get annoying for large directories.  So, I like to have a text-mode file manager on hand to help sort through, mark, and blow away useless cruft while identifying useful cruft.

I grew up with Midnight Commander, and I’m a creature of habit, so I set about trying to compile a copy of mc statically, so I only had to do it once for all the x86_64 clusters I deal with (and incidentally dodge the problems of differing versions of dependency libraries, lack of development libraries, and so on).  Turns out that’s not so simple.  One would hope that

./configure --disable-shared --enable-static

would do it.  Not so much, at least not with mc.  The mc binary would still be huge, but would still contain dynamic links to glib, pcre, curses, and a host of other things.

Various mailing list posts circa 2002 were unhelpful, especially those of the form “edit the Makefiles and add -static to LIBS“.  Though a scientist, after a few hours the scientific process breaks down and I make more than one change at once.  In other words, the following, while it worked, probably contains thoroughly unnecessary duplication of flags:

./configure --with-glib-static --without-x --disable-shared --enable-static \
    CC='gcc -static -static-libgcc -fno-exceptions' \
    CXX='g++ -static -static-libgcc -fno-exceptions' \
    LDFLAGS='-Wl,-static -static -lc' \
    LIBS='-lc'

The keys to success were:

1. Specifying CC, CXX, LDFLAGS, and LIBS as command-line arguments to configure rather than environment variables.  In the latter case, configure only seems to honor them occasionally.
2. Putting the static linkage options into what will run as the compiler.
3. When -static-libgcc is specified, an -fno-exceptions option may be necessary to avoid pulling in stack unwinding code that seems to be available only in a shared library.
4. Adding LIBS='-lc', because otherwise LDFLAGS='-Wl,-static' seemed to cause the linker to miss the crt0 startup routines, which a tailing '-lc' fixes.

That generated a completely static executable.  Installation wasn’t terribly straightforward, since I didn’t want to install a second copy of mc on my development box (even in my home directory), tar it up, and then clean it up.  (I don’t claim this is logical; just that I simply didn’t want to do it after spending far too long trying to get a static executable.)   I wound up pulling everything from my (properly installed, local) copy of mc, both from /etc/mc and /usr/share/mc, pooling it in one directory (~/etc/mc) on the clusters, and pointing mc to it with MC_DATADIR=$HOME/etc/mc (via an export MC_DATADIR=... in my .bashrc).

It should be noted that this hack depends on the presence of static link libraries for glib, ncurses/slang, and perhaps pcre.  That took a little doing as well (since my development box runs Gentoo, that meant twiddling with USE flags and re-emerging some packages).

Keyboard and mouse broken after xorg-server update

So, in a case of RTFM (or perhaps RTFF, read the f-ing forum), keyboard and mouse on my home router were out of commission after an update of Xorg server.  I’d even run revdep-rebuild without any library name to try to sort out exactly what was meant by the “You MUST run revdep-rebuild after this update” message the dbus update gave me.  Everything checked out clean.

But (who knew?) an Xorg server version bump might break various other modules, especially driver modules. A quick search of the Gentoo Forums gave a particularly elegant one-liner:

emerge -av1 $(qlist -I -C x11-drivers)

which reads, “emerge, verbosely and asking for confirmation, any installed package from the category x11-drivers”.  qlist is part of portage-utils.

That would include display drivers, but stripping those out left x11-drivers/xf86-input-evdev (and -keyboard and -mouse, for good measure).  My keyboard and mouse immediately worked again.

*headdesk*

Been a while

So, almost two years ago I created this blog for two purposes:

1. To record — for posterity — all the dirty hacks clever solutions I came up with in the course of systems administration and programming projects.
2. To make those publicly available, since it seemed that every problem I wound up solving was usually a pain and almost never documented clearly in a manual, man page, mailing list, blog, &c.

And for almost two years, I’ve not run across things which have driven me sufficiently crazy to cause me to take time out to post.  But it seems time to make a return.

Mercurial, Eclipse, Hard Links

For one of my major projects, I use Mercurial for version control, and Eclipse with PyDev for my primary development environment.  I frequently clone a development trunk to create new development branches (usually one per feature set I’m working on at a time).  I had a little hiccup of paranoia about what it means to clone a repository, since at least some of the newly-cloned files are not physical copies, but hard links.  And I didn’t know what Eclipse did with hard links.

A quick look at the output of hg clone --help reveals the following:

• Running hg clone copies the tracked files and hard links the metadata (.hg directory).
• hg clone --pull copies everything and hard links nothing.
• cp -al copies nothing and hard links everything, which means your editor had better break hard links or else changes will be saved in two repositories at once.

In other words, an invocation of hg clone followed by editing some tracked files will always do the right thing, regardless of what your editor does.  Since I always clone repositories using hg clone (disk space isn’t too tight at the moment), it was never an issue, even using a rather complex Java IDE.

That got me wondering, though, what would have happened if I cloned using cp -al. It turns out that by default,

• emacs breaks hard links.  Can be overridden by adding (setq backup-by-copying-when-linked t) to one’s “.emacs” file.
• vim preserves hard links.  Can be overriden by set bkc=no in .vimrc or at the vim command line
• Eclipse preserves hard links.  This was tested empirically rather than looked up in documentation, and there isn’t an easily-located setting to change this.
• OS X TextEdit breaks hard links.  Also tested, and I didn’t even look for a way to change it.

The moral of the story:  when in doubt, hg clone.

Schema management with SQLAlchemy

At work I’m developing a database system for tracking quantum mechanics calculations. It’s been done before, but a) this focuses more on job tracking than job submission and analysis, b) it’s really more about the command-line side of things with a web-based search, c) it’s more about some home-grown dynamics calculations than general-purpose “QM for the masses”, and d) it’s what a professor hired me to do, so I don’t ask questions.

The core of the system is Python-based, with database handling care of SQLAlchemy and a web interface using Pylons. Pylons is a mixed blessing. It’s relatively lean and eminently customizable, but it has the feel of a large number of developing (i.e. beta) projects glued together with chewing gum. They’re very high-quality beta, and the chewing gum is well-placed. For the most part, it’s better than inventing or re-inventing a web framework. My main gripe is that the documentation is scattered everywhere, and the bulk of Pylons-specific documentation is not a reference work, but a set of FAQs and cookbook recipes. That said, it’s the least intrusive framework I investigated, and I’m sticking with it for now. If I have spare time, perhaps I’ll contribute some development effort or some documentation.

The question of the day relates not to Pylons but SQLAlchemy. I’m working on a feature enhancement which will require some minor database restructuring, which raises the broader issue of schema migration. So far, there have only been beta deployments of this software, so hand-coded migration scripts have been much handier than anything else. As soon as the package is released to the project director’s collaborators, it’ll need to be a more automatic process.

So, just having gone public on this blog, and not knowing if there will be any readership, I pose the question: what are people’s experiences with schema migration? I’m poking around sqlalchemy-migrate. It seems well-thought-out, but I have two concerns:

• The version management is not exactly lightweight. I currently have a one-row entry in the DB for schema version. I’d rather not have to add another table to the DB, or another entire tree in the source code.
• The fewer dependent packages I have to make scientific end-users install, the less trouble support will be. It’d be great not to have to include another package in the installation instructions, and it’s one less thing that can break.

So, as I said…comments on experiences with schema migration would be helpful.

Un-quarantine downloaded files on OS X

Though I agree in principle with the idea of marking downloaded files as hazardous, it can be quite annoying when, say, extracting piles and piles of W3C documentation for future reference — especially when opening some index.html pops up a dialog box about a downloaded application. The “downloaded application” marker OS X puts on downloaded files is in fact an extended attribute.

The attribute in question is “com.apple.quarantine”, as shown below:

$ ls -l@
total 1560
drwxr-xr-x  25 mzwier  staff     850 Jul 22 18:04 html40
-rw-r--r--@  1 mzwier  staff  369830 Jul 22 18:00 html40.tgz
	com.apple.quarantine	    42

The tool to manage extended attribute data is (logically), “xattr”. xattr has no man page, but an informative-enough help option (this directly from xattr --help, reprinted for reference and discussion):

$ xattr --help
usage: xattr [-l] file [file ...]
       xattr -p [-l] attr_name file [file ...]
       xattr -w attr_name attr_value file [file ...]
       xattr -d attr_name file [file ...]

The first form lists the names of all xattrs on the given file(s).
The second form (-p) prints the value of the xattr attr_name.
The third form (-w) sets the value of the xattr attr_name to attr_value.
The fourth form (-d) deletes the xattr attr_name.

options:
  -h: print this help
  -l: print long format (attr_name: attr_value)

So, to lift the quarantine on a specific file, the proper move is

xattr -d com.apple.quarantine FILE

To lift the quarantine on a whole directory tree (say, of documentation), the move is

find DIRNAME -print0 | xargs -0 xattr -d com.apple.quarantine

piggybacking on standard tricks with find and xargs.

Automatic Backup of USB Drive

I’ve started to keep some regularly-updated critical information on my USB thumb drive. Apple’s Time Machine doesn’t seem to be willing to back up external disks (or at least VFAT-formatted external disks). I figured I’d piggyback on Time Machine’s hourly backups by using rsync to copy the thumb drive periodically to my MacBook’s hard drive. OS X 10.5 still supports cron, but I keep running across references to launchd. After some googling, I came up with the following script:

#!/bin/bash
test -e /Volumes/MCZ &&
  rsync -avr --delete /Volumes/MCZ $HOME/Metabackup/ >& $HOME/Metabackup/MCZ.log

and the following plist describing the new “service” I wanted launchd to handle:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>KeepAlive</key>
        <false/>
        <key>Label</key>
        <string>znet.backups.mczthumb</string>
        <key>ProgramArguments</key>
        <array>
                <string>/bin/bash</string>
                <string>/Users/mzwier/bin/backup-thumb.sh</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>StartCalendarInterval</key>
        <dict>
                <key>Minute</key>
                <integer>52</integer>
        </dict>
</dict>
</plist>

This is exactly equivalent to the cron line

52 * * * * /bin/bash /Users/mzwier/bin/backup-thumb.sh

I saved the plist in ~/Library/LaunchAgents (as znet.backups.mczthumb) and ran launchctl:

launchd% load /Users/mzwier/Library/LaunchAgents/znet.backups.mczthumb

Worked on the first try. It may be useful to perform an update every time the drive is mounted, but that’s a challenge for another day.