I grew up with Midnight Commander, and I’m a creature of habit, so I set about trying to compile a copy of mc statically, so I only had to do it once for all the x86_64 clusters I deal with (and incidentally dodge the problems of differing versions of dependency libraries, lack of development libraries, and so on).  Turns out that’s not so simple.  One would hope that

./configure --disable-shared --enable-static

would do it.  Not so much, at least not with mc.  The mc binary would still be huge, but would still contain dynamic links to glib, pcre, curses, and a host of other things.

Various mailing list posts circa 2002 were unhelpful, especially those of the form “edit the Makefiles and add -static to LIBS“.  Though a scientist, after a few hours the scientific process breaks down and I make more than one change at once.  In other words, the following, while it worked, probably contains thoroughly unnecessary duplication of flags:

./configure --with-glib-static --without-x --disable-shared --enable-static \
    CC='gcc -static -static-libgcc -fno-exceptions' \
    CXX='g++ -static -static-libgcc -fno-exceptions' \
    LDFLAGS='-Wl,-static -static -lc' \
    LIBS='-lc'

The keys to success were:

1. Specifying CC, CXX, LDFLAGS, and LIBS as command-line arguments to configure rather than environment variables.  In the latter case, configure only seems to honor them occasionally.
2. Putting the static linkage options into what will run as the compiler.
3. When -static-libgcc is specified, an -fno-exceptions option may be necessary to avoid pulling in stack unwinding code that seems to be available only in a shared library.
4. Adding LIBS='-lc', because otherwise LDFLAGS='-Wl,-static' seemed to cause the linker to miss the crt0 startup routines, which a tailing '-lc' fixes.

That generated a completely static executable.  Installation wasn’t terribly straightforward, since I didn’t want to install a second copy of mc on my development box (even in my home directory), tar it up, and then clean it up.  (I don’t claim this is logical; just that I simply didn’t want to do it after spending far too long trying to get a static executable.)   I wound up pulling everything from my (properly installed, local) copy of mc, both from /etc/mc and /usr/share/mc, pooling it in one directory (~/etc/mc) on the clusters, and pointing mc to it with MC_DATADIR=$HOME/etc/mc (via an export MC_DATADIR=... in my .bashrc).

It should be noted that this hack depends on the presence of static link libraries for glib, ncurses/slang, and perhaps pcre.  That took a little doing as well (since my development box runs Gentoo, that meant twiddling with USE flags and re-emerging some packages).

But (who knew?) an Xorg server version bump might break various other modules, especially driver modules. A quick search of the Gentoo Forums gave a particularly elegant one-liner:

emerge -av1 $(qlist -I -C x11-drivers)

which reads, “emerge, verbosely and asking for confirmation, any installed package from the category x11-drivers”.  qlist is part of portage-utils.

That would include display drivers, but stripping those out left x11-drivers/xf86-input-evdev (and -keyboard and -mouse, for good measure).  My keyboard and mouse immediately worked again.

*headdesk*

1. To record — for posterity — all the dirty hacks clever solutions I came up with in the course of systems administration and programming projects.
2. To make those publicly available, since it seemed that every problem I wound up solving was usually a pain and almost never documented clearly in a manual, man page, mailing list, blog, &c.

And for almost two years, I’ve not run across things which have driven me sufficiently crazy to cause me to take time out to post.  But it seems time to make a return.

A quick look at the output of hg clone --help reveals the following:

• Running hg clone copies the tracked files and hard links the metadata (.hg directory).
• hg clone --pull copies everything and hard links nothing.
• cp -al copies nothing and hard links everything, which means your editor had better break hard links or else changes will be saved in two repositories at once.

In other words, an invocation of hg clone followed by editing some tracked files will always do the right thing, regardless of what your editor does.  Since I always clone repositories using hg clone (disk space isn’t too tight at the moment), it was never an issue, even using a rather complex Java IDE.

That got me wondering, though, what would have happened if I cloned using cp -al. It turns out that by default,

• emacs breaks hard links.  Can be overridden by adding (setq backup-by-copying-when-linked t) to one’s “.emacs” file.
• vim preserves hard links.  Can be overriden by set bkc=no in .vimrc or at the vim command line
• Eclipse preserves hard links.  This was tested empirically rather than looked up in documentation, and there isn’t an easily-located setting to change this.
• OS X TextEdit breaks hard links.  Also tested, and I didn’t even look for a way to change it.

The moral of the story:  when in doubt, hg clone.

The core of the system is Python-based, with database handling care of SQLAlchemy and a web interface using Pylons. Pylons is a mixed blessing. It’s relatively lean and eminently customizable, but it has the feel of a large number of developing (i.e. beta) projects glued together with chewing gum. They’re very high-quality beta, and the chewing gum is well-placed. For the most part, it’s better than inventing or re-inventing a web framework. My main gripe is that the documentation is scattered everywhere, and the bulk of Pylons-specific documentation is not a reference work, but a set of FAQs and cookbook recipes. That said, it’s the least intrusive framework I investigated, and I’m sticking with it for now. If I have spare time, perhaps I’ll contribute some development effort or some documentation.

The question of the day relates not to Pylons but SQLAlchemy. I’m working on a feature enhancement which will require some minor database restructuring, which raises the broader issue of schema migration. So far, there have only been beta deployments of this software, so hand-coded migration scripts have been much handier than anything else. As soon as the package is released to the project director’s collaborators, it’ll need to be a more automatic process.

So, just having gone public on this blog, and not knowing if there will be any readership, I pose the question: what are people’s experiences with schema migration? I’m poking around sqlalchemy-migrate. It seems well-thought-out, but I have two concerns:

• The version management is not exactly lightweight. I currently have a one-row entry in the DB for schema version. I’d rather not have to add another table to the DB, or another entire tree in the source code.
• The fewer dependent packages I have to make scientific end-users install, the less trouble support will be. It’d be great not to have to include another package in the installation instructions, and it’s one less thing that can break.

So, as I said…comments on experiences with schema migration would be helpful.

The attribute in question is “com.apple.quarantine”, as shown below:

$ ls -l@
total 1560
drwxr-xr-x  25 mzwier  staff     850 Jul 22 18:04 html40
-rw-r--r--@  1 mzwier  staff  369830 Jul 22 18:00 html40.tgz
	com.apple.quarantine	    42

The tool to manage extended attribute data is (logically), “xattr”. xattr has no man page, but an informative-enough help option (this directly from xattr --help, reprinted for reference and discussion):

$ xattr --help
usage: xattr [-l] file [file ...]
       xattr -p [-l] attr_name file [file ...]
       xattr -w attr_name attr_value file [file ...]
       xattr -d attr_name file [file ...]

The first form lists the names of all xattrs on the given file(s).
The second form (-p) prints the value of the xattr attr_name.
The third form (-w) sets the value of the xattr attr_name to attr_value.
The fourth form (-d) deletes the xattr attr_name.

options:
  -h: print this help
  -l: print long format (attr_name: attr_value)

So, to lift the quarantine on a specific file, the proper move is

xattr -d com.apple.quarantine FILE

To lift the quarantine on a whole directory tree (say, of documentation), the move is

find DIRNAME -print0 | xargs -0 xattr -d com.apple.quarantine

piggybacking on standard tricks with find and xargs.

#!/bin/bash
test -e /Volumes/MCZ &&
  rsync -avr --delete /Volumes/MCZ $HOME/Metabackup/ >& $HOME/Metabackup/MCZ.log

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>KeepAlive</key>
        <false/>
        <key>Label</key>
        <string>znet.backups.mczthumb</string>
        <key>ProgramArguments</key>
        <array>
                <string>/bin/bash</string>
                <string>/Users/mzwier/bin/backup-thumb.sh</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>StartCalendarInterval</key>
        <dict>
                <key>Minute</key>
                <integer>52</integer>
        </dict>
</dict>
</plist>

This is exactly equivalent to the cron line

52 * * * * /bin/bash /Users/mzwier/bin/backup-thumb.sh

I saved the plist in ~/Library/LaunchAgents (as znet.backups.mczthumb) and ran launchctl:

launchd% load /Users/mzwier/Library/LaunchAgents/znet.backups.mczthumb

Worked on the first try. It may be useful to perform an update every time the drive is mounted, but that’s a challenge for another day.