Monday, February 4, 2008

Writing good homepages, or why I am complaining today.

I re-discovered one of my pet peeves today while surfing around the Internets. Here is the setup: I work using a queue in Firefox. Whenever I see an interesting link in an article or whatever I am reading, I open it in a new tab to read later. The side effect of this is that I might start reading one article, open a new tab, and then read through the next five pages before getting to the newest tab. So of course I forget what I opened, and probably why, so it is sort of important that the new page be somewhat self-explanatory.

Then I read the Galleon page:



The first thing on the page, and incidentally the only content above the fold, is a changelog. And not even a useful one, at that: just some cryptic bugs that could be part of a video player, web browser, or fancy-shmancy version of Solitaire. So if you can't tell from the screenshot above what Galleon is (and I suspect that would be the case), it is an open-source replacement for TiVo desktop, letting you stream from your Linux or Mac (or Windows) machine to your TiVo, something that is otherwise reserved strictly for the Microsoft-running crowd.

The actual description of what Galleon does is at the bottom of the homepage. Also in that section is the features list, which is not included anywhere else in the site, nor is it mentioned in the navigation box on the left. So if you are like me, you get to the page and think "gee, now I have to navigate and figure out what this page is about any why I opened it..." Also, since changelogs usually live on their own, I was extremely perplexed as to where I might find a features page. That is, until I scrolled nearly two screens-full down the "changelog" page.

So here are the nits I have with the Galleon website:
  1. Nobody cares about the actual changelog if they are downloading from the website. Almost any modern project-management website package or hosting (like Trac or Google Code) will let you directly link in to your source tree. So if somebody cares to read the changelog, include a link in the navigation directly to the HEAD version of the changelog. Otherwise,

  2. Nobody scrolls on the first page. If you have separate sections of your website, then there is no reason to include important information below the fold on the homepage. Users know this, so they will not scroll.

  3. Summarize the point of your website, using the fewest words possible, using a large, attention-grabbing font directly on the homepage. See examples from Tumblr, Flickr (log out first, if necessary), ma.gnolia, and Jott. See anti-examples from Del.icio.us and of course Galleon.tv.

I don't want to be unfairly critical, but the simplest thing to keep in mind is to assume that a visitor is looking at your website with absolutely zero context. There are some special cases, one being "if you are here, you know why." I would actually say that del.icio.us might fall in to this category. You could also open the homepage of your local library and have a pretty good idea of how to use it. There are also patterns that are well-understood or self-explanatory. See digg, and to a lesser degree, reddit.

The ultimate question to ask about a homepage design is, "can I spend five seconds reading the homepage and decide whether I want to be at this website?" If you cannot get your message across that quickly, you are sending the wrong (oversized) message. And if you are not currently sending your message that quickly, then chances are you are sending too much background noise along with your signal.




(For some research background, see the Stanford Guidelines for Web Credibility (specifically #7 and its supporting research) and G Lindgaard, G Fernandes, C Dudek, J Brown's paper on fast first impressions (or its mention in the journal Nature).)

Tuesday, January 29, 2008

Awww come ON!

So I was going through some old email when I dug this one up:



From: me
To: ThinkGeek customer service
Subj: Das Keyboard II question

Hi,

Is Das Keyboard II available in a Dvorak layout, or do I have to pop the
keys off and switch them myself? My fear is that if I pop several off, I may
mix them up because they don't have any markings!

Thanks,
David


The product in question...


From: ThinkGeek customer service
To: me
Subj: Re: Das Keyboard II question

Thank you for your interest in our products, David! The Das Keyboard II is not available in a Dvorak layout as such, but you can use a third-party utility to remap the key layout without having to remove anything.

M*** S********
ThinkGeek Customer Service

Wednesday, January 23, 2008

Money really CAN buy anything!

This gem is courtesy the Purdue Usenet server...

From: redacted 
Newsgroups: purdue.forsale.misc
Subject: Grass-fed Goat Meat
Date: Sat, 19 Jan 2008 15:19:49 -0500
Organization: Purdue University

One 20-month-old Boer goat doe for sale. The doe weighs approximately
75 pounds which should yield between 37 and 45 pounds of packaged meat.
It was raised primarily on pasture/hay with small amounts of grain as
a supplement.

For more information on the benefits of grass-fed meat, see the
following links:
http://www.americangrassfed.org
http://news.ucanr.org/newsstorymain.cfm?story=531
http://www.eatwild.com/healthbenefits.htm

Options:
A) $100 live with free delivery in Lafayette
B) $175 butchered, you pick up packaged meat at the butcher (Monon
Meat Packing Co. in Monon, IN)
C) $190 butchered, we deliver packaged meat in the Lafayette area

Inquiries: (xxx) xxx-xxxx or redacted _at_ example.com


o_O

Ubuntu <⁄3 Mac OS X

After my last post, I was delighted to see this pop up today:



Connection Failed. Well, digging into the syslog on the Ubuntu machine shows this:

Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21034 exited 1
Jan 23 10:23:12 localhost afpd[21035]: ASIP session:548(5) from 192.168.1.111:53577(8)
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21035 done
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21036 exited 1
Jan 23 10:23:12 localhost afpd[21037]: ASIP session:548(5) from 192.168.1.111:53579(8)
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21037 done
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21038 exited 1
Jan 23 10:23:12 localhost afpd[21039]: ASIP session:548(5) from 192.168.1.111:53581(8)
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21039 done
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21040 exited 1
Jan 23 10:23:12 localhost afpd[21041]: ASIP session:548(5) from 192.168.1.111:53583(8)
Jan 23 10:23:12 localhost afpd[6312]: server_child[1] 21041 done

Hmm. No likey. So I did /etc/init.d/netatalk restart, and tried again. Connection Failed, again. Checking the syslog again, this is what I see:

Jan 23 10:26:45 localhost atalkd[21115]: restart (2.0.3)
Jan 23 10:26:46 localhost atalkd[21115]: zip_getnetinfo for eth2
Jan 23 10:27:06 localhost last message repeated 2 times
Jan 23 10:27:16 localhost atalkd[21115]: config for no router
Jan 23 10:27:17 localhost atalkd[21115]: ready 0/0/0
Jan 23 10:27:29 localhost afpd[21124]: Registering CNID module [last]
Jan 23 10:27:29 localhost afpd[21124]: Registering CNID module [cdb]
Jan 23 10:27:29 localhost afpd[21124]: Registering CNID module [dbd]
Jan 23 10:27:29 localhost afpd[21124]: Loading ConfigFile
Jan 23 10:27:29 localhost afpd[21124]: Finished parsing Config File
Jan 23 10:27:29 localhost papd[21126]: restart (2.0.3)
Jan 23 10:27:29 localhost papd[21126]: CUPS support enabled (1.3)
Jan 23 10:27:35 localhost afpd[21124]: localhost:AFPServer@* started on 65280.126:128 (2.0.3)
Jan 23 10:27:35 localhost afpd[21124]: dsi_tcp: hostname 'color' resolves to loopback address
Jan 23 10:27:35 localhost afpd[21124]: dsi_tcp: '192.168.1.53' on interface 'eth2' will be used instead.
Jan 23 10:27:35 localhost afpd[21124]: ASIP started on 192.168.1.53:548(5) (2.0.3)
Jan 23 10:27:35 localhost afpd[21124]: DSIConfigInit: Error registering afp://192.168.1.53/?NAME=localhost&ZONE= with SRVLOC
Jan 23 10:27:35 localhost afpd[21124]: uam: loading (/usr/lib/netatalk/uams_clrtxt.so)
Jan 23 10:27:35 localhost afpd[21124]: uam: uams_clrtxt.so loaded
Jan 23 10:27:35 localhost afpd[21124]: uam: loading (/usr/lib/netatalk/uams_randnum.so)
Jan 23 10:27:35 localhost afpd[21124]: uam: uam not found (status=-1)
Jan 23 10:27:35 localhost afpd[21124]: uam: "Cleartxt Passwrd" available
Jan 23 10:30:10 localhost afpd[21124]: server_child[1] 21140 exited 1
Jan 23 10:30:10 localhost afpd[21141]: ASIP session:548(5) from 192.168.1.111:53657(8)

Okay, daemon is starting back up... etc, etc... and then notice "uam: "Cleartext Passwrd" available." Hrm, interesting. I went to great lengths to compile against OpenSSL.

Then it hit me: I did a software update, and that probably updated netatalk to the Ubuntu precompiled version. No problem, I'll just re-do the compilation and installation steps from the last post.

Sure enough, after starting the new process and looking at syslog:

Jan 23 10:38:41 localhost afpd[21252]: uam: loading (/usr/lib/netatalk/uams_dhx.so)
Jan 23 10:38:41 localhost afpd[21252]: uam: uams_dhx.so loaded
Jan 23 10:38:41 localhost afpd[21252]: uam: loading (/usr/lib/netatalk/uams_clrtxt.so)
Jan 23 10:38:41 localhost afpd[21252]: uam: uams_clrtxt.so loaded
Jan 23 10:38:41 localhost afpd[21252]: uam: loading (/usr/lib/netatalk/uams_randnum.so)
Jan 23 10:38:41 localhost afpd[21252]: uam: uams_randnum.so loaded
Jan 23 10:38:41 localhost afpd[21252]: uam: "2-Way Randnum exchange" available
Jan 23 10:38:41 localhost afpd[21252]: uam: "Randnum exchange" available
Jan 23 10:38:41 localhost afpd[21252]: uam: "Cleartxt Passwrd" available
Jan 23 10:38:41 localhost afpd[21252]: uam: "DHCAST128" available
Well that certainly looks better than plain-text only. Et viola:



Of course, the real solution is to use dpkg-divert which will tell the package management system in Ubuntu not to overwrite my custom version of netatalk. Maybe I will do that once I am not running late for class.

Saturday, January 19, 2008

OS X <3 Ubuntu

Update: in newer Debian packages, new authentication methods are available. Under Ubuntu 9.04, I edited /etc/netatalk/afpd.conf to specify -uamlist=uams_dhx2_pam.so,uams_dhx2.so, added the service to avahi, and I was able to mount my homedir over AFP. No re-compling necessary!

One project I have been sitting on for a while was to get file sharing by AFP and Bonjour working between by MacBook Pro and my Ubuntu machine. Well, faced with the prospect of actually doing homework, I suddenly found the time to do it today.

The end steps were really simple... the main sources I consulted were two threads on the Ubuntu forums, and one page on the Gentoo wiki. Update: see also this excellent guide.

The instructions boil down to this:

1. Build the package netatalk from source, enabling ssl.

I was not able to get my MacBook to connect to my Linux machine after I did an "apt-get install netatalk". A little digging showed that the Debian license (and hence the Ubuntu license) do not allow building with OpenSSL. Okay, an easy fix [1]:

$ mkdir -p ~/src/netatalk
$ cd ~/src/netatalk
$ sudo aptitude install devscripts cracklib2-dev dpkg-dev libssl-dev
$ apt-get source netatalk
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ DEB_BUILD_OPTIONS=ssl sudo dpkg-buildpackage -us -uc
$ sudo debi
$ echo "netatalk hold" | sudo dpkg --set-selections

(Update: added netatalk hold, thanks to Damon Timm's guide.) Nice.

2. Install Avahi [2]

sudo apt-get install avahi-daemon avahi-discover avahi-utils libnss-mdns service-discovery-applet mdns-scan


Done.

3. Create a service file for AFP [2].

sudo echo >> /etc/avahi/afpd.service
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
</service-group>
^D


At this point, the box should pop up in the Leopard finder:


Just for kicks, here are some bonus Avahi service files:

/etc/avahi/services/ssh.service
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_ssh._tcp</type>
<port>22</port>
</service>
</service-group>


/etc/avahi/services/sftp.service
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_sftp-ssh._tcp</type>
<port>22</port>
</service>
</service-group>


Here is iTerm happily showing my SSH and SFTP services on both my MacBook (goto) and Ubuntu machine (color):



[1] http://ubuntuforums.org/showthread.php?t=410274
[2] http://ubuntuforums.org/showthread.php?t=218630
[3] http://gentoo-wiki.com/HOWTO_Share_Directories_via_AFP

Thursday, January 17, 2008

SVN on Mac

One nit that I haven't quite gotten over after switching to a Mac over a year ago is the lack of GUI support for SVN. One of the main reasons that I switched in the first place was the combination of a Unix base (the Darwin/XNU kernel) with some nifty keyboard tricks in the window manager using Quicksilver. All in all, it felt (and still feels) a lot like what I want out of a computer: fast
access to a lot of stuff via keyboard. Of course, Apple claims that their usability studies show that mousing is, on average, just as fast or faster than typing. But I get cranky when I have to use the mouse or (shudder) touchpad.

But I digress.

Despite my affinity for everything typing, there are still some times that I just want to be brain-dead and mousing, or maybe I am doing an initial checkout (or import) and I want to see a screen full of options instead of working through the command line.

So that leads me to the journey of countless others. A quick Google search of any permutation of "Mac," "SVN," "GUI," etc. gives a ton of hits, mostly blog posts complaining about the lack of a GUI client for SVN on the Mac.

A while back, I found one neat solution, however: SCPlugin. It's hosted on Tigris.org, the same site that hosts TortoiseSVN (my favorite SVN client on Windows). SCPlugin is great because it integrates with Finder, which prevents me from having to use another application. Sure, Finder has its myriad shortcomings, but I use Terminal for most of my file manipulation, anyhow. Finder does what it needs to just fine for me.

Of course, since I do most of my file manipulation on the command line, the svn binary from MacPorts is indispensable. But for those few moments when I need to do something and I don't care to read through the SVN book, it's pretty useful.

Now the minuses: First and foremost, SCPlugin will not work with an HTTPS server with a self-signed certificate. I imagine that, for most small developers who have a private SVN repo, or any project which does not want to be hosted on SourceForge/Google Code/some other SVN provider, this is how they work. It is very easy to set up Apache+SVN+SSL with a self-signed cert, but SCPlugin chokes on it. At least 0.7.1 bwhatever that I am using. It is an open bug, so at least the light is on it. The alternative could be svn+ssh, but that is not supported in SCPlugin. Sigh. Similarly, it does not support personal SSL certs. While I don't use this, I imagine that it would be a pretty large barrier to entry in a lot of workplaces.

Another option I found is Mac SVN, but it does not appear to be in active development (the timestamp on the last package is January 2007, over a year ago). There is also Syncro SVN, although it is $59.

So for now maybe I can struggle with the command line when I am dealing with the self-signed-cert servers, and/or cough up the cash for an actual SSL cert for my own server. Technically that is the best solution, but it's not the best free solution.

Friday, January 4, 2008

Email threading headaches (threadaches?)

Abstract: "Waaaaah Outlook, waaaaah Blackberry."

I am working on a project which has me sorting email roughly into threads. Not necessarily in a tree thread, but more of a flat threading (a'la Gmail). The goal is to ingest email by script and then use XML-RPC to either attach them to an existing email thread ticket in Trac on another machine, or create a new thread ticket.

The easy answer is simply to examine the "In-Reply-To" and/or "References" headers and keep an index of Message IDs, handily discussed in RFC 822 (In-Reply-To and References), RFC 2822 (In-Reply-To and References), and RFC 1036 (References, USENET-style). In fact, ├╝ber-geek, Netscaper, nightclub owner, and all-around smart guy Jamie Zawinski posted his own guide to message threading. Jamie's article is fantastically interesting especially because he gives some analysis of historical message archives to determine what "real" headers look like.

So this is great, right? Well, almost. Jamie's guide even gives the warning that
"[o]f course these numbers are very much dependent on the sample set, which, in this case, was probably skewed toward Unix users, and/or toward people who had been on the net for quite some time (due to the age of the archives I checked.)"
Problem the first: my project deals in large part with Microsoft Exchange and Blackberry users. Okay, well I have my own corpus of email to mine, and I can verify that Outlook/Exchange does an exceptional job of following the RFC rules.

Probelm the second: Blackberry via Exchange, however, does not play nicely with In-Reply-To and References headers.

Here is an example message structure. This is the shape of the thread that I am examining:
  • Subject: Query
    From A, with no agent header but including the header "X-MimeOLE: Produced By Microsoft Exchange V6.5" and some message ID, maybe "<0001@exch04.example.lcl>". Generated with Outlook.
    • Subject: Re: Query
      From me, with agent header "User-Agent: Thunderbird 1.5.0.7 (Windows/20060909)", a vaild In-Reply-To and a valid References header, and some message id "<0a3f@example.com>". Generated, obviously, with Thunderbird.
      • Subject: Re: Query
        From P, again no agent header but the same X-MimeOLE header as A's query. There is no In-Reply-To or References header, but there are two headers "Thread-Topic" and "Thread-Index." Has a valid Message ID. This message was sent via Blackberry using their enterprise server, I think, or maybe via the desktop sync (Poor Man's Push) application. Not sure.
        • Subject: Re: Query
          From me, with a valid and correct In-Reply-To and References header, a logical Message ID.
      • Subject: Re: Query
        From A, again using Outlook. This Exchange message does have a valid In-Reply-To and References headers, and has "Thread-Topic" and "Thread-Index" headers.
Okay, so what do we learn from this exercise? Well, Thunderbird respects headers and includes the appropriate identification header fields. And this makes sense, since Thunderbird code ostensibly came from Mr. Zawinski at some point in some way, being that it was a fork of Netscape mail. The second thing is that Outlook also observes the identifier fields. Blackberries, however, not so much. Oh, and since there are no Agent headers, you cannot immediately differentiate a Blackberry/Exchange message from an Outlook/Exchange message. Exchange also includes the "Thread Subject" and "Thread Index" fields. The "Thread Subject" field is static throughout the thread (and is the original message subject), while the Thread Index changes for each message.

One interesting thing to note is that the Blackberry uses base64 encoding for all MIME parts, including the plain text portion of the email. Outlook includes plain text as actual plain text, at least in version 2003.

So here's the rub: how can I get the Blackberry reply back into the message thread? And I don't even need it in the "correct" thread structure, just into a flat thread. Gmail can do it, how?

Off the top of my head, Gmail probably uses the Subject field (sans "Re:", and so on) to find what might be similar messages, and then uses some fuzzy text block matching algorithm to determine whether it's an actual reply. Gmail already does this fuzzy comparison (or something like it) to do its "Show Hidden Text" feature in replies. In this way, Gmail was able to handle the Blackberry reply listed above.

Interestingly, if the Blackberry reply has a different To: address than previously seen in the thread, however, it breaks the threading. So, for example, if I get a message from P and reply with my @gmail.com email, and then P replies using my @example.com email using his Blackberry, the thread is broken in Gmail with the third message. So the behavior is not seemingly entirely consistent in Gmail.

The Blackberry reply-matching is pretty important to my project. Critical, actually. It seems like the only way to do it so far is by doing some complex (and expensive) text matching. The fuzzy search algorithm is attractive, but since I want to use this in some ticketing system like Trac, I would probably need to use some second corpus of text to perform the matching (remember, for me Trac is accessed by XML-RPC between two machines, so exhaustive ticket searches are very expensive).

I went back and looked at more Blackberry messages. The thread example above is originally from 2006, and in the mean time P upgraded his Blackberry from something pre-Curve to an 8830. The behavior is the same, in that it does not respect In-Reply-To or References. I checked emails from two other Blackberry users, with the same results.

So, I will trudge forward and shoot to use the fuzzy text matching if I can. Maybe something will fix itself in the mean time (not likely). But my guess is that the reason Blackberry threading works in Gmail due to its (incredibly) smart thread finding.