The problem
Some time ago, I manually hacked together a working setup to use gnome-keyring-daemon in xubuntu to automatically unlock my ssh key when logging in. This setup broke when I upgraded to Ubuntu Lucid and I have never tried to fix it in Maverick and Natty. With the arrival of Oneiric, I gave the gnome-keyring-daemon another try. Lo and behold - it works!
The fix
To enable the gnome-keyring-daemon, make sure to have GNOME services enabled in the XFCE session settings. Then add an entry to the xfconf database with the following command:
xfconf-query -v -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
That should do the trick. You can manage your keys using the seahorse GUI.