I tried to set this up so I could use my yubikey as a portable authenticator following the Debian guide, and ran in to a bit of an issue.

Despite what I was doing, ssh-agent would auto start, set environment variables and give me lots of trouble.

Killing ssh-agent and manually setting the $SSH_AUTH_SOCK to my gnupgp socket fixed the issue, however I couldn’t get ssh-agent to stop starting and setting that variable on login.

Having gone through some anguish I stumbled across xfce documentation detailing how to stop this behaviour.

Now it works.

Great.