At some point in the past, my system somehow lost the ability to avoid asking me for my key every time I connect to one of my servers. It generally okay, I don't do so much work on that server that I'd have to connect all the time...

However, it is annoying when, once in while, I end up doing a lot of work, back and forth.

The ssh-agent was installed and working. I could add the key manually:

ssh-add ~/.ssh/my-secret-key
ssh-add -l

The ssh-add command allows you to add and remove keys from the ssh-agent. The -l command line option lists the keys currently held by the agent.

Now, why would the key not get automatically installed the first time I was entering it?

I'm not totally sure what happened, but one time when I upgraded, it stopped work. The fact is that this upgrade must have removed the ssh-askpass package. I still had the gnome-ssh-askpass which opens a window to ask for the passphrase, but the ssh-askpass was not installed anymore...

sudo apt-get install ssh-askpass

resolved the problem.

Source: http://unix.stackexchange.com/questions/83986/tell-ssh-to-use-a-graphica...