As I was looking around for the way to specify the correct key for launchpad, I could not find it. Luckily the ssh process is running when it asks for the passphrase so I could see the URL used to access the launchpad server from Bazaar. It is:

bazaar.launchpad.net

The address is not visible by default because you use the lp: protocol instead of a full URI to access launchpad.

  Host bazaar.launchpad.net
    HostName bazaar.launchpad.net
    User <your launchpad username>
    PasswordAuthentication no
    HostbasedAuthentication no
    IdentitiesOnly yes
    IdentityFile /path/to/your/.ssh/id_rsa-launchpad-file

The path to your identity file should be correct. The path I show here is just to let you know how it works. Remember that your .ssh directory needs drwx------ for permissions (chmod 700) and your file should also be limited to the owner -rw------- (chmod 600).