Depending on how your server is set up you might be required to log into it using a public key (for me I needed this for my Amazon instances). This is fine with SSH, you can just link to it directly when like so:

ssh user@example.com -i /path/to/example.pem

I had a problem with SFTP since it doesn't have the -i option. I got around this issue by associating the public key file to the ssh connection.

Create a <config> file in your .ssh folder, and write an entry similar to the following example:

host example.com
IdentityFile ~/.ssh/example.pem

You should then be able to log in through ssh and sftp without having to specify a public key file since it's automatically associated with the host.