When you create an Amazon EC2 instance, you’re given a .PEM private key allowing for passwordless entry to your server. Losing this key can be pretty costly but below I’ll show how to get you back in again.

The Problem

We’ve lost our PEM key or the one we have isn’t working:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$  ssh -vvv -i /path/to/my.pem ubuntu@host.com
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
...
debug2: key: /path/to/my.pem (0x0), explicit
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/me/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /path/to/my.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA 99:99:aa:9a:aa:99:99:a9:aa:99:99:99:99:9a:99:aa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

The Plan

We need to set a new authorized_key on our server. To do this we’ll:

• Create a temporary new EC2 instance (E2) with a new keypair
• Mount our servers EBS volume to E2
• Set the authorized_key in our EBS volume to use our new key
• Reattach the EBS to our original EC2 and log in.

The Implementation

I don’t like big wordy tutorials so here’s a tl;dr of all steps involved:

• Create a snapshot of your EC2’s (E) EBS volume (V)
• Create a new volume (V2) from the snapshot
• Start new t2.micro EC2 Ubuntu instance (E2), using a new key pair
• Attach V2 to E2, as /dev/xvdf (or /dev/sdf)
• SSH in to E2

• 1 2 3

  sudo mount /dev/xvdf1 /mnt/tmp -t ext4 cp ~/.ssh/authorized_keys /mnt/tmp/home/ubuntu/.ssh/authorized_keys sudo umount /mnt/tmp

• Detach V2 from E2
• Stop E
• Detach V from E
• Attach V2 to E as /dev/sda1
• Start E
• Login as before, using your new .pem file
• If all is well and you’re in, delete E2 and V

In my personal case, the above didn’t help and I was still getting the error Permission denied (publickey). I had to also copy E2‘s sshd_config because I’d borked E‘s and it was the actual reason I couldn’t SSH in.

So before the umount line above, also do:

1
2
3

sudo cp /etc/ssh/sshd_config /mnt/tmp/etc/ssh/sshd_config
mkdir /mnt/tmp/home/ubuntu/.ssh/bak
mv /mnt/tmp/home/ubuntu/.ssh/id_rsa /mnt/tmp/home/ubuntu/.ssh/id_rsa.pub /mnt/tmp/home/ubuntu/.ssh/known_hosts /mnt/tmp/home/ubuntu/.ssh/bak

Hope this helps.

Thanks to yegor256 for his helpful post on Stack Overflow.