SSH

ssh — SSH stands for Secure Shell. OpenSSH SSH client (remote login program).

The SSH protocol uses encryption to secure the connection between a client and a server. All user authentication, commands, output, and file transfers are encrypted to protect against attacks in the network.

Uses for SSH protocol:

Providing secure access for users and automated processes.
Interactive and automated file transfers
Issuing remote commands
Managing network infrastructure and other mission-critical system components.

1 How Does the SSH Protocol Work
- 1.1 How Does Authentication Work?
2 SSH Key Pairs
3 SSH Configuration File
4 File Transfers over SSH
5 How to SSH without Keypairs, and use Password Authentication instead
6 SSH Troubleshooting
7 How to regain SSH access to an EC2 instance using User Data
8 How to set up SSH Agent Forwarding (for Bastion hosts)

How Does the SSH Protocol Work

The protocol works in the client-server model, which means that the connection is established by the SSH client connecting to the SSH Server.

After the setup phase the SSH protocol uses strong encryption and hashing algorithms to ensure the privacy and integrity of the data that is exchanged between the client and server.

SSH Client initiates the connection by contacting the SSH Server
SSH Server sends the server public key to the SSH Client
Negotiate parameters and open secure channel
User login to server host operating system

How Does Authentication Work?

Initializing a connection in SSH consists of:

Negotiating the version of protocol to use.
Negotiating cryptoigraphic algorithms and other options to use
Negotiating a one-time session key for encrypting the rest of the session
Authenticating the server host using its host key
Authenticating the user using a password, public key authentication, or other means.

After this, data can be exchanged, including terminal data, graphics, and files.

Public Key Authentication - The key-based authentication mechanism in SSH is called public key authentication.

Essentially, some session-specific data is signed using the private identity key. The signature is then sent to the server that checks if the key used for signing is configured as an authorized key. The server then verifies the digital signature using the public key in the authorized key. The identity key is never sent to the server.
The essential thing in public key authentication is that it allows one server to access another server without having to type in a password.
This powerful feature is why it is so widely used for file transfers (using the SFTP protocol) and configuration management. It is also commonly used by system administrators for single sign-on.

SSH Key Pairs

Strong authentication with SSH keys

The idea is to have a cryptographic key pair (Public and private keys), and configure the public key on the server to authorize access and grant anyone who has a copy of the private key to access the server.
Private Keys (identify keys) are typically stored in a user's ~/.ssh directory on the client machine.
The public key "fingerprint" is in the ~/.ssh/authorized_keys file on the server.

Different Type of SSH Keys

RSA is well-regarded and supported everywhere. It is considered quite secure. Common key sizes go up to 4096 bits and as low as 1024. The key size is adjustable. You should choose RSA.
DSA is not in common use anymore, as poor randomness when generating a signature can leak the private key. In the past, it was guaranteed to work everywhere as per RFC 4251, but this is no longer the case. DSA has been standardized as being only 1024 bits (in FIPS 186-2, though FIPS 186-3 has increased that limit). OpenSSH 7.0 and newer actually disable this algorithm. (WEAK)
ECDSA is newer and is based on DSA. It has the same weaknesses as DSA, but it is generally thought to be more secure, even at smaller key sizes. It uses the NIST curves (P256).
Ed25519, while not one you listed, is available on newer OpenSSH installations. It is similar to ECDSA but uses a superior curve, and it does not have the same weaknesses when weak RNGs are used as DSA/ECDSA. It is generally considered to be the strongest mathematically.

What Do SSH Keys Look Like

A Private Key (identity key) will look something like this; Will have ---BEGIN, ---END:

A Public key fingerprint or authorized key will look like this:

ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQDSgM36Y7GQvqom4E4/YuByhwl6oLY1Zn+SvqlMyK0jIjSJ6OttTD03utNKAtSvAk8r+t7GJRSRlmQ
6XpDxVBgTmOP6/IV5FQ9KMzruELwVPkK5OzJUUhUPb1WL4zjERf3wOGRRYagFPYwglUfW/g1E/4U21edRKWevK900pW3EEWkr/ZJN4LiDgx 
fn8ZXARRbv+YhZ0S//p/XeCtw8uzbBNsqsBL+2zIrQqIOOyO5M1XvnsLzDVGnZPhJmF7/7BwRR2bJATfK/j5VVe3ZK8RkuDoh7TzMn2hvqm
WcxDn1H+x2hAUOp4+zh+XF/NNeljKTnj8CUVvcGu8bhK2OrUm/F ec2-user@ip-172-31-33-239.ec2.internal

(Note, I had to break these lines up for the sake of showing how it looks on this wiki. Really the whole things is one long string)

Permissions

The proper permissions and other notes

drwx------ 1 ardika ardika 4096 Jan 13 14:26 .ssh/                       # 700
-rw------- 1 ardika ardika 1671 Jan  6 21:59 .ssh/private_key.pem        # 600
-rw-r--r-- 1 ardika ardika 404 Jun 13  2019 .ssh/public_key.pub          # 644
-rw------- 1 ec2-user ec2-user 391 Jan 11 22:58 .ssh/authorized_keys     # 600

SSH Configuration File

/etc/ssh/sshd_config - The SSH configuration file.

Configurations you can adjust:
- Listening port number
- RSAAuthentication
- RSAAuthentication
- PubkeyAuthentication
- PasswordAuthentication
- PermitRootLogin

Below are some "how-to's" when modifying the sshd_config file to enable/disable certain features of the SSH service.

File Transfers over SSH

SCP - Secure Copy Protocol

- SCP can only be used for transferring files, and is not interactive (everything has to be specified on the command line).

SFTP - Secure File Transfer Protocol

- More elaborate, and allows interactive commands to do things like creating directories, deleting directories and files (all subject to system permissions, ofc), etc.

- Allows for a range of operations on remote files - it is more like a remote file system protocl. An SFTP client's extra capabilities compared to an SCP client include resuming interrupted transfers, directory listings, and remote file removal.

Both SCP and SFTP utilize the same SSH encryption during file transfer with the same general level of overhead, SCP is usually much faster than SFTP at transferring files, especially on high latency networks. This is because SCP implements a more efficient transfer algorithm, one which does not require waiting for packet confirmations. This leads to faster speed but comes at the expense of not being able to interrupt a transfer, so unlike SFTP, SCP transfer cannot be canceled without terminating the session.

How to SSH without Keypairs, and use Password Authentication instead

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-password-login/

https://www.serverkaka.com/2018/08/enable-password-authentication-aws-ec2-instance.html

SSH Troubleshooting

Below are some of the common errors or issues you may run into with SSH:

Server refused our key

You might be unable to log in to an EC2 instance if:

You're using an SSH private key but the corresponding public key is not in the authorized_keys file.
You don't have permissions for your authorized_keys file.
You don't have permissions for the .ssh folder.
Your authorized_keys file or .ssh folder isn't named correctly.
Your authorized_keys file or .ssh folder was deleted.
Your instance was launched without a key, or it was launched with an incorrect key.

To connect to your EC2 instance after receiving the error "Server refused our key," you can update the instance's user data to append the specified SSH public key to the authorized_keys file, which sets the appropriate ownership and file permissions for the SSH directory and files contained in it.

Server refused public-key signature despite accepting key!

Could be due to:

Could be due to changed permissions on the .ssh directory, the authorized_keys file, or even the home directories. Check permissions.
Firewall

Permission denied

Check permissions! Make sure you are using the correct user name, correct key, correct host, etc.

Connection Timeout || Connection Timed out

A connection timeout indicates that the client attempted to establish a network socket to the SSH server, but the server failed to respond within the timeout period.

Ensure the destination host is correct
Check firewall/security groups.
Check the ports you are using or the service is listening on ($ grep -i port /etc/ssh/sshd_config).
Check that the service is actually running and bound to the expected port.

Connection Refused

This means the request is routed to the host, but the host does not successfully accept the request.

Ensure the destination host is correct
Check firewall/security groups.
Check the ports you are using or the service is listening on ($ grep -i port /etc/ssh/sshd_config).
Check that the service is actually running and bound to the expected port.

Too Many Authentication Failures

This could happen if you have (default on my system) five or more DSA/RSA identity files stored in your .ssh directory. In this case if the -i option isn't specified at the command line the ssh client will first attempt to login using each identity (private key) and next prompt for password authentication. However, sshd drops the connection after five bad login attempts (again default may vary).

I was able to solve this issue by editing the ~/.ssh/config file on the local machine.

Added this line:

Host *
         IdentitiesOnly=yes

And it worked.

Copied my private key to another EC2 instance/Bastion, but now when I ssh to another instance I'm getting asked for a passphrase when I shouldn't be

https://serverfault.com/questions/379938/ec2-instance-always-ask-me-to-enter-passphrase-for-the-pem-during-connection

If sharing a private key between 2 or more ec2 instances and if you try to establish a ssh connection from a ssh connection on ec2, use notepad to open .pem file on your local machine and copy the contents to the new .pem file you're creating in the ssh terminal. It will work 100% and won't ask you for any passphrase.

If you open your local .pem file with other text editors i.e. VSCode you will be asked for the passphrase when trying to use your new .pem file.

To check if the key is OK run: openssl rsa -check -in test.pem -noout like so:

# Bad key:
openssl rsa -check -in .ssh/ec2_nva_key.pem -noout
unable to load Private Key
139794998015904:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:815:

# Good key:
openssl rsa -check -in .ssh/ec2_nva_key.pem -noout
RSA key ok

How to connect to a RHEL 8 system running FIPS using PuTTY

RHEL 8: https://access.redhat.com/solutions/4906221

More FIPS (How can I make RHEL 6 or RHEL 7 FIPS 140-2 compliant? https://access.redhat.com/solutions/137833 )

How to regain SSH access to an EC2 instance using User Data

https://aws.amazon.com/premiumsupport/knowledge-center/user-data-replace-key-pair-ec2/

1. Create a new key pair.

2. If you create the private key in the Amazon EC2 console, retrieve the public key for the key pair.

3. Open the Amazon EC2 console.

4. Stop your instance.

5. Choose Actions, Instance Settings, and then choose View/Change User Data.

6. Copy the following script into the View/Change User Data dialog box:

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0 

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt" 

#cloud-config
cloud_final_modules:
- [users-groups, once]
users:
  - name: username
    ssh-authorized-keys: 
    - PublicKeypair

Replace username with your user name, such as ec2-user. You can enter the default user name, or enter a custom user name, if one was previously set up for the instance. For a list of default user names, see General prerequisites for connecting to your instance.

Replace PublicKeypair with the public key retrieved in step 2. Be sure to enter the entire public key, starting with ssh-rsa.

7. Choose Save.

8. Start your instance.

9. After the cloud-init phase is complete, validate that the public key was replaced.

Important: Because the script contains a key pair, remove the script from the User Data field.

Note: Notice the [users-groups, once] -- This will only run the user-data script once, which is the next boot up. If you want to run this again, you can change once to always. So be sure to remove the script from the User Data Field once you regain access.

ANOTHER GUIDE USING USER DATA SPECIFICALLY FOR PERMISSIONS DENIED ERROR (This didn't work for me after testing a couple times though): https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-fix-permission-denied-errors/

References: [+] https://aws.amazon.com/premiumsupport/knowledge-center/ec2-server-refused-our-key/ [+] https://serverfault.com/questions/716033/gitlab-server-refused-public-key-signature-despite-accepting-key-on-a-valid [+] https://serverfault.com/questions/837981/too-many-authentication-failures-for-ec2-user

How to set up SSH Agent Forwarding (for Bastion hosts)

Really good article, very easy:

Securely Connect to Linux Instances Running in a Private Amazon VPC:

https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/

Configuring SSH Agent Forwarding in Windows Subsystem for Linux (Ubuntu 18.04):

https://stfc-cloud-docs.readthedocs.io/en/latest/howto/ConfigureSSHAgentForwardingInWindowsSubsystemForLinux.html

To update Ubuntu 18.04 LTS run:

sudo apt-get update
sudo apt-get upgrade

This left me with a nice up to date install. You can then run the following commands to start and add your key to the ssh agent:

# To get the environment variables set in the user's shell environment and start the agent:
eval $(ssh-agent -s)
ssh-add \<path-to-your-ssh-private-key\>

Then you can ssh to your Bastion host with agent forwarding using:

ssh -A ec2-user@<public_IP>

Then you can SSH to your instances in the private subnet:

ssh ec2-user@<private_IP>

NOTE: You can view the private keys that were added to the agent:

$ ssh-add -l
2048 SHA256:naYvKCfcR+qJXL9A6YiohfaxpjjKEK4G4dW2rAcxLgg .ssh/ohio_key_for_home.pem (RSA)

Another good link: https://www.ssh.com/ssh/agent

Navigation menu

SSH

Contents