Bastion server

Logging into servers through the bastion

To improve the security of our infrastructure it’s not possible to connect directly to a production server with SSH. Instead, all connections must come from a small server called the “bastion”, which only allows connections from a few whitelisted networks and logs any connection attempt.

To log into a server through the bastion, use one of the following methods:

  • Use SSH’s -J flag:

    ssh -J <username>@bastion.infra.rust-lang.org <username>@servername.infra.rust-lang.org

  • Configure your SSH client to always jump through the bastion when connecting to a host:

    • Add this snippet to your SSH configuration file (usually located in ~/.ssh/config):

      Host servername.infra.rust-lang.org
    ProxyJump <username>@bastion.infra.rust-lang.org

    • Use SSH:

      ssh <username>@servername.infra.rust-lang.org

Please remember the bastion server only allows connections from a small list of IP addresses. Infra team members with AWS access can change the whitelist, but it’s good practice to either have your own bastion server or a static IP address.

The SSH keys authorized to log into each account are stored in the simpleinfra repository. Additionally, people with sensitive 1password access can use the master key stored in the vault to log into every account, provided their connection comes from any whitelisted IP.

Common maintenance procedures

Adding a new user to the bastion server

To add a new user to the bastion you need to add its key to a file named <username>.pub in ansible/roles/common/files/ssh-keys, and change the Ansible playbook adding the user to the list of unprivileged users. Please leave a comment clarifying which servers the user will have access to.

Once that’s done apply the playbook and add a new whitelisted IP address.

Editing a whitelisted IP

Due to privacy reasons, all the static IP addresses of team members with access to the bastion are stored on AWS SSM Parameter Store instead of public git repositories. When running the following commands, replace USERNAME and IP_ADDRESS with the proper values.

Adding a whitelisted IP

  1. Run: 
    aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" --region us-west-1
  2. Add the username to the list in terraform/bastion/firewall.tf (local variable allowed_users).
  3. apply the Terraform configuration.

Updating a whitelisted IP

  1. Run:

    aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" --region us-west-1

  2. apply the Terraform configuration.

Removing a whitelisted IP

  1. Run:

    aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" --region us-west-1

  2. Remove the username from the list in terraform/bastion/firewall.tf (local variable allowed_users).

  3. apply the Terraform configuration.