Safe shell (SSH) is a regular means of accessing server infrastructure within the IT business. In contrast, no single commonplace answer exists for managing person SSH entry in an automatic and safe method. Allow us to discover how we tackled this problem at GoodData.
Our Necessities
Given the varied and globally distributed platform infrastructure we function at GoodData, safe shell (SSH) is a core means of accessing our programs so as to carry out needed operations and debugging. It’s due to this fact very important for us to have an SSH authorization methodology that’s:
- safe in accordance with the newest business requirements;
- sturdy in its efficiency; and
- straightforward to make use of for our workers.
Having utilized FreeIPA to offer SSH entry administration for a number of years, a strategic choice was made in late 2020 to modify to Okta because the company-wide id administration answer. This transition introduced us with a chance to refresh our entry management setup and guarantee its consistency with present safety requirements.
Trying to find a Resolution
We recognized inadequate scalability as the first shortcoming of our earlier SSH setup. With FreeIPA, each SSH login try requires the person’s public SSH key to be verified in opposition to the server, rendering FreeIPA a single level of failure and stopping any person logins to focus on machines in case our FreeIPA servers skilled an outage or upkeep.
We got down to discover and implement an answer that might keep away from this pitfall; one that might be sturdy sufficient to assist each day operations on the dimensions we require, i.e. managing entry to a whole bunch of servers distributed throughout the globe.
Moreover, deprecating long-lived SSH credentials from the ecosystem was one other objective we deemed essential. There was an inherent safety danger stemming from static SSH keys being current on workers’ gadgets, as they may probably be stolen and used for entry to our infrastructure. Whereas we had partially mitigated this danger utilizing multi-factor authentication, we had been striving for a super answer that might solely depend on short-lived, dynamic credentials. This manner, any leaked credential wouldn’t pose a safety danger, as a result of it might expire in a short time.
After in depth analysis, we chosen HashiCorp Vault to offer SSH entry administration through its dynamic SSH certificates engine.
Technical design
Since Vault had already been current in our infrastructure as a retailer for static key-value credentials, extending its position by including the SSH certificates signing backend was pretty simple.
Person Authentication
For person entry, Vault has been built-in with Okta utilizing the OIDC authentication methodology. We outlined a number of person login roles similar to various ranges of entry granted to distinct teams of customers; every position is tied to a selected OIDC declare worth offered by Okta. Each Vault login position, in flip, permits signing SSH certificates by a corresponding SSH backend position. The SSH certificates key ID configuration ensures that person position data is preserved within the ensuing signed certificates.
For instance, if a person first.final belongs to group1, they’ll login to Vault utilizing login position group1. Vault coverage may also enable them to signal a certificates utilizing SSH position group1, and the ensuing certificates’s key ID might be okta-first.final:group1.
SSH Person Provisioning
In absence of a centralized SSH authorization server, a problem stays: learn how to decide which customers are allowed to hook up with a selected server? To resolve this, we have now outlined two distinct SSH login situations we would have liked to assist:
- login to externally uncovered leap stations; and
- additional hops to internally accessible servers.
On leap stations, we require the customers to log in below a private non-privileged account, so as to present a secure atmosphere to entry programs utilizing private credentials (similar to AWS IAM Id Heart logins or studying secrets and techniques from Vault itself). Due to this fact, a easy Python script is periodically run on the leap station servers to fetch an inventory of related customers from Okta API and create a system account for every of them.
On all different servers, a single shared person account is ample for all customers logging in through SSH; due to this fact, no periodic person synchronization from Okta is required.
By this splitting of use circumstances, the necessity to ship Okta credentials is proscribed to a handful of leap station nodes whereas we nonetheless maintain the extent of person entry separation we require.
Person Authorization
The method of authorizing customers makes use of the sshd AuthorizedPrincipalsCommand directive; three items of setup are pre-delivered by our configuration administration instrument to facilitate this:
- Vault SSH certificates authority public key;
- checklist of Vault SSH roles allowed to log in to a given server; and
- an authorization script to be referred to as upon person login.
When a person makes an attempt to log in to a server, AuthorizedPrincipalsCommand known as, with the %i (certificates’s key ID) and %u (username) parameters offered to our authorization script. The script parses the person’s Vault position from the important thing ID (making use of its static format described above) and compares it in opposition to the checklist of roles allowed to log in. On leap stations, it moreover verifies whether or not the person is logging in below their very own private account identify. A second-factor verification can be hooked into the login course of for the leap stations case.
Advantages and Classes
Utilizing Vault’s SSH certificates engine, we have now been capable of transition to a way more sturdy and safe login system than earlier than. Customers throughout the corporate admire the system’s stability; since authorization is finished with data statically out there to SSH servers and no stay communication with Vault is required at login time, the person expertise is far smoother than beforehand. Safety has additionally been improved by eliminating static, long-lived SSH keys in favor of shortly expiring SSH certificates.
Now we have encountered minor challenges with the brand new setup. Having a shared system account for all customers on a majority of servers implies that auditing person actions is barely tougher; we have now labored round this by parsing the certificates key ID into an atmosphere variable upon login. Instructions similar to who aren’t helpful since introducing the brand new setup. Retraining customers to the brand new Vault-based SSH login workflow was an preliminary hurdle we have now been capable of overcome with time.
General, although, we conclude that the brand new means of doing SSH at GoodData supplies extra safety and a greater person expertise than the previous answer primarily based on static SSH keys.