Enterprise SSH Key Management

serjsk8

serjsk8

Hello,
I'd like to hear your opinion and share experience.
This post is not related to FreeBSD.

Background:
We have a enterprise OnPrem environment with the following setup:
  • Users: 1,000+
  • Infrastructure: 400+ Windows/Unix VMs.
  • Authentication: Microsoft Active Directory for user accounts, with passwords expiring every 45 days.
We need to implement automated SSH certificate management that supports short-lived certificates.
Security team requires regular certificate rotation.

I'm looking at:
- HashiCorp Vault

Has anyone implemented something like this?
What's the easiest way to automatically issue short-lived SSH certificates to many users?
What works best in practice?
 
Vault is common (used a lot in projects I work with). Unfortunately I'm just a user, so I can't say how easy the automation part is.
However, any centralized secrets storage system worth it's salt should offer you policies that can be enforced.
 
AD is kerberos in the backend, with magic microsoft pixie dust on top. it is hypothetically possible to integrate ssh with that.
 
cracauer@ said:
Kerberos tickets for ssh.
Click to expand...
atax1a said:
it is hypothetically possible to integrate ssh with that.
Click to expand...
This. But its tricky if you need service accounts logging in remotely. Works fine for the typical 'user' accounts though, nice added benefit, single sign-on if implemented correctly.

Cross-domain trusts work fine, cross-forest trusts only if they're two way.

Also, since ADS is basically a bastardized combination of Kerberos, LDAP and DNS, you can actually store your SSH keys in AD.
 
You want SSSD2 then, no SSH keys with Kerberos. Though, I don't use SSSD2 yet, Kerberos perfectly works with `.k5login`. We even use keytabs from CI/CD jobs..
 
serjsk8 said:
Hello,
I'd like to hear your opinion and share experience.
This post is not related to FreeBSD.

Background:
We have a enterprise OnPrem environment with the following setup:
  • Users: 1,000+
  • Infrastructure: 400+ Windows/Unix VMs.
  • Authentication: Microsoft Active Directory for user accounts, with passwords expiring every 45 days.
We need to implement automated SSH certificate management that supports short-lived certificates.
Security team requires regular certificate rotation.

I'm looking at:
- HashiCorp Vault

Has anyone implemented something like this?
What's the easiest way to automatically issue short-lived SSH certificates to many users?
What works best in practice?
Click to expand...

With one (or dual) SSH CA's you can sign user keys and create a certificate with an expiration date. An SSH CA requires no extra software, only configuration distributed to your servers. SSH will check the expiration date on login from trusted CA's.

The issue is how often you rotate those certs. There are many options for controlling which users can access what systems (ie: roles) using a CA. You can even sign hardware tokens (ie: Yubikeys) because the SSH keys are a private/public pair of KEYS, not CERTS. A cert can only inform a server about the keys, but certs are optional.

You could either setup a method to renew certs for users at the CA, or you could use the SSH key blacklists to disable key access. If you distribute the key blacklist to all servers at a routine interval, you can disable a key without using a certificate expiration date. There are also some scripts for pulling the blacklist dynamically.

The bigger concern to me is how are you distributing the user keys to authorized_keys? If this is dynamic or refreshed frequently, it may be a better idea to just manage SSH keys, and avoid certificates all together.

Perhaps the product acts as an SSH CA? If so you should be able to configure your servers to trust signed keys (with a certificate) the product issues.
 
Thanks all for reply!
cracauer@ said:
Kerberos tickets for ssh.
Click to expand...
SirDice said:
Also, since ADS is basically a bastardized combination of Kerberos, LDAP and DNS, you can actually store your SSH keys in AD.
Click to expand...
Yes, I tried that option.
I think it's one of the possibilities.
Our users use a Windows or Linux "jump box."
You can get your Kerberos ticket there, and Putty, for example, supports Kerberos tickets.
But I found that this option is very sensitive to DNS settings.
There were many times when I couldn't connect to the remote server because SSH rejected my ticket.

RussellASC said:
The bigger concern to me is how are you distributing the user keys to authorized_keys?
Click to expand...
I'm also thinking about how to do this.
 
SirDice said:
AuthorizedKeysCommand can be leveraged to LDAP query AD and get the keys. Bit more tricky to do though. AD is in essence LDAP with Kerberos authentication.
Click to expand...
Yes, I'm also trying this scenario.
But for some reason, our security team isn't happy with it. 🤔
Although, in my opinion, it could be a good solution.
Perhaps we need to think about it in more detail.
 
Sometimes I think our security department lives in a completely different world or on another planet. :)
 
serjsk8 said:
Yes, I'm also trying this scenario.
But for some reason, our security team isn't happy with it. 🤔
Although, in my opinion, it could be a good solution.
Perhaps we need to think about it in more detail.
Click to expand...
They are likely upset because the SSH keys are forever. They insist on credential rotation when they don't understand how it works.

If they had a clue, they'd be asking you to rotate the SSH keys routinely, not the certificate.

Mallcops with audit reports, not real security. Bet they want you to make admin specific users instead of letting admins use root directly too.

My advice is to look at the options for SSH CA's. You can use the roles to specify the user "X" can login to account "X" on any machine in group "Y", without having to push the key to authorized_keys on every server. This is done by using the SSH CA to sign the SSH key pair, and produces a limited time certificate. The destination systems must be configured to honor keys (with cert) signed by the SSH CA.

This gives you routine expiration and no need to go touch authorized_keys for every user, everywhere.

I would recommend a long duration on the certificate (ie: 1 year), and use the "RevokedKeys /etc/ssh/revoked.sshca" option to manage keys that should no longer be used. You ideally can push that file routinely via configuration management, or just a cron job.

If you want to discuss how to best secure SSH overall, we should have a different thread. ;]
 
SirDice said:
Hmm, ours might be living right next to yours. The CISO certainly does.
Click to expand...
Unfortunately my experience is that most cyber security has completely given up on making things secure, and instead just wants to surveil everything to CYA for insurance purposes.

Gotta love installing "security" agents on every system as root, that have vulnerabilities and are network services increasing your attack surface.
 
DtxdF said:
security/opkssh
Click to expand...
I checked it out, and it looks good.
But from my perspective, connecting a closed enterprise environment to an "OpenID Connect" provider would be... very problematic.

RussellASC said:
They are likely upset because the SSH keys are forever. They insist on credential rotation when they don't understand how it works.
Click to expand...
Yes, you right.

RussellASC said:
Mallcops with audit reports, not real security. Bet they want you to make admin specific users instead of letting admins use root directly too.
Click to expand...
I've lost count of how many different agents we install on our servers. These agents generate all sorts of reports that no one studies.

RussellASC said:
If you want to discuss how to best secure SSH overall, we should have a different thread. ;]
Click to expand...
Of course I'm interested!
And I'd to continue this conversation and hear about your experiences.
We can continue here or in another thread...
 
serjsk8 said:
I checked it out, and it looks good.
But from my perspective, connecting a closed enterprise environment to an "OpenID Connect" provider would be... very problematic.
Click to expand...
I think the idea is that you run the OIDC provider internal to the enterprise, against your own credentials database.
 
You must log in or register to reply here.
Back
Top