Epithet, Briefly 30 Nov 2023

Since Bob asked, I'm cleaning up a system to make using short-lived ssh certificates easy, simple, and secure.

The gist is to use a custom ssh-agent which, in turn, uses some modern authn service (OIDC, Keycloak, Okta, Google Sign-In, etc) to authenticate you. It then passes that auth token up to a CA which relies on a policy service to verify the auth token and respond with cert params (time, principals, extensions, whatever is needed). The CA uses the policy provided attributes to generate a short lived cert, which it gives back to the agent. Voila, you can access things for a couple minutes.

There is a real sequence diagram of what is going on in the repo.

CA

The CA is dirt simple and is the only thing that will ever see the CA private key. The same CA service should be usable by more or less anyone, maybe with some variety in terms of where the private key is stored, but that is it.

The CA receives an opaque auth token + pubkey from the agent and responds with a cert. The existing implementation uses a policy service to decide whether to issue the cert, and what attributes to give it.

The actual CA was kept as simple as possible, because it is a CA. You don't want to muck with it much.

Policy

The policy service consumes the opaque auth tokens from the CA and reponds with basic info, assuming the auth is valid:

{
    "identity":"brianm@skife.org",
    "principals":["oncall_fluffy", "oncall_wibble", "wheel"],
    "expiration":"2m",
    "extensions": {
        "permit-pty":true,
        "permit-port-forwarding":true
    }
}

The policy server (single endpoint, takes a post with the context for the request) is nice as it can be dynamic and make use of risk models and various other signals to decide what to do: eg, "is it a deploy server asking for a key? Is there actually a deploy in progress matching the identity requesting the cert? Okay, give it a 5 minute cert" on the sophisticated side; simply checking an OIDC token for validity on the simpler side.

Agent

The ssh agent generates a new keypair on startup and never lets the private key leave memory. It has a small GRPC API over a domain socket for interfacing with authentication mechanisms, such as a helper to simply feed it tokens over stdin for simple cases. We popped a browser (or used a cli tool) to do the Okta dance.

Michael and I made this at a previous company. We spun up the CA in one lambda function, the policy server in another, and authed against Okta. I want it for my own stuff now and it doesn't look like anyone there is maintaining it anymore, so I have been bringing it back to life.

---

SSH Certificate Notes 29 Nov 2023

There are lots of tutorials out there, but I want to compile my notes so if I walk away from playing with them again for a few years, I can pick it up again!

SSH Keys for different things are all the same (modulo chosen algo type)

User keys, host keys, CA keys -- they are all the same. Don't let ssh-keygen docs, with all of its options and whatnot, convince you otherwise.

Use AuthorizedPrincipalsFile to manage access

Don't go and create a local user account for everyone who may be accessing the host, just have a shared local user, probably even root, where you differentiate who did what using the identity on the cert they authed with.

AuthorizedPrincipalsFile is a file that lists the allowed principals for a given local user. Use groups as principals, not individual users, and list the groups which may use the local user in the file for the given local user. The groups an actual user is in are then added to that user's cert.

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

The %u is the normal user placeholder, so /etc/ssh/auth_principals/root would control the principals that may log in as root. The file takes one principal per line. Copilot really wants me to add that you can use wildcards in the file, but that idea is making me queasy. Principals as groups sounds better to me.

The overall sshd_config looks something like:

Port {{ port }}
Protocol 2
AcceptEnv LANG LC_*
LoginGraceTime 120

UsePAM no 
PasswordAuthentication no
IgnoreRhosts yes
PubkeyAuthentication yes

HostKey {{ path }}/ssh_host_ed25519_key
TrustedUserCAKeys {{ path }}/ca.pub
AuthorizedPrincipalsFile {{ path }}/auth_principals/%u

Testing with sshd

You can run sshd as a normal user, as long as you only try to log in as that same user.

Invocation looks like, /opt/homebrew/sbin/sshd -d -D -f /tmp/my_ssh/sshd_config. This will start it without forking behavior, process a single login, and give us debug output.

The sshd_config may need StrictModes no depending on the directories in play and the permissions on them.

Various invocations

ssh with weird agent socket

Given I am doing this to test agent + cert muckery, invoking ssh to talk to the sshd mentioned above looks like:

ssh -v \
    -o UserKnownHostsFile=/dev/null \
    -o StrictHostKeyChecking=no \
    -F /dev/null \
    -l $USER \
    -o IdentityAgent=/path/to/agent.sock \
    -p 2222 \
    localhost

Same, but for a local cert instead of agent (assuming user_key is the private key, and has user_key.pub and user_key-cert.pub alongside it):

ssh -v \
    -o UserKnownHostsFile=/dev/null \
    -o StrictHostKeyChecking=no \
    -F /dev/null \
    -l $USER \
    -i path/to/user_key \
    -p 2222 \
    localhost

Given a CA key, generate a user cert locally

ssh-keygen -s  path/to/ca -I wobble -n {{ principals }} .path/to/user_key

Add a comma seperated (no spaces) list of the principals (groups) for {{ principals }} in that invocation (needs to align with the auth principals on sshd). The -I is basically irrevalent for testing purposes, is just a cert identifier.

---

Using More FreeBSD 17 Nov 2023

I do some volunteer sysadmining for a local non-profit, mostly network management and related things. Between there, and my home setup, I have gradually moved from defaulting to Ubuntu back towards FreeBSD over the last year. Nothing wrong with Ubuntu, but I kept finding FreeBSD to simply be a little bit easier and a lot better documented. For a dev box, I'll stick with linux—all the tooling optimizes there first, in particular the container ecosystem is too useful to walk away from. For sysadmin'y stuff though, FreeBSD is where it's at. At a minimum, it's rc system is much easier to work with than systemd :-)

My general toolkit for running anything is a pair of refurbished HP Prodesks sharing a CARP virtual interface using devd events if I need to do anything on failover. It's shockingly easy and just works.

(side note: I tested 14.0-RELEASE on a pair of them and discovered that if a server is hosting a VIP and needs to to use that virtual IP when it is the backup, say to find DNS, you need to disable net.inet.ip.source_address_validation in sysctl, but then it works fine. I probably shouldn't do this, but I'll sort that out later.)

(side note 2: With those old HP Prodesks, you need to go into the BIOS and disable UEFI (switch to use "legacy boot" for the FreeBSD install to work). You don't need to for Ubuntu.)

---