Photo by Vlad Hilitanu on Unsplash

YugabyteDB Security

In this blog, I attempt to set up TLS for yugabytedb in a hard way. I hope this can clarify and help troubleshoot TLS issues. To simplify and focus on TLS, I first set up a minimum pre-requisites. Then I step through the client or/and server configurations to reach levels of security requirements (no authentication, password authentication, encryption in transit, verify-ca, verify-full, and mTLS. In this process, I “guess” (think out loud) to troubleshoot errors I encountered.

0. Vanilla single node YugabyteDB

Here is a script to get a single node YugabyteDB running…

1. No authentication

After having a vanilla YugabyteDB installed and running, you should be able to login to ysqlsh prompt like the following:

As you can see, ysqlsh login as yugabyte (superuser), and no password prompted. That’s because the gflag, ysql_enable_auth, is set to false in default and the ysql_hba file looks like this:

“trust” means no question asks, you can be whoever you want.

2. Turn on Authentication

No authentication is NO to OLTP. Let’s try to turn on the basic authentication. Add the following gflag to the tserver config file and restart the tserver:

Now ysqlsh prompt is asking password for user yugabyte (hint: the default password for superuser yugabyte is yugabyte).

Let’s see what has changed in ysql_hba.

It changed to md5. Two recommendations about the default authentication: 1) change the default password and 2) replace md5 with scram-sha-256. “md5” hash algorithm is no longer considered secure and YugabyteDB supports scram-sha-256. To change the default password, follow the steps.

Add two flags in tserver flagfile. This will turn off the authentication, so yugabyte can login without password (yugabyte password is using md5 hash).

In ysqlsh prompt, change yugabyte password by:

Since you turned on the scram-sha-256 password, the new password is hashed with scram-sha-256 algorithm. Now turn on basic authentication by modifying the flag file, then restart the tserver.

3. TLS encryption with Let’s Encryption dns01

The first level of TLS security is “encryption in transit”. To setup YugabyteDB with encryption in transit, you need x509 certificate for the VM. Here I use Let’s Encrypt dns01 challenge and certbot to setup server certificate/key pair. The certbot version in yum repo is old and the certbot official site recommends using snapd to get the current version. Here, I use google clouddns to setup the yugabytedb fqdn.

Once this is done, you can check /etc/letsencrypt/live/$domain/ and find the server certificate (fullchain.pem) and key pair (privkey.pem).

You can check the cert content and make sure the vm fqdn matches the subject in the certificate. In my case, I have a wildcard certificate that can be shared by other vms in the same sub-domain.

Now we have the server certificate and keypair, we can turn on TLS encryption on tserver. Add the following flags in the tserver flagfile.

Copy certs and key to /opt/yugabyte as the flag (certs_for_client_dir) we specified.

Restarting tserver failed with the ERROR in the log.

ERROR: could not load server certificate file "/etc/letsencrypt/live/ysung.fans/node.0.0.0.0.crt": Permission denied

It looks like (I guess) tserver is looking for cert/key file that matches the binding IP of the process. I changed the cert/key naming to match the file naming. Restart failed again due to:

2021-11-05 18:11:59.517 UTC [89601] FATAL: could not load root certificate file "/opt/yugabyte/ca.crt": No such file or directory 2021-11-05 18:11:59.517 UTC [89601] LOG: database system is shut down

This does not make sense because I don’t need ca.crt when I just want data encryption in transit. According to PostgreSQL doc, the ca.crt is the certificate of CA who signs the client certificate. Since I am not doing a client certificate, why do I need this in my server-side flag?

Anyway, I add the freeIPA ca.crt (not Let’s encrypt CA-chain) to the /opt/yugabyte.

After the ca.crt copied, the tserver started without error.

Let’s check what tserver added to ysql_pg.conf.

In sum, to turn on TLS encryption you need:

1. Server certificate and keypair
2. ca.crt??
3. two flags (certs_for_client_dir and use_client_to_server_encryption)

4. TLS server verification

The purpose of yugabytedb server verification is to prevent server identity spoofing. According to PostgreSQL doc, both verify-ca and verify-full are for server identity verification. So verify-ca and verify-full are client-side configurations, aka the client verifies the identity of the yugabytedb server by checking if the CA (signed server’s certificate) is trustworthy (verify-ca) or if the yugabytedb CN or SAN matches the hostname FQDN (verify-full).

What does a yugabytedb client need to verify when connecting to tserver? According to the TLS handshake protocol, a client initiates a TLS connection, then the tserver will send back its certificate. “verify-ca” happens when the client receives the server’s certificate and check if the issuer is trustworthy. So the client will need the CA certificate which signs the server’s certificate in the client ca trust store.

Since the server certificate is signed by Let’s Encrypt, the client will need to download the Let’s Encrypt CA chain. I follow the PostgreSQL client configuration steps:

Once those files were in place, I tried the ysqlsh command with sslmode=verify-ca like this:

The connection was refused! I checked the log and found the error, unknown CA. I guess the client system (ybnode1) doesn’t trust the Let’s Encrypt CA chain.

2021-11-03 19:48:59.957 UTC [61006] LOG: database system was shut down at 2021-11-03 19:48:56 UTC I1103 19:48:59.962649 61007 webserver.cc:172] Starting webserver on 0.0.0.0:13000 I1103 19:48:59.962894 61007 webserver.cc:177] Document root: /opt/yugabyte-2.9.1.0/www I1103 19:48:59.963232 61007 webserver.cc:167] Webserver listen spec is 0.0.0.0:13000 I1103 19:48:59.963698 61007 webserver.cc:266] Webserver started. Bound to: http://0.0.0.0:13000/ 2021-11-03 19:48:59.970 UTC [60978] LOG: database system is ready to accept connections 2021-11-03 19:49:02.325 UTC [61012] LOG: could not accept SSL connection: tlsv1 alert unknown ca

The following is the centos way to add the CA chain to the system.

Let’s try again.

Great! I have verify-ca working. In sum, you need the following to get verify-ca mode from server-side and client-side.

• server-side

1. The server certificate and keypair
2. ca.crt??
3. two flags (certs_for_client_dir and use_client_to_server_encryption)

• client-side

1. root.crt (who signs the yugabytedb server)
2. add root.crt to system ca trust store

Next, let’s check what happens when we just switch to sslmode=verify-full.

Ahh…when using verify-full, the client, ysqlsh, not only check if the ca is trustworthy but also attempt to match the “CN” or “SAN” in server certificate with server FQDN. As I use Google CloudDNS, my yugabytedb server FQDN (ybnode1.ysung.fans) is associated with external IP. At the same time, yugabytedb is listening on eth0 with internal IP (GCP VM doesn’t know its external IP).

When ysqlsh tries to resolve the server identity from FQDN (ybnode1.ysung.fans), but the server hostname is controlled by GCP internal DNS. ‘nslookup ybnode1’ shows the external IP.

The workaround is resetting the hostname and making the internal resolution to the same FQDN.

Also, remember to change the “listen addresses” in both flagfiles from localhost to fqdn.

Let’s try again with verify-full.

In sum, verify-ca and verify-full are for clients to verify server identity. In addition to verify-ca configuration requirements, you will need server FQDN to match the “CN” or “SubjectAltName” in the server certificate.

5. TLS mutual trust (mTLS)

In the beginning, we shortly discuss user authentication using scram-sha-256. It is a password-based authentication. It is tedious and not secure to manage database users with password-based authentication. Refer to pg_hba doc for all support authentication in yugabytedb. In the last part, I will show one of the authentication, client certification which fits the cloud-native scenario.

We spent a lot of time verifying server identity. Yugabtedb also supports client verification (authentication). As I mentioned in part3, turning on TLS encryption doesn’t need ca.crt in server-side. But I added the ca.crt to /opt/yugabyte to get yugabyted started. The ca.crt is actually the CA who signs the user certificate.

I have a freeIPA user, pguser1, created with its x509 certificate and key pair created.

Download the user certificate to the yugabyte user home. For clarification, the root.crt under /opt/yugabyte/.yugabytedb is the certification of CA who signs our yugabytedb server.

To user client cert in yugabytedb, you need to create a database role that matches the “CN” in client certification. In my case, it is pguser1.

Next, we need to add hba rule to tserver flagfile and restart the server.

Now, if you run the following ysqlsh command, you will find that you log in as pguser1 without a password although you preset the pguser1 role. This implies that whoever has this user certificate and keypair can access yugabytedb as the role.

To improve the security, you can combine scram-sha-256 with the client certificate by modifying the hba flag and restarting the server.

If you try login in, you should be prompted with a password.

The hba rules now look like this.

In sum, you need the following to get yugabytedb mTLS.

• server-side

1. all requirements from server authentication
2. ca.crt who signs the client certificate
3. hba rules

• client-side

1. root.crt who signs server certificate
2. client key and certificate

mTLS is pretty much a “must-have” in cloud-native as services come and go all the time. In the last section, I learned how to set up mTLS. Next, we can try multi-tenancy mTLS to accept users from different groups/orgs/companies to a single yugabyteDB. In theory, we can concat multiple ca.crt (user certificate issuers) to yugabytedb server (/opt/yugabyte/ca.crt). This way, yugabytedb can verify the user certificate as long as its issuer is “trusted” by the yugabytedb. The next step is using hba rules to segregate db access and map function to map “CN” in the certificate to different yugabytedb roles.

That’s for today. Stay tuned.