MTLS over k8s/nginx ingress controller

  1. Setup test service and ingress
kubectl get pods -n ingress-nginx
kubectl create deployment test --image=httpd --port=80
kubectl expose deployment test
kubectl create ingress test-localhost --class=nginx --rule="test.localdev.me/*=test:80"
kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 8080:80

curl http://test.localdev.me:8080/

2. Enable TLS

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt -subj "/CN=test.localdev.me/O=test.localdev.me"
kubectl create secret tls self-tls --key server.key --cert server.crt

kubectl edit ingress >
  tls:
  - hosts:
    - test.localdev.me
    secretName: self-tls

sudo kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 443:443

curl -kv https://test.localdev.me/

3. Enable MTLS

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority'
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt
openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client'
openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt
kubectl edit ingress >
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"

curl -vk https://test.localdev.me/

Ref:

OpenSSL basics

self-signed certificate:

# Generate a self-signed certificate and a new private key
openssl req -x509 -days 1825 -nodes -newkey rsa:2048 -keyout KEY.key \
-out CERT.crt -subj "/C=US/ST=Chicago/L=Town/O=Abc Inc/CN=example.com"

# Generate a self-signed certificate using existing private key
openssl req -x509 -days 365 -key KEY.key -out CERT.crt -subj "/C=US/O=IBM/CN=ibm.com"

# Create a public key
openssl rsa -in myprivate.key -pubout > mypubkey.pub
openssl rsa -in myprivate.key -out mypubkey.pub -pubout -outform PEM

## List self-signed certificates
openssl x509 -text -noout -in /tmp/mycert.pem
openssl x509 -inform der -text -in /tmp/mycert.cer
openssl pkcs12 -in /tmp/mycert.p12 -password pass:mypass -nokeys

## Create self-signed certificate using CA
openssl genrsa -out user1.key 2048
openssl req -new -key user1.key -out user1.csr -subj "/CN=User1/O=development"
sudo openssl x509 -req -in user1.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user1.crt -days 45

# Create a local Certifcate Authority (CA) and install it in the default trust store
openssl genrsa -out ca.key 2048                                     # create a CA private key
openssl req -new -key ca.key -out ca.csr -subj "/CN=Example-CA"  # create a CA certificate signing request
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt            # Create a CA cert
sudo cp ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Create a certificate and sign is using local CA
openssl genrsa -out example.key 2048                                    # Create a cert key
openssl req -new -key example.key -out example.csr -subj "/C=AU/ST=WA/L=Perth/O=Example/CN=*.example.com"     # Create a cert signing request
openssl x509 -req -in admin.csr -CA ca.crt -CAKey ca.key -CAcreateserial -out example.crt     # Create a cert and sign using CA
openssl verify -CAfile ca.crt example.crt                                  # Verify certificate

Certificate signing request (CSR):

# Generate CSR and create a new private key
openssl req -new -nodes -newkey rsa:2048 -keyout NEW_KEY.key -out CERT.csr \
-subj "/C=OM/ST=Muscat/L=Muscat/O=Abc/OU=IT/CN=abc.com/emailAddress=admin@abc.com"

# Generate CSR using an existing private key
openssl genrsa -out MYKEY.key 2048            # Generate a private key
openssl req -new -key MYKEY.key -out CERT.csr \
-subj "/C=OM/ST=Muscat/L=Muscat/O=Abc/OU=IT/CN=abc.com"

# List CSR
openssl req -text -noout -in CERT.csr

Verify CSR, Certificate and Private key:

## Verify CSR, private key and certificate (md5 hash values must be same)
openssl req -in MyCsr.csr -outform pem -pubkey -noout | sha256sum
openssl x509 -in MyCert.crt -outform pem -pubkey -noout | sha256sum
openssl pkey -in MyKey.key -outform pem -pubout | sha256sum

openssl req -in MyCsr.csr  -modulus -noout | openssl md5 | cut -d' ' -f2
openssl x509 -in MyCert.crt -modulus -noout | openssl md5 | cut -d' ' -f2
openssl rsa -in MyKey.key -modulus  -noout | openssl md5 | cut -d' ' -f2

## List CSR and private key
openssl req -noout -text -in www_mydomain_com.csr   # List CSR
openssl rsa -check -in /tmp/mycert.key              # List and check private key
openssl x509 -noout -text -in /tmp/mycert.pem       # List certificate

## Remove password from private key
openssl rsa -in mykey.key -out mykey.key

# Verify cert and chain
openssl verify -CAfile CA.pem cert.pem
openssl verify -verbose -x509_strict -CAfile ca.pem [-CApath .] cert.pem

# Error Can't load /home/ubuntu/.rnd into RNG
sudo sed -i '0,/RANDFILE/{s/RANDFILE/\#&/}' /etc/ssl/openssl.cnf

pfx | p12:

## Verify pk12 or pfx cert
openssl pkcs12 -info -nodes -in cert.pfx | openssl x509 -noout -text

## Convert pem to p12|pfx format
openssl pkcs12 -export -inkey cert.key -in cert.pem -out [cert.p12|cert.pfx] [-password pass:pass1234 -name mycert]
openssl pkcs12 -info -in cert.p12 -password pass:pass1234 -nokeys

## Convert pfx to pem (nodes: don't encrypt)
openssl pkcs12 -in CertificateBundle.p12 -out CertificateBundle.pem -nodes

## Convert OPENSSH PRIVATE KEY to RSA PRIVATE KEY (PEM) format
cp ~/.ssh/id_rsa ~/.ssh/id_rsa.bak
ssh-keygen -p -N "" -m pem -f ~/.ssh/id_rsa

## Export private key and certificate and CA chain from pfx
openssl pkcs12 -nodes -nocerts -in cert.pfx -out cert.key     # Export private key
openssl pkcs12 -nokeys -clcerts -in cert.pfx -out cert.pem    # Export certificate
openssl pkcs12 -nokeys -cacerts -in cert.pfx -out ca.pem      # Export ca certificate chain

## Verify cert and chain
openssl rsa -in cert.key -check             # List private key
openssl x509 -in cert.pem -text -noout      # List certificate
openssl x509 -in ca.pem -text -noout        # List CA
openssl verify -verbose -x509_strict -CAfile ca.pem -CApath . cert.pem

## To import CA certs along with server cert into p12, create a single pem file as
cat cert.pem intermediate.pem root.pem > ca.pem  # In cert chain file, the cert.pem need to be first, followed by intermediate.pem and Root.pem would be in the last

Verify server certificate expiry date and accepted client certificates:

curl -k -v --cert example.p12 --key example.pem https://example.com

# List all server certs
echo|openssl s_client -connect localhost:443 -showcerts | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
echo | openssl s_client -connect google.com:443 2>/dev/null | openssl x509 -text -noout

# Show expiry dates of server SSL certificate
echo|openssl s_client -connect localhost:443 2&> /dev/null | openssl x509 -dates -noout

# Saves sever SSL certificate
openssl s_client -showcerts -connect www.example.com:443 > cert.der

base64:

# Encode username and password using base64
echo -n 'admin:password' | base64                 # Encode data
echo "YWRtaW46cGFzc3dvcmQ=" | base64 --decode     # Decode data

# Generate a random 32 chars key
head -c 32 /dev/urandom | base64

# Generate a random string of length 20
openssl rand -base64 20

# Encode keys
grep certificate-authority-data ~/.kube/config | \
cut -d " " -f 6 | base64 -d > ca.pem

Encrypt and decrypt files:

# Create a secret test file
$ echo "My top secret text" > secrets.txt

# Encrypt the file
$ openssl aes-256-cbc -a -salt -in secrets.txt -out secrets.txt.enc -k password

# Decrypt the file
$ openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new -k password

# Verify
$ cat secrets.txt.new
My top secret text

keytool:

## List certificates in a keystore
$JAVA_HOME/keytool -list -v -keystore mykeystore.jks -storepass pass123 -noprompt [-alias root]

## Convert/Import p12 certificate into JKS keystore
$JAVA_HOME/bin/keytool -importkeystore -srckeystore mycert.p12 \
-srcstoretype pkcs12 -srcstorepass pass123 -destkeystore mykeystore.jks \
-deststorepass pass123 [-deststoretype jks -alias mycert -destailas mycert]

## Import trusted Intermediate or Root chain certificate into JKS keystore
$JAVA_HOME/bin/keytool -import -trustcacerts -alias intermediate \
-file intermediate.cer -keystore mykeystore.jks -storepass pass123 -noprompt

References:

<pre class="wp-block-syntaxhighlighter-code">https:&#047;&#047;www.sslshopper.com/csr-decoder.html

<blockquote class="wp-embedded-content" data-secret="zUUmx5jhTw"><a href="https://mihail.stoynov.com/2009/03/12/certificates-keystores-java-keytool-utility-and-openssl/">Certificates, keystores, java keytool utility and openssl</a></blockquote><iframe class="wp-embedded-content" sandbox="allow-scripts" security="restricted" style="position: absolute; clip: rect(1px, 1px, 1px, 1px);" title="&#8220;Certificates, keystores, java keytool utility and openssl&#8221; &#8212; Mihail Stoynov's blog" src="https://mihail.stoynov.com/2009/03/12/certificates-keystores-java-keytool-utility-and-openssl/embed/#?secret=7q2x0gn6yA#?secret=zUUmx5jhTw" data-secret="zUUmx5jhTw" width="584" height="329" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe>

https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs

https://www.sslshopper.com/article-most-common-openssl-commands.html

http://curl.haxx.se/docs/sslcerts.html

http://www.rackspace.com/knowledge_center/article/generate-a-csr-with-openssl

https://redkestrel.co.uk/articles/CSR-FAQ/

https://www.digicert.com/easy-csr/openssl.htm

https://www.sslshopper.com/certificate-key-matcher.html

https://geekflare.com/openssl-commands-certificates/</pre>