![](/img/guoping.png)
Exposing applications using Ingress and TLS termination on Kubernetes in HPE GreenLake for Private Cloud Enterprise
March 20, 2024Overview
HPE GreenLake for Private Cloud Enterprise: Containers, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.
Once applications are deployed in a K8s cluster, the next step is to create services that expose these applications. By default, K8s services are created with the ClusterIP type, which supports internal connectivity among different components of the applications. However, these services are not directly accessible from outside the cluster. It can be challenging to securely expose the deployed applications over HTTPS. This involves generating and managing SSL/TLS certificates for multiple applications deployed in the cluster. These certificates are crucial for secure communication between services. Installing them and managing them correctly is essential to avoid access issues and security risks.
To address exposing applications over HTTPS, K8s provides the concept of Ingress. An Ingress acts as an entry point for external traffic into the cluster. It can be configured with TLS termination. However, setting up K8s Ingress with TLS termination is intricate. It involves creating a K8s Secret to host the certificate and referencing the secret in the Ingress resource. It also requires an additional load balancer configuration in the cluster.
This blog post outlines the comprehensive steps used to expose applications using Ingress and TLS termination on K8s in HPE GreenLake for Private Cloud Enterprise. MetalLB is deployed to the cluster to set up the load balancer. It enables external access to services within the cluster. Cert-manager is used for creating and managing SSL/TLS certificates. The generated certificate is stored as a K8s Secret object. This secret can be mounted by application Pods or used by an Ingress controller. The Nginx Ingress controller is deployed and configured in the cluster. It handles SSL certificates and facilitates secure access to applications in the backend.
Despite the complexities, securely exposing applications in a K8s cluster over HTTPS is attainable. It can be achieved by leveraging Ingress and TLS termination, along with a suite of suitable tools and utilities deployed within the K8s cluster in HPE GreenLake for Private Cloud Enterprise.
Prerequisites
Before starting, make sure you have the following:
- A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise
- The kubectl CLI tool, together with the kubeconfig file for accessing the K8s cluster
- The helm CLI tool, version 3.12.0 or later
- A domain and a list of subdomains to generate the SSL certificate and host the applications in the cluster
Set up the load balancer with MetalLB
You can install MetalLB and set up the load balancer in the K8s cluster by following the instructions found in the blog post Setting up the load balancer with MetalLB.
Here is the deployed MetalLB to the namespace metallb-system in the cluster:
$ kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/controller-57b4fdc957-56wv8 1/1 Running 0 22m pod/speaker-c7sgk 1/1 Running 0 22m pod/speaker-dtlpm 1/1 Running 0 22m pod/speaker-gxccz 1/1 Running 0 22m pod/speaker-pwl87 1/1 Running 0 22m pod/speaker-rvvkz 1/1 Running 0 22m pod/speaker-wxd5n 1/1 Running 0 22m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/webhook-service ClusterIP 10.102.54.20 <none> 443/TCP 22m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/speaker 6 6 6 6 6 kubernetes.io/os=linux 22m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/controller 1/1 1 1 22m NAME DESIRED CURRENT READY AGE replicaset.apps/controller-57b4fdc957 1 1 1 22m
You can see the range of virtual IP addresses, "10.6.115.251-10.6.115.254", defined in the CRD resource IPAddressPool, and the layer 2 service IP address announcement in the CRD resource L2Advertisement:
$ kubectl get ipaddresspools -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES cfe-pool true false ["10.6.115.251-10.6.115.254"] $ kubectl get l2advertisements -n metallb-system NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES cfe-l2advert ["cfe-pool"]
Deploy Nginx Ingress controller
In order for an Ingress to work in the cluster, there must be an Ingress controller being deployed and running. It's the Ingress controller that accesses the certificate and the routing rules defined on the Ingress resource and makes them part of its configuration.
A variety of Ingress controllers are available for deployment in the cluster, including Traefik, HAProxy and Nginx Ingress controller. Execute the command below to install the Nginx Ingress controller to the cluster using Helm:
$ helm upgrade --install ingress-nginx ingress-nginx \ > --repo https://kubernetes.github.io/ingress-nginx \ > --namespace ingress-nginx --create-namespace Release "ingress-nginx" does not exist. Installing it now. NAME: ingress-nginx LAST DEPLOYED: Wed Mar 6 18:30:55 2024 NAMESPACE: ingress-nginx STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the load balancer IP to be available. You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - pathType: Prefix backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key> type: kubernetes.io/tls
The Nginx Ingress controller is deployed to the namespace ingress-nginx in the cluster. Type the following command to check the deployment details:
$ kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-controller-548768956f-8bz2q 1/1 Running 0 15m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.108.173.7 10.6.115.251 80:32734/TCP,443:32265/TCP 15m service/ingress-nginx-controller-admission ClusterIP 10.108.100.150 <none> 443/TCP 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 15m NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-548768956f 1 1 1 15m
The service ingress-nginx-controller gets deployed as the service type of LoadBalancer with the EXTERNAL-IP assigned as 10.6.115.251. This IP address will be used for setting up domain and subdomain name resolution.
Generate a self-signed certificate using cert-manager
You can deploy cert-manager to the K8s cluster and generate a self-signed certificate by following the instructions found in the blog post Generating self-signed certificates using cert-manager.
Here is the cert-manager deployed to the namespace cert-manager in the cluster:
$ kubectl get all -n cert-manager NAME READY STATUS RESTARTS AGE pod/cert-manager-59fbb6655d-h7sqb 1/1 Running 0 18s pod/cert-manager-cainjector-69548575fb-7fqd2 1/1 Running 0 18s pod/cert-manager-webhook-57b78f476d-mp45s 1/1 Running 0 16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.107.221.97 <none> 9402/TCP 20s service/cert-manager-webhook ClusterIP 10.104.243.185 <none> 443/TCP 19s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 18s deployment.apps/cert-manager-cainjector 1/1 1 1 18s deployment.apps/cert-manager-webhook 1/1 1 1 17s NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-59fbb6655d 1 1 1 19s replicaset.apps/cert-manager-cainjector-69548575fb 1 1 1 19s replicaset.apps/cert-manager-webhook-57b78f476d 1 1 1 18s
Below is the deployed self-signed custom resource definition (CRD) Issuer in the namespace nginx-apps in which you want to generate the certificate:
$ kubectl get issuer -n nginx-apps NAME READY AGE cfe-selfsigned-issuer True 115s
Here is the generated self-signed certificate in the namespace nginx-apps:
$ kubectl get certificate -n nginx-apps NAME READY SECRET AGE cfe-selfsigned-tls True cfe-tls-key-pair 2m23s
The K8s Secret 'cfe-tls-key-pair' is created automatically in the same namespace as part of certificate deployment:
$ kubectl get secrets -n nginx-apps cfe-tls-key-pair NAME TYPE DATA AGE cfe-tls-key-pair kubernetes.io/tls 3 2m25s
Type the following command to check the commonName and the dnsNames in the generated certificate:
$ kubectl describe certificate cfe-selfsigned-tls -n nginx-apps Name: cfe-selfsigned-tls Namespace: nginx-apps Labels: <none> Annotations: <none> API Version: cert-manager.io/v1 Kind: Certificate Metadata: Creation Timestamp: 2024-03-06T17:58:51Z Generation: 1 Managed Fields: API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:commonName: f:dnsNames: f:isCA: f:issuerRef: .: f:kind: f:name: f:secretName: Manager: kubectl-client-side-apply Operation: Update Time: 2024-03-06T17:58:51Z API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:revision: Manager: cert-manager-certificates-issuing Operation: Update Subresource: status Time: 2024-03-06T17:58:52Z API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:conditions: .: k:{"type":"Ready"}: .: f:lastTransitionTime: f:message: f:observedGeneration: f:reason: f:status: f:type: f:notAfter: f:notBefore: f:renewalTime: Manager: cert-manager-certificates-readiness Operation: Update Subresource: status Time: 2024-03-06T17:58:52Z Resource Version: 2128063 UID: 977eaa8a-1612-489b-a34d-0e78ab113096 Spec: Common Name: example.com Dns Names: green.nginx.example.com blue.nginx.example.com nginx.example.com example.com Is CA: true Issuer Ref: Kind: Issuer Name: cfe-selfsigned-issuer Secret Name: cfe-tls-key-pair Status: Conditions: Last Transition Time: 2024-03-06T17:58:52Z Message: Certificate is up to date and has not expired Observed Generation: 1 Reason: Ready Status: True Type: Ready Not After: 2024-06-04T17:58:52Z Not Before: 2024-03-06T17:58:52Z Renewal Time: 2024-05-05T17:58:52Z Revision: 1 Events: <none>
Deploy sample Nginx applications
In order to configure and validate the Ingress TLS termination, three sample Nginx applications will be deployed to the cluster using the YAML manifest files from the GitHub repo ingress-demo:
$ tree ingress-demo/ ingress-demo/ ├── apps │ ├── nginx-blue.yaml │ ├── nginx-green.yaml │ └── nginx-main.yaml ├── ingress-host-based-selfsigned.yaml ├── ingress-path-based-selfsigned.yaml └── README.md
Each YAML manifest file in the folder 'apps' defines the Deployment and the Service resource.
Type the following commands to deploy those Nginx applications to the namespace nginx-apps:
$ cd ingress-demo/ $ kubectl apply -f apps/nginx-main.yaml -n nginx-apps service/nginx-main created deployment.apps/nginx-main created $ kubectl apply -f apps/nginx-green.yaml -n nginx-apps service/nginx-green created deployment.apps/nginx-green created $ kubectl apply -f apps/nginx-blue.yaml -n nginx-apps service/nginx-blue created deployment.apps/nginx-blue created
Type the command shown below to check the details of each application deployment:
$ kubectl get all -n nginx-apps NAME READY STATUS RESTARTS AGE pod/nginx-blue-78647f4c4b-z8wq9 1/1 Running 0 10s pod/nginx-green-8956bbd9f-zz7hk 1/1 Running 0 22s pod/nginx-main-64bfd77895-tf7xd 1/1 Running 0 31s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-blue ClusterIP 10.108.51.116 <none> 80/TCP 15s service/nginx-green ClusterIP 10.106.115.65 <none> 80/TCP 23s service/nginx-main ClusterIP 10.108.33.44 <none> 80/TCP 32s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-blue 1/1 1 1 15s deployment.apps/nginx-green 1/1 1 1 24s deployment.apps/nginx-main 1/1 1 1 32s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-blue-78647f4c4b 1 1 1 15s replicaset.apps/nginx-green-8956bbd9f 1 1 1 24s replicaset.apps/nginx-main-64bfd77895 1 1 1 32s
Three Nginx services, nginx-main, nginx-blue and nginx-green, are deployed as the ClusterIP type. They provide internal connectivity and can solely be accessed from within the cluster.
Type the following command to check that all the application service endpoints have been populated:
$ kubectl get endpoints -n nginx-apps NAME ENDPOINTS AGE nginx-blue 10.192.3.78:80 1m nginx-green 10.192.4.45:80 1m nginx-main 10.192.4.44:80 1m
Set up Ingress TLS
The Ingress resource with TLS has to be created. Here is the sample Ingress TLS resource ingress-host-based-selfsigned.yaml, available from the GitHub repo ingress-demo:
$ cat ingress-host-based-selfsigned.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-host-based-selfsigned annotations: ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/issuer: "nginx-selfsigned-issuer" spec: ingressClassName: nginx tls: - hosts: - nginx.example.com secretName: cfe-tls-key-pair rules: - host: nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-main port: number: 80 - host: blue.nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-blue port: number: 80 - host: green.nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-green port: number: 80
In the above sample YAML manifest file, there is the 'tls' block that contains the hostname 'nginx.example.com' and the secret cfe-tls-key-pair created in the certification step. There is also the 'rules' block in which a list of routing rules is defined per host, e.g., host nginx.example.com will be routed to the application service nginx-main in the backend.
Type the following command to deploy the Ingress resource to the namespace nginx-apps:
$ kubectl apply -f ingress-host-based-selfsigned.yaml -n nginx-apps ingress.networking.k8s.io/ingress-host-based-selfsigned created
Check the details of the TLS and Rules settings by typing the command shown below:
$ kubectl describe ingress ingress-host-based-selfsigned -n nginx-apps Name: ingress-host-based-selfsigned Labels: <none> Namespace: nginx-apps Address: Ingress Class: nginx Default backend: <default> TLS: cfe-tls-key-pair terminates nginx.example.com Rules: Host Path Backends ---- ---- -------- nginx.example.com / nginx-main:80 (10.192.4.44:80) blue.nginx.example.com / nginx-blue:80 (10.192.3.78:80) green.nginx.example.com / nginx-green:80 (10.192.4.45:80) Annotations: cert-manager.io/issuer: nginx-selfsinged-issuer ingress.kubernetes.io/ssl-redirect: true Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 20s nginx-ingress-controller Scheduled for sync Normal CreateCertificate 20s cert-manager-ingress-shim Successfully created Certificate "cfe-tls-key-pair"
Access deployed Nginx applications
Before accessing the deployed Nginx applications, you need set up the domain and the subdomain name resolution. For the sample domain name nginx.example.com, and its subdomains, blue.nginx.example.com and green.nginx.example.com, the workstation host file has been used for DNS resolution.
Type the following commands to check that this is done correctly:
$ host nginx.example.com nginx.example.com has address 10.6.115.251 $ host green.nginx.example.com green.nginx.example.com has address 10.6.115.251 $ host blue.nginx.example.com blue.nginx.example.com has address 10.6.115.251
You can then validate the Ingres TLS configuration of the deployed Nginx applications using the browser.
Start the browser and type the URL nginx.example.com. It will be redirected over HTTPS with the warning message 'Your connection is not private':
This is due to the fact that the self-signed certifcate is generated in cert-manager and configured in the K8s Ingress resource.
Click Not secure and start the Certificate Viewer to check the certificate:
Click Proceed to nginx.example.com (unsafe). You will then go to the Nginx MAIN page:
Type the URL green.nginx.example.com to the browser. It will be redirected over HTTPS with the same warning message 'Your connection is not private':
Click Proceed to green.nginx.example.com (unsafe). You will then go to the Nginx GREEN page:
The same thing occurs when you type the URL blue.nginx.example.com to the browser. The access will be redirected over HTTPS with the same warning message 'Your connection is not private':
Click Proceed to blue.nginx.example.com (unsafe). You will then go to the Nginx BLUE page:
You have successfully configured the Ingress with the generated TLS certificate and exposed the deployed applications with TLS termination.
Conclusion
This blog post provided a comprehensive guide on how to expose applications and make them accessible securely via HTTPS in a K8 cluster in HPE GreenLake for Private Cloud Enterprise. It detailed the process of configuring TLS termination on an Ingress controller, utilizing a K8s Ingress resource and a self-signed TLS certificate generated with cert-manager. While the emphasis of this post was on self-signed certificates, the outlined procedure is equally applicable to any type of certificates. This flexibility allows customers to follow the steps using their own CA certificates or any commercially issued certificates for Ingress TLS termination, ensuring secure exposure of their applications in the K8s cluster over HTTPS.
Please keep coming back to the HPE Developer Community blog to learn more about HPE GreenLake for Private Cloud Enterprise.
Tags
Related
![](/img/guoping.png)
Deploying Super Mario game on Kubernetes in HPE GreenLake for Private Cloud Enterprise
Mar 29, 2024![](/img/guoping.png)
Getting started with volume snapshots on a Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise
Jan 23, 2024![](/img/guoping.png)
How to backup and restore stateful applications on Kubernetes using Kasten K10 in HPE GreenLake for Private Cloud Enterprise
Jan 26, 2024![](/img/sudhi-1.jpeg)
HPE GreenLake for Private Cloud Enterprise: Mastering cloud migration with the 6Rs approach
Jun 13, 2024![](/img/guoping.png)
Adding a monitoring stack to a Kubernetes cluster using Prometheus and Grafana in HPE GreenLake for Private Cloud Enterprise
Jan 25, 2024![](/img/guoping.png)