Ingress Controllers on Wercker Clusters

This guide explains how to set up an example ingress controller on an existing Wercker cluster. The ingress controller setup is composed of four parts:

  • Default backend deployment—The default backend deployment handles default routes for health checks and 404 responses. This is done by using a stock image that serves the minimum required routes for a default backend.
  • Default backend service—The default backend service exposes the default backend deployment for consumption by the ingress controller deployment.
  • Ingress controller deployment—The ingress controller deployment deploys an image that contains the binary for the ingress controller and nginx. The binary manipulates and reloads the /etc/nginx/nginx.conf configuration file when an Ingress is created in Kubernetes. Nginx upstreams point to services that match specified selectors.
  • Ingress controller service—The ingress controller service exposes the ingress controller deployment as a LoadBalancer type service. Because Wercker Clusters uses an Oracle Cloud Infrastructure Services integration/cloud-provider, a load balancer will be dynamically created with the correct nodes configured as a backend set.

Setting Up the Ingress Controller

Create the Default Backend

Create the file nginx-default-backend-deployment.yaml for the default backend deployment using the following snippet:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

Create the file nginx-default-backend-service.yaml for the default backend deployment using the following snippet:

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend

Using the files above, create the deployment and the service by executing these commands:

kubectl create -f nginx-default-backend-deployment.yaml
kubectl create -f nginx-default-backend-service.yaml

Create the Ingress Controller Nginx Deployment and Service

Create the file nginx-ingress-controller-deployment.yaml for the nginx ingress controller deployment using the following snippet:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-controller
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source 
      # IP correctly regardless of docker configuration.
      # However, it is not a hard dependency of the nginx-ingress-controller
      # itself, and it may cause issues if port 10254 already is taken on 
      # the host.
      # Since hostPort is currently broken on CNI 
      # (https://github.com/kubernetes/kubernetes/issues/31307), we have to
      # use hostNetwork where CNI is used, like with kubeadm.
      # hostNetwork: true
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.10
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --update-status=false

Create the file nginx-ingress-controller-service.yaml for the nginx ingress controller service using the following snippet:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller
  namespace: default
  labels:
    k8s-app: nginx-ingress-controller
spec:
  type: LoadBalancer
  ports:
  - port: 80
    nodePort: 30021
    name: http
  - port: 443
    nodePort: 30022
    name: https
  selector:
    k8s-app: nginx-ingress-controller

Create the nginx ingress controller deployment and service:

kubectl create -f nginx-ingress-controller-deployment.yaml
kubectl create -f nginx-ingress-controller-service.yaml

View the service list. The EXTERNAL-IP for the nginx-ingress-controller is <pending> until the load balancer has been fully created in Oracle Cloud Infrastructure.

$ kubectl get svc
NAME                       CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default-http-backend       10.96.208.64   <none>          80/TCP                       1h
nginx-ingress-controller   10.96.121.145  <pending>       80:30021/TCP,443:30022/TCP   1s
$ kubectl get svc
NAME                       CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default-http-backend       10.96.208.64   <none>          80/TCP                       48m
nginx-ingress-controller   10.96.45.134   129.146.11.154  80:30021/TCP,443:30022/TCP   47m

Using an Ingress with a Service

Create the docker-hello-world Service Definition

Create the file hello-world-ingress.yaml using the snippet below. This snippet uses a publicly available "hello, world" image from Docker Hub. You can substitute another image of your choice that can be run in a similar manner.

The service's type is ClusterIP—not LoadBalancer—because this service will be proxied by the ingress nginx controller. This service does not need public access directly to it. Instead, the public access will be routed from the load balancer to the ingress controller, and from the ingress controller to the upstream service.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: docker-hello-world
  labels:
    app: docker-hello-world
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: docker-hello-world
    spec:
      containers:
      - name: docker-hello-world
        image: scottsbaldwin/docker-hello-world:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: docker-hello-world-svc
spec:
  selector:
    app: docker-hello-world
  ports:
    - port: 8088
      targetPort: 80
  type: ClusterIP

Create this new deployment and service:

kubectl create -f hello-world-ingress.yaml

Create the TLS Secret

A TLS secret is used for SSL termination on the ingress controller. To generate the secret for this example, a self-signed certificate is used. While this is ok for testing, for production, use a certificate signed by a Certificate Authority.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"
kubectl create secret tls tls-secret --key tls.key --cert tls.crt

Note: Under Windows, you may need to replace "/CN=nginxsvc/O=nginxsvc" with "//CN=nginxsvc\O=nginxsvc" (for example, this is necessary when running the openssl command from a Git Bash shell).

Create the Ingress Resource

Create the file ingress.yaml and populate it with this snippet:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-world-ing
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - secretName: tls-secret
  rules:
  - http:
      paths:
      - backend:
          serviceName: docker-hello-world-svc
          servicePort: 8088

Create the resource:

kubectl create -f ingress.yaml

Verify Everything

Before continuing, verify that all of the components have been successfully created and are operating as expected.

Service Ports

At this point, the docker-hello-world-svc service should be running as a ClusterIP service, and the nginx-ingress-controller service should be running as a LoadBalancer service.

$ kubectl get svc --all-namespaces
NAMESPACE     NAME                       CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default       docker-hello-world-svc     10.96.83.247    <none>           8088:31295/TCP               16s
default       kubernetes                 10.96.0.1       <none>           443/TCP                      1h
kube-system   kube-dns                   10.96.5.5       <none>           53/UDP,53/TCP                1h
default       default-http-backend       10.96.208.64    <none>           80/TCP                       1h
default       nginx-ingress-controller   10.96.121.145   129.146.11.154   80:30021/TCP,443:30022/TCP   5m

cURL the Load Balancer

Use the EXTERNAL-IP for the nginx-ingress-controller service (for example, 129.146.11.154) to curl an http request:

$ curl -I http://129.146.11.154
HTTP/1.1 301 Moved Permanently
Via: 1.1 10.68.69.10 (McAfee Web Gateway 7.6.2.10.0.23236)
Date: Thu, 07 Sep 2017 15:20:16 GMT
Server: nginx/1.13.2
Location: https://129.146.11.154/
Content-Type: text/html
Content-Length: 185
Proxy-Connection: Keep-Alive
Strict-Transport-Security: max-age=15724800; includeSubDomains;

The output shows a 301 redirect and a Location header that suggest that http traffic is being redirected to https. Either cURL against the https url or add the -L option to automatically follow the location header. The -k option instructs cURL to not verify the SSL certificates.

$ curl -ikL http://129.146.11.154
HTTP/1.1 301 Moved Permanently
Via: 1.1 10.68.69.10 (McAfee Web Gateway 7.6.2.10.0.23236)
Date: Thu, 07 Sep 2017 15:22:29 GMT
Server: nginx/1.13.2
Location: https://129.146.11.154/
Content-Type: text/html
Content-Length: 185
Proxy-Connection: Keep-Alive
Strict-Transport-Security: max-age=15724800; includeSubDomains;

HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
Server: nginx/1.13.2
Date: Thu, 07 Sep 2017 15:22:30 GMT
Content-Type: text/html
Content-Length: 71
Connection: keep-alive
Last-Modified: Thu, 07 Sep 2017 15:17:24 GMT
ETag: "59b16304-47"
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<h1>Hello webhook world from: docker-hello-world-1732906117-0ztkm</h1>

The last line of the output shows the HTML that is returned from the pod whose hostname is docker-hello-world-1732906117-0ztkm.

Issue the cURL request again to see the hostname in the HTML output change; this demonstrates that load balancing is occurring.

$ curl -k https://129.146.11.154
<h1>Hello webhook world from: docker-hello-world-1732906117-6115l</h1>
$ curl -k https://129.146.11.154
<h1>Hello webhook world from: docker-hello-world-1732906117-7r89v</h1>
$ curl -k https://129.146.11.154
<h1>Hello webhook world from: docker-hello-world-1732906117-0ztkm</h1>

Inspect nginx.conf

The ingress controller manipulates the nginx.conf file within the pod for the nginx-ingress-controller. Find the pod name and use it with a kubectl exec command to show the contents of nginx.conf.

$ kubectl get po
NAME                                       READY     STATUS    RESTARTS   AGE
default-http-backend-726995137-s4lwh       1/1       Running   0          1h
docker-hello-world-6bd6f7dfc8-2htm4        1/1       Running   0          1h
docker-hello-world-6bd6f7dfc8-6jzhs        1/1       Running   0          1h
docker-hello-world-6bd6f7dfc8-9pkw7        1/1       Running   0          1h
nginx-ingress-controller-110676328-h86xg   1/1       Running   0          1h

$ kubectl exec -it nginx-ingress-controller-110676328-h86xg -- cat /etc/nginx/nginx.conf

In the output, look for proxy_pass. There will be one for the default backend and another that looks similar to:

proxy_pass http://default-docker-hello-world-svc-8088;

This shows that nginx is proxying requests to an upstream called default-docker-hello-world-svc-8088. The upstream definition can be found in the output, and looks something like:

upstream default-docker-hello-world-svc-8088 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;
    server 10.244.31.5:80 max_fails=0 fail_timeout=0;
    server 10.244.71.5:80 max_fails=0 fail_timeout=0;
    server 10.244.67.5:80 max_fails=0 fail_timeout=0;
}

The upstream is proxying to three hosts that are listening on port 80. These hosts and port are found in the service description as well:

$ kubectl describe svc/docker-hello-world-svc
Name:                   docker-hello-world-svc
Namespace:              default
Labels:                 <none>
Annotations:            <none>
Selector:               app=docker-hello-world
Type:                   ClusterIP
IP:                     10.96.83.247
Port:                   <unset> 8088/TCP
Endpoints:              10.244.31.5:80,10.244.67.5:80,10.244.71.5:80
Session Affinity:       None
Events:                 <none>