IP Whitelisting Using Istio Policy On Kubernetes Microservices

Recently, we explored Preserving the Source IP address on AWS Classic Loadbalancer and Istio’s envoy using the proxy protocol in our first Part. Continuing to the second part of this series, we will look at How can we apply IP whitelisting on the Kubernetes microservices!

Problem Statement:

There are some microservices behind an internet-facing loadbalancer that we want to have limited access to, based on source IP address. This will prevent our microservices from unauthorized access.

The environment took for implementing this scenario:

1. Kubernetes Environment (Kubernetes v-1.15.3)
2. Service Mesh using Istio

This blog is divided into solution for Version 1.4 and 1.5/1.6

Solution For Version 1.4

In Istio’s component called Mixer, you can apply IP whitelisting using Mixer Policy. The Envoy sidecar logically calls Mixer before each request to perform precondition checks. Therefore in precondition checks, we apply a policy to restrict and allow access to our microservices.

So we have to perform these two steps to reach our goal:
1. Enabling Istio Policy.
2. Build and use policy.

Enabling Policy Enforcement

The mixer policy is deprecated in Istio 1.5

In the default Istio installation profile, policy enforcement is disabled. To install Istio with policy enforcement on, use the --set values.global.disablePolicyChecks=false and --set values.pilot.policy.enabled=true install option.

Ex:

istioctl manifest apply --set values.global.disablePolicyChecks=false --set values.pilot.policy.enable=true

Or if you already setup Istio, you can use these steps:

1. kubectl -n istio-system get cm istio -o yaml > policy-enforce.yaml
2. vim  policy-enforce.yaml
3. Update to disablePolicyChecks: true to disablePolicyChecks: false
4. kubectl apply -f policy-enforce.yaml
5. kubectl -n istio-system get cm istio -o jsonpath="{@.data.mesh}" | grep disablePolicyChecks

Output:
disablePolicyChecks: false

Background Of Istio’s Mixer Components

I suggest going through the mixer components once before moving to create and apply policy.

Build And Use Policy In 1.4

Once Policy Enforcement is enabled, you have to create Istio instance, handler, and rule as follow:

apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: sourceip
  namespace: istio-system
spec:
  compiledTemplate: listentry
  params:
     value: request.headers["x-forwarded-for"]
---
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: whitelistip
  namespace: istio-system
spec:
  compiledAdapter: listchecker
  params:
    overrides: ["1.2.*.*/32","3.4.*.*/32","33.4.*.*/32"]
    blacklist: false
    entryType: IP_ADDRESSES
---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: checkip
  namespace: istio-system
spec:
  match: source.labels["app"] == "istio-ingressgateway"
  actions:
  - handler: whitelistip
    instances: [ sourceip ]

1. In kind: instance, we create a template of type listentry and pass the attribute request.headers[“x-forwarded-for”]. Attribute request.headers[“x-forwarded-for”] provide an actual source IP to handler and handler will use it to validate with IP addresses that we will store in adapter listchecker (in next step).

2. In kind: handler, we create an adapter of type listchecker and provide all the authorized IP addresses.
entryType: IP_ADDRESSES means, we are defining that attributes which adapter get from instance are of type IP_ADDRESSESS

3. In kind: rule, we define a match condition, which means, when to validate the IP address in handler with IP address provided by instance.

You can check whether Instance, Handler, and Rule are created or not.

1. kubectl get instance -n istio-system
2. kubectl get handler -n istio-system
3. kubectl get rule -n istio-system

Solution For Version 1.5/1.6

Build And Use Policy In 1.6

Policy Enforcement is by default enabled in V-1.5/1.6

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: whitelistip
 namespace: istio-system
spec:
 selector:
   matchLabels:
     istio: ingressgateway
 action: ALLOW
 rules:
 - from:
   - source:
       ipBlocks: ["3.55.*.*/32"]

Optional: Creating Custom HTTP Header Using Envoy Lua Filter

If your loadbalancer is behind some other proxy server like Cloudflare, you have to create custom Http header instead using ‘x-forwarded-for’, because the ‘x-forwarded-for header’ contains 2 or more IPs (one of Cloudflare and other of your classic load balancer), means attribute “request.headers” contains List data type instead of the string data type.

In order to create your custom Http header, you can use Envoy Filter. Below is the example EnvoyFilter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingressgateway-user-ip
  namespace: istio-system
spec:
  workloadLabels:
    app: istio-ingressgateway
  filters:
    - listenerMatch:
        portNumber: 443
        listenerType: ANY
      filterName: envoy.lua
      filterType: HTTP
      filterConfig:
        inlineCode: |
          function envoy_on_request(request_handle)
            local xff_header = request_handle:headers():get("X-Forwarded-For")
            local first_ip = string.gmatch(xff_header, "(%d+.%d+.%d+.%d+)")();
            request_handle:headers():add("x-client-ip", first_ip);
          end

    - listenerMatch:
        portNumber: 80
        listenerType: ANY
      filterName: envoy.lua
      filterType: HTTP
      filterConfig:
        inlineCode: |
          function envoy_on_request(request_handle)
            local xff_header = request_handle:headers():get("X-Forwarded-For")
            local first_ip = string.gmatch(xff_header, "(%d+.%d+.%d+.%d+)")();
            request_handle:headers():add("my-custom-header", first_ip);
          end

In the above example, I have defined function func envoy_on_request(request_handle) which will always execute when getting request on ingress-gateway proxy for ports 80 and 443. Also, I have defined a custom header named “my-customer-header” so you can use request.headers[“my-custom-header”] in listener instead of request.headers[“x-forwarded-for”].

Run the following command to check whether Lua script is applied or not on the ingress-gateway.

istioctl proxy-config listeners $(kubectl get pods -n istio-system -lapp=istio-ingressgateway -o=jsonpath='{.items[0].metadata.name}') -n istio-system  -o json

Output:

    {
        "name": "0.0.0.0_80",
        "address": {
            "socketAddress": {
                "address": "0.0.0.0",
                "portValue": 80
         ....
         ....
        
                            "httpFilters": [
                                {
                                    "name": "envoy.lua",
                                    "config": {
                                        "inlineCode": "function envoy_on_request(request_handle)\n  local xff_header = request_handle:headers():get(\"X-Forwarded-For\")\n  local first_ip = string.gmatch(xff_header, \"(%d+.%d+.%d+.%d+)\")();\n  request_handle:headers():add(\"my-custom-header\", first_ip);\nend\n"
                                    }
                                },
     ....
     .....

After creating custom http header “my-custom-header”, you can use this in Policy as follow:

In Version 1.4

apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: whitelistip
  namespace: istio-system
spec:
  compiledAdapter: listchecker
  params:
    overrides: ["3.44.*.*/32"]
    blacklist: false
    entryType: IP_ADDRESSES
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: sourceip
  namespace: istio-system
spec:
  compiledTemplate: listentry
  params:
   value: request.headers["my-custom-header"] | "0.0.0.0"
  
---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: checkip
  namespace: istio-system
spec:
  match: source.labels["app"] == "istio-ingressgateway"
  actions:
  - handler: whitelistip
    instances: [ sourceip ]
---

In Version 1.5/1.6

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: whitelistip
 namespace: istio-system
spec:
 selector:
   matchLabels:
     istio: ingressgateway
 action: ALLOW
 rules:
 - when:
   - key: request.headers[my-custom-header]
     values: ["3.44.*.*"]

What’s next?

In the coming posts, we will see “kibana dashboard for Ingress gateway logs using EFK”.

More on Mixer components?

Mixer Configuration Model
Mixer Compiled In Adapter Dev Guide
Istio’s mixer policy enforcement with custom adapters