Capturing full requests in Istio with envoy and Lua

Have you ever thought “I wish I could see what this Application is trying to send!” when debugging some applications which refuse to behave?

If you use Istio Service Mesh it is easier than ever to do it.

Istio uses Envoy Proxy sidecar to route all traffic in and out of the application. All we have to do is configure the sidecar proxy to save all details of the request before forwarding it for further inspection.

All of it is possible with the integration of the envoy+Lua. Envoy with Lua scripting – it’s a swiss army knife on steroids.

EnvoyFilter

Istio uses “EnvoyFilter” type to alter the configuration of the envoy proxy. This mechanism allows you to update the configuration of any envoy proxy in your service mesh. It uses labels/selectors to narrow down your changes to only the necessary subset of sidecars.

Lua is a widely supported scripting language, with Nginx, Envoy and other popular Proxy servers having native support for it.

This is an example of the filter which will change the configuration of the Envoy sidecar proxy of all Pods with the label “app: shell” in the namespace “graph”

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: shell-lua
  namespace: graph
spec:
  workloadSelector:
    labels:
      app: shell
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value: # lua filter specification
       name: envoy.lua
       typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            function envoy_on_request(request_handle)
                local file, err = io.open ('/etc/istio/proxy/log.txt',"a")
                local buffered = request_handle:body()
                local dataString = tostring(buffered:getBytes(0, buffered:length()))
                local path = request_handle:headers():get(":path")
                local method = request_handle:headers():get(":method")
                local request_headers = request_handle:headers()
                if file==nil then
                  print("Couldnt open file: "..err)
                else
                  file:write("\n-----------------------------------\n")
                  file:write("Path: ",path,"\n")
                  file:write("Method: ",method,"\n")
                  file:write("Headers:\n")
                  for key, value in pairs(request_headers) do
                    file:write(key,": ",value,"\n")
                  end
                  file:write("Body:\n")
                  file:write(dataString)
                  file:write("\n-----------------------------------\n")
                  file:close()
                end    
            end

What this will do? It will configure Envoy to capture requests coming from the “app: shell” application and store every request on the local filesystem of istio-proxy sidecar container. (istio-proxy runs with ReadOnly filesystem, so we used the path where envoy keeps its dynamic configuration and writable)

Now if we jump into a “shell” pod and run a few commands (we use graph command line utility to interact with our Indexer).

If all works, we will be able to see all requests generated by the command line in the log.txt file on the istio-proxy container:

bash-3.2$ kubectl exec shell -c istio-proxy -it bash -n graph
istio-proxy@shell:/$ cat /etc/istio/proxy/log.txt 
-----------------------------------
Path: /
Method: POST
Headers:
:authority: indexer-agent:8000
:path: /
:method: POST
:scheme: http
content-type: application/json
accept: */*
content-length: 1497
user-agent: node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
accept-encoding: gzip,deflate
x-forwarded-proto: http
x-request-id: 59dbbec2-829f-95d7-b584-a13665c5d34e
x-envoy-decorator-operation: indexer-agent.graph.svc.cluster.local:8000/*
Body:
{"query":"{\n  indexerRegistration {\n    url\n    address\n    registered\n    location {\n      latitude\n      longitude\n      __typename\n    }\n    __typename\n  }\n  indexerDeployments {\n    subgraphDeployment\n    synced\n    health\n    fatalError {\n      handler\n      message\n      __typename\n    }\n    node\n    chains {\n      network\n      latestBlock {\n        number\n        __typename\n      }\n      chainHeadBlock {\n        number\n        __typename\n      }\n      earliestBlock {\n        number\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n  indexerAllocations {\n    id\n    allocatedTokens\n    createdAtEpoch\n    closedAtEpoch\n    subgraphDeployment\n    signalledTokens\n    stakedTokens\n    __typename\n  }\n  indexerEndpoints {\n    service {\n      url\n      healthy\n      tests {\n        test\n        error\n        possibleActions\n        __typename\n      }\n      __typename\n    }\n    status {\n      url\n      healthy\n      tests {\n        test\n        error\n        possibleActions\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n  indexingRules(merged: true) {\n    identifier\n    identifierType\n    allocationAmount\n    allocationLifetime\n    autoRenewal\n    parallelAllocations\n    maxAllocationPercentage\n    minSignal\n    maxSignal\n    minStake\n    minAverageQueryFees\n    custom\n    decisionBasis\n    requireSupported\n    __typename\n  }\n}","variables":{}}
-----------------------------------

Troubleshooting

Tail istio-proxy logs of the container you are targeting to see if it is throwing any errors

bash-3.2$ kubectl logs -f shell -c istio-proxy -n graph

You will be able to see something like this if you have errors in your Lua:

2022-10-31T14:41:51.648701Z     error   envoy lua       script log: [string "function envoy_on_request(request_handle)..."]:7: bad argument #1 to 'write' (string expected, got nil)

Make sure you are using the correct Labels/Selectors, you don’t want to apply your filter to the wrong workload. Check this by running:

bash-3.2$ \kubectl get pods -l app=shell -n graph
NAME    READY   STATUS    RESTARTS   AGE
shell   2/2     Running   0          32h