Kubernetes

This page describes two alternatives for tracing an API, via packet capture, in Kubernetes. The following section gives a brief introduction to help you decide which is better for your goals. This is followed by a detailed walkthrough of each method.

If you’re just getting started, we recommend running Akita in the way described in the first alternative: As a sidecar for a single service, and only for a short duration that you run to test trace collection. The steps described here will only work if the traffic to the API endpoint is unencrypted HTTP; for example, if a load balancer is the HTTPS endpoint, then traffic to the service itself will be unencrypted.

Alternative 1: Run akitasoftware/cli as a sidecar

The akitasoftware/cli Docker container can be run as a sidecar within a particular pod, to monitor the APIs implemented by that pod. This provides observability for a single service, thus limiting the complexity of what gets traced and how to filter it.

However, in production and staging Kubernetes environments, configuring a sidecar will restart the pod, so this option is more disruptive. It also means that there is one akita process running for each instance of a service being monitored.

This is the recommended method if you want to test Akita on a short-running service, such as in a test environment, as it ensures that the trace quits as soon as the pod is stopped.

Alternative 2: Run akitasoftware/cli using host networking, to monitor any of the pods running on a node.

The akitasoftware/cli Docker container can also be attached to the host network, and see all network traffic on the host. This can be accomplished by starting a new daemonset (or a single new pod), so it is a less intrusive change to the configuration, leaving existing deployments as-is. A single pod can also monitor all the services on the node, so fewer changes are needed to monitor all your services.
However, local security policies might prohibit the use of host networking.

This is the recommended method for continuous monitoring in a production or staging environment.

Running Akita as a Sidecar

In this section, we show how to configure a deployment to include the Akita CLI as a sidecar that captures API traffic for the duration of its pod's lifetime.

Thanks to Bruno Felix, who experimented with this method and provided us with his examples!

Step 1: add your Akita credentials as a Kubernetes secret

Here is a YAML file that can be used with kubectl apply to create or update the secret:

apiVersion: v1
kind: Secret
metadata:
  name: akita-secrets
type: Opaque
data:
  api-key-id: <base64 encoded key id>
  api-key-secret: <base64 encoded key secret>

Replace the templates with the base64 encoded versions of your AKITA_API_KEY_ID and AKITA_API_KEY_SECRET. If you have these values as environment variables already, then in a shell execute the following commands:

$ echo -n $AKITA_API_KEY_ID | base64
$ echo -n $AKITA_API_KEY_SECRET | base64 -w 0

Otherwise, you can get the non-base64-encoded versions from ~/.akita/credentials.yaml, and encode them in the same way.

Step 2: edit a deployment template to add the Akita Software container as a sidecar

For this example, we’ll have the Akita learn session continue for the entire pod lifetime. Alternatives to this behavior are explained below.

Find the file that describes the deployment of the service you want to monitor, or a standalone definition of a pod. It will have a section called “template” that describes the containers to run inside the pod, that will look something like this:

template:
    metadata:
      labels:
        app: my-example-app
    spec:
      containers:
      - name: app
        image: bfwaterdog/app-example
        ports:
        - containerPort: 5000

Add a second container in the same spec that specifies the akitasoftware/cli container, along with its command-line arguments, like this:

- name: akita
        image: akitasoftware/cli:latest
        env:
          - name: AKITA_API_KEY_ID
            valueFrom:
              secretKeyRef:
                name: akita-secrets
                key: api-key-id
          - name: AKITA_API_KEY_SECRET
            valueFrom:
              secretKeyRef:
                name: akita-secrets
                key: api-key-secret
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "AKITA_PID=$(pgrep akita) && kill -2 $AKITA_PID && tail -f /proc/$AKITA_PID/fd/1"]
        args:
          - learn
          - --service
          - your-service-name
          - --filter
          - port 5000

This configuration says to send the trace of port 5000 to Akita, using the service name your-service-name. You should replace port 5000 with the port on which your service receives API requests (before any remapping by Kubernetes), and your-service-name with the name of the service you created via the Akita UI. The env section pulls the Akita CLI environment variables out of the secret you configured earlier. The preStop section runs a script inside the container to gracefully kill the Akita CLI, allowing it to finish the trace and create a spec.

Step 3: apply the modified deployment template

Use kubectl apply to create the new deployment, or modify an existing one. You can also use kubectl run to create an ephemeral pod that uses Akita as a sidecar.

Find the pod that was created in response to the change, and you can use

kubectl logs <pod name> akita

to see the Akita CLI output.

If you terminate or restart the pod, the Akita CLI will automatically create a spec based on the trace it has collected. You can find this spec in the Akita UI. You can also use the akita apispec command to generate a spec from an incomplete trace.

Building continuous models rather than one per pod

The learn command creates one model per pod, and only at the end of that pod's lifetime. Once you see Akita at work, a better option is to enable continuous monitoring using the akita apidump command:

args:
          - apidump
          - --service
          - your-service-name

This will create a series of models based on all traces collected from a single service, starting within a few minutes after traffic first arrives.

To split up your traces, you can use the --deployment flag with a value such as "staging" or "production".

Optional: use kubectl debug to attach Akita to an already-existing pod

The kubectl debug command is an alpha feature in Kubernetes v1.18 and later. It is likely not enabled by default in your environment. However, it permits you to start a new ephemeral container in an existing pod, so lets you run the Akita CLI as a sidecar in a nondisruptive way.

If the service you want to monitor is running in pod myservice-xyzabc, on port 500, then you can start monitoring its API with

kubectl debug -it myservice-xyzabc \
   --image=akitasoftware/cli:latest \
   --env="AKITA_API_KEY_ID=$AKITA_API_KEY_ID" \
   --env="AKITA_API_KEY_SECRET=$AKITA_API_KEY_SECRET" \
   -- learn --service my-service-name --filter port 5000

This command will show you the normal akita learn output and you can use Ctrl+C to stop the trace as if you were running normally from the command line.

Running Akita using Host Networking

In this section, we show how to configure a daemonset with the Akita CLI that attaches to the host network to collect an API trace.

Step 1: add your Akita credentials as a Kubernetes secret

Here is a YAML file that can be used with kubectl apply to create or update the secret:

apiVersion: v1
kind: Secret
metadata:
  name: akita-secrets
type: Opaque
data:
  api-key-id: <base64 encoded key id>
  api-key-secret: <base64 encoded key secret>

Replace the templates with the base64 encoded versions of your AKITA_API_KEY_ID and AKITA_API_KEY_SECRET. If you have these values as environment variables already, then in a shell execute the following commands:

$ echo -n $AKITA_API_KEY_ID | base64
$ echo -n $AKITA_API_KEY_SECRET | base64 -w 0

Otherwise, you can get the non-base64-encoded versions from ~/.akita/credentials.yaml, and encode them in the same way.

Step 2: create a Daemonset definition to run the Akita container using host networking

A Daemonset controller ensures that each node is running an instance of the pod. We will use this to capture all traffic, from all services, in a Kubernetes environment. Use the example YAML below:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: akita-capture
spec:
  selector:
    matchLabels:
      name: akita-capture 
  template:
    metadata:
      labels:
        name: akita-capture
    spec:
      containers:
        - image: akitasoftware/cli:latest
          imagePullPolicy: Always
          name: akita
          args:
          - apidump
          - --service
          - <your service name here>
          - --rate-limit
          - "200"
          env:
          - name: AKITA_API_KEY_ID
            valueFrom:
              secretKeyRef:
                name: akita-secrets
                key: api-key-id
          - name: AKITA_API_KEY_SECRET
            valueFrom:
              secretKeyRef:
                name: akita-secrets
                key: api-key-secret
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always

This configuration will create a trace covering one hour, create an API model from the trace at the end of that hour, and then restart. Because no --filter argument is specified to akita apidump, every service that can be parsed by the Akita CLI will appear in the same trace.

In this example, we set a per-node rate limit of 200 requests/minute. If the rate of packet captures is higher than this, the Akita CLI will start performing statistical sampling of the packets it sees. We recommend always setting a rate limit in production environments.

Optional: tag your traces and models

You can optionally add some tags to help you identify the source of the traces. akita apidump takes a --tags argument which contains a comma-separated list of key=value pairs, like the following example:

args:
   ...
   - --tags
   - owner=mark,namespace=staging

See [#tagging-traces-and-specs-with-kubernetes-information]("tagging traces and specs with Kubernetes" below)

A tag that is treated specially by Akita is the deployment name. You can specify the --deployment flag or use the AKITA_DEPLOYMENT environment variable to separate traces from test, staging, and production into different models. See Continuously monitor your APIs.

Optional: remove Kubernetes APIs

You may want to filter out the Kubernetes control traffic itself, which you can do by adding --filter "not port 80" to the akita learn command line:

args:
   ...
   - --filter
   - not port 80

Step 3: apply the Daemonset configuration and wait for an API model to be generated

Apply the configuration you created with kubectl apply -f <yaml file>. You should see pods named akita-capture-xxxxx scheduled on all of the eligible nodes. You can use kubectl logs to verify that they have started a trace correctly.

Once the traces have started, you can find them listed on the Akita web console under the "traces" section of the service page. The first model should show up within 15 minutes of traffic arriving.

Alternative: create a pod definition to run the Akita container using host networking

If you want to look at only a particular service, or a particular node, you can directly create a single pod using the example shown below. This YAML file that creates a single pod named akita-capture, which collects API traces from a service running on port 50100, for 10 minutes:

apiVersion: v1
kind: Pod
metadata:
  name: akita-capture
spec:
  containers:
  - image: akitasoftware/cli:latest
    name: akita
    args:
    - learn
    - --filter
    - dst port 50100
    - -c
    - sleep 600
    - -u
    - root
    - --service
    - my-service-name
    env:
    - name: AKITA_API_KEY_ID
      valueFrom:
        secretKeyRef:
          name: akita-secrets
          key: api-key-id
    - name: AKITA_API_KEY_SECRET
      valueFrom:
        secretKeyRef:
          name: akita-secrets
          key: api-key-secret
  dnsPolicy: ClusterFirst
  hostNetwork: true
  restartPolicy: Never

This example will use normal Kubernetes scheduling to decide where to run the new pod. To control which host is used add a node selector to the rule, like this:

spec:
  nodeName: your-node-name

or use a pod affinity rule.

Once the collection has finished, you will have to delete the pod manually.

Next Steps

Once you have a specification generated from your Kubernetes-hosted API, you can view it on the Akita web site, or compare two specifications to look for deltas. See Generate API Models from Traffic or Diff API Models.

👍

The examples in this document use the ":latest" tag for the Akita CLI container, which is the easiest way to get started. However, for long-term use we recommend specifying a particular version of the container, so that your environment is predictable and upgrades can be deterministically applied.

Tagging traces and specs with Kubernetes information

You can use the --tags argument to akita learn or akita apispec to specify key-value pairs that help you identify the source of a trace. The Akita CLI (in versions 0.16.1 and later) also recognizes the following environment variables and converts them into reserved tags. You can make this information available to the Akita container via the Downward API.

All of these tags are optional; however, some of them have support in the Akita web console.

Environment variable

Tag

Suggested use

AKITA_DEPLOYMENT

x-akita-deployment

Identify the Kubernetes cluster, for example as "staging" or "deployment".

AKITA_DEPLOYMENT_COMMIT

x-akita-git-commit

Records a Git commit hash; can be used to record the deployed version for a monorepo, or the last Terraform commit.

AKITA_AWS_REGION

x-akita-aws-region

Amazon Web Services region in which the cluster is running, if applicable.

AKITA_K8S_NAMESPACE

x-akita-kubernetes-namespace

Kubernetes namespace in which the capture agent is running.

AKITA_K8S_NODE

x-akita-kubernetes-node

Identifier of the node on which the capture agent is running.

AKITA_K8S_HOST_IP

x-akita-kubernetes-host-ip

Host IP of the node.

AKITA_K8S_POD

x-akita-kubernetes-pod

Name of the pod which captured the trace.

AKITA_K8S_POD_IP

x-akita-kubernetes-pod-ip

IP address of the pod.

AKITA_K8S_DAEMONSET

x-akita-kubernetes-daemonset

Name of the Daemonset used to create the pod.

Adding the following code in the env section of the Daemonset or pod definition will extract all the available information from the Kubernetes Downward API. The other variables listed in the table would need to be configured to static values, or filled at creation time by a Terraform module.

env:
        - name: AKITA_K8S_NODE
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: AKITA_K8S_POD
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: AKITA_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: AKITA_K8S_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: AKITA_K8S_HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP

Did this page help you?