Kubernetes

Setup your collectors!

Netsil collectors can be run in both containerized as well as non-containerized environments. Only one collector is needed per host (VM or bare metal OS). Please follow the environment specific installation instructions below.

Installation

(Consult Supported Versions and Collector Requirements before installing.)

  • Save the manifest below as netsil-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: netsil

Next, create the namespace with the command:

kubectl create -f netsil-ns.yaml
  • Save the manifest below as collector.yaml.
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  namespace: netsil
  name: collector
  labels:
    app: netsil
    component: collector
spec:
  minReadySeconds: 0
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: netsil
        component: collector
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: collector
        image: netsil/collectors:latest
        command: ["/bin/bash","-c","while true ; do NETSIL_SP_HOST=$NETSIL_SERVICE_HOST /opt/netsil/collectors/start.sh ; echo Exiting, possibly to upgrade ; sleep 5 ; done"]
        securityContext:
          capabilities:
            add:
            - NET_RAW
            - NET_ADMIN
        env:
        # Please DO NOT prepend http:// or https:// to the NETSIL_SERVICE_HOST value
        - name: NETSIL_SERVICE_HOST
          value: ${your_netsil_ip}
        - name: NETSIL_ORGANIZATION_ID
          value: ${organizationId}
        - name: SAMPLINGRATE
          value: "100"
        - name: DEPLOY_ENV
          value: "docker"
        - name: KUBERNETES
          value: "yes"
        - name: SD_BACKEND
          value: "docker"
        resources:
          requests:
            memory: "512Mi"
            cpu: "1000m"
          limits:
            memory: "1Gi"
            cpu: "2000m"
        volumeMounts:
        - name: cgroup
          mountPath: /host/sys/fs/cgroup/
          readOnly: true
        - name: proc
          mountPath: /host/proc/
          readOnly: true
        - name: docker-sock
          mountPath: /var/run/docker.sock
          readOnly: true
      volumes:
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup/
      - name: proc
        hostPath:
          path: /proc/
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock

Now, install the collectors as a DaemonSet with the command:

kubectl create -f collector.yaml

Installing on Kubernetes pre-1.6

  • If you are running a Kubernetes version below 1.6, please download a separate collector.yaml manifest.
  • In the manifest, please uncomment the NETSIL_SERVICE_HOST parameter and replace it with ${your_netsil_ip}; in addition, uncomment the NETSIL_ORGANIZATION_ID parameter and replace it with ${organizationId}:

    - name: NETSIL_SERVICE_HOST
      value: ${your_netsil_ip}
    - name: NETSIL_ORGANIZATION_ID
      value: ${organizationId}
    
  • In addition, add the following environment variables, where <kube-dns-address> is the location of your kubernetes dns nameserver. Without these two variables, the kubernetes_state metrics from the kubernetes integration will not work.

    - name: OVERWRITE_RESOLVCONF
      value: yes
    - name: K8S_NAMESERVER
      value: <kube-dns-address>
    

Installing on Kubernetes 1.7+

  • In order to be picked up properly, hostnames may require an additional environment variable to be defined in your collector.yaml, shown below:
    - name: KUBERNETES_KUBELET_HOST
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    

RBAC Setup

  • If you have RBAC enabled in your kubernetes cluster, you may need to give the collector proper permissions before installation.
  • First, add the field serviceAccountName: netsil to the template spec of your collectors manifest like so:

    ...
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccountName: netsil
      containers:
      - name: collector
        image: netsil/collectors:latest
    ...
    
  • Next, download rbac-setup.yaml and run the following command:

     kubectl create -f rbac-setup.yaml
    

This will give the collector the proper authentication and authorization to work with your k8s cluster's RBAC

  • Finally, redeploy the collectors manifest with kubectl.

Installing Integrations

Creating and Persisting the Config File

Below, we list some methods of creating the config file such that it will be persisted across container restarts.

If the service you are integrating with is running as a container, its integration likely supports autoconf. Thus, your integrations config file should reside in the /etc/netsil-dd-agent/conf.d/auto_conf directory of your collectors. Also, you should use%%host%% and %%port%% in place of any hardcoded host and port parameters in the config file.

Remember to use the Configuration section from the instructions page for your integration as reference for the name and contents of the config file.

ConfigMaps

  • If you wish to use ConfigMaps to configure collector integrations, please follow the directions below:
  • In your collector.yaml, add the following block to the volumeMounts section:

    - name: configmap-volume
      mountPath: /conf.d/
    - name: configmap-auto-conf
      mountPath: /etc/netsil-dd-agent/auto_conf/
    
  • In addition, add the following block to the volumes section:

    - name: configmap-volume
      configMap:
        name: integrations
    - name: configmap-auto-conf
      configMap:
        name: auto-conf
    
  • Next, download integrations-configmap.yml and auto-configmap.yml

  • To add more integrations, append them to integrations-configmap.yml. To add more integrations of the autoconf variety, append them to auto-configmap.yaml. Now, run the following command:

    kubectl create -f integrations-configmap.yaml
    kubectl create -f auto-configmap.yaml
    
  • Finally, install the collectors with the command

    kubectl create -f collector.yaml
    

Custom Images

Create a custom container image that derives from the base collector image provided by Netsil. The following is a template Dockerfile:

  # Collectors follow the same versioning scheme as the AOC, replace x.x.x with your AOC version
  FROM netsil/collectors:stable-x.x.x

  # Copy the ".yaml" file(s) at collector build time
  COPY *.yaml /etc/netsil-dd-agent/conf.d/auto_conf

Volume-mounting

Volume-mount the configuration directory from the host, such that the yaml configuration fields can be provided directly on the host file system. You will need to provide the following parameter to the container's run command:

  -v /etc/netsil-dd-agent/conf.d/:/etc/netsil-dd-agent/conf.d/:ro

Running the Integration

After you have persisted the config file, the integration should start automatically when the collector image itself is run.

Checking Configuration

Check to make sure that all yaml files are valid with the following command:

kubectl exec -n netsil <collector-pod> /etc/init.d/netsil-collectors configcheck

Checking Runtime

Check to make sure that the integration is running correctly with the following command:

kubectl exec -n netsil <collector-pod> /etc/init.d/netsil-collectors info

The output of the info command should contain a section similar to the following:

    Checks
    ======
      [...]
      <name-of-integration>
      ----------
          - instance #0 [OK]
          - Collected 8 metrics & 0 events

Reporting Troubleshooting Information

If you are having issues with your collectors, you can run an inspect command which will gather troubleshooting information about the collectors, as well as any necessary logs. Follow the steps below to do so.

First, run

kubectl exec -n netsil <collector-pod> /etc/init.d/netsil-collectors inspect

This will generate a tarball in the /tmp directory whose filename begins with netsil-collectors-inspection. You can obtain this tarball with the following commands

# To get the exact name
kubectl exec -n netsil <collector-pod> ls /tmp

# To obtain the tarball
kubectl cp -n netsil <collector-pod>:/tmp/<filename-of-inspect-tarball> <filename-of-inspect-tarball>

Please send this tarball to us, via email or slack.

Uninstallation

You can uninstall the collectors with

kubectl delete -f collector.yaml