Kafka Integration

Infrastructure Integration

Instructions

Follow the installation guide below for your given collector environment.

During installation, use the configuration section below as reference.

After installation, the infrastructure datasources in the table below will be available in the AOC.

Installation Guide

Installing this integration consists of creating a yaml file in the filesystem of your collectors. Click below for instructions on how to do so for your given collectors environment.

Docker

Kubernetes

Mesos-Marathon

Debian

Ubuntu

RHEL/CentOS

Configuration

  1. Ensure that JMX Remote is enabled on Kafka brokers.
  2. Edit kafka.yaml to configure the agent. Please refer kafka.yaml for all available configuration options.
    instances:
      - host: localhost
        port: 9999 # This is the JMX port on which Kafka exposes its metrics (usually 9999)
        tags:
          kafka: broker
    
    init_config:
      is_jmx: true
    
      conf:
        # Producers (v0.9.0.x to v0.10.2.x)
        - include:
            domain: 'kafka.producer'
            bean_regex: 'kafka\.producer:type=producer-metrics,client-id=.*'
            attribute:
              response-rate:
                metric_type: gauge
                alias: kafka.producer.response_rate
        - include:
            domain: 'kafka.producer'
            bean_regex: 'kafka\.producer:type=producer-metrics,client-id=.*'
            attribute:
              request-rate:
                metric_type: gauge
    
        # Consumers (v0.9.0.x to v0.10.2.x)
        - include:
            domain: 'kafka.consumer'
            bean_regex: 'kafka\.consumer:type=consumer-fetch-manager-metrics,client-id=.*'
            attribute:
              bytes-consumed-rate:
                metric_type: gauge
                alias: kafka.consumer.bytes_in
        - include:
            domain: 'kafka.consumer'
            bean_regex: 'kafka\.consumer:type=consumer-fetch-manager-metrics,client-id=.*'
            attribute:
              records-consumed-rate:
                metric_type: gauge
                alias: kafka.consumer.messages_in
    

In addition, you can edit kafka_consumer.yaml as well.

Infrastructure Datasources

Datasource Available Aggregations Unit Description
kafka.net.bytes_out avg max min sum byte/second Outgoing byte rate.
kafka.net.bytes_in avg max min sum byte/second Incoming byte rate.
kafka.net.bytes_rejected avg max min sum byte/second Rejected byte rate.
kafka.messages_in avg max min sum message Incoming message rate.
kafka.request.fetch.failed avg max min sum request Number of client fetch request failures.
kafka.request.fetch.failed_per_second avg max min sum request/second Rate of client fetch request failures per second.
kafka.request.produce.time.avg avg max min sum request/second Average time for a produce request.
kafka.request.produce.time.99percentile avg max min sum request/second Time for produce requests for 99th percentile.
kafka.request.produce.failed_per_second avg max min sum request/second Rate of failed produce requests per second.
kafka.request.produce.failed avg max min sum request Number of failed produce requests.
kafka.request.fetch.time.avg avg max min sum request/second Average time per fetch request.
kafka.request.fetch.time.99percentile avg max min sum request/second Time for fetch requests for 99th percentile.
kafka.request.update_metadata.time.avg avg max min sum millisecond Average time for a request to update metadata.
kafka.request.update_metadata.time.99percentile avg max min sum millisecond Time for update metadata requests for 99th percentile.
kafka.request.metadata.time.avg avg max min sum millisecond Average time for metadata request.
kafka.request.metadata.time.99percentile avg max min sum millisecond Time for metadata requests for 99th percentile.
kafka.request.offsets.time.avg avg max min sum millisecond Average time for an offset request.
kafka.request.offsets.time.99percentile avg max min sum millisecond Time for offset requests for 99th percentile.
kafka.request.handler.avg.idle.pct avg max min sum fraction Average fraction of time the request handler threads are idle.
kafka.replication.isr_shrinks avg max min sum node/second Rate of replicas leaving the ISR pool.
kafka.replication.isr_expands avg max min sum node/second Rate of replicas joining the ISR pool.
kafka.replication.leader_elections avg max min sum event/second Leader election rate.
kafka.replication.unclean_leader_elections avg max min sum event/second Unclean leader election rate.
kafka.replication.under_replicated_partitions avg max min sum Number of unreplicated partitions.
kafka.log.flush_rate avg max min sum flush/second Log flush rate.
kafka.consumer.delayed_requests avg max min sum request Number of delayed consumer requests.
kafka.consumer.expires_per_second avg max min sum eviction/second Rate of delayed consumer request expiration.
kafka.expires_sec avg max min sum eviction/second Rate of delayed producer request expiration.
kafka.follower.expires_per_second avg max min sum eviction/second Rate of request expiration on followers.
kafka.producer.delayed_requests avg max min sum request Number of producer requests delayed.
kafka.producer.expires_per_seconds avg max min sum eviction/second Rate of producer request expiration.
kafka.producer.request_rate avg max min sum request/second Number of producer requests per second.
kafka.producer.response_rate avg max min sum response/second Number of producer responses per second.
kafka.producer.request_latency_avg avg max min sum millisecond Producer average request latency.
kafka.producer.bytes_out avg max min sum byte/second Producer bytes out rate.
kafka.producer.message_rate avg max min sum message/second Producer message rate.
kafka.producer.io_wait avg max min sum nanosecond Producer I/O wait time.
kafka.consumer.max_lag avg max min sum offset Maximum consumer lag.
kafka.consumer.fetch_rate avg max min sum request The minimum rate at which the consumer sends fetch requests to a broker.
kafka.consumer.bytes_in avg max min sum byte/second Consumer bytes in rate.
kafka.consumer.messages_in avg max min sum message/second Rate of consumer message consumption.
kafka.consumer.zookeeper_commits avg max min sum write/second Rate of offset commits to ZooKeeper.
kafka.consumer.kafka_commits avg max min sum write/second Rate of offset commits to Kafka.
kafka.broker_offset avg max min sum offset Current message offset on broker.
kafka.consumer_lag avg max min sum offset Lag in messages between consumer and broker.
kafka.consumer_offset avg max min sum offset Current message offset on consumer.