Infrastructure Integration

Configuration

  1. Use Go's expvar package to expose your memory information

    package ...
    
    import (
        ...
        "net/http"
        "expvar"
        ...
    )
    
    // If your application has no http server running for the DefaultServeMux,
    // you'll have to have a http server running for expvar to use, for example
    // by adding the following to your init function
    func init() {
        go http.ServeAndListen(":8080", nil)
    }
    
    ...
    
    // You can also expose variables that are specific to your application
    // See http://golang.org/pkg/expvar/ for more information
    
    var (
        exp_points_processed = expvar.NewInt("points_processed")
    )
    
    func processPoints(p RawPoints) {
        points_processed, err := parsePoints(p)
        exp_points_processed.Add(points_processed)
        ...
    }
    
    ...
    
  2. Configure the agent by editing /etc/netsil-dd-agent/conf.d/go_expvar.yamlin the collectors.

Example:

    init_config:
    instances:
      # Most memstats metrics are exported by default
      # See http://godoc.org/runtime#MemStats for their explanation
      # Note that you can specify a `type` for the metrics. One of:
      #  * counter
      #  * gauge (the default)
      #  * rate (note that this will show up as a gauge that is meant to be seen as a "per second rate")

      - expvar_url: http://localhost:8080
        # namespace: examplenamespace         # The default metric namespace is 'go_expvar', define your own
        # tags:
        #   - "application_name:myapp"
        #   - "optionaltag2"
        # metrics:
        #   # These metrics are just here as examples.
        #   # Most memstats metrics are collected by default without configuration needed.
        #   - path: memstats/PauseTotalNs
        #     alias: go_expvar.gc.pause_time_in_ns
        #     type: rate
        #     tags:
        #       - "metric_tag1:tag_value1"
        #       - "metric_tag2:tag_value2"
        #   - path: memstats/Alloc            # metric will be reported as a gauge by default
        #   - path: memstats/Lookups
        #     type: rate                      # metric should be reported as a rate instead of the default gauge
        #   - path: memstats/Mallocs          # with no name specified, the metric name will default to a path based name
        #     type: counter                   # report as a counter instead of the default gauge
        #   - path: memstats/Frees
        #     type: rate
        #   - path: memstats/BySize/1/Mallocs # You can get nested values by separating them with "/"
        #   - path: myvariable
        #     alias: go_expvar.my_custom_name
        #     type: gauge
        #   - path: routes/get_.*/count       # You can use a regex when you want to report for all elements matching a certain pattern
  1. Check and make sure that all yaml files are valid with following command:

    /etc/init.d/netsil-collectors configcheck
    
  2. Restart the Agent using the following command:

    /etc/init.d/netsil-collectors restart
    
  3. Execute the info command to verify that the integration check has passed:

    /etc/init.d/netsil-collectors info
    

    The output of the info command should contain a section similar to the following:

    Checks
    ======
      [...]
      go_expvar
      ----------
          - instance #0 [OK]
          - Collected 8 metrics & 0 events
    

Infrastructure Datasources

Datasource Available Aggregations Unit Description
go_expvar.memstats.alloc avg max min sum byte Bytes allocated and not yet freed
go_expvar.memstats.frees avg max min sum operation Number of frees
go_expvar.memstats.heap_alloc avg max min sum byte Bytes allocated and not yet freed
go_expvar.memstats.heap_idle avg max min sum byte Bytes in idle spans
go_expvar.memstats.heap_inuse avg max min sum byte Bytes in non-idle spans
go_expvar.memstats.heap_objects avg max min sum item Total number of allocated objects
go_expvar.memstats.heap_released avg max min sum byte Bytes released to the OS
go_expvar.memstats.heap_sys avg max min sum byte Bytes obtained from system
go_expvar.memstats.lookups avg max min sum operation Number of pointer lookups
go_expvar.memstats.mallocs avg max min sum operation Number of mallocs
go_expvar.memstats.num_gc avg max min sum garbage collection Number of garbage collections
go_expvar.memstats.pause_ns.95percentile avg max min sum nanosecond 95th percentile of recent GC pause durations
go_expvar.memstats.pause_ns.avg avg max min sum nanosecond Average of recent GC pause durations
go_expvar.memstats.pause_ns.count avg max min sum sample/second Number of submitted GC pause durations
go_expvar.memstats.pause_ns.max avg max min sum nanosecond Max GC pause duration
go_expvar.memstats.pause_ns.median avg max min sum nanosecond Median GC pause duration
go_expvar.memstats.pause_total_ns avg max min sum nanosecond Total GC pause duration over lifetime of process
go_expvar.memstats.total_alloc avg max min sum byte Bytes allocated (even if freed)