Stream Processor

Stream Processor is one of the key functional components in Netsil AOC. Following is a brief description of each key component:

  • Collector: Collects network packets, infrastructure metrics and custom metrics. Ships the collected packets and metrics to stream processor.
  • Stream Processor: Receives data from collectors. Processes the data into compact timeseries metrics which are then shipped to the AOC.
  • AOC: Indexes and stores time series. Serves a user interface and APIs for querying, alerting etc.

The collectors are installed on the hosts which are to be monitored. The stream processor can be installed alongside the collector, alongside the AOC, or as a standalone installation. This guide describes the standalone installation for stream processor.

Standalone Stream Processor (beta)

Stream processor(s) can be run as a standalone component for processing network packets and metrics at scale outside the host(s) being monitored. The standalone stream processor registers itself at AOC, and the collectors would be automatically configured by the AOC to send data to the stream processor(s) installed within their subnet space. The automatic subnet match behavior can be overridden by providing the exact address of the stream processor where the collector should connect via a configuration parameter on the collector, see parameter NETSIL_SP_HOST_OVERRIDE at collector configuration.

Stream Processor Collector Acrhicture Different Traffic Collection Modes

Prerequisites

Resource Requirements

You will need to provide a machine with sufficient resources to run a standalone stream processor:

Recommended Minimum
vCPUs 4 (or more) 2
Memory 8 GiB (or more) 4 GiB
Disk 16 GiB (or more) 8 GiB

Ports and Firewall Rules

Inbound

Please open the inbound ports listed below. The "Your Private Subnet" source refers to the subnet where you are installing the collectors.

Port Protocol Source Description Default requirement (yes/no)
2005 TCP Your Private Subnet RPCAP control channel yes
3005 TCP Your Private Subnet RPCAP data channel yes
2005 UDP Your Private Subnet RPCAP data channel no

Outbound

The following ports need to be accessible on the AOC from the standalone stream processor:

Port Protocol Source Description Default requirement (yes/no)
2000 HTTPS Public Metrics and events channel yes

Configuration

The AOC address needs to be configured on the standalone stream processor via NETSIL_SP_HOST. Refer the collector configuration section for the full set of configuration parameters.

Installation

The standalone stream processor is part of the collectors package. To run it, set the NETSIL_ROLE variable as sp in the collector configuration. In this mode, only the stream processor runs and none of the collector processes such as the traffic-collector or netsil-dd-agent would run.

Docker

Run the command below, making sure to provide the address of your AOC installation and your organization id.

docker run -td \
       --name=netsil_sp \
       --net=host \
       --ulimit core=0 \
       -e DEPLOY_ENV="docker" \
       -e NETSIL_ROLE=sp \
       -e NETSIL_SP_HOST=${your_netsil_ip} \
       -e NETSIL_ORGANIZATION_ID=${organizationId} \
       netsil/collectors:latest

Debian and RHEL

Install the collectors package with the parameter NETSIL_ROLE set as sp. Follow the instructions at these links for installing the package for debian and for rhel.

Collector Configuration

To make the collectors use standalone stream processor they must be started with NETSIL_ROLE parameter set as collector. The parameter NETSIL_SP_HOST_OVERRIDE can be provided as the address of the standalone stream processor. It is preferable to use the private ip address of the stream porcessor to prevent high bandwidth traffic over the public network.

To enable automatic subnet based load balancing between collectors and stream processors do not provide NETSIL_SP_HOST_OVERRIDE parameter to the collectors. The local ip address of the stream processor host machine is used for load balancing.

NOTE: The collectors still need to talk to AOC for sending infratructure metrics and getting auto-updates. Specify the AOC address via NETSIL_SP_HOST as usual.

Considerations for collector installation on bandwidth intensive instances

Here are a few points to consider regarding network bandwidth consumption on network intensive instances. By tweaking settings such as sampling rate and compression, one can strike a balance between fidelity, network overheads and local CPU overheads:

  • Enable Static Sampling: We recommend that you enable collector sampling on network heavy instances. Even with sampling enabled, the AOC should sufficiently report error rates, latency and throughput trends. The sampling parameter is provided as a percentage of total traffic (1-100), so a sampling rate of 50 would sample 50% of the flows. By default, the sampling is turned off.

  • Run the collectors in local stream processing mode. In this mode the network overheads are smaller at the cost of higher CPU overhead. the network flows are processed on the same host in this mode.