netdata/helmchart

Netdata Helm chart for Kubernetes deployments

Based on the work of varyumin (https://github.com/varyumin/netdata).

Introduction

This chart bootstraps a Netdata deployment on a Kubernetes cluster using the Helm package manager.

By default, the chart installs:

• A Netdata child pod on each node of a cluster, using a Daemonset
• A Netdata k8s state monitoring pod on one node, using a Deployment. This virtual node is called netdata-k8s-state.
• A Netdata parent pod on one node, using a Deployment. This virtual node is called netdata-parent.

Disabled by default:

• A Netdata restarter CronJob. Its main purpose is to automatically update Netdata when using nightly releases.

The child pods and the state pod function as headless collectors that collect and forward all the metrics to the parent pod. The parent pod uses persistent volumes to store metrics and alarms, handle alarm notifications, and provide the Netdata UI to view metrics using an ingress controller.

Please validate that the settings are suitable for your cluster before using them in production.

Prerequisites

• A working cluster running Kubernetes v1.9 or newer.
• The kubectl command line tool, within one minor version difference of your cluster, on an administrative system.
• The Helm package manager v3.8.0 or newer on the same administrative system.

Required Resources and Permissions

Netdata is a comprehensive monitoring solution that requires specific access to host resources to function effectively. By design, monitoring solutions like Netdata need visibility into various system components to collect metrics and provide insights. The following mounts, privileges, and capabilities are essential for Netdata's operation, and applying restrictive security profiles or limiting these accesses may significantly impact functionality or render the monitoring solution ineffective.

Installing the Helm chart

You can install the Helm chart via our Helm repository, or by cloning this repository.

Installing via our Helm repository (recommended)

To use Netdata's Helm repository, run the following commands:

helm repo add netdata https://netdata.github.io/helmchart/
helm install netdata netdata/netdata

See our install Netdata on Kubernetes documentation for detailed installation and configuration instructions. The remainder of this document assumes you installed the Helm chart by cloning this repository, and thus uses slightly different helm install/helm upgrade commands.

Install by cloning the repository

Clone the repository locally.

git clone https://github.com/netdata/helmchart.git netdata-helmchart

To install the chart with the release name netdata:

helm install netdata ./netdata-helmchart/charts/netdata

The command deploys ingress on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list.

Uninstalling the Chart

To uninstall/delete the my-release deployment:

 helm delete netdata

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following table lists the configurable parameters of the netdata chart and their default values.

General settings

1

"Recreate"

[]

"netdata/netdata"

"{{ .Chart.AppVersion }}"

"Always"

"alpine"

"latest"

"Always"

false

[]

{}

"ClusterIP"

19999

{}

""

[]

""

null

""

true

See values.yaml for defaults

"/"

"Prefix"

"netdata.k8s.local"

false

{}

{}

[]

[]

[]

true

true

true

"netdata"

{}

false

"00 06 * * *"

"rancher/kubectl"

".auto"

"Always"

"Never"

{}

"Forbid"

60

3

3

""

""

Service Discovery

"netdata/agent-sd"

"v0.2.10"

"Always"

true

"netdata-child-sd-config-map"

"config.yml"

""

{}

See values.yaml for defaults

Parent

"netdata-parent"

true

19999

{}

0

3

30

1

1

0

3

30

1

1

201

201

201

300

{}

[]

{}

""

{}

[]

{}

{}

"Default"

true

"-"

"15Gi"

true

"-"

"1Gi"

See values.yaml for defaults

false

""

""

[]

[]

[]

Child

true

"{{ .Values.parent.port }}"

{}

{}

0

3

30

1

1

0

3

30

1

1

30

{}

See values.yaml for defaults

{}

""

{}

true

{}

true

"ClusterFirstWithHostNet"

true

"/var/lib/netdata-k8s-child"

false

"https://localhost:10250"

See values.yaml for defaults

{}

[]

false

""

""

[]

[]

K8s State

"netdata-k8s-state"

true

"{{ .Values.parent.port }}"

{}

0

3

30

1

1

0

3

30

1

1

30

{}

[]

{}

""

{}

true

{}

"ClusterFirstWithHostNet"

true

"-"

"1Gi"

{}

[]

See values.yaml for defaults

false

""

""

[]

[]

Netdata OpenTelemetry

false

"netdata-otel"

"{{ .Values.parent.port }}"

"ClusterIP"

4317

{}

""

""

[]

""

""

{}

0

3

30

1

1

0

3

30

1

1

201

201

201

30

{}

[]

{}

""

{}

true

{}

"ClusterFirst"

true

"-"

"10Gi"

See values.yaml for defaults

{}

[]

false

""

""

[]

[]

OpenTelemetry Collector

false

"daemonset"

"otel/opentelemetry-collector-k8s"

true

true

false

See values.yaml for defaults

See values.yaml for defaults

true

true

See values.yaml for defaults

See values.yaml for defaults

Example to set the parameters from the command line:

$ helm install ./netdata --name my-release \
    --set notifications.slack.webhook_url=MySlackAPIURL \
    --set notifications.slack.recipient="@MyUser MyChannel"

Another example, to set a different ingress controller.

By default kubernetes.io/ingress.class set to use nginx as an ingress controller, but you can set Traefik as your ingress controller by setting ingress.annotations.

$ helm install ./netdata --name my-release \
    --set ingress.annotations=kubernetes.io/ingress.class: traefik

Alternatively to passing each variable in the command line, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install ./netdata --name my-release -f values.yaml

Tip: You can use the default values.yaml

Note:: To opt out of anonymous statistics, set the DO_NOT_TRACK environment variable to non-zero or non-empty value in parent.env / child.env configuration (e.g.,: DO_NOT_TRACK: 1) or uncomment the line in values.yml.

Configuration files

To deploy additional netdata user configuration files, you will need to add similar entries to either the parent.configs or the child.configs arrays. Regardless of whether you add config files that reside directly under /etc/netdata or in a subdirectory such as /etc/netdata/go.d, you can use the already provided configurations as reference. For reference, the parent.configs the array includes an example alarm that would get triggered if the python.d example module was enabled. Whenever you pass the sensitive data to your configuration like the database credential, you can take an option to put it into the Kubernetes Secret by specifying storedType: secret in the selected configuration. By default, all the configurations will be placed in the Kubernetes configmap.

Note that in this chart's default configuration, the parent performs the health checks and triggers alarms but collects little data. As a result, the only other configuration files that might make sense to add to the parent are the alarm and alarm template definitions, under /etc/netdata/health.d.

Tip: Do pay attention to the indentation of the config file contents, as it matters for the parsing of the yaml file. Note that the first line under var: | must be indented with two more spaces relative to the preceding line:

  data: |-
    config line 1 #Need those two spaces
        config line 2 #No problem indenting more here

Persistent volumes

There are two different persistent volumes on parent node by design (not counting any Configmap/Secret mounts). Both can be used, but they don't have to be. Keep in mind that whenever persistent volumes for parent are not used, all the data for specific PV is lost in case of pod removal.

1. database (/var/cache/netdata) - all metrics data is stored here. Performance of this volume affects query timings.
2. alarms (/var/lib/netdata) - alarm log, if not persistent pod recreation will result in parent appearing as a new node in netdata.cloud (due to ./registry/ and ./cloud.d/ being removed).

In case of child instance it is a bit simpler. By default, hostPath: /var/lib/netdata-k8s-child is mounted on child in: /var/lib/netdata. You can disable it, but this option is pretty much required in a real life scenario, as without it each pod deletion will result in a new replication node for a parent.

Collecting logs with OpenTelemetry

Netdata can ingest, store, and visualize your cluster's container logs through OpenTelemetry. This relies on two components — both disabled by default:

• netdataOpentelemetry — Netdata's OpenTelemetry receiver. It runs as its own Deployment (the netdata-otel node) and listens for OTLP data on port 4317. It receives, stores, and displays the logs in the Netdata UI. It does not collect logs itself.
• otel-collector — the upstream OpenTelemetry Collector, bundled as an optional subchart. It runs as a DaemonSet (one pod per node), reads each node's local container log files, and forwards them over OTLP to netdataOpentelemetry.

netdataOpentelemetry is the destination; otel-collector is what feeds it. The collector is bundled only as a convenient, working default, and is off by default because it is not the only option. If you already run a log pipeline (Fluent Bit, Vector, an existing Collector, or any OTLP-capable agent), leave otel-collector disabled and point that pipeline's OTLP exporter at the netdata-otel service on port 4317 instead.

Log flow:

container stdout/stderr
   │  the container runtime writes them to node-local files (/var/log/pods/…)
   ▼
otel-collector DaemonSet  (one pod per node, needs host log access)
   │  reads each node's log files, pushes over OTLP
   ▼
netdata-otel:4317  →  stored and shown in the Netdata UI

Securing the endpoint with TLS

By default the netdata-otel receiver listens on port 4317 in plaintext — TLS is disabled (tls_cert_path, tls_key_path, and tls_ca_cert_path are unset in netdataOpentelemetry.configs.otel.data). The steps below turn it on with a self-signed certificate. TLS affects both sides: the receiver must serve the certificate, and every client (including the bundled otel-collector) must be switched to TLS, or it will stop delivering data.

1. Generate a self-signed certificate and key (Linux, openssl):

openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout tls.key -out tls.crt -days 365 \
  -subj "/CN=netdata-otel"

2. Create a Kubernetes TLS secret from those files, in the chart's namespace:

kubectl create secret tls netdata-otel-tls \
  --cert=tls.crt --key=tls.key \
  --namespace <your-namespace>

3. Mount the secret into the receiver and point the config at it. The certificate paths live inside netdataOpentelemetry.configs.otel.data, which is a single block — supply it in full with the two tls_*_path values filled in (keep the metrics and logs sections in sync with the chart's values.yaml). Mounting the secret alone does nothing until these paths are set:

netdataOpentelemetry:
  extraVolumes:
    - name: otel-tls
      secret:
        secretName: netdata-otel-tls
  extraVolumeMounts:
    - name: otel-tls
      mountPath: /etc/netdata/otel-certs
      readOnly: true
  configs:
    otel:
      data: |
        endpoint:
          path: "0.0.0.0:4317"
          tls_cert_path: /etc/netdata/otel-certs/tls.crt
          tls_key_path: /etc/netdata/otel-certs/tls.key
          tls_ca_cert_path: null
        metrics:
          print_flattened: false
          buffer_samples: 10
          throttle_charts: 100
          chart_configs_dir: otel.d/v1/metrics
        logs:
          journal_dir: otel/v1
          size_of_journal_file: "100MB"
          number_of_journal_files: 10
          size_of_journal_files: "1GB"
          duration_of_journal_files: "7 days"
          duration_of_journal_file: "2 hours"
          store_otlp_json: false

4. Switch every client to TLS. A TLS listener rejects plaintext connections, so any sender must be reconfigured — otherwise logs silently stop arriving. For the bundled otel-collector, enable TLS on its OTLP exporter. Because the certificate is self-signed, skip verification with insecure_skip_verify — this keeps the connection encrypted but does not validate the certificate chain (only the tls block is overridden; the exporter's endpoint is kept from the chart defaults):

otel-collector:
  config:
    exporters:
      otlp:
        tls:
          insecure: false
          insecure_skip_verify: true

Apply the same change to any external OTLP client — Fluent Bit, Vector, or another Collector — pointing at the netdata-otel service.

For production, replace the self-signed certificate with one issued by a trusted CA, give it a CN/SAN that matches the netdata-otel service DNS name, and have clients trust that CA via tls_ca_cert_path instead of skipping verification.

Service discovery and supported services

Netdata's service discovery, which is installed as part of the Helm chart installation, finds what services are running on a cluster's pods, converts that into configuration files, and exports them, so they can be monitored.

Applications

Service discovery currently supports the following applications via their associated collector:

• ActiveMQ
• Apache
• Bind
• CockroachDB
• Consul
• CoreDNS
• Elasticsearch
• Fluentd
• FreeRADIUS
• HDFS
• Lighttpd
• Logstash
• MySQL
• NGINX
• OpenVPN
• PHP-FPM
• RabbitMQ
• Tengine
• Unbound
• VerneMQ
• ZooKeeper

Prometheus endpoints

Service discovery supports Prometheus endpoints via the Prometheus collector.

Annotations on pods allow a fine control of the scraping process:

• prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation excludes the pod from the scraping process.
• prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
• prometheus.io/port: Scrape the pod on the indicated port instead of the pod's declared ports.

Configure service discovery

If your cluster runs services on non-default ports or uses non-default names, you may need to configure service discovery to start collecting metrics from your services. You have to edit the default ConfigMap that is shipped with the Helmchart and deploy that to your cluster.

First, copy netdata-helmchart/charts/netdata/sdconfig/child.yml to a new location outside the netdata-helmchart directory. The destination can be anywhere you like, but the following examples assume it resides next to the netdata-helmchart directory.

cp netdata-helmchart/charts/netdata/sdconfig/child.yml .

Edit the new child.yml file according to your needs. See the Helm chart configuration and the file itself for details. You can then run helm install/helm upgrade with the --set-file argument to use your configured child.yml file instead of the default, changing the path if you copied it elsewhere.

helm install --set-file sd.child.configmap.from.value=./child.yml netdata ./netdata-helmchart/charts/netdata
helm upgrade --set-file sd.child.configmap.from.value=./child.yml netdata ./netdata-helmchart/charts/netdata

Now that you pushed an edited ConfigMap to your cluster, service discovery should find and set up metrics collection from your non-default service.

Custom pod labels and annotations

Occasionally, you will want to add specific labels and annotations to the parent and/or child pods. You might want to do this to tell other applications on the cluster how to treat your pods, or simply to categorize applications on your cluster. You can label and annotate the parent and child pods by using the podLabels and podAnnotations dictionaries under the parent and child objects, respectively.

For example, suppose you're installing Netdata on all your database nodes, and you'd like the child pods to be labeled with workload: database so that you're able to recognize this.

At the same time, say you've configured chaoskube to kill all pods annotated with chaoskube.io/enabled: true, and you'd like chaoskube to be enabled for the parent pod but not the childs.

You would do this by installing as:

$ helm install \
  --set child.podLabels.workload=database \
  --set 'child.podAnnotations.chaoskube\.io/enabled=false' \
  --set 'parent.podAnnotations.chaoskube\.io/enabled=true' \
  netdata ./netdata-helmchart/charts/netdata

Contributing

If you want to contribute, we're humbled!

• Take a look at our Contributing Guidelines.
• This repository is under the Netdata Code Of Conduct.
• Chat about your contribution and let us help you in our forum!