TopGit theo dõi fluent/fluent-operator trên GitHub trong nhóm Developer Tools, đã đạt 688 sao. Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
Facilitate the management of Fluent Bit and Fluentd in Kubernetes
Fluent Operator
Fluent Operator provides great flexibility in building a logging layer based on Fluent Bit and Fluentd.
Once installed, the Fluent Operator provides the following features:
Fluent Bit Management: Deploy and destroy Fluent Bit DaemonSet automatically.
Fluentd Management: Deploy and destroy Fluentd StatefulSet automatically.
Custom Configuration: Select input/filter/output plugins via labels.
Dynamic Reloading: Update configuration without rebooting Fluent Bit and Fluentd pods.
Multi-tenant log isolation: Fluentd supports multi-tenant log isolation through the label-router plugin.
Pluggable deployment components: Either Fluent Bit or Fluentd can be deployed separately.
Table of contents
Fluent Operator
Table of contents
Overview
Fluent Bit
Fluentd
Get Started
Prerequisites
Install
Deploy Fluent Operator with YAML
Deploy Fluent Operator with Helm
Fluent Operator Walkthrough
Collect Kubernetes logs
Deploy the Kubernetes logging stack with YAML
Deploy the Kubernetes logging pipeline with Helm
Collect audit logs
Fluentd
Plugins
Fluent Bit
Fluentd
Best Practice
Plugin Grouping
Monitoring
Custom Parser
Misc
Custom Plugin
Roadmap
Development
Requirements
Running
Contributing
Documentation
Manifests
Community
About the community
Contact us
Videos and blogs
Overview
Although both Fluent Bit and Fluentd can collect, process(parse and filter) and then forward log to the final destinations, still they have strengths in different aspects.
Fluent Bit is a good choice as a logging agent because of its lightweight and efficiency, while Fluentd is more powerful to perform advanced processing on logs because of its rich plugins.
Fluent Bit only mode: If you just need to collect logs and send logs to the final destinations, all you need is Fluent Bit.
Fluent Bit + Fluentd mode: If you also need to perform some advanced processing on the logs collected or send to more sinks, then you also need Fluentd.
Fluentd only mode: If you need to receive logs through networks like HTTP or Syslog and then process and send the log to the final sinks, you only need Fluentd.
Fluent Operator includes CRDs and controllers for both Fluent Bit and Fluentd which allows you to config your log processing pipelines in the 3 modes mentioned above as you wish.
Fluent Bit will be deployed as a DaemonSet while Fluentd will be deployed as a StatefulSet. The whole workflow could be described as below:
Fluent Bit
The following CRDs are defined for Fluent Bit:
FluentBit: Defines the Fluent Bit DaemonSet and its configs. A custom Fluent Bit image ghcr.io/fluent/fluent-operator/fluent-bit is required to work with FluentBit Operator for dynamic configuration reloading.
ClusterFluentBitConfig: Select cluster-level input/filter/output plugins and generates the final config into a Secret.
FluentBitConfig: Selects namespace-level filter/output/parser and cluster-level parser plugins and generates final config into a Secret.
Each ClusterInput, ClusterParser, ClusterFilter, ClusterMultilineParser, ClusterOutput represents a Fluent Bit config section, which are selected by ClusterFluentBitConfig via label selectors. Fluent Operator watches those objects, constructs the final config, and finally creates a Secret to store the config which will be mounted into the Fluent Bit DaemonSet. The entire workflow looks like below:
To enable Fluent Bit to pick up and use the latest config whenever the Fluent Bit config changes, a wrapper called Fluent Bit watcher is added to restart the Fluent Bit process as soon as Fluent Bit config changes are detected. This way, the Fluent Bit pod needn't be restarted to reload the new config. The Fluent Bit config is reloaded in this way because there is no reloading interface in Fluent Bit itself. Please refer to this known issue for more details.
Fluentd
The following CRDs are defined for Fluentd:
Fluentd: Defines the Fluentd Statefulset and its configs. A custom Fluentd image kubesphere/fluentd is required to work with Fluentd Operator for dynamic configuration reloading.
FluentdConfig: Select cluster-level or namespace-level scope input/filter/output plugins and generates the final config into a Secret.
ClusterFluentdConfig: Select cluster-level input/filter/output plugins and generates the final config into a Secret.
Kubernetes v1.16.13+ is necessary for running Fluent Operator.
Install
Deploy Fluent Operator with YAML
Install the latest stable version
kubectl create namespace fluent
kubectl apply -f https://github.com/fluent/fluent-operator/releases/latest/download/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
Install the development version
kubectl create namespace fluent
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-operator/master/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
The default container runtime is containerd. Set --set containerRuntime=docker or --set containerRuntime=crio if your cluster uses a different runtime.
Helm installs CRDs on first install but does not upgrade or delete them on subsequent helm upgrade or helm uninstall calls. To update CRDs when upgrading the chart:
Advanced installation — full Helm lifecycle management of CRDs via the separate fluent-operator-fluent-bit-crds and fluent-operator-fluentd-crds charts:
# Step 1: install Fluent Bit CRDs as a separately managed release
helm install fluent-operator-fluent-bit-crds fluent/fluent-operator-fluent-bit-crds -n fluent --create-namespace
# Step 2: install Fluentd CRDs as a separately managed release (optional)
helm install fluent-operator-fluentd-crds fluent/fluent-operator-fluentd-crds -n fluent
# Step 3: install the operator, skipping CRDs (already installed above)
helm install fluent-operator fluent/fluent-operator -n fluent --skip-crds
With this approach, CRDs are upgraded with helm upgrade fluent-operator-fluent-bit-crds / helm upgrade fluent-operator-fluentd-crds and can be protected from deletion with --set additionalAnnotations."helm\.sh/resource-policy"=keep.
See the MIGRATION-v4.md guide for more information on upgrading from v3.x.
Fluent Operator Walkthrough
For more info on various use cases of Fluent Operator, you can refer to Fluent-Operator-Walkthrough.
Collect Kubernetes logs
This guide provisions a logging pipeline including the Fluent Bit DaemonSet and its log input/filter/output configurations to collect Kubernetes logs including container logs and kubelet logs.
Note that you need a running Elasticsearch v5+ cluster to receive log data before start. Remember to adjust output-elasticsearch.yaml to your own es setup. Kafka and Fluentd outputs are optional and are turned off by default.
Deploy the Kubernetes logging stack with YAML
kubectl apply -f manifests/logging-stack
# You can change the namespace in manifests/logging-stack/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/ | kubectl apply -f -
Deploy the Kubernetes logging pipeline with Helm
The Kubernetes logging pipeline (container log collection via Fluent Bit) is enabled by default (Kubernetes: true). To also deploy Fluentd as a log aggregation layer:
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2020.04.26 uwQuoO90TwyigqYRW7MDYQ 1 1 99937 0 31.2mb 31.2mb
Success!
Collect audit logs
The Linux audit framework provides a CAPP-compliant (Controlled Access Protection Profile) auditing system that reliably collects information about any security-relevant (or non-security-relevant) event on a system. Refer to manifests/logging-stack/auditd, it supports a method for collecting audit logs from the Linux audit framework.
kubectl apply -f manifests/logging-stack/auditd
# You can change the namespace in manifests/logging-stack/auditd/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/auditd/ | kubectl apply -f -
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2021.04.06 QeI-k_LoQZ2h1z23F3XiHg 5 1 404879 0 298.4mb 149.2mb
Fluentd
Fluentd acts as a log forward layer that receives logs from Fluent Bit or other Apps through the network.
For more info on various use cases of Fluent Operator Fluentd CRDs, you can refer to Fluent-Operator-Walkthrough.
Plugins
Fluent Bit
Here you can find supported Fluent Bit plugins.
Fluentd
Here you can find supported Fluentd plugins.
Best Practice
Plugin Grouping
Here you can find the plugin group information.
Monitoring
Here you can use the built-in HTTP Server in Fluent Bit.
Custom Parser
Here you can customize parser in Fluent Bit.
Misc
If you want to learn more about Fluent-Operator, please refer to the misc.
Custom Plugin
Here you can learn how to use custom plugin in FluentBit and Fluentd.
Roadmap
Support containerd log format
Add Fluentd CRDs as the log aggregation layer with group name fluentd.fluent.io
Add FluentBit Cluster CRDs with new group name fluentbit.fluent.io
Rename the entire project to Fluent Operator
Support more Fluentd & FluentBit plugins
Development
Requirements
golang v1.16+.requirement
kubectl v1.19.2+.
kubebuilder v3.1+ (the project is build with v3.1.0)
Access to a Kubernetes cluster v1.19.2+
Running
Make sure you have a KUBECONFIG file loaded for your local developement kubernetes cluster.
Install CRDs: make install
export NAMESPACE=something to set the namespace for the operator
Run: make run
Contributing
Documentation
API Doc is generated automatically. To modify it, edit the comment above struct fields, then run go run cmd/doc-gen/main.go.
Manifests
Most files under the folder manifests/setup are automatically generated from config. Don't edit them directly, run make manifests instead, then replace these files accordingly.
Community
About the community
Fluent Operator was previously sponsored and open-sourced by the KubeSphere Team and maintained by the Fluent community.
This project was accepted by the Fluent community as a sub-project in 2021 and is currently a vendor-neutral and community-driven project.
Contact us
You are welcome to join us in #fluent-operator on the CNCF Slack.
Videos and blogs
From FluentBit Operator to Fluent Operator: The Long Way to Come - Benjamin Huo
What is Fluent Operator and how do you use it - Henrik Rexed
Under the Hood with Fluent Bit Operator: Kubernetes-native Log Processor - Feynman Zhou & Dhruv Kela
fluent/fluent-operator có 688 sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/fluent/fluent-operator. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
fluent/fluent-operator có những chủ đề gì?
GitHub topics của fluent/fluent-operator: "fluentbit", "fluentd", "hacktoberfest", "kubernetes", "kubesphere", "logging", "operator". TopGit xếp repo vào nhóm Developer Tools.
fluent/fluent-operator còn đang phát triển không?
Commit gần nhất trên fluent/fluent-operator là 10 ngày trước (theo timestamp GitHub). Repo có 333 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
fluent/fluent-operator là gì?
fluent/fluent-operator (fluent/fluent-operator) là dự án Go trên GitHub. Theo mô tả gốc: Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
fluent/fluent-operator so với các dự án Developer Tools khác thế nào?
fluent/fluent-operator được TopGit xếp vào nhóm Developer Tools, với 688 sao GitHub và viết bằng Go. Xem trang chủ đề Developer Tools trên TopGit để so sánh với các dự án tương tự theo số sao và mức độ hoạt động.
fluent/fluent-operator viết bằng ngôn ngữ gì?
fluent/fluent-operator chủ yếu viết bằng Go. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Vì sao fluent/fluent-operator được xếp vào nhóm Developer Tools?
TopGit xếp fluent/fluent-operator vào nhóm Developer Tools dựa trên GitHub topics và mô tả của repo (gắn thẻ: "fluentbit", "fluentd", "hacktoberfest"). Việc phân loại dựa trên metadata thật của repo, không phải đoán theo cảm tính biên tập.
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về fluent-operator?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về fluent-operator.