digitalocean/firebolt — 719★ trên GitHub (Go). Golang framework for streaming ETL, observability data pipeline, and event processing apps
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
A golang framework for streaming event processing & data pipeline apps
Introduction
Firebolt has a simple model intended to make it easier to write reliable pipeline applications that process a stream of data.
It can be used to build systems such as:
logging/observability pipelines
streaming ETL
event processing pipelines
Every application's pipeline starts with a single source, the component that receives events from some external system. Sources
must implement the node.Source interface.
We provide one built-in source:
kafkaconsumer - Events come from a Kafka topic, and are passed to the root nodes as []byte
The processing of your application is executed by its nodes which form a processing tree. Data - events - flow down
this tree. A parent node passes results down to it's child nodes. Nodes may process events synchronously or
asynchronously. A synchronous node type node.FanoutNode returns a slice of results for fanout or 'demultiplexing' use
cases. Each node must implement the node.SyncNode, node.FanoutNode, or node.AsyncNode interfaces accordingly.
We provide two built-in node types:
kafkaproducer - Events are produced onto a kafka topic by an asynchronous producer.
elasticsearch - Events are bulk indexed into Elasticsearch.
Firebolt has both run and compile-time dependencies on librdkafka, see Developing
Example: Logging Pipeline
At DigitalOcean, our first use of Firebolt was in our logging pipeline. This pipeline consumes logs from just about
every system we run. The diagram below depicts the source and nodes in this application.
This system uses the built-in kafkaconsumer source (in yellow) and kafkaproducer and elasticsearch nodes (in green).
The blue nodes are custom to this application.
What does Firebolt do for me?
Firebolt is intended to address a number of concerns that are common to near-realtime data pipeline applications, making it easy
to run a clustered application that scales predictably to handle large data volume.
It is not an analytics tool - it does not provide an easy way to support 'wide operations' like record grouping, windowing,
or sorting that require shuffling data within the cluster. Firebolt is for 'straight through' processing pipelines that are
not sensitive to the order in which events are processed.
Some of the concerns Firebolt addresses include:
kafka sources Minimal configuration and no code required to consume from a Kafka topic, consumer lag metrics included
kafka sinks Same for producing to a Kafka topic
loose coupling Nodes in the pipeline are loosely coupled, making them easily testable and highly reusable
simple stream filtering Filter the stream by returning nil in your nodes
convenient error handling Send events that fail processing to a kafka topic for recovery or analysis with a few lines of config
outage recovery: offset management Configurable Kafka offset management during recovery lets you determine the maximum "catch up" to attempt after an outage, so you can quickly get back to realtime processing.
outage recovery: parallel recovery After an outage, process realtime data and "fill-in" the outage time window in parallel, with a rate limit on the recovery window.
monitorability Firebolt exposes Prometheus metrics to track the performance of your Source and all Nodes without writing code. Your nodes can expose their own custom internal metrics as needed.
leader election Firebolt uses Zookeeper to conduct leader elections, facilitating any processing that may need to be conducted on one-and-only-one instance.
Documentation
Configuration The configuration file format
Execution How Firebolt processes your data
Registry Adding node types to the registry
Sample Application Code Example code for running the Firebolt executor
Sources Implementing and using sources
Sync Nodes Implementing and using synchronous nodes
Fanout Nodes Implementing and using fanout nodes
Async Nodes Implementing and using asynchronous nodes
Leader Election Starting leader election and accessing election results
Messaging How to send and receive messages between the components of your system
Metrics What metrics are exposed by default, and how to add custom metrics to your nodes
Built-In Types
Kafka Producer Node for producing events onto a Kafka topic
Elasticsearch Node for indexing documents to an Elasticsearch cluster
Developing
Firebolt depends on librdkafka v1.3.0 or later. To get started building a
firebolt app (or working on firebolt itself), install it following the
instructions here.
TopGit chưa ghi nhận license cho digitalocean/firebolt. Phần lớn repo public trên GitHub là mã nguồn mở, nhưng điều khoản khác nhau từng repo — mở file LICENSE để xác nhận.
digitalocean/firebolt là gì?
digitalocean/firebolt (digitalocean/firebolt) là dự án Go trên GitHub. Theo mô tả gốc: Golang framework for streaming ETL, observability data pipeline, and event processing apps
Đọc thêm về digitalocean/firebolt ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/digitalocean/firebolt là nguồn chính thức.
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về firebolt?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về firebolt.