Được TopGit lập chỉ mục từ metadata GitHub: sogou/workflow có 14.4k sao, viết chủ yếu bằng C++. C++ Parallel Computing and Asynchronous Networking Framework
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
As Sogou`s C++ server engine, Sogou C++ Workflow supports almost all back-end C++ online services of Sogou, including all search services, cloud input method, online advertisements, etc., handling more than 10 billion requests every day. This is an enterprise-level programming engine in light and elegant design which can satisfy most C++ back-end development requirements.
You can use it:
To quickly build an HTTP server:
#include <stdio.h>
#include "workflow/WFHttpServer.h"
int main()
{
WFHttpServer server([](WFHttpTask *task) {
task->get_resp()->append_output_body("<html>Hello World!</html>");
});
if (server.start(8888) == 0) { // start server on port 8888
getchar(); // press "Enter" to end.
server.stop();
}
return 0;
}
As a multifunctional asynchronous client, it currently supports HTTP, Redis, MySQL and Kafka protocols.
MySQL protocol supports MariaDB, TiDB as well.
To implement client/server on user-defined protocol and build your own RPC system.
srpc is based on it and it is an independent open source project, which supports srpc, brpc, trpc and thrift protocols.
To build asynchronous workflow; support common series and parallel structures, and also support any DAG structures.
As a parallel computing tool. In addition to networking tasks, Sogou C++ Workflow also includes the scheduling of computing tasks. All types of tasks can be put into the same flow.
As an asynchronous file IO tool in Linux system, with high performance exceeding any system call. Disk file IO is also a task.
To realize any high-performance and high-concurrency back-end service with a very complex relationship between computing and networking.
To build a micro service system.
This project has built-in service governance and load balancing features.
Wiki link : PaaS Architecture
Compiling and Running Environment
This project supports Linux, macOS, Windows, Android and other operating systems.
Windows version is currently released as an independent branch, using iocp to implement asynchronous networking. All user interfaces are consistent with the Linux version.
Supports all CPU platforms, including 32 or 64-bit x86 processors, big-endian or little-endian arm processors, loongson processors.
Master branch requires OpenSSL 1.1 or above, and BoringSSL is fully compatible. If you don't like SSL, you may checkout the nossl branch.
Uses the C++11 standard and therefore, it should be compiled with a compiler which supports C++11. Does not rely on boost or asio.
No other dependencies. However, if you need Kafka protocol, some compression libraries should be installed, including lz4, zstd and snappy.
Get Started (Linux, macOS):
git clone https://github.com/sogou/workflow
cd workflow
make
cd tutorial
make
Sogou C++ Workflow has been packaged for Debian Linux and ubuntu 22.04.
To install the Workflow library for development purposes:
sudo apt-get install libworkflow-dev
To install the Workflow library for deployment:
sudo apt-get install libworkflow1
With dnf on Fedora Linux:
Sogou C++ Workflow has been packaged for Fedora Linux.
To install the Workflow library for development purposes:
sudo dnf install workflow-devel
To install the Workflow library for deployment:
sudo dnf install workflow
With xmake
If you want to use xmake to build workflow, you can see xmake build document
Tutorials
Client
Creating your first task：wget
Implementing Redis set and get：redis_cli
More features about series：wget_to_redis
Server
First server：http_echo_server
Asynchronous server：http_proxy
Parallel tasks and Series
A simple parallel wget：parallel_wget
Important topics
About error
About timeout
About global configuration
About DNS
About exit
Computing tasks
Using the build-in algorithm factory：sort_task
User-defined computing task：matrix_multiply
Use computing task in a simple way: go task
Asynchronous File IO tasks
Http server with file IO：http_file_server
User-defined protocol
A simple user-defined protocol: client/server
Use TLV message
Other important tasks/components
About timer
About counter
About resource pool
About module
About DAG
Service governance
About service governance
More documents about upstream
Connection context
About connection context
Built-in clients
Asynchronous MySQL client：mysql_cli
Asynchronous Kafka client: kafka_cli
Programming Paradigm
Program = Protocol + Algorithm + Workflow
Protocol
In most cases, users use built-in common network protocols, such as HTTP, Redis or various rpc.
Users can also easily customize user-defined network protocol. In the customization, they only need to provide serialization and deserialization functions to define their own client/server.
Algorithm
In our design, the algorithm is a concept symmetrical to the protocol.
If protocol call is rpc, then algorithm call is an apc (Async Procedure Call).
We have provided some general algorithms, such as sort, merge, psort, reduce, which can be used directly.
Compared with a user-defined protocol, a user-defined algorithm is much more common. Any complicated computation with clear boundaries should be packaged into an algorithm.
Workflow
Workflow is the actual business logic, which is to put the protocols and algorithms into the flow graph for use.
The typical workflow is a closed series-parallel graph. Complex business logic may be a non-closed DAG.
The workflow graph can be constructed directly or dynamically generated based on the results of each step. All tasks are executed asynchronously.
Structured Concurrency and Task Abstraction
Our system contains five basic tasks: communication, computation, file IO, timer, and counter.
All tasks are generated by the task factory, and users organize the concurrency structure by calling interfaces, such as series, parallel, DAG, etc.
In most cases, the tasks generated by the user through the task factory is a complex task which encapsulates multiple asynchronous processes, but it is transparent to the user.
For example, an HTTP request may include many asynchronous processes (DNS, redirection), but for user, it is just a networking task.
File sorting seems to be an algorithm, but it actually includes many complex interaction processes between file IO and CPU computation.
If you think of business logic as building circuits with well-designed electronic components, then each electronic component may be a complex circuit.
The task abstraction mechanism greatly reduces the number of tasks users need to create and the depth of callbacks.
Any task runs in a SeriesWork and the tasks in the same SeriesWork shares the series context, which simplifies data transfer between asynchronous tasks.
Callback and Memory Reclamation Mechanism
All calls are executed asynchronously, and there is almost no operation that occupies a thread.
Explicit callback mechanism. Users are aware that they are writing asynchronous programs.
A set of object lifecycle mechanisms greatly simplifies memory management for asynchronous programs.
The lifecycle of any task created by the framework is from creation until the callback function finishes running. There is no risk of leakage.
If a task is created but the user does not want to run it, the user needs to release it through the dismiss() interface.
Any data in the task, such as the response of the network request, will also be recycled with the task. At this time, the user can use std::move() to move the required data.
The project doesn’t use std::shared_ptr to manage memory.
We try to avoid user's derivations, and encapsulate user behavior with std::function instead, including:
The callback of any task.
Any server's process. This conforms to the FaaS (Function as a Service) idea.
The realization of an algorithm is simply a std::function. But the algorithm can also be implemented by derivation.
If used deeply, one will find that everything can be derived.
Any Other Questions?
You may check the FAQ and issues list first to see if you can find the answer.
You are very welcome to send the problems you encounter in use to issues, and we will answer them as soon as possible. At the same time, more issues will also help new users.
sogou/workflow thuộc nhóm Data trên TopGit, cùng 8 topic GitHub. Trang Trending và Topics liệt kê các repo cùng số sao và cùng ngôn ngữ để so sánh.
Đọc thêm về sogou/workflow ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/sogou/workflow là nguồn chính thức.
sogou/workflow có bao nhiêu sao?
sogou/workflow có 14.4k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/sogou/workflow. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
sogou/workflow có những chủ đề gì?
GitHub topics của sogou/workflow: "consul", "dag", "http", "kafka", "mysql", "redis", "sogou", "tasking". TopGit xếp repo vào nhóm Data.
sogou/workflow có phải mã nguồn mở không?
Có — sogou/workflow phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/sogou/workflow.
sogou/workflow có website riêng không?
TopGit chưa ghi nhận URL trang chủ cho sogou/workflow. Phần README ở tab phía trên thường có link demo, hoặc xem mô tả GitHub của repo.
sogou/workflow còn đang phát triển không?
Commit gần nhất trên sogou/workflow là 18 ngày trước (theo timestamp GitHub). Repo có 2.6k fork — một chỉ báo về mức độ quan tâm của cộng đồng.
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về workflow?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về workflow.
