Indexed by TopGit from live GitHub metadata: pingcap/tispark has 889 stars, written primarily in Scala. TiSpark is built for running Apache Spark on top of TiDB/TiKV
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
TiSpark is a thin layer built for running Apache Spark on top of TiDB/TiKV/TiFlash to answer complex OLAP queries. It enjoys the merits of both the Spark platform and the distributed clusters of TiKV/TiFlash while seamlessly integrated to TiDB.
The figure below show the architecture of TiSpark.
TiSpark integrates well with the Spark Catalyst Engine. It provides precise control of computing, which allows Spark to read data from TiKV efficiently. It also supports index seek, which significantly improves the performance of the point query execution.
It utilizes several strategies to push down computing to reduce the size of dataset handling by Spark SQL, which accelerates query execution. It also uses the TiDB built-in statistical information for the query plan optimization.
From the perspective of data integration, TiSpark + TiDB provides a solution that performs both transaction and analysis directly on the same platform without building and maintaining any ETLs. It simplifies the system architecture and reduces the cost of maintenance.
In addition, you can deploy and utilize the tools from the Spark ecosystem for further data processing and manipulation on TiDB. For example, using TiSpark for data analysis and ETL, retrieving data from TiKV as a data source for machine learning, generating reports from the scheduling system and so on.
TiSpark relies on the availability of TiKV clusters and PDs. You also need to set up and use the Spark clustering platform.
Most of the TiSpark logic is inside a thin layer, namely, the tikv-client library.
Doc TOC
User Guide
Dev Guide
Benchmark
About mysql-connector-java
We will not provide the mysql-connector-java dependency because of the limit of the GPL license.
The following versions of TiSpark's jar will no longer include mysql-connector-java.
TiSpark > 3.0.1
TiSpark > 2.5.1 for TiSpark 2.5.x
TiSpark > 2.4.3 for TiSpark 2.4.x
Now, TiSpark needs mysql-connector-java for writing and auth. Please import mysql-connector-java manually when you need to write or auth.
you can import it by putting the jar into spark jars file
you can also import it when you submit spark job like
TiDB starts to support view since tidb-3.0. TiSpark currently does not supportview. Users are not be able to observe or access data through view with TiSpark.
Spark config spark.sql.runSQLOnFiles should not be set to false, or you may got Error in query: Table or view not found error.
Using the style of "{db}.{table}.{colname}" in the condition is not supported, e.g. select * from t where db.t.col1 = 1.
Null in aggregration is not supported, e.g. select sum(null) from t group by col1.
The dependency tispark-assembly should not be packaged into JAR of JARS file (for example, build with spring-boot-maven-plugin), or you will get ClassNotFoundException. You can solve it by adding spark-wrapper-spark-version in your dependency or constructing another forms of jar file.
TiSpark doesn't support GBK character set.
TiSpark doesn't support the whole collations rule. Currently, TiSpark only supports the following collations: utf8_bin, utf8_general_ci, utf8_unicode_ci, utf8mb4_bin, utf8mb4_general_ci and utf8mb4_unicode_ci.
If spark.sql.ansi.enabled is false an overflow of sum(bigint) will not cause an error but “wrap” the result, or you can cast bigint to decimal to avoid the overflow.
TiSpark supports retrieving data from table with Expression Index, but the Expression Index will not be used by the planner of TiSpark.
Follow us
Twitter
@PingCAP
Forums
For English users, go to TiDB internals.
For Chinese users, go to AskTUG.
License
TiSpark is under the Apache 2.0 license. See the LICENSE file for details.
The most recent commit recorded on pingcap/tispark was 26 days ago, based on the GitHub push timestamp. The repository has 251 forks — one of the better signals of community interest.
How many stars does pingcap/tispark have?
pingcap/tispark has 889 GitHub stars — refresh the page for the live number, or check github.com/pingcap/tispark. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is pingcap/tispark written in?
pingcap/tispark is written primarily in Scala. GitHub's language field is based on the largest share of bytes in the default branch.
What topics is pingcap/tispark associated with?
GitHub's repository topics for pingcap/tispark: "bigdata", "spark", "tidb", "tikv". TopGit's editorial category is open-source.
Where do I read more about pingcap/tispark?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/pingcap/tispark is the definitive source.
Read full README in the tab above.
Want a second opinion on tispark?
Ask an AI that can read this page — one click and you get its take on tispark.