An AI-powered entry in TopGit's GitHub warehouse: microsoft/SynapseML, 5.3k stars, AI Tools, Scala. Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
SynapseML (previously known as MMLSpark) is an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. SynapseML provides simple, composable, and distributed APIs for a wide variety of different machine learning tasks such as text analytics, vision, anomaly detection, and many others. SynapseML is built on the Apache Spark distributed computing framework and shares the same API as the SparkML/MLLib library, allowing you to seamlessly embed SynapseML models into existing Apache Spark workflows.
With SynapseML, you can build scalable and intelligent systems to solve challenges in domains such as anomaly detection, computer vision, deep learning, text analytics, and others. SynapseML can train and evaluate models on single-node, multi-node, and elastically resizable clusters of computers. This lets you scale your work without wasting resources. SynapseML is usable across Python, R, Scala, Java, and .NET. Furthermore, its API abstracts over a wide variety of databases, file systems, and cloud data stores to simplify experiments no matter where data is located.
SynapseML publishes runtime-specific JVM artifacts: Spark 3.5 uses Scala 2.12,
while Spark 4.0 and 4.1 use Scala 2.13. See the
installation matrix before selecting a Maven
coordinate.
Topics
Links
Build
Version
Docs
Support
Binder
Usage
Table of Contents
Synapse Machine Learning
Features
Documentation and Examples
Setup and installation
Microsoft Fabric
Synapse Analytics
Databricks
Python Standalone
Spark Submit
SBT
Apache Livy and HDInsight
Docker
R
Building from source
Papers
Learn More
Contributing & feedback
Other relevant projects
Features
Vowpal Wabbit on Spark
The Cognitive Services for Big Data
LightGBM on Spark
Spark Serving
Fast, Sparse, and Effective Text Analytics
Leverage the Microsoft Cognitive Services at Unprecedented Scales in your existing SparkML pipelines
Train Gradient Boosted Machines with LightGBM
Serve any Spark Computation as a Web Service with Sub-Millisecond Latency
HTTP on Spark
ONNX on Spark
Responsible AI
Spark Binding Autogeneration
An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration
Distributed and Hardware Accelerated Model Inference on Spark
Understand Opaque-box Models and Measure Dataset Biases
Automatically Generate Spark bindings for PySpark and SparklyR
Isolation Forest on Spark
CyberML
Conditional KNN
Distributed Nonlinear Outlier Detection
Machine Learning Tools for Cyber Security
Scalable KNN Models with Conditional Queries
Documentation and Examples
For quickstarts, documentation, demos, and examples please see our website.
Setup and installation
SynapseML installation has two parts: the language wrapper and the JVM
artifacts loaded by Spark. Installing synapseml from PyPI does not add the
JVM artifacts. A Python wrapper can import successfully while its JVM class is
missing; using a _2.12 artifact with Spark 4 can produce errors such as
LightGBMClassifier does not exist in the JVM.
Choose one complete published build from the Spark runtime. master is the
canonical Spark 3.5 development line; Spark 4.0 and Spark 4.1 are maintained on
their corresponding branches.
Code line
Spark runtime
Scala
Python baseline
Release tag
Python package
Maven coordinate
master
Spark 3.5.x
2.12
Python 3.11
v1.1.3
synapseml==1.1.3
com.microsoft.azure:synapseml_2.12:1.1.3
spark4.0
Spark 4.0.1+ (<4.1)
2.13
Python 3.12
v1.1.3-spark4.0
synapseml==1.1.3
com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
spark4.1
Spark 4.1.x
2.13
Python 3.13
v1.1.3-spark4.1
synapseml==1.1.3
com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1
Always configure
https://mmlspark.blob.core.windows.net/maven, where the Spark 4 artifacts are
published. See the full installation guide for platform-specific details.
Latest master snapshot
The latest successful master build targets Spark 3.5 and Scala 2.12. This
copy-ready command reads the current snapshot version published by CI and starts
Spark with that exact JVM build:
--packages is supported for production Spark jobs as well as notebooks. For
example, use it with spark-submit on EMR rather than downloading a single
JAR manually:
The top-level synapseml_2.12 artifact is an aggregate artifact: its POM
references the SynapseML module JARs and their runtime dependencies, but the
artifact JAR itself does not contain implementation classes. In particular,
IsolationForest is in synapseml-core_2.12 and also requires the LinkedIn
Isolation Forest dependency. Spark resolves this complete dependency closure
when --packages or spark.jars.packages is used.
If a cluster image or pre-installed environment must be manually provisioned,
use a Maven-compatible dependency resolver during setup to install the complete
runtime dependency closure for the selected coordinate, and verify that the JARs
are added to both the driver and executor JVM classpaths (for example via
spark.driver.extraClassPath and spark.executor.extraClassPath or standard
cluster classpath configuration).
Manual JAR placement is error-prone:
Merely having a JAR localized or added to Python/YARN/Livy search paths does
not ensure that its classes are loaded into the Spark driver's JVM classpath.
Copying only the aggregate artifact or omitting transitive dependencies will
cause ClassNotFoundException or NoClassDefFoundError (often surfaced via
Py4J as <class> does not exist in the JVM) even if the Python module imports
successfully.
Always pin a single coordinate matching the cluster's Spark and Scala versions.
SBT
Choose the dependency matching your Spark runtime.
To install SynapseML from within a Jupyter notebook served by Apache Livy, the
following Spark 3.5 / Scala 2.12 configure magic can be used. You will need to
start a new session after this configure cell is executed.
Excluding certain packages from the library may be necessary due to current issues with Livy 0.5.
The easiest way to evaluate SynapseML is via our pre-built Docker container. To
do so, run the following command:
docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release jupyter notebook
Navigate to http://localhost:8888/ in your web browser to run the sample
notebooks. See the documentation for more on Docker use.
To read the EULA for using the docker image, run docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula
R
To try out SynapseML using the R autogenerated wrappers see our
instructions. Note: This feature is still under development
and some necessary custom wrappers may be missing.
Building from source
SynapseML has recently transitioned to a new build infrastructure.
For detailed developer docs, please see the Developer Readme.
If you are an existing SynapseML developer, you will need to reconfigure your
development setup. We now support platform-independent development and
better integrate with IntelliJ and SBT.
If you encounter issues, please reach out to our support email!
Papers
Large Scale Intelligent Microservices
Conditional Image Retrieval
MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales
Flexible and Scalable Deep Learning with SynapseML
Large-Scale Automatic Audiobook Creation
Learn More
Visit our website.
Watch our keynote demos at the Spark+AI Summit 2019, the Spark+AI European Summit 2018, the Spark+AI Summit 2018 and SynapseML at the Spark Summit.
See how SynapseML is used to help endangered species.
Explore generative adversarial artwork in our collaboration with The MET and MIT.
Explore our collaboration with Apache Spark on image analysis.
Contributing & feedback
This project has adopted the Microsoft Open Source Code of Conduct. For more
information see the Code of Conduct FAQ or contact
[email protected] with any additional
questions or comments.
See CONTRIBUTING.md for contribution guidelines.
To give feedback and/or report an issue, open a GitHub
Issue.
Other relevant projects
Vowpal Wabbit
LightGBM
DMTK: Microsoft Distributed Machine Learning Toolkit
Recommenders
JPMML-SparkML plugin for converting SynapseML LightGBM models to PMML
Microsoft Cognitive Toolkit
Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
The most recent commit recorded on microsoft/SynapseML was 3 months ago, based on the GitHub push timestamp. The repository has 873 forks — one of the better signals of community interest.
How many stars does microsoft/SynapseML have?
microsoft/SynapseML has 5.3k GitHub stars — refresh the page for the live number, or check github.com/microsoft/SynapseML. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is microsoft/SynapseML open source?
Yes — microsoft/SynapseML ships under the MIT license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/microsoft/SynapseML.
What else is in the AI Tools space?
microsoft/SynapseML is tracked by TopGit under the AI Tools category, alongside 20 GitHub-tagged topics. Trending and Topics pages list peer repositories of comparable stars and language.
What topics is microsoft/SynapseML associated with?
The project maintains a homepage at http://aka.ms/spark. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about microsoft/SynapseML?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/microsoft/SynapseML is the definitive source.
Read full README in the tab above.
Still deciding about SynapseML?
One click hands the question to an AI along with this page — see what it says about SynapseML.