An open-source entry in TopGit's GitHub warehouse: deepinsight/mxnet-operator, 43 stars, Go. Tools for ML/MXNet on Kubernetes.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
(Forks initally from tf-operator project https://github.com/jlewi/mlkube.io.git now moved to https://github.com/tensorflow/k8s)
Requirements
Custom Resources require Kubernetes 1.7
Motivation
Distributed MXNet training jobs require managing multiple sets of MXNet replicas.
There are three kinds of replicas sets which act as different roles in the job:
Server (1-n instance)
Worker (1-n instance)
Scheduler(only one instance in a training cluster).
K8s makes it easy to configure and deploy each set of MXNet replicas. Various tools like
helm and ksonnet can
be used to simplify generating the configs for a MXNet job. Helm will be supported a few days later.
However, in addition to generating the configs we need some custom control logic because
K8s built-in controllers (Jobs, ReplicaSets, StatefulSets, etc...) don't provide the semantics
needed for managing MXNet jobs.
To solve this we define a
K8S Custom Resource
and Operator to manage a MXNet
job on K8s.
MxJob provides a K8s resource representing a single, distributed, mxnet training job.
The Spec and Status (defined in mx_job.go
are customized for MXNet. The spec allows specifying the Docker image and arguments to use for each MXNet
replica (i.e. scheduler, worker, and parameter server). The status provides relevant information such as the number of
replicas in various states.
Using a CRD gives users the ability to create and manage mxnet Jobs just like builtin K8s resources. For example to
create a job
kubectl create -f examples/mx_job_local.yaml
To list jobs
kubectl get mxjobs
NAME KINDS
example-dist-job MxJob.v1beta1.mlkube.io
Design
The code is closely modeled on Coreos's etcd-operator.
The MxJob Spec(defined in mx_job.go
reuses the existing Kubernetes structure PodTemplateSpec to describe MXNet processes.
We use PodTemplateSpec because we want to make it easy for users to
configure the processes; for example setting resource requirements or adding volumes.
We expect
helm or ksonnet could be used to add syntactic sugar to create more convenient APIs for users not familiar
with Kubernetes.
Leader election allows a K8s deployment resource to be used to upgrade the operator.
Installing the CRD and operator on your k8s cluster
The status information reported by the operator is hacky and not well thought out. In particular, we probably
need to figure out what the proper phases and conditions to report are.
Failure/Termination Semantics
The semantics for aggregating status of individual replicas into overall MxJob status needs to be thought out.
Dead/Unnecessary code
There is a lot of code from earlier versions (including the ETCD operator) that still needs to be cleaned up.
Testing
There is minimal testing.
Unittests
There are some unittests.
E2E tests
The helm package provides some basic E2E tests.
Building the Operator
Resolve dependencies (if you don't have glide install, check how to do it here)
glide install
Runing the Operator Locally
Running the operator locally (as opposed to deploying it on a K8s cluster) is convenient for debugging/development.
We can configure the operator to run locally using the configuration available in your kubeconfig to communicate with
a K8s cluster.
* MY_POD_NAMESPACE is used because the CRD is namespace scoped and we use the namespace of the controller to
set the corresponding namespace for the resource.
TODO(jlewi): Do we still need to set MY_POD_NAME? Why?
Go version
On ubuntu the default go package appears to be gccgo-go which has problems see issue golang-go package is also really old so install from golang tarballs instead.
Vendoring
You may need to remove the vendor directory of dependencies that also vendor dependencies as these may produce conflicts
with the versions vendored by mlkube; e.g.
Does deepinsight/mxnet-operator have a project website?
No homepage URL was recorded for deepinsight/mxnet-operator in TopGit's last sync. The README tab above frequently contains screenshots and demo links, or check the repository description on GitHub.
How many stars does deepinsight/mxnet-operator have?
deepinsight/mxnet-operator has 43 GitHub stars — refresh the page for the live number, or check github.com/deepinsight/mxnet-operator. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is deepinsight/mxnet-operator open source?
Yes — deepinsight/mxnet-operator ships under the Apache-2.0 license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/deepinsight/mxnet-operator.
What is deepinsight/mxnet-operator?
deepinsight/mxnet-operator (deepinsight/mxnet-operator) is a Go project on GitHub. From the project's own README: Tools for ML/MXNet on Kubernetes.
Where do I read more about deepinsight/mxnet-operator?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/deepinsight/mxnet-operator is the definitive source.
Read full README in the tab above.
Is mxnet-operator worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of mxnet-operator.