An open-source entry in TopGit's GitHub warehouse: Yelp/mrjob, 2.6k stars, Python. Run MapReduce jobs on Hadoop or Amazon Web Services
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows you
to buy time on a Hadoop cluster on an hourly basis. mrjob has basic support for Google Cloud Dataproc (Dataproc)
which allows you to buy time on a Hadoop cluster on a minute-by-minute basis. It also works with your own
Hadoop cluster.
Some important features:
Run jobs on EMR, Google Cloud Dataproc, your own Hadoop cluster, or locally (for testing).
Write multi-step jobs (one map-reduce step feeds into the next)
Easily launch Spark jobs on EMR or your own Hadoop cluster
Duplicate your production environment inside Hadoop
Upload your source tree and put it in your job's $PYTHONPATH
Run make and other setup scripts
Set environment variables (e.g. $TZ)
Easily install python packages from tarballs (EMR only)
Setup handled transparently by mrjob.conf config file
Automatically interpret error logs
SSH tunnel to hadoop job tracker (EMR only)
Minimal setup
To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY
To run on Dataproc, set $GOOGLE_APPLICATION_CREDENTIALS
No setup needed to use mrjob on your own Hadoop cluster
Installation
pip install mrjob
As of v0.7.0, Amazon Web Services and Google Cloud Services are optional
depedencies. To use these, install with the aws and google targets,
respectively. For example:
pip install mrjob[aws]
A Simple Map Reduce Job
Code for this example and more live in mrjob/examples.
.. code-block:: python
"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re
WORD_RE = re.compile(r"[\w']+")
class MRWordFreqCount(MRJob):
def mapper(self, _, line):
for word in WORD_RE.findall(line):
yield (word.lower(), 1)
def combiner(self, word, counts):
yield (word, sum(counts))
def reducer(self, word, counts):
yield (word, sum(counts))
if name == 'main':
MRWordFreqCount.run()
Try It Out!
::
# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on Dataproc
python mrjob/examples/mr_word_freq_count.py README.rst -r dataproc > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts
Setting up EMR on Amazon
create an Amazon Web Services account <http://aws.amazon.com/>_
Get your access and secret keys (click "Security Credentials" on
your account page <http://aws.amazon.com/account/>_)
Set the environment variables $AWS_ACCESS_KEY_ID and
$AWS_SECRET_ACCESS_KEY accordingly
Setting up Dataproc on Google
Create a Google Cloud Platform account <http://cloud.google.com/>_, see top-right
Learn about Google Cloud Platform "projects" <https://cloud.google.com/docs/overview/#projects>_
Select or create a Cloud Platform Console project <https://console.cloud.google.com/project>_
Enable billing for your project <https://console.cloud.google.com/billing>_
Go to the API Manager <https://console.cloud.google.com/apis>_ and search for / enable the following APIs...
Google Cloud Storage
Google Cloud Storage JSON API
Google Cloud Dataproc API
Under Credentials, Create Credentials and select Service account key. Then, select New service account, enter a Name and select Key type JSON.
Install the Google Cloud SDK <https://cloud.google.com/sdk/>_
Advanced Configuration
To run in other AWS regions, upload your source tree, run make, and use
other advanced mrjob features, you'll need to set up mrjob.conf. mrjob looks
for its conf file in:
The contents of $MRJOB_CONF
~/.mrjob.conf
/etc/mrjob.conf
See the mrjob.conf documentation <https://mrjob.readthedocs.io/en/latest/guides/configs-basics.html>_ for more
information.
Introduction to Recommendations and MapReduce with mrjob <http://aimotion.blogspot.com/2012/08/introduction-to-recommendations-with.html>_
(source code <https://github.com/marcelcaraciolo/recsys-mapreduce-mrjob>__)
Social Graph Analysis Using Elastic MapReduce and PyPy <http://postneo.com/2011/05/04/social-graph-analysis-using-elastic-mapreduce-and-pypy>_
Thanks to Greg Killion <mailto:[email protected]>_
(ROMEO ECHO_DELTA <http://www.romeoechodelta.net/>_) for the logo.
TopGit's last sync did not record any GitHub topics for Yelp/mrjob. GitHub topics appear in the right sidebar of a repository page; that's the authoritative place to check.
How active is development on Yelp/mrjob?
The most recent commit recorded on Yelp/mrjob was 4 months ago, based on the GitHub push timestamp. The repository has 580 forks — one of the better signals of community interest.
How many stars does Yelp/mrjob have?
Yelp/mrjob has 2.6k GitHub stars — refresh the page for the live number, or check github.com/Yelp/mrjob. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is Yelp/mrjob written in?
Yelp/mrjob is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
Where can I see Yelp/mrjob in action?
The project maintains a homepage at http://packages.python.org/mrjob/. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about Yelp/mrjob?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/Yelp/mrjob is the definitive source.
Read full README in the tab above.
Curious whether mrjob is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about mrjob.
/mrjob