dmlc/gluon-nlp

GluonNLP: Your Choice of Deep Learning for NLP

GluonNLP is a toolkit that helps you solve NLP problems. It provides easy-to-use tools that helps you load the text data, process the text data, and train models.

See our documents at https://nlp.gluon.ai/master/index.html.

Features

• Easy-to-use Text Processing Tools and Modular APIs
• Pretrained Model Zoo
• Write Models with Numpy-like API
• Fast Inference via Apache TVM (incubating) (Experimental)
• AWS Integration via SageMaker

Installation

First of all, install the MXNet 2 release such as MXNet 2 Alpha. You may use the following commands:

# Install the version with CUDA 10.2
python3 -m pip install -U --pre "mxnet-cu102>=2.0.0a"

# Install the version with CUDA 11
python3 -m pip install -U --pre "mxnet-cu110>=2.0.0a"

# Install the cpu-only version
python3 -m pip install -U --pre "mxnet>=2.0.0a"

To install GluonNLP, use

python3 -m pip install -U -e .

# Also, you may install all the extra requirements via
python3 -m pip install -U -e ."[extras]"

If you find that you do not have the permission, you can also install to the user folder:

python3 -m pip install -U -e . --user

For Windows users, we recommend to use the Windows Subsystem for Linux.

Access the Command-line Toolkits

To facilitate both the engineers and researchers, we provide command-line-toolkits for downloading and processing the NLP datasets. For more details, you may refer to GluonNLP Datasets and GluonNLP Data Processing Tools.

# CLI for downloading / preparing the dataset
nlp_data help

# CLI for accessing some common data processing scripts
nlp_process help

# Also, you can use `python -m` to access the toolkits
python3 -m gluonnlp.cli.data help
python3 -m gluonnlp.cli.process help

Run Unittests

You may go to tests to see how to run the unittests.

Use Docker

You can use Docker to launch a JupyterLab development environment with GluonNLP installed.

# GPU Instance
docker pull gluonai/gluon-nlp:gpu-latest
docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 --shm-size=2g gluonai/gluon-nlp:gpu-latest

# CPU Instance
docker pull gluonai/gluon-nlp:cpu-latest
docker run --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 --shm-size=2g gluonai/gluon-nlp:cpu-latest

For more details, you can refer to the guidance in tools/docker.