Điểm qua allenai/RL4LMs: 2.4k sao trên GitHub, viết chủ yếu bằng Python, thuộc nhóm AI Tools. A modular RL library to fine-tune language models to human preferences
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
A modular RL library to fine-tune language models to human preferences
We provide easily customizable building blocks for training language models including implementations of on-policy algorithms, reward functions, metrics, datasets and LM based actor-critic policies
Paper Link: https://arxiv.org/abs/2210.01241
Website Link: https://rl4lms.apps.allenai.org/
Thoroughly tested and benchmarked with over 2000 experiments :fire: (GRUE benchmark :trophy:) on a comprehensive set of:
7 different Natural Language Processing (NLP) Tasks:
Summarization
Generative Commonsense Reasoning
IMDB Sentiment-based Text Continuation
Table-to-text generation
Abstractive Question Answering
Machine Translation
Dialogue Generation
Different types of NLG metrics (20+) which can be used as reward functions:
All of these building blocks can be customizable allowing users to train transformer-based LMs to optimize any arbitrary reward function on any dataset of their choice.
Recent updates (v0.2.0) on 23-Nov-22
Added daily dialog task
Fixed compatibility issues with some Seq2seq models such as BART, blendorbot etc
Implemented data parallel support
Refactored policy classes
Recent updates (v0.2.1)
Minor logging updates
Install
Local Installation
git clone https://github.com/allenai/RL4LMs.git
cd RL4LMs
pip install -e .
Docker
We provide also a Dockerfile for development using docker containers containing all the dependencies.
docker build . -t rl4lms
Additional dependencies
Optionally, coreNLP libraries are required for certain metric computations (eg. SPICE) which can be downloaded through cd rl4lms/envs/text_generation/caption_metrics/spice && bash get_stanford_models.sh
Quick Start - Train PPO/NLPO using pre-defined YAML configs
We provide a simple training API that can be invoked via train script that allows to train PPO, NLPO or a supervised model by using a config file (YAML).
For example, to train T5-base on CNN/DM summarization on PPO using Rouge-1 as reward function, you can run:
Config file contains details about hyper-parameter settings for building blocks which are described below:
Dataset/Task: Dataset containing samples with input prompts and reference sentences. Available datasets are found in the class DataPoolRegistry in registry. (See how to create your own dataset here)
Tokenizer - A pre-trained tokenizer that is used to (de)tokenize input and output sequences with settings for padding and truncation
tokenizer:
model_name: t5-base
padding_side: left
truncation_side: left
pad_token_as_eos_token: False
Reward Function: Reward function which computes token-level scores at each time step of MDP. Available reward functions can be found in the class RewardFunctionRegistry. (See how to create your own reward function here)
reward_fn:
id: rouge
args:
rouge_type: "rouge1"
Environment: Configures a gym-style text generation environment which simulates MDP episodes. Rollouts are generated using train samples from dataset consisting of input and reference texts.
Further, we wrap our env with SubProcVecEnv from stable-baselines that processes n_envs episodes in parallel using multi-processing to compute step-wise rewards.
Further configuration settings include:
max_episode_length : max length of the episode
max_prompt_length - maximum length of the input text to consider
terminate_on_eos - whether to terminate the episode as soon as EOS action is performed
prompt_truncation_side - truncation side for the prompt text
context_start_token - id for context token (corresponds to initial token given to decoder in encoder-decoder models)
On-policy alg: We provide implementations of 4 on-policy algorithms: PPO, NLPO, A2C and TRPO adapted from stable-baselines3 tailored to work with NLP tasks which can be used out-of-the-box with either a causal policy or a seq2seq LM policy. (See how to create your own on-policy algorithm or policy)
We also provide a supervised trainer for benchmarking purposes. Supervised Warm start models are already uploaded to Huggingface Hub and specified in the respective config files.
Hyper-parameters for the algorithm can be specified at alg/args.
Further, all RL algorithms use adaptive KL controller to keep the LM close to original LM by setting initial KL co-efficient (alg/kl_div/coeff) and target KL (alg/kl_div/target_kl).
We support two types of LM policy: causal LM policy (for decoder only models) and seq2seq LM policy (for encoder-decoder models). Further for NLPO, we also provide maskable variants of these. Policy implementations can be found here in and it can be attached to algorithms by specifying alg/policy/id and alg/policy/args
Trainer Config: We provide an On-policy trainer - a feature-complete wrapper that instantiates building blocks from their corresponding configs and provides an outer training loop consisting of train and eval iterations train_evaluation/n_iters.
Each iteration corresponds to performing updates with alg/args/n_steps x env/n_envs of the chosen algorithm.
For every eval_every iters, LM is evaluated on validation split using metrics listed in train_evaluation/metrics with generation kwargs provided in train_evaluation/generation_kwargs (this overrides rollout alg/policy/generation_kwargs for inference purposes only)
RL4LMs provide complete customizability - with respect to adding new tasks/datasets, reward functions, evaluation metric, on-policy algorithms and actor-critic policies.
Adding dataset
Users can create their own datasets by sub-classing TextGenPool just by overriding prepare(cls, split: str, **args) -> 'TextGenPool': method to return an instance of TextGenPool. An example is shown below:
from rl4lms.data_pools.text_generation_pool import Sample, TextGenPool
class MyDataPool(TextGenPool):
@classmethod
def prepare(cls, split: str):
..
samples = []
for ix, item in enumerate(..):
sample = Sample(id=f"{split}_{ix}",
prompt_or_input_text=item["document"],
references=[item["target"]]
)
samples.append(sample)
pool_instance = cls(samples)
return pool_instance
Adding reward function
Custom reward funtions can be implemented easily by sub-classing RewardFunction (a callable) which takes observation ($s$), next observation ($s'$), action ($a$), done (indicating whether episode is finished) and meta info (containing other information about textual input). Here, Observation is a data class object consisting of generated text (at a particular step), prompt text, context text (at that step), reference text which can be used to compute token-level or sentence level rewards.
:bulb:
In addition to traditional NLG metrics, for quick prototyping, we provide two synthetic reward functions which trains LMs to generate numbers in increasing order and generate dates. These can be used to quickly test different algorithms and policies. Corresponding configs can be found here (numbers, dates)
Adding custom metrics
Users can create their own evaluation metric which then will be used to periodically evaluate the model on validation split of dataset. This can be done by sub-classing BaseMetric which takes prompt texts, generated texts, reference texts, meta_infos, current LM model, split name as inputs and returns a dict with metric name as key and value consisting of tuple of sentence-level scores and corpus level scores. An example is as follows:
In addition to supported on-policy algorithms (PPO, NLPO, A2C,TRPO), users can implement their own on-policy algorithms with ease by sub-classing stable-baselines3's OnPolicyAlgorithm. Since we provide wrappers for on-policy algorithms that handles rollouts using LM policies, environment, computing rewards etc, users just need to implement train() method with custom loss functions.
from stable_baselines3.common.on_policy_algorithm import OnPolicyAlgorithm
class MyOnPolicyAlgorithm(OnPolicyAlgorithm):
def __init__(**args):
super().__init__(**args)
def train(self) -> None:
# train for n_epochs epochs
for epoch in range(self.n_epochs):
# Do a complete pass on the rollout buffer
for rollout_data in self.rollout_buffer.get(self.batch_size):
# compute loss
Adding custom policies
We provide LM based actor-critic policy implementations that wraps causal LM and seq2seq LMs. These can be also extended (for eg: use a different critic architecture) by overriding appropriate methods (eg. evaluate_actions())
Registry
Finally, just register your custom components by adding them to corresponding registry, after which they can be used directly from configs similar to pre-defined components :wave:
Crowdsourcing templates
We have provided the crowdsourcing templates we used on mechanical turk, along with example inputs in scripts/crowdworking_templates. You might find these a helpful starting point either for evaluating your own model's generations, or for gathering training data for a learned reward function.
Logging and Experiment Results
Additionally, we support WANDB logging and warm-starting of training by storing checkpoints and other training artifacts in a user-specified path. This is especially useful for running preemptible jobs on large, scheduled clusters.
Artifacts include (1) jsonl file containing rollout infos at specified intervals (2) jsonl file containing training infos at specified intervals (3) jsonl file containing validation metrics at specified intervals (4) jsonl file containing test metrics before and after training (5) json file with validation predictions at specified intervals (6) json file with test predictions before and after training (7) trained LM model (8) config json used to run the experiment
@inproceedings{Ramamurthy2022IsRL,
title={Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization},
author={Rajkumar Ramamurthy and Prithviraj Ammanabrolu and Kiant{\'e} Brantley and Jack Hessel and Rafet Sifa and Christian Bauckhage and Hannaneh Hajishirzi and Yejin Choi},
journal={arXiv preprint arXiv:2210.01241},
url={https://arxiv.org/abs/2210.01241},
year={2022}
}
Questions/Discussion/Ideas?
For discussion, questions, ideas exchange, join our slack channel
allenai/RL4LMs có 2.4k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/allenai/RL4LMs. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
allenai/RL4LMs có phải mã nguồn mở không?
Có — allenai/RL4LMs phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/allenai/RL4LMs.
allenai/RL4LMs còn đang phát triển không?
Commit gần nhất trên allenai/RL4LMs là 2.7 năm trước (theo timestamp GitHub). Repo có 200 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
allenai/RL4LMs dùng license gì?
allenai/RL4LMs phát hành theo license Apache-2.0. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
allenai/RL4LMs là gì?
allenai/RL4LMs (allenai/RL4LMs) là dự án Python trên GitHub. Theo mô tả gốc: A modular RL library to fine-tune language models to human preferences
allenai/RL4LMs viết bằng ngôn ngữ gì?
allenai/RL4LMs chủ yếu viết bằng Python. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Cùng nhóm AI Tools còn repo nào?
allenai/RL4LMs thuộc nhóm AI Tools trên TopGit, cùng 9 topic GitHub. Trang Trending và Topics liệt kê các repo cùng số sao và cùng ngôn ngữ để so sánh.
Đọc thêm về allenai/RL4LMs ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/allenai/RL4LMs là nguồn chính thức.
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về RL4LMs?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về RL4LMs.