Một mục mã nguồn mở trong kho dữ liệu GitHub của TopGit: allenai/OLMo, 6.7k sao, Python. Modeling, training, eval, and inference code for OLMo
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
⚠️ NOTICE ⚠️ This repository is out of date with our more recent releases and is no longer active. For the latest Olmo releases and updates, please visit: https://github.com/allenai/OLMo-core/
OLMo is a repository for training and using AI2's state-of-the-art open language models. It is designed by scientists, for scientists.
Installation
First, install PyTorch following the instructions specific to your operating system.
For training and fine-tuning, we recommend installing from source:
git clone https://github.com/allenai/OLMo.git
cd OLMo
pip install -e .[all]
You can also install from PyPI with:
pip install ai2-olmo
Pretraining
OLMo pretraining follows a two-stage training procedure.
In the first stage, we train on large amounts of mostly web-based data: OLMo-mix-1124
In the second stage, we train on a smaller amount of high-quality, targeted data: Dolmino-mix-1124
You can find all the checkpoints, at minimum every 1000 training steps in OLMo core and Hugging Face format:
Variant
OLMo Format (Stage 1)
OLMo Format (Stage 2)
Hugging Face Format
OLMo-2 1B
OLMo-2 1B
OLMo-2 1B
Hugging Face for the 1B variant
OLMo-2 7B
OLMo-2 7B
OLMo-2 7B
Hugging Face for the 7B variant
OLMo-2 13B
OLMo-2 13B
OLMo-2 13B
Hugging Face for the 13B variant
OLMo-2 32B
OLMo-2 32B
OLMo-2 32B
Hugging Face for the 32B variant
Note: The 32B variant was trained on our new trainer. To train or fine-tune OLMo-2 32B, visit OLMo-core.
Steps to reproduce
To reproduce any of the training processes described below, run this:
The training configs below refer to training data that gets streamed in live over HTTP.
To reproduce at large scale, we recommend downloading the files locally and changing the paths to point to your
local file system.
Note: You need to upgrade PyTorch to 2.5.x to run.
Stage 1
Stage 1 is the biggest stage, where we train on 4T or 5T tokens on largely web-based data.
OLMo2 1B
OLMo2 7B
OLMo2 13B
Number of tokens
4 Trillion
4 Trillion
5 Trillion
Checkpoint
stage1-step1907359-tokens4001B
stage1-step928646-tokens3896B
stage1-step596057-tokens5001B
Training config
OLMo2-1B-stage1.yaml
OLMo2-7B-stage1.yaml
OLMo2-13B-stage1.yaml
WandB
wandb.ai/OLMo2-1B
wandb.ai/OLMo2-7B
wandb.ai/OLMo2-13B
You can find the .csv.gz files containing the training data here.
Stage 2 for the 1B
For the 1B model, we have trained three times with different data order on 50B high quality tokens, used last checkpoint of seed 42 as final checkpoint.
Checkpoint
Training config
WandB
random seed 42069
stage2-ingredient1-step23852-tokens51B
OLMo2-1B-stage2-seed42069.yaml
wandb.ai/OLMo2-1B
random seed 666
stage2-ingredient2-step23852-tokens51B
OLMo2-1B-stage2-seed666.yaml
wandb.ai/OLMo2-1B
random seed 42 (main)
stage2-ingredient3-step23852-tokens51B
OLMo2-1B-stage2-seed42.yaml
wandb.ai/OLMo2-1B
Stage 2 for the 7B
For the 7B model, we train three times with different data order on 50B high quality tokens, and then average ("soup") the models.
Checkpoint
Training config
WandB
random seed 42
stage2-ingredient1-step11931-tokens50B
OLMo2-7B-stage2-seed42.yaml
wandb.ai/OLMo2-7B
random seed 42069
stage2-ingredient2-step11931-tokens50B
OLMo2-7B-stage2-seed42069.yaml
wandb.ai/OLMo2-7B
random seed 666
stage2-ingredient3-step11931-tokens50B
OLMo2-7B-stage2-seed666.yaml
wandb.ai/OLMo2-7B
final souped model
main
no config, we just averaged the weights in Python
The training configs linked here are set up to download the latest checkpoint after stage 1, and start training from there.
Stage 2 for the 13B
For the 13B model, we train three times with different data order on 100B high quality tokens, and one more time
on 300B high quality tokens. Then we average ("soup") the models.
Checkpoint
Training config
WandB
random seed 1110, 100B
stage2-ingredient1-step11931-tokens100B
OLMo2-13B-stage2-seed1110-100B.yaml
wandb.ai/OLMo2-13B
random seed 2662, 100B
stage2-ingredient2-step11931-tokens100B
OLMo2-13B-stage2-seed2662-100B.yaml
wandb.ai/OLMo2-13B
random seed 6209, 100B
stage2-ingredient3-step11931-tokens100B
OLMo2-13B-stage2-seed6209-100B.yaml
wandb.ai/OLMo2-13B
random seed 2662, 300B
stage2-ingredient4-step11931-tokens300B
OLMo2-13B-stage2-seed2662-300B.yaml
wandb.ai/OLMo2-13B
final souped model
main
no config, we just averaged the weights in Python
The training configs linked here are set up to download the latest checkpoints after stage 1, and start training from there.
Note: You can find all the information about the 32B in the OLMo-core repository.
Instruction tuned variants
For instruction tuned variants of these models, go to
OLMo2 1B Instruct
OLMo2 7B Instruct
OLMo2 13B Instruct
OLMo2 32B Instruct
Inference
You can use our Hugging Face integration to run inference on the OLMo Transformers checkpoints:
from transformers import AutoModelForCausalLM, AutoTokenizer
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-2-0425-1B")
message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
# optional verifying cuda
# inputs = {k: v.to('cuda') for k,v in inputs.items()}
# olmo = olmo.to('cuda')
response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
Alternatively, with the Hugging Face pipeline abstraction:
The quantized model is sensitive to input types and CUDA handling. To avoid potential issues, we recommend explicitly converting input IDs to CUDA using: inputs.input_ids.to('cuda')
Evaluation
Additional tools for evaluating OLMo models are available at the OLMo Eval and olmes repositories.
Modal.com Hosting
An example script is provided for hosting an OLMo 2 model on Modal.com using the OpenAI API in ./scripts/olmo2_modal_openai.py.
To run that:
Follow the instructions under Getting Started in the Modal.com Guide to install
the Modal library and command line tools.
Follow the instructions under Secrets in the Modal.com Guide to create a Modal secret named "example-secret-token"
that defines a value for the variable MODAL_TOKEN for your server.
Then run
modal deploy ./scripts/olmo2_modal_openai.py
You can check your endpoint using curl similar to the following:
curl -X POST \
-H "Authorization: Bearer [the secret token from above]" \
-H "Content-Type: application/json" \
-d @body.json \
https://[the web endpoint modal creates above]/v1/chat/completions
@misc{olmo20242olmo2furious,
title={2 OLMo 2 Furious},
author={Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamish Ivison and Pang Wei Koh and Jiacheng Liu and Saumya Malik and William Merrill and Lester James V. Miranda and Jacob Morrison and Tyler Murray and Crystal Nam and Valentina Pyatkin and Aman Rangapur and Michael Schmitz and Sam Skjonsberg and David Wadden and Christopher Wilhelm and Michael Wilson and Luke Zettlemoyer and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi},
year={2024},
eprint={2501.00656},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.00656},
}
Có — allenai/OLMo phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/allenai/OLMo.
allenai/OLMo có trang demo không?
Dự án có trang chủ ở https://allenai.org/olmo. Tab "Readme" ở trang này thường có ảnh chụp và hướng dẫn bắt đầu nhanh.
allenai/OLMo dùng license gì?
allenai/OLMo phát hành theo license Apache-2.0. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
allenai/OLMo là gì?
allenai/OLMo (allenai/OLMo) là dự án Python trên GitHub. Theo mô tả gốc: Modeling, training, eval, and inference code for OLMo
Đọc thêm về allenai/OLMo ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/allenai/OLMo là nguồn chính thức.
Đọc đầy đủ README ở tab phía trên.
OLMo có đáng để bạn bỏ thời gian?
ChatGPT, Claude và Perplexity đều đọc được trang này. Hỏi thử xem họ nghĩ gì về OLMo.