wandb/wandbot

WandBot

WandBot is a support assistant designed for Weights & Biases' Experiment Tracking and Weave.

What's New

Evaluation

English

Note

• v1.3.1 uses:
  • claude Sonnet-3.7 for the response synthesizer, updated from gpt-4o-2024-11-20
  • an updated index that exludes korean and japanese versions of the docs as well as excludes the blog posts from Fully Connected.
• 1.3.0rc with gpt-4-preview judge and 1.3.0rc with gpt-4o judge are the same wandbot system evaluated with different judges.
• The ~2.5% improvement between 1.3.0rc (gpt-4o judge) and 1.3.0 prod is mostly due to using reranker-v3.5 (from 2.0) and flash-2.0-001 (from gpt-4o). However evals previous to the v1.3.0 prod eval had 10-12 errors (out of 490 total calls), so there might be some noise in the results.

Japanese

Features

• WandBot uses:
  • a hosted ChromaDB vector store
  • OpenAI's v3 embeddings
  • Gemini flash-2.0 for query enhancement
  • GPT-4o for response synthesis
  • Cohere's re-ranking model
• It features periodic data ingestion and report generation, contributing to the bot's continuous improvement. You can view the latest data ingestion report here.
• The bot is integrated with Discord and Slack, facilitating seamless integration with these popular collaboration platforms.
• Performance monitoring and continuous improvement are made possible through logging and analysis with Weights & Biases Weave
• Has a fallback mechanism for model selection

Installation

The project is built with Python version 3.12 and utilizes uv for dependency management. Follow the steps below to install the necessary dependencies:

uv venv
uv sync
source .venv/bin/activate
uv pip install .

Usage

Running WandBot

Before running the wandbot, ensure the following environment variables are set:

OPENAI_API_KEY
COHERE_API_KEY
WANDB_API_KEY
WANDBOT_API_URL="http://localhost:8000"
WANDB_TRACING_ENABLED="true"
LOG_LEVEL=INFO
WANDB_PROJECT="wandbot-dev"
WANDB_ENTITY= <your W&B entity>

If you're running the slack or discord apps you'll also need the following keys/tokens set as env vars:

SLACK_EN_APP_TOKEN
SLACK_EN_BOT_TOKEN
SLACK_EN_SIGNING_SECRET
SLACK_JA_APP_TOKEN
SLACK_JA_BOT_TOKEN
SLACK_JA_SIGNING_SECRET
DISCORD_BOT_TOKEN

Then install the app to install all dependencies in a virtual env:

uv pip install .

Local Development

Start the wandbot locally using the following commands:

# Run API server only with uvicorn
uv run uvicorn wandbot.api.app:app --host 0.0.0.0 --port 8000

# Run Slack and Discord bots locally (requires API to be running)
# Make sure to activate your virtual environment first
source wandbot_venv/bin/activate  # or wherever your venv is located
($VIRTUAL_ENV/bin/python -m wandbot.apps.slack -l en) & \
($VIRTUAL_ENV/bin/python -m wandbot.apps.slack -l ja) & \
($VIRTUAL_ENV/bin/python -m wandbot.apps.discord)

The app will initialize automatically when started.

Modal Deployment

Wandbot can be deployed on Modal for serverless hosting with auto-scaling and low latency. The deployment is split into two separate Modal apps:

• wandbot-api: The FastAPI server
• wandbot-bots: Discord and Slack bots

The modal api url is set in modal_config.py and you can also overwrite this by settting WANDBOT_MODAL_API_URL in .env.

# Set up your environment
uv venv
uv sync

# Deploy everything (API server and bots)
./modal/deploy_all.sh

# Or deploy individually from project root
modal deploy modal/modal_app.py    # Deploy API only
modal deploy modal/modal_bots.py   # Deploy bots only

The deploy_all.sh script will:

1. Deploy the API server
2. Deploy the bot application
3. Start the bots immediately (they run for 24 hours)
4. Set up an hourly cron job to ensure bots stay running

See MODAL_DEPLOYMENT.md for detailed Modal deployment instructions.

For more detailed instructions on installing and running the bot, please refer to the run.sh file located in the root of the repository.

Executing these commands will launch the API, Slackbot, and Discord bot applications, enabling you to interact with the bot and ask questions related to the Weights & Biases documentation.

Running the Evaluation pipeline

Eval Config

The eval config can be found here and includes cli args to set the number of trials, weave parallelism, weave logging details and debug mode : wandbot/src/wandbot/evaluation/eval_config.py

The following evaluation sets are used:

English evaluation dataset

• ref: weave:///wandbot/wandbot-eval/object/wandbot_eval_data:eCQQ0GjM077wi4ykTWYhLPRpuGIaXbMwUGEB7IyHlFU

Japanese evaluation dataset

• ref: weave:///wandbot/wandbot-eval-jp/object/wandbot_eval_data_jp:oCWifIAtEVCkSjushP0bOEc5GnhsMUYXURwQznBeKLA

Dependencies

Ensure wandbot is installed by installing the production depenencies, activate the virtual env that was created and then install the evaluation dependencies

uv venv
uv sync
source .venv/bin/activate
uv pip install .

Environment variables

Make sure to set the environment variables (i.e. LLM provider keys etc) from the .env file.

Launch the wandbot app You can either use uvicorn or gunicorn to launch N workers to be able to serve eval requests in parallel. Note that weave Evaluations also have a limit on the number of parallel calls make, set via the WEAVE_PARALLELISM env variable, which is set further down in the eval.py file using the n_weave_parallelism flag. Launch wandbot with 8 workers for faster evaluation.

uvicorn

uv run uvicorn wandbot.api.app:app \
--host 0.0.0.0 \
--port 8000 \
--workers 8 \
--timeout-keep-alive 75 \
--loop uvloop \
--http httptools

Testing: You can test that the app is running correctly by making a request to the chat/query endpoint, you should receive a response payload back from wandbot after 30 - 90 seconds:

curl -X POST \
   http://localhost:8000/chat/query \
  -H 'Content-Type: application/json' \
  -d '{"question": "How do I log a W&B artifact?"}'

Debugging For debugging purposes during evaluation you can run a single instance of the app by chaning the uvicorn command above to use --workers 1

Run the evaluation

Launch W&B Weave evaluation in the root wandbot directory. Ensure that you're virtual envionment is active. By default, a sample will be evaluated 3 times in order to account for both the stochasticity of wandbot and our LLM judge.

• For debugging, pass the --debug flag to only evaluate on a small number of samples.
• To adjust the number of parallel evaluation calls weave makes use the --n_weave_parallelism flag when calling eval.py
• see eval_config.py for all evaluation options.

source wandbot_venv/bin/activate

uv run src/wandbot/evaluation/eval.py

When running evals before prod, we always run 5 trials per eval sample. It can also be a good idea to reduce weave parallelism in order to avoid api rate limiting issues which might skew eval results:

caffeinate uv run src/wandbot/evaluation/eval.py --experiment_name <insert eval name> --n_trials 5 --n_weave_parallelism 4

Debugging, only running evals on 1 sample and for 1 trial:

uv run src/wandbot/evaluation/eval.py  --debug --n_debug_samples=1 --n_trials=1

Evaluate on Japanese dataset:

uv run src/wandbot/evaluation/eval.py  --lang ja

To only evaluate each sample once:

uv run src/wandbot/evaluation/eval.py  --n_trials 1

Data Ingestion

The data ingestion module pulls code and markdown from Weights & Biases repositories and ingests them into vectorstores for the retrieval augmented generation pipeline.

Ensure that the api keys for the embeddings model provider and the remote vector store provider are set before running the script. See debugging steps below for more tips.

To clone the repos you will need to have a .ssh key at ~/.ssh/id_rsa or generate one if you don't have one already:

ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa

Then run the ingestion pipeline script:

uv run src/wandbot/ingestion/__main__.py

Note:

Pay special attention to the configs in src/wandbot/configs/vector_store_config.py and src/wandbot/configs/ingestion_config as this is where important settings such as the embedding model, embedding dimensions and hosted vs local vector db are set. Ensure your VECTOR_STORE_API_KEY env var is set in .env.

You will notice that the data is ingested into the data/cache directory and stored in three different directories raw_data, vectorstore with individual files for each step of the ingestion process.

These datasets are also stored as wandb artifacts in the project defined in the environment variable WANDB_PROJECT and can be accessed from the wandb dashboard.

Ingestion pipeline debugging

To help with debugging, you can use the steps and include_sources flags to specify only sub-components of the pipeline and only certain documents sources to run. For example if you wanted to stop the pipeline before it creates the vector db and creates the artifacts and W&B report AND you only wanted to process the Weave documentation, you would do the following:

uv run src/wandbot/ingestion/__main__.py --steps prepare preprocess --include_sources "weave_documentation" --debug

Note on updating hosted Chroma vector db

A. If you compute a diff between the old dev docs and the new ones

1. You could use delete() then add(), on the same ids if you have consistent ids across updates
2. You could call update() or upsert() on the same ids, but if you changed any metadata schemas and want to drop old keys, you'll have to explicitly do that.

B. If you don't compute a diff or want a simple way to do this

1. You could delete everything in the collection and add it
2. You could create a new collection and insert the new data into that.

Release Checklist

[] Evaluation: Run evaluations, ensure no performance drop or justify why drop is acceptable/expected [] Evaluation: Prefix eval name in Weave evals with "Prod vX.X.X -" for easy identification (in wandbot/wandbot-eval) [] Knowlege Update: If knowlege base is updated then update the ingestion report with "Prod vX.X.X -" for easy identification (in wandbot/wandbot-dev) [] Deployment Testing: Clone the repo to a staging env (e.g. test Replit app). Install and run via shell, sure no issues, ping local endpoint and ensure no errors. [] Deployment Testing: Then create a staged deployment and again ensure no errors. Ping staged endpoint to ensure correct response is received. [] Deployment: Clone the repo to a prod environment. Deploy updated version. Test via cli and slackbot that the endpoint is working and the correctg response is received. [] [] GitHub: Update evaluation table at top of README with latest eval score, weave Eval link and data ingestion Report link [] GitHub: Update git tag [] GitHub: Create gthub release