IBM/AssetOpsBench
IBM/AssetOpsBench — an open-source project — sits at 2.4k GitHub stars. AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Snapshot
Top contributors
Show top contributors
AssetOpsBench
AI Agents for Industrial Asset Operations & Maintenance
A unified, open framework for building, orchestrating, and evaluating domain-specific AI agents in Industry 4.0.
📄 Paper · 🤗 Dataset · 🎮 Playground · 📢 IBM Blog · 🎥 Video · 📊 Kaggle · 🚀 Colab
[!IMPORTANT] 🎉 AssetOpsBench is officially accepted at KDD 2026 (Datasets & Benchmarks Track), Jeju, South Korea, alongside our hands-on tutorial Building Reliable Industrial Agents with MCP. See Publications for the full list of 2025–2026 work.
At a Glance
| 9 Asset classes | 141+ Scenarios | 5 + 1 Domain agents + utility server | 2 Orchestration frameworks | 20+ University extensions | 500+ Competition submissions |
Built for: maintenance engineers, reliability specialists, facility planners, and Industry 4.0 researchers. Powered by: LLMs + Time Series Foundation Models, orchestrated over live sensor data and Industry 4.0 records (FMEA, work orders, alerts). Now with: simplified interface and native MCP (Model Context Protocol) support.
Quick Start
# Clone and install
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .
# Try a scenario (to be enabled)
python -m assetopsbench.run --scenario "List all sensors of Chiller 6 in MAIN site"
Or jump in instantly:
- 🚀 Run on Colab — no install required (illustration of LLM Agent)
- 🎮 Try the HF Playground — interactive demo
- 📖 Read INSTRUCTIONS.md — full setup, MCP servers, plan-execute runner
[!NOTE] Active development is on
main. The codebase used for various publication venues continues to be maintained on separate branches, for example, ACL 2026IndustryAssetEQAand prior experimental work is maintained onmain-0.x.
What is AssetOpsBench?
AssetOpsBench is a unified framework for developing, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance. It provides reproducible scenarios, agent tooling, and evaluation pipelines for multi-step workflows in simulated industrial environments.
Domain-Specific MCP Servers
| MCP Servers | Important tools |
|---|---|
| IoT | sites, asset_ids, asset_detail, assets, find_assets_by_sensors, installed_sensors, measured_sensors, latest_reading, history, sensor_stats |
| FMSR | get_failure_modes, generate_failure_modes, add_failure_modes |
| TSFM | Tasks/evidence: list_tasks, profile_series, characterize_series, data_quality; model catalog: list_models, search_models, find_models, resolve_model, model_template, register_model, register_finetuned, hf_stats; feature catalog: list_features, search_features, extract_features, select_features; run/eval ledger: recipe_template, run_recipe, run_tabular_recipe, run_plan, evaluate, list_runs, list_results |
| WO | Read: list_workorders, get_workorder, get_workorder_tasks, get_workorder_costs, get_workorder_actuals_vs_planned, get_workorder_kpis, get_schedule_calendar, get_my_assigned_workorders, get_failure_codes; write: generate_work_order, update_workorder, approve_workorder, assign_technician, close_workorder, cancel_workorder |
| Vibration | get_vibration_data, list_vibration_sensors, compute_fft_spectrum, compute_envelope_spectrum, assess_vibration_severity, calculate_bearing_frequencies, diagnose_vibration |
| Shared utility server | |
| Utilities | json_reader, get_sensor_catalog, get_asset_catalog, get_failure_mode_catalog, current_date_time, current_time_english |
Five domain servers (IoT, FMSR, TSFM, WO, Vibration) plus one shared utility server, documented in full — arguments, categories, and backing services — in docs/mcp-servers.md.
The full TSFM MCP surface currently contains 41 tools covering model cards, feature cards, recipe execution, evaluation, and result/run lookup. See docs/mcp-servers.md for the complete reference.
The WO MCP surface contains 15 tools (9 read + 6 write) covering the full Maximo-style work-order lifecycle — query, create, approve, assign, close, cancel — plus costs, KPIs, and scheduling, backed by CouchDB with IBM Maximo mxwo field names. Set AOB_READONLY=1 to expose the 9 read tools only. See docs/mcp-servers.md for the complete reference.
Agent Frameworks
- Plan Execute — plan-and-execute sequential workflow to work with any LLM
- Deep Agent — planning, sub-agents, and virtual filesystem for long-horizon tasks
- Claude Agent — ReAct-based orchestrator using Claude with agent-as-tool delegation
- OpenAI Agent — ReAct-based orchestrator using OpenAI models with agent-as-tool delegation
MCP Environment
The src/ directory contains MCP servers and a plan-execute runner built on the Model Context Protocol. See INSTRUCTIONS.md for setup.
Example Scenarios
| Domain | Example Task |
|---|---|
| IoT | "List all assets in MAIN site" |
| FMSR | "List known failure modes for asset class pump" |
| TSFM | "Find a forecasting model for Chiller 6 and check it can be loaded" |
| WO | "Generate a work order for Chiller 6 anomaly detection" |
Some tasks focus on a single domain, others are multi-step end-to-end workflows. Explore all scenarios on Hugging Face.
Leaderboards
- To be revised (WIP with latest models)
- Evaluated with 7 Large Language Models
- Trajectories scored using LLM Judge (Llama-4-Maverick-17B)
- 6-dimensional criteria measuring reasoning, execution, and data handling
Example: MetaAgent leaderboard
Publications
12+ contributions across 7 top venues in 2025–2026 from the team behind AssetOpsBench.
⭐ KDD 2026 — Jeju, South Korea (click to expand)
- [D&B] AssetOpsBench: A Benchmark for Industrial Asset Operations Agents · D. Patel, S. Lin, et al. · 📄 Paper
- [Tutorial] Building Reliable Industrial Agents with MCP: A Hands-on AssetOpsBench Tutorial for AI-Driven Operations · D. Patel, C. Shyalika, et al.
ACL 2026 - San Diego, USA
- [Industry] IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance · C. Shyalika, D. Patel, A. Sheth · arXiv:2604.23446
ICLR 2026 - Brazil
- [Main] Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring · N. Martinez, F. O'Donncha, W. M. Gifford, N. Zhou, D. C. Patel, R. Vaculin
AAAI 2026 — Singapore
- [Demo] AssetOpsBench-Live: Privacy-Aware Online Evaluation of Multi-Agent Performance in Industrial Operations · D. Patel, N. Zhou, S. Lin, J. T. Rayfield, C. Shyalika, S. R. Yarrabothula · 🎥 Demo
- [Main] SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search · Y. Zhang, G. Ganapavarapu, S. Jayaraman, B. Agrawal, D. Patel, A. Fokoue · 💻 Code
- [Bridge] Knowledge-Guided AI for Industrial Asset Health Monitoring · S. Lin, D. Patel
- [Tutorial] From Inception to Productization: Hands-on Lab for the Lifecycle of Multimodal Agentic AI in Industry 4.0 · C. Shyalika, S. Ahuja, S. Lin, R. Wickramarachchi, D. Patel, A. Sheth · 🌐 Website · 📊 Slides
- [Workshop(AABA4ET)] Agentic Code Generation for Heuristic Rules in Equipment Monitoring · F. Lorenzi, A. Langbridge, F. O'Donncha, J. Rayfield, B. Eck, S. Rosato
IAAI 2026 - Singapore
- [Deployed] Deployed AI Agents for Industrial Asset Management: CodeReAct Framework for Event Analysis and Work Order Automation · N. Zhou, D. Patel, A. Bhattacharyya
- [Emmerging] Diversity Meets Relevancy: Multi-Agent Knowledge Probing for Industry 4.0 Applications · C. Constantinides, D. Patel, S. Kimbleton, N. Garg, M. Paracha
NeurIPS 2025 — San Diego, USA
- [D&B Track] FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes · C. Constantinides, D. Patel, S. Lin, C. Guerrero, S. D. Patil, J. Kalagnanam · 📄 arXiv · 💻 Code
- [Social] Building Reliable Agentic Benchmarks: Insights from AssetOpsBench (invited talk, 2000+ registered) · D. Patel · 📅 Luma
EMNLP 2025 — Suzhou, China
- [Main] ReAct Meets Industrial IoT: Language Agents for Data Access · J. T. Rayfield, S. Lin, N. Zhou, D. C. Patel
- [Main] Generalized Embedding Models for Industry 4.0 Applications · C. Constantinides, S. Lin, D. C. Patel · 📄 arXiv
- [Findings] Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring · S. Lin, D. Patel, C. Constantinides · 📄 ACL Anthology · 💻 Code
Tutorials & Technical Material
📘 Hands-on guides from our team:
- ReActXen IoT Agent (EMNLP 2025)
- FailureSensorIQ (NeurIPS 2025)
- AssetOpsBench Lab (AAAI 2026)
- SPIRAL (AAAI 2026)
- AssetOpsBench Technical Material
AI Competitions
AssetOpsBench powers public AI agent competitions that bring together researchers, students, and practitioners worldwide.
🔴 Live — IJCAI 2026
Industrial Automation Challenge: Benchmarking Physics-Grounded LLMs for Task Reasoning
A new challenge co-located with IJCAI 2026 that pushes LLM agents on physics-grounded industrial reasoning.
- 🌐 Challenge site: ai-industrial-challenge-ijcai
- 📋 IJCAI 2026 competitions: 2026.ijcai.org/competitions
✅ Completed — CODS 2025
AssetOpsBench-Live: AI Agentic Challenge
Launched in September 2025 at CODS 2025, the competition evaluated multi-agent systems on live industrial scenarios.
- 🏆 Competition page: codabench.org/competitions/10206
- 👥 365 participants · 500+ agent submissions
Talks & Events
| Date | Event |
|---|---|
| 2026-08 | KDD 2026 — AssetOpsBench paper + MCP tutorial · Jeju, South Korea |
| 2026-05-10 | NUS Seminar: AssetOpsBench Applications |
| 2025-12 | NeurIPS 2025 Social: Building Reliable Agentic Benchmarks (2000+ registered) |
| 2025-10-03 | 2-Hour Workshop: AI Agents and Their Role in Industry 4.0 Applications · NJIT ACM |
| 2025-09-01 | CODS 2025 Competition Launch — AssetOpsBench-Live |
| 2025-06-01 | AssetOpsBench v1.0 released — 141 industrial scenarios |
University Projects & Extensions
AssetOpsBench is being extended by university research groups exploring new asset classes, evaluation paradigms, and agentic architectures. To list your project, open a PR.
- Calibrated Coordination Reduces Overconfident Errors in Multi-Agent LLM Systems – Confidence-weighted aggregation and abstention framework for reducing hallucinated confidence events in multi-agent industrial troubleshooting and operational decision-making benchmarks. Chand Sahil Mansuri, Sadamori Kojaku, Binghamton University.
- Internalizing MCP Tool Knowledge in Small LLMs via QLoRA Fine-Tuning — HPML project using AssetOpsBench to fine-tune ~4B models to internalize MCP tool knowledge and reduce prompt schema overhead. Ayal Yakobe, Columbia University · repo
- SPIN — Structural LLM Planning via Iterative Navigation for Industrial Tasks. Yusuke Ozaki, University at Albany · paper · repo
- Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents — Automated scenario generation, transformer asset integration, and scenario quality evaluation. Rohith Kanathur, Sagar Chethan Kumar, Columbia University · repo · paper
- AgentOpsBench — High-throughput battery analytics MCP server with DNN prognostics (RUL prediction) and 3.3× latency optimization. Siddharth Gowda, Rushin Bhatt, Aryaman Agrawal, Winston Li, Columbia University · repo
- Skill-Knowledge-Augmented Agents on AssetOpsBench — Confidence-gated skill execution with scoped knowledge plugins for industrial fault diagnosis. Vera Mazeeva, Sanskruti Shejwal, Shrey Arora, Mana Abbaszadeh, Columbia University · repo
- Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines. Krish Veera, Alimurtaza Mustafa Merchant, Sajal Kumar Goyla, Shambhawi Bhure, Columbia University · paper · repo
- Towards Multi-Turn Dialog Systems for Industrial Asset Operations and Maintenance - Improved response quality and reduced redundant tool calls and multi-turn latency. Chengrui Li, Rujing Li, Yitong Bai, Rui Li, Columbia University ·paper· repo
- Skills and Knowledge Plugin MCP Servers for Optimized Industrial O&M Agents - reducing planning overhead and improving retrieval grounding in industrial asset maintenance agents through an MCP Skills Server that exposes reusable multi-step operational workflows and a Knowledge Plugin Server that enables injection of context-specific documentation. Andrew Li, Kirthana Natarajan, Thai On, Trisha Maturi, Yeshitha Bhuvanesh, Columbia University · repo
- Profiling and Optimizing the TSFM MCP Server - Developed a reproducible benchmarking harness, stage-level profiling system, and interchangeable model interface that identified preprocessing and inference bottlenecks, achieving up to 12.8× faster forecasting and 12.2% lower fine-tuning latency while supporting forecasting, fine-tuning, and anomaly detection workflows. Tomas Pasiecznik, Sam Colman, Byeolah Kwon, Sally Go, Columbia University · repo
- Profiling and Optimizing the AssetOpsBench Plan-Execute Pipeline - Provides the first systematic performance characterization of the AssetOpsBench plan-execute pipeline to quantify the latency-accuracy tradeoff of thinking mode on Gemma 4 26B for industrial asset operations tasks. Implemented and evaluated scenario-based routing optimizations to balance the tradeoff. Shen Li, Charles Xu, Ann Li, Caroline Cahill, Columbia University · repo
- Performance Optimzation of the TSFM Agent in an Industrial Agentic Benchmark - Developed an optimization framework for IBM's TinyTimeMixer(TTM) model by implementing model pre-loading, torch.compile graph fusion, and replacing Huggingface abstractions with direct batched model calls. We achieved 3.3X reduction in workflow latency and 68% decrease in total execution time while maintaining zero-shot forecast quality on industrial sensor data. Alisha Vinod, Jonathan Ang, Sanjaii Vijayakumar, Thomas Ajai, Columbia University . repo
- Visual Inspection Agent for AssetOpsBench - Adds a vision modality to AssetOpsBench via an MCP-connected Visual Inspection Agent and 22 hand-authored visual inspection scenarios across pumps, induction motors, power transformers, and wind turbine blades. Benchmarks AWQ W4A16 quantization and vLLM serving optimizations on Qwen2.5-VL-7B and Llama-3-LLaVA-NeXT-8B, with an LLM-as-a-judge scoring pipeline for accuracy evaluation. Amaan Sheikh, Aman Upganlawar, Madhav Rajkondawar, Yang-Jung (Eric) Chen, Columbia University · repo
- Agentic AI Workflows for Naval Operations and Maintenance — Exploring AssetOpsBench for evaluating agentic AI workflows, with future extensions using digital-twin-generated synthetic data. Priyam Dalmia, Chin-Teng Lin, Fred Chang, University of Technology Sydney
- Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operations — Treats a typed knowledge graph as the data layer for AssetOpsBench agents: holding the model and orchestration fixed, the same GPT-4 rises from 65% to 82–83% with LLM-generated Cypher and to 99% with deterministic graph handlers on the 139-scenario snapshot, plus Generation-Augmented Knowledge (GAK) — provenance-tagged enrichment — for the non-deterministic FMSR scenarios. Accepted at Agents+Graph @ VLDB 2026. Madhulatha Mandarapu, Sandeep Kunkunuru (VaidhyaMegha / Samyama) · paper · repo
Call for Scenario Contribution
We are expanding AssetOpsBench to cover a broader range of industrial challenges. We invite researchers and practitioners to contribute new scenarios, particularly in:
- Asset Classes: Turbines, HVAC systems, Pumps, Transformers, CNC Machines, Robotics, Engines
- Task Domains: Prognostics and Health Management, Remaining Useful Life (RUL) estimation, Root Cause Analysis (RCA), Diagnostic Analysis, Predictive Maintenance
How to contribute:
- Define your scenario following our Utterance Guideline and Ground Truth Guideline
- Explore the Hugging Face dataset for examples
- For external public sources and starter asset-class mappings, see External Industrial Dataset Guide
- Submit a Pull Request or open an Issue with the tag
new-scenario - Contact us with questions:
- Dhaval Patel — [email protected]
- Nianjun Zhou — [email protected]
Infrastructure Support
Model API access for AssetOpsBench 2.0 is enabled by TokenRouter (PaleBlueDot AI), a unified API platform providing access to leading AI models through a single API endpoint.
Contributors
Thanks to these wonderful people ✨
 ShuxinLin 💻 |
 DhavalRepo18 💻 |
 ChathurangiShyalika 💻 |
 nianjunz 💻 |
 PUSHPAK-JAISWAL 💻 |
 LGDiMaggio 💻 |
 Ayan Das 💻 |
 jdsheehan 💻 |
 bradleyjeck 💻 |
 jtrayfield 💻 |
 florenzi002 💻 |
 flamehaven01 💻 |
 kushwaha001 💻 |
 Rohith-Kanathur 💻 |
 Mohit Gupta 💻 |
 jellyfishing2346 💻 |
 anncli 💻 |
 ozatamago 💻 |
 tomaspasie 💻 |
 Sans-Shej 💻 |
 sanjaiiv04 💻 |
 SajalGoyla 💻 |
 priyamDalmia 💻 |
 krishrveera 💻 |
 kmn01 💻 |
 Coderlicr 💻 |
 BrianBai093 💻 |
 AyalYakobe 💻 |
 amaan784 💻 |
 Dev-Scodes5 💻 |
 jack-pfeifer 💻 |
 sandeepkunkunuru 💻 |
 srutanik 💻 |
 thedgarg31 💻 |
Star History
If AssetOpsBench is useful to your work, please ⭐ star the repo, 🍴 fork it, and tell us what you're building.
Related repositories
REA is an MCP server and CLI that gives AI coding agents the ability to reverse engineer binaries, JavaScript apps, .NET assemblies, and websites. It hooks into Hopper, Ghidra, or IDA Pro for native binary work, or runs static JavaScript analysis with just Node.js. Every investigation returns an Evidence bundle that includes what was found, what was not, and the reasoning path. Setup registers REA with supported agents and walks you through connecting an analysis engine. The CLI also works standalone for terminal-first workflows.
Master programming by recreating your favorite technologies from scratch.
A curated meta-list of curated lists organized by technology domain. The repository acts as a directory pointing to hundreds of specialized awesome lists covering programming languages, platforms, frameworks, and tooling. All content is community-contributed under the CC0 public domain dedication.
Public APIs is a community-curated GitHub repository listing free, publicly accessible APIs across a wide range of categories, with auth type, HTTPS, and CORS noted for each entry. It's a browsable reference, not a library to install.
Quick answers
Does IBM/AssetOpsBench have a project website?
No homepage URL was recorded for IBM/AssetOpsBench in TopGit's last sync. The README tab above frequently contains screenshots and demo links, or check the repository description on GitHub.
How active is development on IBM/AssetOpsBench?
The most recent commit recorded on IBM/AssetOpsBench was 2 months ago, based on the GitHub push timestamp. The repository has 352 forks — one of the better signals of community interest.
Is IBM/AssetOpsBench open source?
Yes — IBM/AssetOpsBench ships under the Apache-2.0 license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/IBM/AssetOpsBench.
What license does IBM/AssetOpsBench use?
IBM/AssetOpsBench is released under the Apache-2.0 license. Always verify the LICENSE file directly on GitHub for the authoritative terms — license strings can be edited out of sync with a project's actual stance.
What topics is IBM/AssetOpsBench associated with?
GitHub's repository topics for IBM/AssetOpsBench: "ai-for-physical-assets", "condition-based-maintenance", "hvac-maintenance", "iot", "llm-agents", "model-context-protocol", "predictive-maintenance", "time-series-forecasting". TopGit's editorial category is open-source.
Where do I read more about IBM/AssetOpsBench?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/IBM/AssetOpsBench is the definitive source.
Read full README in the tab above.
Still deciding about AssetOpsBench?
One click hands the question to an AI along with this page — see what it says about AssetOpsBench.