danielmiessler/Substrate
As an AI tool, danielmiessler/Substrate has picked up 932 stars on GitHub (TypeScript). An Open-source Framework for Human Understanding, Meaning, and Progress.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Snapshot
Top contributors
Show top contributors
Substrate
Infrastructure for Human Knowledge & Progress
What Is It • Data • Quick Start • Docs • Contribute • Roadmap
🎯 What Is Substrate?
→ Watch Introduction Video • → Read Blog Post
A Shared Foundation for Human Progress
Think of substrate as the base layer—the common ground where we can all work together to understand problems and drive solutions forward. Instead of fragmented discussions and reinventing the wheel, Substrate gives us one place to:
- Document problems → What's actually broken, with evidence
- Track solutions → What works, what doesn't, with results
- Connect progress → Link people, organizations, and projects actually moving things forward
- Build on each other → Arguments and ideas that build on shared evidence
- Measure outcomes → Did it work? What actually changed?
The Purpose: Accelerate Human Progress
We can't solve problems we don't understand. We can't build on solutions we can't find. Substrate provides:
- 🎯 Shared understanding → One place to understand what's wrong and what works
- 🚀 Faster progress → Build on existing knowledge instead of starting over
- 📊 Evidence-based action → Ground decisions in authoritative data, not opinions
- 🔗 Connected knowledge → See how problems, solutions, people, and data interconnect
- 🌍 Collective intelligence → Human insight + AI analysis working together toward progress
An open-source framework connecting 17+ knowledge components:
graph TB
subgraph "🌍 The Substrate"
A[🧩 Problems]
B[💡 Solutions]
C[📊 Data Sources]
D[🗣️ Arguments]
E[📋 Claims]
F[👥 People]
G[🏢 Organizations]
H[🚀 Projects]
I[📈 Plans]
J[🎯 Values]
K[💭 Ideas]
end
L[👤 Human Contributors] --> A
L --> B
L --> D
M[🤖 AI Analysis] --> A
M --> B
M --> D
A -.connects to.-> B
B -.connects to.-> C
D -.connects to.-> C
E -.connects to.-> C
F -.connects to.-> G
G -.connects to.-> H
H -.connects to.-> B
A --> N[🔍 Shared Understanding]
B --> N
C --> N
D --> N
🏗️ Structured Components
|
📊 Authoritative Data
|
🚀 What's New
[!TIP] October 2025 - Massive Data Expansion!
We just added 8 authoritative data sources measuring human wellbeing beyond traditional economic metrics. Now track financial stress, health crises, social connection, worker agency, and environmental health.
📅 Latest Updates (Click to expand)
🔥 October 27, 2025 - Human Wellbeing Infrastructure
5 NEW US Wellbeing Data Sources:
| Added | Source | What It Measures |
|---|---|---|
| 🆕 | DS-00004 FRED Economic Wellbeing | Debt stress, unemployment, inequality |
| 🆕 | DS-00005 CDC WONDER Mortality | Overdoses, suicides, deaths of despair |
| 🆕 | DS-00006 Census ACS Social | Living alone, commute times, isolation |
| 🆕 | DS-00007 BLS JOLTS Labor | Worker "permission to quit" index |
| 🆕 | DS-00008 EPA Air Quality | PM2.5, ozone, environmental health |
Why This Matters:
"If we measure only GDP and unemployment, we will miss the slow-motion collapse of human thriving happening in plain sight."
These sources capture:
- 💰 Leading indicators - Financial stress predicts recessions
- 🎭 Behavioral truth - Actions reveal what surveys miss
- 🏗️ Structural determinants - Air quality you can't "self-care" out of
- 🚨 Crisis detection - County-level suffering visibility
- 🆓 Worker agency - Quit rate = economic freedom measure
October 25, 2025 - Core Data Updates
- ✅ Pulitzer Prize, GDP, inflation data refreshed
- ✅ Knowledge Worker Salaries validation completed
October 16-18, 2025 - Infrastructure
- 🏗️ Library science methodology (8-dimension evaluation)
- ⚡ TypeScript automation with auto-discovery
- 📊 Central logging and health monitoring
- 📚 Complete documentation suite
📜 Previous Updates (Click to expand)
September 2024 - Community Growth
- ✅ Claims, Arguments, and Values frameworks
- ✅ 6+ community contributors
- ✅ 10+ merged pull requests
- ✅ TELOS integration
August 2024 - Framework Expansion
- ✅ Extended component types (Plans, Ideas, Models)
- ✅ Improved documentation structure
- ✅ Enhanced contribution guidelines
July 2024 - Foundation
- ✅ Single-repo structure with 17+ object types
- ✅ Core framework and documentation
- ✅ Public launch with initial datasets
- ✅ Community contribution framework
→ View Full Changelog
📊 Data & Evidence
[!NOTE] All data sources include complete library science cataloging with 8-dimension evaluation: Authority, Currency, Objectivity, Accuracy, Methodology, Coverage, Reliability, and Provenance.
[!IMPORTANT] We know data can be controversial. That's exactly why we:
- 📊 Collect from multiple sources - Cross-reference data from different authoritative providers
- 🔍 Provide complete transparency - Every source fully documented with provenance, methodology, and limitations
- 📝 Full logging - All data pulls logged with timestamps, source versions, and processing steps
- 🔓 Open source everything - TypeScript update scripts show exactly how data is fetched and transformed
You can verify, audit, and challenge our data. That's the point.
Core Datasets (Data/)
| Dataset | Time Span | Data Points | Source | Status |
|---|---|---|---|---|
| 🇺🇸 US GDP | 1929-2025 | 96 years + 314 quarters | FRED/BEA | ✅ Active |
| 💵 US Inflation | 1947-2025 | 945 months | FRED/BLS | ✅ Active |
| 🦠 Bay Area COVID Wastewater | 2022-2025 | 161 weeks | CDPH | ✅ Active |
| 🏆 Pulitzer Prize Winners | 1918-2024 | 249 winners | Wikidata | ✅ Active |
| 💼 Knowledge Worker Salaries | Global | Multi-region | Research | ✅ Active |
Wellbeing Data Sources (Data-Sources/) 🆕
🌍 Global Health & Development (Click to expand)
| ID | Name | Coverage | Indicators | Update |
|---|---|---|---|---|
| DS-00001 | WHO Global Health Observatory | 194 countries | 2,000+ health indicators | Quarterly |
| DS-00002 | UN SDG Indicators | 193 countries | 231 SDG targets | Biannual |
| DS-00003 | World Bank Open Data | Global | Development metrics | Varies |
🇺🇸 US Human Wellbeing (October 2025)
| Source | Key Indicators | Why It Matters | Update Freq |
|---|---|---|---|
| DS-00004 FRED Economic |
• Debt service ratio • Credit card delinquency • Financial stress index • Underemployment (U-6) • Consumer sentiment • GINI inequality | 💰 Economic Security Foundation Debt >12% = stress, >14% = crisis. Leading indicator of recession. | Weekly to Annual |
| DS-00005 CDC WONDER |
• Drug overdose deaths • Opioid-specific deaths • Suicide rates • All-cause mortality | 🚨 Crisis Detection Deaths of despair reveal community breakdown surveys miss. County-level visibility. | Annual (1-2yr lag) |
| DS-00006 Census ACS |
• Living alone (isolation) • Commute times (time poverty) • No internet (digital divide) • Household demographics | 🤝 Social Connection Structural isolation and time poverty constrain flourishing. Census tract granularity. | Annual |
| DS-00007 BLS JOLTS |
• Quit rate (worker agency) • Job openings • Hire rate • Layoff/discharge rate | 🆓 "Permission to Quit" People only quit when they have options. Measures economic freedom and dignity. | Monthly |
| DS-00008 EPA Air Quality |
• PM2.5 (particulate matter) • Ozone (O3) • SO2, CO, NO2 • AQI exceedances | 🌫️ Environmental Health You can't "self-care" out of breathing toxic air. Structural wellbeing constraint. | Real-time |
Composite Wellbeing Indices
[!TIP] Combine multiple data sources to create powerful wellbeing metrics:
- 💸 Financial Stress Composite - Debt + delinquency + evictions + stress index
- 🚨 Crisis Alert Composite - Overdoses + suicides + long-term unemployment
- 🤝 Community Health Composite - Living alone + commute + digital divide (inverted)
- 🆓 Worker Agency Index - Quit rate + job openings / unemployment
- 🌫️ Environmental Health Index - PM2.5 + ozone (inverted)
→ Wellbeing Implementation Guide | → Data Directory
🤖 Human & AI Collaboration
Substrate provides the pieces. You and your AI create the connections.
👤 Humans Contribute
|
🤖 AI Analyzes
|
Everything is designed for dual consumption:
- ✅ Human-readable - Markdown and CSV anyone can open
- ✅ Machine-parseable - Consistent formats AI can query
- ✅ Fully documented - Complete methodology for every dataset
- ✅ Linked with IDs - Unambiguous entity references
Use Cases
🔍 OSINT & Investigation
- Cross-reference public records, corporate filings, government data
- Link people → organizations → transactions → policies
- Surface connections invisible in isolated databases
- Build evidence chains from claims to verifiable records
- Pattern detection for fraud, corruption, illicit networks
📊 Research & Analysis
- Track claims against authoritative data sources
- Evaluate argument quality based on evidence backing
- Compare solutions across different implementations
- Measure progress toward stated goals with real metrics
- Cross-correlate economic, health, social, environmental data
🌐 Public Accountability
- Verify political claims against documented evidence
- Track campaign promises → policy outcomes → measured results
- Link donations → voting records → policy positions
- Monitor government spending against stated objectives
- Environmental justice analysis (who breathes toxic air?)
🚀 Quick Start
View the Data (No Installation Required)
All datasets are available as CSV and Markdown files you can browse directly:
# Clone the repository
git clone https://github.com/danielmiessler/Substrate.git
cd Substrate
# Explore core datasets
ls Data/*/
# Explore wellbeing data sources
ls Data-Sources/*/
Run the Automation (Optional)
[!WARNING] Requires Bun runtime. Install:
curl -fsSL https://bun.sh/install | bash
# Install dependencies
bun install
# Update a specific dataset
cd Data/US-GDP
bun run update.ts
# Update a wellbeing source (requires API key)
export FRED_API_KEY="your_key_here"
cd Data-Sources/DS-00004—FRED_Economic_Wellbeing
bun run update.ts
# Update all datasets
bun run scripts/update-all.ts
Get API Keys (Free)
| Data Source | Get Key | Rate Limit |
|---|---|---|
| FRED Economic | fred.stlouisfed.org/docs/api | 120 req/min |
| Census ACS | api.census.gov/data/key_signup | 500 req/day |
| EPA Air Quality | Email: [email protected] | 10 req/min |
| BLS JOLTS | bls.gov/developers/home | 500 req/day |
| CDC WONDER | No key required | Fair use |
→ Complete Getting Started Guide | → Quick Reference
💡 Contribute
[!IMPORTANT] Anyone can submit components. No gatekeeping on ideas—just structured formats.
What You Can Add
|
📋 Problems
💡 Solutions
📈 Plans
|
🗣️ Arguments
📊 Data
💭 Ideas
|
👥 People & Orgs
📏 Metrics
🎯 Values
|
How to Submit
- Fork the repository on GitHub
- Add your component in the appropriate directory (
Problems/,Solutions/, etc.) - Follow the format in that directory's README
- Submit a Pull Request
[!NOTE] We're building a web interface to make this easier for non-technical contributors!
→ Contribution Guidelines
📚 Documentation
Getting Started
|
Technical
|
🗺️ Roadmap
✅ Completed
Phase 1: Foundation (July 2024)
- ✅ Single-repo structure with 17+ object types
- ✅ Core framework and documentation
- ✅ Public launch with initial datasets
- ✅ Community contribution framework
Phase 2: Community (Aug-Sep 2024)
- ✅ Claims, Arguments, and Values frameworks
- ✅ 6+ community contributors
- ✅ 10+ merged pull requests
- ✅ TELOS integration
Phase 3: Data Infrastructure (Oct 2025) 🔥
- ✅ 13 authoritative data sources (5 core + 8 wellbeing)
- ✅ Library science methodology (8-dimension evaluation)
- ✅ TypeScript automation system with Bun runtime
- ✅ 6,000+ lines of documentation across all sources
- ✅ Comprehensive wellbeing indicators (economic, health, social, labor, environmental)
- ✅ Free API access with rate limiting and retry logic
🚧 Planned
Phase 4: Enhanced Access (Q4 2025 - Q1 2026)
- 🎨 Web-based contribution interface (no coding required)
- 📊 Interactive data visualizations
- 🔌 RESTful API for programmatic access
- 📱 Mobile-friendly exploration
Phase 5: Dataset Expansion (2026)
- 🌍 Additional international sources (UNICEF, OECD, IHME)
- ⚡ Real-time data feeds integration
- 🗳️ Community-driven dataset requests
- 🤝 Partnerships with research institutions
Phase 6: Intelligence Layer (2026+)
- 🤖 Automated relationship discovery
- 📈 Confidence scoring for AI-suggested links
- 🎯 Pattern detection algorithms
- 🔔 Email/Slack notifications for data updates
- 📚 Machine-readable catalog (DCAT/CKAN compliance)
🔗 Integration with TELOS
Substrate provides evidence. TELOS provides intention.
| TELOS (Goals & Strategy) | Substrate (Evidence & Solutions) |
|---|---|
|
|
Together: Complete system for intention + evidence-based action.
🙏 Credits
Created By
Daniel Miessler • July 2024
Special Thanks
Inspiration & Contributions:
- Jonathan Dunn @xssdoctor - Similar vision and collaboration
- Joel Parish - Structure wisdom and guidance
- Joseph Thacker - Continuous flow of innovative ideas
Community Contributors:
@ThatNateGuy • @JaymanW • @karai114 • @DesertEaglePWN • @ktfth
📄 License
MIT License - see the LICENSE file for details.
📊 Repository Stats
Data: 13 sources • 1,700+ data points • 107-year span (1918-2025)
Code: TypeScript automation • 6,000+ lines documentation • GitHub Actions
Community: 6+ contributors • 10+ PRs merged • 17 object types
⬆ Back to Top
Built with ❤️ for human understanding and progress
Powered by TypeScript • Bun • Open Data
Star History
Related repositories
Superpowers is an open-source (MIT) skills library and bootstrap instruction set that turns a coding agent's ad-hoc habits into a fixed pipeline: brainstorm a spec, write a plan, build under TDD, review, then close out the branch. Skills trigger automatically once installed, and the same methodology works across eleven different agent harnesses, each requiring its own install step.
Firecrawl is an API and set of SDKs for turning any URL into Markdown, structured JSON, or a screenshot, handling JS rendering and proxy rotation itself. It's built for feeding AI agents and LLM pipelines clean web data, with dedicated search, crawl, map, and agent endpoints, an MCP server, and SDKs for nine languages.
AutoGPT is an open-source platform designed for building, deploying, and running AI agents that can carry out complete workflows. Users can define tasks in plain English or use a visual builder to shape each step. The project offers two primary paths: a managed, hosted AutoGPT Platform that handles infrastructure and model access for a fee, and a self-hosting option that is free but requires users to provide their own infrastructure and model API keys. Agents can run on demand, on schedules, or from triggers, connecting to over 45 platforms and hundreds of AI models. It's presented as a tool to automate various functions, from executive operations and sales research to marketing campaign drafts and incident triage in engineering.
prompts.chat is the largest open-source prompt library for AI, formerly called Awesome ChatGPT Prompts. It hosts curated prompts in CSV and Markdown, available as a public website, Hugging Face dataset, or self-hosted instance. The project supports multiple LLM providers including ChatGPT, Claude, Gemini, Llama, and Mistral. Self-hosting uses a Next.js setup wizard that configures authentication via GitHub, Google, or Azure AD, with PostgreSQL as the recommended database. CLI access, an MCP server, and a Claude Code plugin extend its reach into developer workflows. The codebase is MIT-licensed while prompt data falls under CC0. Its 166k GitHub stars make it an AI resource on the platform with 166k GitHub stars, and it has been cited by Harvard, Columbia, and Forbes.
Quick answers
How active is development on danielmiessler/Substrate?
The most recent commit recorded on danielmiessler/Substrate was 5 days ago, based on the GitHub push timestamp. The repository has 129 forks — one of the better signals of community interest.
How many stars does danielmiessler/Substrate have?
danielmiessler/Substrate has 932 GitHub stars — refresh the page for the live number, or check github.com/danielmiessler/Substrate. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is danielmiessler/Substrate written in?
danielmiessler/Substrate is written primarily in TypeScript. GitHub's language field is based on the largest share of bytes in the default branch.
What license does danielmiessler/Substrate use?
danielmiessler/Substrate is released under the MIT license. Always verify the LICENSE file directly on GitHub for the authoritative terms — license strings can be edited out of sync with a project's actual stance.
What topics is danielmiessler/Substrate associated with?
GitHub's repository topics for danielmiessler/Substrate: "ai", "humans", "meaning", "purpose". TopGit's editorial category is AI Tools.
Where can I see danielmiessler/Substrate in action?
The project maintains a homepage at https://www.youtube.com/watch?v=ky7ejowc_qY. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about danielmiessler/Substrate?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/danielmiessler/Substrate is the definitive source.
Read full README in the tab above.
Is Substrate worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of Substrate.