llm-wiki-kit

LLM wiki kit

First committed by Ivy-Chen1999

Published in Ivy-Chen1999/llm-wiki-kit. Publisher page

20 GitHub starsUpdated 2026-06-23

What it is

Clone-and-go agent skills + a starter Obsidian vault for an AI-maintained personal wiki. Based on Karpathy's LLM Wiki pattern.

— the description by Ivy-Chen1999

AutoGineer overview, from the brain’s own files

llm-wiki-kit implements the LLM Wiki pattern, which shifts from traditional RAG (searching raw documents) to an agent-driven process that summarizes and structures information into linked Obsidian notes. By processing sources once, the system creates a persistent, queryable knowledge base. The kit includes 20 agent skills for tasks like ingesting URLs, mining conversation history, synthesizing notes, and maintaining wiki health, alongside a pre-configured Obsidian vault template.

How it works

Memory model: LLM wiki

AutoGineer overview, from the brain’s own files

The system organizes data into three layers: raw sources, the wiki (markdown notes with YAML frontmatter), and agent skills. Users interact with an agent that loads these skills to perform actions. The vault uses a flat structure with specific folders for notes, raw staging, journals, and system files. Skills are linked to the agent's environment, allowing the agent to automatically process, link, and maintain the wiki based on natural language commands.

Folders
Folder	Role	Files
skills	agent-instructions	24
personal-skills	folder	4
vault-template	folder	12

The agent writes

skills/llm-wiki/references/karpathy-pattern.md evidence: README.md

Requirements

Apps: Obsidian

Set it up

This brain runs code automatically: 1 script. Review install.sh before opening the folder in Claude Code. No step below turns a hook on for you.
- install.sh: sh install.sh
Download the brain as a ZIP and unzip it where you keep your notes or project: https://mp-staging.autogineer.com/agent-brains/ivy-chen1999/llm-wiki-kit/download
cd llm-wiki-kit && claude
AutoGineer scanned these files at commit f511a35.
Not scanned by AutoGineerThe author’s README also describes installing it from a clone of its repository. Read its install section: AutoGineer did not scan that path. https://github.com/Ivy-Chen1999/llm-wiki-kit/blob/f511a3520e1902a53c2776c209e6fc62d3cfa8da/README.md#setup
First committed by Ivy-Chen1999. Published in Ivy-Chen1999/llm-wiki-kit. Licence: MIT License.

Usage

AutoGineer overview, from the brain’s own files

Ingest articles, PDFs, or web pages into the wiki.
Query the wiki to retrieve information with source links.
Synthesize connections between existing notes.
Audit the wiki for orphans, broken links, or stale content.
Automate the capture of chat history from Claude or CLI sessions.
Maintain a controlled tag taxonomy and visual graph structure.

“Save this link to my wiki”: The agent fetches the URL, processes the content, and creates a structured note in the wiki.
“What do I know about X?”: The agent searches the wiki notes and provides an answer with links to the source material.
“Audit my wiki”: The agent checks for orphans, broken links, and contradictions, reporting on the health of the knowledge base.
“Color my graph by category”: The agent updates the Obsidian graph visualization based on the categories defined in the note frontmatter.

Documentation by Ivy-Chen1999

llm-wiki-kit

A set of agent skills and a starter Obsidian vault for keeping a personal wiki that an LLM helps maintain. Based on Andrej Karpathy's LLM Wiki pattern.

The idea: instead of searching raw documents every time you have a question (RAG), the agent processes a source once and writes it into structured, linked Obsidian notes. Questions are then answered from those notes. You pick the sources and ask the questions; the agent does the summarizing, linking, and filing.

How it's organized

There are three layers:

Raw sources — your original material (articles, PDFs, notes, screenshots). Never modified.
The wiki — markdown notes with YAML frontmatter and [[wikilinks]], kept in the vault.
The skills — the instruction files in skills/ that tell the agent how to maintain the wiki.

What's in this repo

llm-wiki-kit/
├── skills/          20 agent skills
├── vault-template/  an empty vault you can copy and start using
├── .env.example     config (vault path, sources, search)
├── install.sh       links the skills into your agent's skills directory
├── setup-qmd.sh     optional: installs and indexes local search
└── LICENSE

Skills

Skill	What it does
`llm-wiki`	Explains the pattern and retrieval approach. Start here.
`wiki-setup`	Initializes a new vault (structure, system files, config).
`wiki-ingest`	Turns a raw source into wiki notes.
`ingest-url`	Fetches a URL and ingests it.
`data-ingest`	Ingests structured/tabular data.
`claude-history-ingest`	Mines Claude conversation history into the wiki.
`codex-history-ingest`	Mines Codex CLI session history into the wiki.
`wiki-query`	Answers questions from the wiki, with links to source notes.
`wiki-research`	Runs web research and files the results into the wiki.
`wiki-update`	Revises existing notes as things change.
`wiki-synthesize`	Builds notes that connect several existing notes.
`cross-linker`	Finds and adds missing `[[wikilinks]]`.
`tag-taxonomy`	Keeps tags to a controlled list.
`wiki-lint`	Checks for orphans, broken links, contradictions, stale notes.
`wiki-status`	Reports what's new and what's ready to ingest.
`wiki-dashboard`	Generates a vault overview.
`wiki-rebuild`	Rebuilds the index and cache from the notes.
`wiki-export`	Exports the wiki to other formats.
`wiki-capture`	Drops rough notes into the staging area.
`graph-colorize`	Colors the Obsidian graph by category.

Requirements

Obsidian, to read and edit the vault.
An agent that loads skills from a directory — Claudian (the Obsidian + Claude plugin), Claude Code, or similar.
Optional: qmd, a local search engine (BM25 + vector). Without it, the skills use Grep instead, so it's not required. See setup-qmd.sh or run npm install -g @tobilu/qmd.

Setup

Two commands. The installer is interactive and does everything else (links the skills, asks where the vault should live, creates it, writes the config):

git clone https://github.com/Ivy-Chen1999/llm-wiki-kit.git
cd llm-wiki-kit
./install.sh

Then follow the final instructions it prints: open the vault folder in Obsidian, and ask your agent to ingest a source or answer a question.

Notes:

The installer links skills into ~/.claude/skills, which Claude (Claude Code and the Claudian Obsidian plugin) reads automatically. For another agent: ./install.sh --skills-dir <that agent's skills dir>.
It offers to install local search (qmd) at the end. Saying no is fine — the skills fall back to Grep. You can run ./setup-qmd.sh later instead.
Re-running is safe: existing skills, vault, and config are left untouched.

Working with more than one vault

The skills resolve the target vault in this order (highest first):

an OBSIDIAN_VAULT_PATH environment variable (per-session override),
a .env at the current working directory / vault root (vault-scoped),
~/.obsidian-wiki/config (the global default).

So to keep a vault self-contained, drop a .env at its root pointing OBSIDIAN_VAULT_PATH at itself (and give it its own QMD_WIKI_COLLECTION). Work from that vault and the skills target only it — the global default and your other vaults are untouched.

How to use it — just say what you want

You don't memorize skill names. Skills trigger automatically from what you say — Claude matches your message to the right one. Open your vault in Claude and talk normally:

You want to…	Just say something like…
Add an article / file / folder	"add this to my wiki", "ingest these docs"
Add a web page	"save this link to my wiki", paste a URL + "add this"
Ask what you know	"what do I know about X?", "find everything on Y"
Quick answer (no deep read)	"quick answer: …", "just scan for …"
Save the current chat	"save this conversation to my wiki"
Connect related notes	"find missing links", "my wiki feels disconnected"
Find cross-cutting themes	"what concepts keep coming up together?"
Check health / fix issues	"audit my wiki", "what needs fixing?"
See what's been ingested	"what's the status of my wiki?"
Research a topic from the web	"research X and file it", "deep dive on Y"
Tidy up tags	"clean up my tags", "normalize tags"
Color the graph	"color my graph by category"
Start fresh	"archive and rebuild the wiki"

If a phrasing doesn't trigger the right skill, just name it ("use wiki-ingest") — but normal language usually works.

Vault layout

The vault-template/ uses a flat layout: all notes live in notes/, and the category is set in frontmatter (category: concept|entity|reference|insight|synthesis).

my-wiki/
├── notes/        the notes
├── raw/          staging area for unprocessed sources
├── journal/      daily notes
├── Templates/    note and daily-note templates
└── _system/
    ├── index.md  table of contents (maintained by the skills)
    ├── hot.md    short summary of recent activity
    ├── log.md    append-only change log
    └── tags.md   tag list

If you prefer one folder per category, wiki-setup can generate that layout instead.

Make it your own

The 20 skills above are the shared core. The skills that pay off most are usually personal — shaped around how you work. You're encouraged to add your own:

a retro skill that writes a project post-mortem into the vault,
a digest skill that pulls your todos + field news into a dated entry,
your own ingest house-style, or a dispatcher that routes "/wiki do X".

personal-skills/ has a blank template and two worked examples (retro, digest) to copy and adapt. Personal skills live in ~/.claude/skills/, not in this repo — so updating the shared core (git pull) never touches them.

Credits

The pattern comes from Andrej Karpathy's LLM Wiki (the three-layer structure and the idea of processing a source once instead of re-searching it each time). His original write-up is included in skills/llm-wiki/references/karpathy-pattern.md.
Built for Obsidian.
The skills were taken from a personal vault and generalized: hardcoded paths, personal collection names, and non-English text removed.

License

MIT — see LICENSE.

skills/codex-history-ingest/SKILL.md

name: codex-history-ingest description: > Ingest Codex CLI conversation history into the Obsidian wiki. Use this skill when the user wants to mine their past Codex sessions for knowledge, import their ~/.codex folder, extract insights from previous coding sessions, or says things like "process my Codex history", "add my Codex conversations to the wiki", or "what have I discussed in Codex before". Also triggers when the user mentions .codex sessions, rollout files, session_index.jsonl, or Codex transcript logs.

Codex History Ingest — Conversation Mining

You are extracting knowledge from the user's past Codex sessions and distilling it into the Obsidian wiki. Session logs are rich but noisy: focus on durable knowledge, not operational telemetry.

This skill can be invoked directly or via the wiki-history-ingest router (/wiki-history-ingest codex).

Before You Start

Resolve the vault path (precedence, highest first): the OBSIDIAN_VAULT_PATH environment variable if set, else a .env in the current working directory (vault-scoped), else ~/.obsidian-wiki/config (global default). Also read CODEX_HISTORY_PATH (defaults to ~/.codex).
Read .manifest.json at the vault root to check what has already been ingested
Read index.md at the vault root to understand what the wiki already contains

Ingest Modes

Append Mode (default)

Check .manifest.json for each source file. Only process:

Files not in the manifest (new session rollouts, new index files)
Files whose modification time is newer than ingested_at in the manifest

Use this mode for regular syncs.

Full Mode

Process everything regardless of manifest. Use after wiki-rebuild or if the user explicitly asks for a full re-ingest.

Codex Data Layout

Codex stores local artifacts under ~/.codex/.

~/.codex/
├── sessions/                          # Session rollout logs by date
│   └── YYYY/MM/DD/
│       └── rollout-<timestamp>-<id>.jsonl
├── archived_sessions/                 # Archived rollout logs
├── session_index.jsonl                # Lightweight index of thread id/name/updated_at
├── history.jsonl                      # Local transcript history (if persistence enabled)
├── config.toml                        # User config (contains history settings)
└── state_*.sqlite / logs_*.sqlite     # Runtime DBs (usually skip)

Key data sources ranked by value

session_index.jsonl — best inventory source for IDs, titles, and freshness
sessions/**/rollout-*.jsonl — rich structured transcript events
history.jsonl — useful fallback/timeline aid if enabled

Avoid ingesting SQLite internals unless the user explicitly asks.

Step 1: Survey and Compute Delta

Scan CODEX_HISTORY_PATH and compare against .manifest.json:

~/.codex/session_index.jsonl
~/.codex/sessions/**/rollout-*.jsonl
~/.codex/archived_sessions/** (optional; only if user asks for archived history)
~/.codex/history.jsonl (optional fallback)

Classify each file:

New — not in manifest
Modified — in manifest but file is newer than ingested_at
Unchanged — already ingested and unchanged

Report a concise delta summary before deep parsing.

Step 2: Parse Session Index First

session_index.jsonl typically has entries like:

{"id":"...","thread_name":"...","updated_at":"..."}

Use it to:

Build a canonical session inventory
Prioritize recent/high-signal sessions
Map rollout IDs to human-readable thread names

Step 3: Parse Rollout JSONL Safely

Each rollout-*.jsonl line is an event envelope with:

{
  "timestamp": "...",
  "type": "session_meta|turn_context|event_msg|response_item",
  "payload": { ... }
}

Extraction rules

Prioritize user intent and assistant-visible outputs
Favor response_item records with user/assistant message content
Use event_msg selectively for meaningful milestones; ignore pure telemetry
Treat session_meta as metadata (cwd, model, ids), not user knowledge

Skip/noise filters

Token accounting events
Tool plumbing with no semantic content
Raw command output unless it contains reusable decisions/patterns
Repeated plan snapshots unless they add novel decisions

Critical privacy filter

Rollout logs can include injected instructions, tool payloads, and sensitive text. Do not ingest verbatim system/developer prompts or secrets.

Remove API keys, tokens, passwords, credentials
Redact private identifiers unless relevant and approved
Summarize instead of quoting raw transcripts

Step 4: Cluster by Topic

Do not create one wiki page per session.

Group by stable topics across many sessions
Split mixed sessions into separate themes
Merge recurring concepts across dates/projects
Use cwd from metadata to infer project scope

Step 5: Distill into Wiki Pages

Route extracted knowledge using existing wiki conventions:

Project-specific architecture/process -> projects/<name>/...
General concepts -> concepts/
Recurring techniques/debug playbooks -> skills/
Tools/services -> entities/
Cross-session patterns -> synthesis/

For each impacted project, create/update projects/<name>/<name>.md (project name as filename, never _project.md).

Writing rules

Distill knowledge, not chronology
Avoid "on date X we discussed..." unless date context is essential
Add summary: frontmatter on each new/updated page (1-2 sentences, <= 200 chars)
Add provenance markers:
- ^[extracted] when directly grounded in explicit session content
- ^[inferred] when synthesizing patterns across events/sessions
- ^[ambiguous] when sessions conflict
Add/update provenance: frontmatter mix for each changed page

Step 6: Update Manifest, Log, and Index

Update `.manifest.json`

For each processed source file:

ingested_at, size_bytes, modified_at
source_type: codex_rollout | codex_index | codex_history
project: inferred project name (when applicable)
pages_created, pages_updated

Add/update a top-level project/session summary block:

{
  "project-name": {
    "source_path": "~/.codex/sessions/...",
    "last_ingested": "TIMESTAMP",
    "sessions_ingested": 12,
    "sessions_total": 40,
    "index_updated_at": "TIMESTAMP"
  }
}

Update special files

Update index.md and log.md:

- [TIMESTAMP] CODEX_HISTORY_INGEST sessions=N pages_updated=X pages_created=Y mode=append|full

hot.md — Read $OBSIDIAN_VAULT_PATH/hot.md (create from the template in wiki-ingest if missing). Update Recent Activity with a one-line summary — e.g. "Ingested 12 Codex sessions; surfaced recurring patterns in CLI tooling and shell scripting." Keep the last 3 operations. Update updated timestamp.

Privacy and Compliance

Distill and synthesize; avoid raw transcript dumps
Default to redaction for anything that looks sensitive
Ask the user before storing personal/sensitive details
Keep references to other people minimal and purpose-bound

Reference

See references/codex-data-format.md for field-level parsing notes and extraction guidance.

48 files

Advantages

Processes sources once into structured notes rather than performing repeated RAG searches. evidence: README.md
Supports custom personal skills that persist outside the shared core repository. evidence: README.md

Limitations

Requires an agent capable of loading skills from a directory, such as Claude Code or the Claudian plugin. evidence: README.md
Local search functionality requires an optional external dependency (qmd) to avoid falling back to Grep. evidence: README.md

Security

Risk level: Medium risk

This brain runs

Install scripts (1)

install.sh: sh install.sh

Agent instructions that run or fetch something (48)

personal-skills/digest.example/SKILL.md: Run web searches for the user's chosen topics (edit this list to your field).
personal-skills/retro.example/SKILL.md: `git log --oneline -20` (for the development arc)
skills/claude-history-ingest/SKILL.md: ├── audit.jsonl # Audit log — tool calls, file reads, commands run
skills/cross-linker/SKILL.md: **Run after every ingest.** New pages are almost always poorly connected.
skills/graph-colorize/SKILL.md: Tell the user to open it once, then re-run.
skills/graph-colorize/SKILL.md: **New category value appears later**: a category not in the mapping gets no entry and its nodes stay default gray — re-run to pick it up.
skills/graph-colorize/SKILL.md: Re-running is safe: each run creates a new backup, only `colorGroups` is rewritten.
skills/ingest-url/SKILL.md: Read `.manifest.json` to check if this URL was already ingested · uses the network
skills/ingest-url/SKILL.md: **Git remote name** — run `git remote get-url origin 2>/dev/null` from the current working directory.
skills/ingest-url/SKILL.md: **If available:** Use `defuddle <url>` (via Bash) to retrieve a clean, stripped-down markdown version of the page. · uses the network
skills/llm-wiki/SKILL.md: Put a `.env` at a vault's root (or run the skill from inside it) and that vault wins, regardless of the global default.
skills/llm-wiki/SKILL.md: **`~/.obsidian-wiki/config`** — the global default, written by `install.sh`.
skills/wiki-export/SKILL.md: Neo4j Cypher `MERGE` statements — paste into Neo4j Browser or run with `cypher-shell`:
skills/wiki-export/SKILL.md: **Re-running is safe** — all output files are overwritten on each run
skills/wiki-ingest/SKILL.md: If your model doesn't support vision, skip image sources and tell the user which files were skipped so they can re-run with a vision-capable model.
skills/wiki-lint/SKILL.md: Also verify after every lint run: are `_system/index.md` and `_system/hot.md` up to date?
skills/wiki-lint/SKILL.md: Run these checks in order.
skills/wiki-lint/SKILL.md: Run the `cross-linker` skill targeted at the fragmented tag — it will surface and insert the missing links
skills/wiki-lint/SKILL.md: Run the `cross-linker` skill first if affinity scores look stale (e.g., `affinity: {}` on a page with many wikilinks)
skills/wiki-lint/SKILL.md: For each pair, run a quick grep to count pages that link to both:
skills/wiki-lint/SKILL.md: Run `/wiki-synthesize` to automatically discover and fill the top gaps
skills/wiki-lint/SKILL.md: - **#systems** — 7 pages, cohesion=0.06 ⚠️ — run cross-linker on this tag
skills/wiki-lint/SKILL.md: | [[Caching]] × [[Consistency]] | 5 pages | Run `/wiki-synthesize` |
skills/wiki-lint/SKILL.md: | [[Testing]] × [[Observability]] | 3 pages | Run `/wiki-synthesize` |
skills/wiki-rebuild/SKILL.md: They can now run:
skills/wiki-research/SKILL.md: **Source Verification (mandatory before filing any note):** For every claim you intend to write into a wiki page, you must be able to point to a URL you successfully fetched with `WebFetch`. · uses the network
skills/wiki-research/SKILL.md: For each angle, run **2-3 `WebSearch` queries** using varied phrasing
skills/wiki-research/SKILL.md: For the top 2-3 results per angle, use `WebFetch` (or `defuddle <url>` if available — cleaner extraction) to get content · uses the network
skills/wiki-research/SKILL.md: Run **up to 5 targeted searches** specifically addressing these gaps.
skills/wiki-research/SKILL.md: Run one final targeted pass (2-3 searches) to find authoritative resolution
skills/wiki-setup/SKILL.md: Install instructions: see `.env.example` (QMD section).
skills/wiki-setup/SKILL.md: Tell the user about these recommended community plugins (they install manually):
skills/wiki-setup/SKILL.md: Run a quick sanity check:
skills/wiki-setup/SKILL.md: Run `wiki-status` to see what's available to ingest
skills/wiki-setup/SKILL.md: Run `wiki-ingest` to add their first sources
skills/wiki-setup/SKILL.md: Run `claude-history-ingest` to mine their Claude conversations
skills/wiki-setup/SKILL.md: Run `codex-history-ingest` to mine their Codex sessions (if they use Codex)
skills/wiki-setup/SKILL.md: Run `wiki-status` again anytime to check the delta
skills/wiki-status/SKILL.md: **Graph delta since last run.** Compare the current link graph to the snapshot stored in the previous `_insights.md`:
skills/wiki-status/SKILL.md: Flag: pages that were isolated last run but now have incoming links ("newly connected: X, Y")
skills/wiki-status/SKILL.md: Flag: pages that lost incoming links since last run ("link target may have been renamed: A, B")
skills/wiki-status/SKILL.md: At the very end, embed a compact graph snapshot as an HTML comment so the next run can diff against it.
skills/wiki-status/SKILL.md: - **#systems** — 7 pages, cohesion 0.06 ⚠️ run cross-linker on this tag
skills/wiki-status/SKILL.md: ## Graph Delta Since Last Run
skills/wiki-synthesize/SKILL.md: Run this for your top candidate concepts and intersect the result sets.
skills/wiki-synthesize/SKILL.md: This gives the user visibility into what the wiki thinks is worth exploring without forcing every synthesis in one run.
skills/wiki-update/SKILL.md: If `~/.obsidian-wiki/config` doesn't exist, tell the user to run `bash setup.sh` from their obsidian-wiki repo first.
skills/wiki-update/SKILL.md: Use `git log <last_commit>..HEAD --oneline` to see what's new.

Medium risk: runs-instruction-instruction-network (skills/ingest-url/SKILL.md:28)
Medium risk: runs-instruction-instruction-network (skills/ingest-url/SKILL.md:60)
Medium risk: runs-instruction-instruction-network (skills/wiki-research/SKILL.md:22)
Medium risk: runs-instruction-instruction-network (skills/wiki-research/SKILL.md:43)
Low risk: scripts-present (install.sh)

Attribution and licence

First committed by

Ivy-Chen1999

Publisher

Yuxin (Ivy) Chen (Community)

Source

Ivy-Chen1999/llm-wiki-kit · commit f511a35

Licence

MIT License (files served here)

ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-12

Contributors

Ivy-Chen1999

AutoGineer is not affiliated with or endorsed by Ivy-Chen1999.

Download

The ZIP holds the files AutoGineer serves, with ATTRIBUTION.md and the licence. Download llm-wiki-kit.zip

What it is

How it works

Requirements

Set it up

Usage

Documentation by Ivy-Chen1999

llm-wiki-kit

How it's organized

What's in this repo

Skills

Requirements

Setup

Working with more than one vault

How to use it — just say what you want

Vault layout

Make it your own

Credits

License

Codex History Ingest — Conversation Mining

Before You Start

Ingest Modes

Append Mode (default)

Full Mode

Codex Data Layout

Key data sources ranked by value

Step 1: Survey and Compute Delta

Step 2: Parse Session Index First

Step 3: Parse Rollout JSONL Safely

Extraction rules

Skip/noise filters

Critical privacy filter

Step 4: Cluster by Topic

Step 5: Distill into Wiki Pages

Writing rules

Step 6: Update Manifest, Log, and Index

Update .manifest.json

Update special files

Privacy and Compliance

Reference

Advantages

Limitations

Security

This brain runs

Attribution and licence

Download

Update `.manifest.json`