Your notes are personal. Here's how to make them searchable with AI without sending a single one to the cloud.

What I've built and why local

I have about 770 notes in my Obsidian vault. Articles, quotes, project plans, book summaries, my Zettelkasten, half-finished ideas, decisions I argued with myself about at 2am. They're messy. They're interlinked with wikilinks that only make sense to me. They're mine.

I wanted to ask questions across them. Not "find the note called X" (Obsidian search handles that fine). I mean questions like "What have I written about fear of shipping?" or "Which ideas from the last three months connect to this project I'm starting?" The kind of question that needs something to read, retrieve, and synthesise across hundreds of notes.

Cloud RAG services exist for this. Upload your documents, get an API key, ask questions. But these are my journals ... my unfiltered thoughts. My career plans and the arguments I had with myself about them. I'm not uploading them to anyone's server.

So I built a local RAG pipeline instead.

RAG (retrieval-augmented generation) means you give a language model a way to search your documents before it answers. Instead of making things up, it retrieves relevant chunks of your actual notes and uses them as context. The "local" part means everything runs on my machine: embeddings generated through Ollama, a local vector store for search, and a local language model for the answers. Nothing leaves my laptop.

This tutorial walks you through building the same thing over your own vault. I'll cover what worked, what didn't, and the specific problems Obsidian-style notes create. Wikilinks, for instance, are a whole thing. I'll also show you the evaluation numbers, including the failures. I tested two different embedding models against a hand-built question set and adopted the winner based on real retrieval metrics. That's the part most tutorials skip, and it's the part that actually teaches you something.

I'm documenting this for myself and to share with others that expressed interest in running an Obsidian powered second brain with a local LLM.

Here's what you'll need to follow along:

  • An Obsidian vault (or any folder of Markdown files; the wikilink handling is Obsidian-specific, but the rest works on plain Markdown)
  • Python 3.11+
  • Ollama installed and running locally
  • Patience with messy data. Your notes, like mine, are not clean. That's the point.

Architecture overview

The pipeline has six stages. Everything after "parse" runs through Ollama on your machine.

Vault (.md files)
  │
  ▼
Parse ── resolve wikilinks, extract frontmatter and tags
  │
  ▼
Chunk ── split by heading (240 tokens max) or whole-note
  │
  ▼
Embed ── mxbai-embed-large via Ollama (local)
  │
  ▼
Store ── ChromaDB (local, cosine similarity)
  │
  ▼
Query ── embed the question, retrieve top-6 chunks (min similarity 0.25)
  │
  ▼
Generate ── qwen2.5:14b via Ollama, grounded in retrieved context
  │
  ▼
Answer + source note citations

No API keys. No cloud vector database. No third-party embeddings service. The only external dependency is Ollama, which runs the models locally. The vector store is a ChromaDB collection persisted to disk in .vask/chroma/.

Here's the quick setup if you want to follow along:

git clone https://github.com/reginahack/vask.git
cd vask
uv sync

ollama pull mxbai-embed-large
ollama pull qwen2.5:14b

uv run vask index --vault ~/path/to/your/obsidian-vault
uv run vask query "How is this vault organised under PARA?"

That last command indexes your vault, embeds every chunk, stores them locally, and you're ready to query. The rest of this tutorial is about what happens inside each stage and why the choices matter.

The first thing that broke when I pointed a Markdown parser at my vault was wikilinks.

Standard Markdown links look like [text](url). Obsidian uses [[Note Name]], [[Note Name|display text]], [[Note Name#Heading]], and ![[Embedded Note]]. These are internal references that make sense inside the Obsidian graph. To an embedding model, they're noise.

I had three options:

  1. Keep the raw **[[...]]** syntax. Fast, but double-bracket notation and internal file paths leak vault structure into every embedding.
  2. Recursively inline embedded notes. Resolve ![[Embedded Note]] by pasting the embedded note's full text into the parent. Sounds thorough, but it explodes chunk sizes and duplicates content across the corpus.
  3. Resolve links to readable display text. Replace [[Note Name]] with Note Name and [[target|alias]] with alias. Keep structured metadata about each link separately.

I went with option 3. Here's the core logic from parser.py:

WIKILINK_PATTERN = re.compile(r"(!)?\[\[([^\]]+)\]\]")

def _replace_wikilinks(
    body: str,
    note_index: NoteIndex,
) -> Tuple[str, List[ResolvedLink], List[ResolvedLink]]:
    links: List[ResolvedLink] = []
    embeds: List[ResolvedLink] = []

    def replace(match: re.Match[str]) -> str:
        is_embed = bool(match.group(1))
        raw_target = match.group(2).strip()
        target, heading, alias = _split_link_parts(raw_target)
        resolved_path = note_index.resolve(target)
        note_name = _link_note_name(target, resolved_path)

        if alias:
            display_text = alias
        elif note_name and heading:
            display_text = "{0} — {1}".format(note_name, heading)
        else:
            display_text = note_name or heading or raw_target

        # ... store ResolvedLink metadata for downstream use ...
        return display_text

    normalized = WIKILINK_PATTERN.sub(replace, body)
    return normalized, links, embeds

The regex (!)?\[\[([^\]]+)\]\] handles both regular links and embeds (the optional ! at the start distinguishes ![[embed]] from [[link]]). The NoteIndex class resolves targets against the actual vault file tree, so [[weekly-review]] resolves to its full path rather than a guess.

Here's what a note looks like before and after parsing:

Before: I use the ![[Daily Reset Checklist]] for this, following the [[weekly-review|weekly review]] habit.

After: I use the Daily Reset Checklist for this, following the weekly review habit.

The embedding sees clean prose. The metadata still tracks every link target, heading reference, alias, and resolved file path, which is useful if you later want to build graph-aware retrieval on top.

Frontmatter gets split off and stored as structured data. Tags are normalised into a flat tuple. The body text that flows into chunking is clean, readable, and free of vault-specific syntax.

This matters because embedding models have no concept of [[ brackets. Feeding raw wikilink syntax into the embedding turns every link target into noise in the vector space. Resolving to display text means the embedding captures what the link means to a reader, not what it looks like in a raw file.

Chunking: strategies tested and the numbers

The obvious first approach is to embed each note as a single chunk. My vault has 770 notes. Most of them are atomic (one idea, one file, a few hundred words). Whole-note chunks should be a strong baseline. For short notes, they are.

They fall apart for long notes.

A project plan with six headings produces one embedding that averages across all six sections. Ask a specific question about section three, and the embedding barely registers it. The signal from that section drowns in the noise of the other five.

So I built a second strategy: split at headings with a configurable token budget (240 tokens by default). Each chunk keeps its heading path as context. Here's the section splitter from chunking.py:

HEADING_PATTERN = re.compile(r"^(#{1,6})\s+(.*\S)\s*$")

def split_sections(text: str) -> List[Section]:
    sections: List[Section] = []
    current_heading: Tuple[str, ...] = ()
    buffer: List[str] = []
    heading_stack: List[str] = []
    level_stack: List[int] = []

    for line in text.splitlines():
        match = HEADING_PATTERN.match(line)
        if match:
            _flush_section(sections, current_heading, buffer)
            level = len(match.group(1))
            title = match.group(2).strip()

            while level_stack and level_stack[-1] >= level:
                level_stack.pop()
                heading_stack.pop()

            level_stack.append(level)
            heading_stack.append(title)
            current_heading = tuple(heading_stack)
            buffer = []
            continue

        buffer.append(line)

    _flush_section(sections, current_heading, buffer)
    return sections

The heading stack tracks nesting, so a chunk under ## Roadmap > ### Full-time conditions carries its full heading path. That path is prepended to the chunk text before embedding:

def _render_chunk_text(heading_path: Tuple[str, ...], body: str) -> str:
    if not heading_path:
        return body.strip()
    heading_line = " > ".join(heading_path)
    return "{0}\n\n{1}".format(heading_line, body.strip())

This means the embedding for a chunk about full-time conditions captures both the content and where it sits in the document structure. The retrieval model knows it's a subsection of a roadmap, not a standalone note about career transitions.

One gotcha: dense chunks

Indexing the real vault broke on a handful of notes. Word-based token estimates said the chunks were within budget, but the raw character count was huge (dense tables, long URLs, code blocks). Ollama's embedding context window overflowed.

The fix was a second pass that splits any chunk over 3,000 characters into smaller records before embedding. The chunking logic stays the same; the safety cap catches the pathological edge cases.

DEFAULT_MAX_CHUNK_CHARACTERS = 3000

def _split_record_for_embedding(
    record: ChunkRecord,
    *,
    max_characters: int = DEFAULT_MAX_CHUNK_CHARACTERS,
) -> List[ChunkRecord]:
    if len(record.text) <= max_characters:
        return [record]
    # ... split oversized text into smaller records, preserving heading path ...

Not elegant. It's the kind of thing you only discover when you index a real vault instead of a toy corpus. That's the point.

Retrieval and generation

The retrieval and generation stack runs entirely through Ollama:

  • Embeddings: mxbai-embed-large. I started with nomic-embed-text and later compared both models against a hand-built evaluation set. I'll show those numbers in the next section, but the short version: mxbai-embed-large improved retrieval hit rate by 7 to 18 percentage points depending on the chunking strategy.
  • Generation: qwen2.5:14b. I originally used llama3.1:8b. In qualitative spot-checks, qwen2.5:14b followed the grounding prompt more cleanly (stopped citing chunk numbers, gave tighter sourced answers). I want to be honest: this was a judgement call from testing queries by hand, not a formal benchmark with scored outputs.
  • Vector store: ChromaDB with cosine similarity, persisted to .vask/chroma/.

Retrieval defaults to top-k 6 with a minimum cosine similarity of 0.25. Broad enough to capture relevant context from several notes, with a floor that filters obvious noise. Both values are overridable per query:

uv run vask query "What's the weekly review approach?" --top-k 10 --min-similarity 0.3

One quality-of-life fix worth noting: the index path was originally relative, which silently broke queries run outside the repo directory. Now vask respects a VASK_INDEX_DIR environment variable for the persisted index location. Install it as a global uv tool (uv tool install --editable .) and you can query or reindex from any working directory:

export VASK_INDEX_DIR="$HOME/Developer/vask/.vask"

The bare command handles both use cases natively: vask "question" to query, vask index --vault "/path/to/your/vault" to reindex. No aliases needed, no cd required. Small thing, but it drops the friction of actually using the pipeline once it's built.

The generation prompt

This is where most RAG tutorials get quietly dishonest. They feed the retrieved context to the model and let it answer freely, which usually means confident-sounding answers that blend your notes with the model's training data. For personal notes, where the whole point is answering from your content and not the internet's, that defeats the purpose.

Here's the system promt from rag.py:

{
    "role": "system",
    "content": (
        "Answer using only the provided vault context. "
        "If the context is insufficient, say so plainly. "
        "Cite source note titles inline in square brackets when you rely on them. "
        "Never use numeric citations such as [1] or [2]. "
        "Do not cite chunk numbers."
    ),
}

Two design choices here:

Explicit grounding. "Answer using only the provided vault context" and "If the context is insufficient, say so plainly." Local models are more reliable when the prompt leaves no room for improvisation. Without this constraint, the model fills gaps with plausible-sounding general knowledge that didn't come from your notes.

Note-title citations, not chunk numbers. The model cites [Weekly Review] inline, not [Chunk 3]. On top of that, every answer gets a deterministic Sources: line appended from the actual retrieval set, built programmatically rather than by the model. So even when the inline citations are imperfect, the reader sees which notes informed the answer.

Evaluation: two models, honest numbers

This is the section most PKM tutorials skip. You get a blog post that says "I built a thing and it works," a screenshot of one good answer, and nothing about how often it fails or why. I'm not doing that.

I built a 28-question evaluation set by hand. Each question targets a specific note (or notes) in my real vault. The question types range from direct lookups ("which note says X?") to paraphrases, metaphors, cross-note queries, and dashboard navigation. I ran both chunking strategies against this set with the same retrieval defaults (top-k 6, min similarity 0.25) and counted whether the correct source note appeared anywhere in the top-6 results.

First pass: chunking comparison

The first evaluation compared whole-note chunks against heading-based chunks using nomic-embed-text as the embedding model. The vault at that point had 760 notes:

Heading-based chunking won by 14 percentage points. The biggest gains came from question types where local context matters: paraphrase queries jumped from 0/3 to 2/3, questions spanning multiple notes went from 0/2 to 2/2, and an archived newsletter surfaced because heading-level chunking kept it separate from the surrounding noise.

67.86% is not a number you'd put on a marketing page. On synthetic benchmarks with clean documents and exact-match questions, retrieval pipelines routinely hit 90%+. But this is a real vault. The notes are messy, densely linked, full of tables and shorthand. The questions are the kind of things you'd actually type into a search box, not the kind an evaluation dataset would generate for you.

Second pass: embedding model comparison

Most tutorials test one configuration and call it done. I wasn't satisfied with 67.86%, so I pulled a second embedding model (mxbai-embed-large), re-indexed the vault, and ran the same 28 questions again. By this point the vault had grown slightly to 770 notes.

Here's the head-to-head:

mxbai-embed-large won on both strategies. The whole-note improvement was dramatic, nearly 18 points. Heading-based chunking, already the stronger strategy, still gained over 7 points.

The best overall result: mxbai-embed-large with heading-based chunking at 75.00%.

A few specific improvements stand out. With mxbai-embed-large, paraphrase queries on whole-note went from 0/3 to 2/3. Metaphor retrieval, which both strategies missed entirely under nomic-embed-text, now hit on whole-note. Compare-and-contrast and archived-note lookups that were zeroes before started returning the right source note.

Some categories did regress. Ambiguous phrasing went from 1/1 to 0/1 on both strategies. A cross-note motif that nomic caught in whole-note mode was missed by mxbai. No model wins everywhere. The aggregate was clear enough to adopt.

The generation model

I also switched from llama3.1:8b to qwen2.5:14b for generation. I want to be upfront about this: the switch was based on qualitative spot-checks, not a scored evaluation. In the queries I tested by hand, qwen2.5:14b did a better job following the grounding prompt (stopped citing chunk numbers, gave tighter sourced answers). But I don't have a formal comparison table for generation quality. Honestly benchmarking generation over personal notes is a harder problem than benchmarking retrieval, and I haven't solved it yet.

Known failure modes and what's next

The failure log I'm drawing from here was built against the original nomic-embed-text baseline, before the switch to mxbai-embed-large. Some of these failures have since been fixed by the stronger embedding model (strategy-level misses dropped from 22 to 15). I'm including the original failure analysis because the types of failures are still instructive, even where the specific examples may no longer apply.

Under the nomic run, 17 of the 28 questions failed for at least one strategy. The 22 total misses broke down like this:

The most common failure, nine cases, was retrieval surfacing a semantically adjacent but wrong note. Ask about Phase 3 failure cases and retrieval returns decision logs and orchestration notes from the same project. Ask about a product's revenue architecture and retrieval returns the strategy notes that mention the same business vocabulary. These aren't random misses. The retrieval is finding something close. Just not close enough.

The standout case: the home dashboard

The most instructive failure was the simplest question: "Which dashboard note has quick links for Projects, Inbox, Reference, and Weekly Reviews all in one table?"

Both strategies missed completely.

The expected source (Home Dashboard.md) is a table-heavy navigation note. It's the kind of note that's obvious to a human: you open it, you see the table, you know it's the home dashboard. But to an embedding model, it's a sparse grid of short labels with no connecting prose. The question asks in natural language what the note answers in table cells and column headers.

This failure points at a structural gap that better chunking won't fix. Dashboard notes, MOC (map of content) notes, and navigation hubs have almost no narrative text. Their meaning lives in their structure: tables, link lists, heading hierarchies. Embedding models don't read structure. They read sentences.

Other patterns

Vocabulary mismatch accounted for three failures. One question asked about a "campfire, classroom, billboard" metaphor for communication channels. The note uses that metaphor once, buried in a longer discussion of channel strategy. The figurative language doesn't match the channel names that dominate the note's embedding.

Thin source notes failed three times. One topic hub note was mostly a list of wikilinks with category labels. When heading-level chunking split those labels into fragments, each fragment was too sparse to match any natural-language question about the taxonomy it describes.

What's next

Most of these failures are near misses, not random noise. The retrieval consistently surfaced notes from the right neighbourhood, just not the right note.

The next project will explore whether vault graph structure (backlinks, link proximity, co-citation patterns) can act as a re-ranking signal on top of embedding similarity. A note that's one link hop away from a strong match should probably rank higher than a semantically similar note from an unrelated corner of the vault.

I'm also interested in whether augmenting thin dashboard notes with synthetic descriptions (a one-sentence summary of what the table contains) would close the gap for structural notes without distorting the rest of the corpus.

I've also looked into using it with Apple's native LLM. I think the best use for that is for orchestration of my vault vs querying it, but I have yet to test it.

These are experiments, not promises. But that's the point of evaluating honestly: you know where to invest next instead of guessing.

Your Action Items

  1. Clone the repo and index your own vault. It takes one command: uv run vask index --vault ~/path/to/vault. Start with the bundled sample vault if you want to confirm the install works first.
  2. Write ten questions you'd actually ask your notes, then check whether the right source note shows up in the top-6 results. Where it doesn't, look at what did show up. The near misses teach you more than the hits.
  3. Pick your worst-performing question and work backwards. Read the source note. Ask yourself: would a single sentence added to that note make it matchable? Sometimes the fix is a one-line summary, not a better algorithm.

Weekly notes

If this was useful, come along for the weekly version. Every week I write you a short letter about what I'm learning: second brain, analog-first capture, and thinking clearly about when to use AI tools and when to do the work yourself. It's like getting a note from a friend who's figuring it out alongside you.

Subscribe to weekly notes →

References

Alibaba Cloud. (2024). Qwen 2.5 [Large language model]. https://qwenlm.github.io/blog/qwen2.5

Chroma. (n.d.). ChromaDB: The AI-native open-source embedding database. https://www.trychroma.com/

Meta. (2024). Llama 3.1 [Large language model]. https://llama.meta.com/

mixedbread ai. (2024). mxbai-embed-large [Embedding model]. https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1

Nomic AI. (2024). Nomic Embed Text [Embedding model]. https://huggingface.co/nomic-ai/nomic-embed-text-v1.5

Ollama. (n.d.). Ollama: Get up and running with large language models locally. https://ollama.com/

Tagged in:

article, p10y, rag, ai, obsidian

Last Update: July 12, 2026