diff --git a/CLAUDE.md b/CLAUDE.md index 3f8cb6f..7ceae78 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ # Memex Chat — CLAUDE.md -Obsidian plugin: Chat with your vault using Claude AI. Semantic TF-IDF + local embedding context retrieval, `@Notizname` mentions, thread history, prompt extension buttons, streaming responses, related notes sidebar. +Obsidian plugin: Chat with your vault using Claude AI. Hybrid TF-IDF + embedding search (Reciprocal Rank Fusion), MemPalace external knowledge injection, `@Notizname` mentions, thread history, prompt extension buttons, streaming responses, related notes sidebar. ## Build @@ -21,6 +21,7 @@ Entry: `src/main.ts` → bundled to `main.js` via esbuild (CJS, ES2018 target). | `src/ChatView.ts` | Main UI — `ChatView extends ItemView`. Thread management, sidebar history, context preview, mode buttons, streaming render, Copy/Save actions. View type: `memex-chat-view`. | | `src/VaultSearch.ts` | TF-IDF search engine. Builds in-memory index over all vault markdown files. Frontmatter property boost (5×). `findSimilarByName()` for unresolved link hints. Exports `SearchResult` interface (includes optional `linked` field). | | `src/EmbedSearch.ts` | Local semantic search via `@xenova/transformers` (ONNX, WASM). Caches per-note `.ajson` vectors under `/.memex-chat/embeddings/`. `searchSimilarToFile()` boosts scores by frontmatter property links (+0.15) and shared tags (+0.05/tag). | +| `src/HybridSearch.ts` | Combines TF-IDF and embedding search via Reciprocal Rank Fusion (RRF, k=60). Runs both engines in parallel; rank-merges results so neither score space needs normalization. TF-IDF excerpts are preserved in merged output. | | `src/RelatedNotesView.ts` | Sidebar panel — `RelatedNotesView extends ItemView`. Shows semantically similar notes for the active file; refreshes on file-open. Displays similarity bar and "verknüpft" badge for property-linked notes. View type: `memex-related-notes`. | | `src/ClaudeClient.ts` | Anthropic API client. `streamChat()` yields `ClaudeStreamChunk` via async generator using native `fetch` + SSE. `chat()` and `fetchModels()` use Obsidian `requestUrl` (no SDK). | | `src/SettingsTab.ts` | `MemexChatSettingsTab` + `MemexChatSettings` interface + `DEFAULT_SETTINGS`. Exports `PromptButton` interface. Folder autocomplete via `attachFolderDropdown()` helper. | @@ -33,8 +34,9 @@ Entry: `src/main.ts` → bundled to `main.js` via esbuild (CJS, ES2018 target). - **Data persistence**: `this.saveData(this.data)` / `this.loadData()` — single object `{ settings, threads }`. Settings merge on load preserves new fields via per-entry spread for `promptButtons`. - **Streaming**: `ClaudeClient.streamChat()` is an async generator using native `fetch` with `stream: true` and SSE parsing (`content_block_delta` events). `ChatView` iterates it and calls `updateLastMessage()` per chunk. `chat()` and `fetchModels()` use `requestUrl` (buffered, fine for non-streaming calls). -- **Context flow**: Query → `VaultSearch.search()` or `EmbedSearch.search()` → context preview → user confirms → `sendMessage()` injects note content into the Claude prompt. Auto-retrieve skipped when prompt extension buttons are active. -- **Active search engine**: `plugin.activeSearch` returns `EmbedSearch` when enabled, else `VaultSearch`. +- **Context flow**: Query → `VaultSearch.search()` or `HybridSearch.search()` → context preview → user confirms → `sendMessage()` injects note content into the Claude prompt. If `useMempalace` is enabled, `queryMempalace()` runs first and its results are prepended (highest priority, closest to query). Auto-retrieve skipped when prompt extension buttons are active. +- **Active search engine**: `plugin.activeSearch` returns `HybridSearch` when embeddings are ready, else `VaultSearch`. +- **MemPalace context**: `ChatView.queryMempalace(query)` calls `/usr/local/bin/mempalace search --results N` via `execFile` with a 10 s timeout. Returns `""` silently if the binary is absent, errors, or times out. Results are labeled `MemPalace (Wissens-Archiv):` and prepended before vault context. - **System prompt layering**: base system prompt → optional `systemContextFile` → active `promptButtons` extension files (each appended with `\n\n---\n`). - **@mention syntax**: `@Notizname` — autocomplete triggers after 2 chars, inserts full basename. Parsing in `handleSend` matches vault filenames directly (handles spaces & special chars). - **Prompt buttons**: `activeExtensions: Set` tracks active button file paths. Mode hint panel shows `helpText` above input; hidden after send. Date-search buttons parse month from query and filter files by `getFileDate()`. @@ -61,6 +63,23 @@ Entry: `src/main.ts` → bundled to `main.js` via esbuild (CJS, ES2018 target). - Obsidian Sync wait: `waitForSyncIdle()` monitors vault events (5 s probe, 15 s quiet) before starting `buildIndex`. - esbuild patches required: `stubNativeModules`, `forceOnnxWeb`, `forceOrtWebBrowserMode`. `import.meta.url` defined as a constant string. +## HybridSearch + +- Combines `VaultSearch` (TF-IDF) and `EmbedSearch` (ONNX embeddings) via Reciprocal Rank Fusion. +- `RRF_K = 60` — standard constant; score = `1/(K + rank_tfidf + 1) + 1/(K + rank_embed + 1)`. +- Runs both engines in parallel (`Promise.all`), fetches `topK * 3` candidates from each. +- TF-IDF excerpts preserved in merged output; embedding results fill in where TF-IDF has no match. +- `plugin.hybridSearch` is set after `embedSearch.buildIndex()` completes; `plugin.activeSearch` returns it over `VaultSearch`. + +## MemPalace Integration + +- Requires `/usr/local/bin/mempalace` CLI installed on the host machine. +- `ChatView.queryMempalace(query)`: calls `mempalace search --results N` via Node `execFile`, 10 s timeout. +- Silent no-op if binary missing, process errors, or stdout empty — never throws. +- Output section header: `MemPalace (Wissens-Archiv):`, prepended before vault context so it sits closest to the query in the prompt. +- Controlled by `settings.useMempalace` (toggle) and `settings.mempalaceResults` (1–10, default 3). +- Status indicator "MemPalace wird abgefragt…" shown during the CLI call. + ## RelatedNotesView - Opens in right sidebar leaf via `plugin.activateRelatedView()` or sparkles ribbon icon. @@ -89,6 +108,8 @@ Entry: `src/main.ts` → bundled to `main.js` via esbuild (CJS, ES2018 target). | `useEmbeddings` | `false` | Enable local semantic embeddings | | `embeddingModel` | `TaylorAI/bge-micro-v2` | ONNX embedding model ID | | `embedExcludeFolders` | `[]` | Vault folders excluded from embedding | +| `useMempalace` | `false` | Inject MemPalace CLI search results as additional context | +| `mempalaceResults` | `3` | Number of MemPalace results per query (1–10) | | `promptButtons` | Draft Check, Monthly Check | Header mode buttons with system prompt extension | ## Prompt Buttons (PromptButton interface) diff --git a/main.js b/main.js index 78b6e17..3e92d38 100644 --- a/main.js +++ b/main.js @@ -33397,7 +33397,7 @@ var RelatedNotesView = class extends import_obsidian4.ItemView { async onOpen() { this.registerEvent(this.app.workspace.on("active-leaf-change", () => this.scheduleRefresh())); this.registerEvent(this.app.workspace.on("file-open", () => this.scheduleRefresh())); - this.render([]); + this.render([], []); this.scheduleRefresh(); } scheduleRefresh(delay = 400) { @@ -33405,7 +33405,6 @@ var RelatedNotesView = class extends import_obsidian4.ItemView { clearTimeout(this.refreshTimer); this.refreshTimer = setTimeout(() => this.refresh(), delay); } - /** Called by the plugin when the embedding index finishes building. */ onIndexReady() { this.scheduleRefresh(0); } @@ -33414,48 +33413,127 @@ var RelatedNotesView = class extends import_obsidian4.ItemView { if (!file || file.extension !== "md") return; const es = this.plugin.embedSearch; - if (!es || !es.isIndexed()) { + const useMempalace = this.plugin.settings.useMempalace; + const embedReady = es?.isIndexed() ?? false; + if (!useMempalace && !embedReady) { this.renderStatus("Embedding-Index wird aufgebaut\u2026"); return; } this.renderStatus("Suche verwandte Notizen\u2026"); - const results = await es.searchSimilarToFile(file); - this.render(results, file.basename); + const topK = this.plugin.settings.mempalaceResults ?? 5; + const [mpResults, nativeResults] = await Promise.all([ + useMempalace ? this.queryMempalace(file.basename, topK) : Promise.resolve([]), + embedReady ? es.searchSimilarToFile(file) : Promise.resolve([]) + ]); + this.render(mpResults, nativeResults, file.basename); } + // ─── MemPalace ──────────────────────────────────────────────────────────── + async queryMempalace(query, topK) { + return new Promise((resolve) => { + try { + const { existsSync } = require("fs"); + const { execFile } = require("child_process"); + if (!existsSync("/usr/local/bin/mempalace")) { + resolve([]); + return; + } + execFile( + "/usr/local/bin/mempalace", + ["search", query, "--results", String(topK)], + { timeout: 8e3 }, + (err, stdout) => { + if (err || !stdout) { + resolve([]); + return; + } + resolve(this.parseMempalace(stdout)); + } + ); + } catch { + resolve([]); + } + }); + } + parseMempalace(output) { + const results = []; + const blocks = output.split(/─{10,}/); + for (const block of blocks) { + const locMatch = block.match(/\[\d+\]\s+(.+?)\n/); + const srcMatch = block.match(/Source:\s+(.+?)(?:\.md)?\s*\n/); + const scoreMatch = block.match(/Match:\s+([\d.]+)/); + if (!locMatch || !srcMatch || !scoreMatch) + continue; + const location = locMatch[1].trim(); + const source = srcMatch[1].trim(); + const score = parseFloat(scoreMatch[1]); + const afterScore = block.slice(block.indexOf(scoreMatch[0]) + scoreMatch[0].length).trimStart(); + const excerpt = afterScore.replace(/\n{3,}/g, "\n\n").trim().slice(0, 240); + results.push({ source, location, score, excerpt }); + } + return results; + } + // ─── Rendering ──────────────────────────────────────────────────────────── renderStatus(msg) { this.contentEl.empty(); this.contentEl.createDiv({ cls: "vc-related-status", text: msg }); } - render(results, forNote) { + render(mpResults, nativeResults, forNote) { this.contentEl.empty(); const header = this.contentEl.createDiv("vc-related-header"); header.createDiv({ cls: "vc-related-title", text: "Verwandte Notizen" }); if (forNote) header.createDiv({ cls: "vc-related-subtitle", text: forNote }); - if (!results.length) { + if (!mpResults.length && !nativeResults.length) { this.contentEl.createDiv({ cls: "vc-related-status", text: forNote ? "Keine Treffer." : "" }); return; } - const list = this.contentEl.createDiv("vc-related-list"); - for (const r of results) { - const item = list.createDiv("vc-related-item"); - const info = item.createDiv("vc-related-info"); - const nameRow = info.createDiv("vc-related-name-row"); - nameRow.createSpan({ cls: "vc-related-name", text: r.title }); - if (r.linked) - nameRow.createSpan({ cls: "vc-related-linked", text: "verkn\xFCpft" }); - const folder = r.file.parent?.path; - if (folder && folder !== "/") { - info.createDiv({ cls: "vc-related-folder", text: folder }); + if (mpResults.length) { + this.contentEl.createDiv({ cls: "vc-related-section-label", text: "MemPalace" }); + const mpList = this.contentEl.createDiv("vc-related-list"); + for (const r of mpResults) { + const item = mpList.createDiv("vc-related-item vc-related-item--mp"); + const info = item.createDiv("vc-related-info"); + const nameRow = info.createDiv("vc-related-name-row"); + nameRow.createSpan({ cls: "vc-related-name", text: r.source }); + nameRow.createSpan({ cls: "vc-related-location", text: r.location }); + if (r.excerpt) { + info.createDiv({ cls: "vc-related-excerpt", text: r.excerpt }); + } + const scoreWrap = item.createDiv("vc-related-score-wrap"); + const pct = Math.round(r.score * 100); + const bar = scoreWrap.createDiv("vc-related-bar"); + bar.createDiv({ cls: "vc-related-bar-fill vc-related-bar-fill--mp" }).style.width = `${pct}%`; + scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); + item.addEventListener("click", () => { + const vaultFile = this.app.vault.getMarkdownFiles().find((f) => f.basename === r.source); + if (vaultFile) + this.app.workspace.openLinkText(vaultFile.path, vaultFile.path, false); + }); + } + } + if (nativeResults.length) { + this.contentEl.createDiv({ cls: "vc-related-section-label", text: "Vault" }); + const list = this.contentEl.createDiv("vc-related-list"); + for (const r of nativeResults) { + const item = list.createDiv("vc-related-item"); + const info = item.createDiv("vc-related-info"); + const nameRow = info.createDiv("vc-related-name-row"); + nameRow.createSpan({ cls: "vc-related-name", text: r.title }); + if (r.linked) + nameRow.createSpan({ cls: "vc-related-linked", text: "verkn\xFCpft" }); + const folder = r.file.parent?.path; + if (folder && folder !== "/") { + info.createDiv({ cls: "vc-related-folder", text: folder }); + } + const scoreWrap = item.createDiv("vc-related-score-wrap"); + const pct = Math.round(r.score * 100); + const bar = scoreWrap.createDiv("vc-related-bar"); + bar.createDiv({ cls: "vc-related-bar-fill" }).style.width = `${pct}%`; + scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); + item.addEventListener("click", () => { + this.app.workspace.openLinkText(r.file.path, r.file.path, false); + }); } - const scoreWrap = item.createDiv("vc-related-score-wrap"); - const pct = Math.round(r.score * 100); - const bar = scoreWrap.createDiv("vc-related-bar"); - bar.createDiv({ cls: "vc-related-bar-fill" }).style.width = `${pct}%`; - scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); - item.addEventListener("click", () => { - this.app.workspace.openLinkText(r.file.path, r.file.path, false); - }); } } }; diff --git a/src/RelatedNotesView.ts b/src/RelatedNotesView.ts index 09f7643..78188b6 100644 --- a/src/RelatedNotesView.ts +++ b/src/RelatedNotesView.ts @@ -3,6 +3,13 @@ import type MemexChatPlugin from "./main"; export const VIEW_TYPE_RELATED = "memex-related-notes"; +interface MpResult { + source: string; // basename without .md + location: string; // "wing / room" + score: number; + excerpt: string; +} + export class RelatedNotesView extends ItemView { private plugin: MemexChatPlugin; private refreshTimer: ReturnType | null = null; @@ -19,7 +26,7 @@ export class RelatedNotesView extends ItemView { async onOpen(): Promise { this.registerEvent(this.app.workspace.on("active-leaf-change", () => this.scheduleRefresh())); this.registerEvent(this.app.workspace.on("file-open", () => this.scheduleRefresh())); - this.render([]); + this.render([], []); this.scheduleRefresh(); } @@ -28,7 +35,6 @@ export class RelatedNotesView extends ItemView { this.refreshTimer = setTimeout(() => this.refresh(), delay); } - /** Called by the plugin when the embedding index finishes building. */ onIndexReady() { this.scheduleRefresh(0); } private async refresh() { @@ -36,58 +42,148 @@ export class RelatedNotesView extends ItemView { if (!file || file.extension !== "md") return; const es = this.plugin.embedSearch; - if (!es || !es.isIndexed()) { + const useMempalace = this.plugin.settings.useMempalace; + const embedReady = es?.isIndexed() ?? false; + + // Nothing can run yet + if (!useMempalace && !embedReady) { this.renderStatus("Embedding-Index wird aufgebaut…"); return; } this.renderStatus("Suche verwandte Notizen…"); - const results = await es.searchSimilarToFile(file); - this.render(results, file.basename); + + const topK = this.plugin.settings.mempalaceResults ?? 5; + + const [mpResults, nativeResults] = await Promise.all([ + useMempalace ? this.queryMempalace(file.basename, topK) : Promise.resolve([] as MpResult[]), + embedReady ? es!.searchSimilarToFile(file) : Promise.resolve([]), + ]); + + this.render(mpResults, nativeResults, file.basename); } + // ─── MemPalace ──────────────────────────────────────────────────────────── + + private async queryMempalace(query: string, topK: number): Promise { + return new Promise((resolve) => { + try { + const { existsSync } = require("fs") as typeof import("fs"); + const { execFile } = require("child_process") as typeof import("child_process"); + if (!existsSync("/usr/local/bin/mempalace")) { resolve([]); return; } + execFile( + "/usr/local/bin/mempalace", + ["search", query, "--results", String(topK)], + { timeout: 8000 }, + (err: Error | null, stdout: string) => { + if (err || !stdout) { resolve([]); return; } + resolve(this.parseMempalace(stdout)); + } + ); + } catch { resolve([]); } + }); + } + + private parseMempalace(output: string): MpResult[] { + const results: MpResult[] = []; + const blocks = output.split(/─{10,}/); + for (const block of blocks) { + const locMatch = block.match(/\[\d+\]\s+(.+?)\n/); + const srcMatch = block.match(/Source:\s+(.+?)(?:\.md)?\s*\n/); + const scoreMatch = block.match(/Match:\s+([\d.]+)/); + if (!locMatch || !srcMatch || !scoreMatch) continue; + + const location = locMatch[1].trim(); + const source = srcMatch[1].trim(); + const score = parseFloat(scoreMatch[1]); + + const afterScore = block.slice(block.indexOf(scoreMatch[0]) + scoreMatch[0].length).trimStart(); + const excerpt = afterScore.replace(/\n{3,}/g, "\n\n").trim().slice(0, 240); + + results.push({ source, location, score, excerpt }); + } + return results; + } + + // ─── Rendering ──────────────────────────────────────────────────────────── + private renderStatus(msg: string) { this.contentEl.empty(); this.contentEl.createDiv({ cls: "vc-related-status", text: msg }); } - private render(results: Array<{ file: TFile; score: number; title: string }>, forNote?: string) { + private render( + mpResults: MpResult[], + nativeResults: Array<{ file: TFile; score: number; title: string; linked?: boolean }>, + forNote?: string + ) { this.contentEl.empty(); const header = this.contentEl.createDiv("vc-related-header"); header.createDiv({ cls: "vc-related-title", text: "Verwandte Notizen" }); if (forNote) header.createDiv({ cls: "vc-related-subtitle", text: forNote }); - if (!results.length) { + if (!mpResults.length && !nativeResults.length) { this.contentEl.createDiv({ cls: "vc-related-status", text: forNote ? "Keine Treffer." : "" }); return; } - const list = this.contentEl.createDiv("vc-related-list"); - for (const r of results) { - const item = list.createDiv("vc-related-item"); + // ── MemPalace section ── + if (mpResults.length) { + this.contentEl.createDiv({ cls: "vc-related-section-label", text: "MemPalace" }); + const mpList = this.contentEl.createDiv("vc-related-list"); + for (const r of mpResults) { + const item = mpList.createDiv("vc-related-item vc-related-item--mp"); - const info = item.createDiv("vc-related-info"); - const nameRow = info.createDiv("vc-related-name-row"); - nameRow.createSpan({ cls: "vc-related-name", text: r.title }); - if (r.linked) nameRow.createSpan({ cls: "vc-related-linked", text: "verknüpft" }); + const info = item.createDiv("vc-related-info"); + const nameRow = info.createDiv("vc-related-name-row"); + nameRow.createSpan({ cls: "vc-related-name", text: r.source }); + nameRow.createSpan({ cls: "vc-related-location", text: r.location }); - // Folder path (dimmed) - const folder = r.file.parent?.path; - if (folder && folder !== "/") { - info.createDiv({ cls: "vc-related-folder", text: folder }); + if (r.excerpt) { + info.createDiv({ cls: "vc-related-excerpt", text: r.excerpt }); + } + + const scoreWrap = item.createDiv("vc-related-score-wrap"); + const pct = Math.round(r.score * 100); + const bar = scoreWrap.createDiv("vc-related-bar"); + bar.createDiv({ cls: "vc-related-bar-fill vc-related-bar-fill--mp" }).style.width = `${pct}%`; + scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); + + item.addEventListener("click", () => { + const vaultFile = this.app.vault.getMarkdownFiles().find((f) => f.basename === r.source); + if (vaultFile) this.app.workspace.openLinkText(vaultFile.path, vaultFile.path, false); + }); } + } - // Similarity bar + percentage - const scoreWrap = item.createDiv("vc-related-score-wrap"); - const pct = Math.round(r.score * 100); - const bar = scoreWrap.createDiv("vc-related-bar"); - bar.createDiv({ cls: "vc-related-bar-fill" }).style.width = `${pct}%`; - scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); + // ── Vault (native) section ── + if (nativeResults.length) { + this.contentEl.createDiv({ cls: "vc-related-section-label", text: "Vault" }); + const list = this.contentEl.createDiv("vc-related-list"); + for (const r of nativeResults) { + const item = list.createDiv("vc-related-item"); - item.addEventListener("click", () => { - this.app.workspace.openLinkText(r.file.path, r.file.path, false); - }); + const info = item.createDiv("vc-related-info"); + const nameRow = info.createDiv("vc-related-name-row"); + nameRow.createSpan({ cls: "vc-related-name", text: r.title }); + if (r.linked) nameRow.createSpan({ cls: "vc-related-linked", text: "verknüpft" }); + + const folder = r.file.parent?.path; + if (folder && folder !== "/") { + info.createDiv({ cls: "vc-related-folder", text: folder }); + } + + const scoreWrap = item.createDiv("vc-related-score-wrap"); + const pct = Math.round(r.score * 100); + const bar = scoreWrap.createDiv("vc-related-bar"); + bar.createDiv({ cls: "vc-related-bar-fill" }).style.width = `${pct}%`; + scoreWrap.createDiv({ cls: "vc-related-pct", text: `${pct}%` }); + + item.addEventListener("click", () => { + this.app.workspace.openLinkText(r.file.path, r.file.path, false); + }); + } } } } diff --git a/styles.css b/styles.css index aef3268..6f72ee5 100644 --- a/styles.css +++ b/styles.css @@ -111,6 +111,38 @@ border-radius: 2px; } +.vc-related-bar-fill--mp { + background: var(--color-purple, #8b5cf6); +} + +.vc-related-section-label { + padding: 5px 12px 3px; + font-size: 10px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.06em; + color: var(--text-faint); + background: var(--background-secondary); + border-bottom: 1px solid var(--background-modifier-border); +} + +.vc-related-location { + font-size: 10px; + color: var(--text-faint); + white-space: nowrap; + flex-shrink: 0; +} + +.vc-related-excerpt { + font-size: 10px; + color: var(--text-muted); + margin-top: 3px; + display: -webkit-box; + -webkit-line-clamp: 2; + -webkit-box-orient: vertical; + overflow: hidden; +} + .vc-related-pct { font-size: 10px; color: var(--text-muted);