From f97e9a82f41240421e0ad938bc2709b2b21f730d Mon Sep 17 00:00:00 2001 From: Ricardo Date: Fri, 27 Feb 2026 17:18:30 +0100 Subject: [PATCH] chore: remove dev plans from published repo Plans moved to central /home/rick/code/indiekit-dev/docs/plans/ --- docs/plans/2026-02-20-federation-hardening.md | 737 --------------- .../2026-02-20-fedify-feature-completeness.md | 650 ------------- docs/plans/2026-02-21-activitypub-reader.md | 882 ------------------ .../2026-02-27-activitypub-deck-layout.md | 398 -------- docs/plans/fedify-migration.md | 540 ----------- 5 files changed, 3207 deletions(-) delete mode 100644 docs/plans/2026-02-20-federation-hardening.md delete mode 100644 docs/plans/2026-02-20-fedify-feature-completeness.md delete mode 100644 docs/plans/2026-02-21-activitypub-reader.md delete mode 100644 docs/plans/2026-02-27-activitypub-deck-layout.md delete mode 100644 docs/plans/fedify-migration.md diff --git a/docs/plans/2026-02-20-federation-hardening.md b/docs/plans/2026-02-20-federation-hardening.md deleted file mode 100644 index 193e9cf..0000000 --- a/docs/plans/2026-02-20-federation-hardening.md +++ /dev/null @@ -1,737 +0,0 @@ -# Federation Hardening Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Fix all spec compliance issues found during the Fedify docs audit — persistent Ed25519 keys, proper `assertionMethods`, Redis message queue, Reject listener, database indexes, and storeRawActivities wiring. - -**Architecture:** Changes are isolated to the plugin's `lib/` layer (federation-setup.js, inbox-listeners.js, activity-log.js) plus package.json for the new `@fedify/redis` dependency. The plugin accepts an optional `redisUrl` from Indiekit config; when present, it uses `RedisMessageQueue` instead of `InProcessMessageQueue`. Ed25519 keys are generated once and persisted to `ap_keys` as JWK. Cloudron deployment config is updated to pass through the Redis URL. - -**Tech Stack:** `@fedify/fedify` ^1.10.0, `@fedify/redis` (new), `ioredis` (new), MongoDB (existing) - ---- - -### Task 1: Persist Ed25519 key pair to database - -**Files:** -- Modify: `lib/federation-setup.js` (lines 139-167, setKeyPairsDispatcher) - -**Context:** Currently `generateCryptoKeyPair("Ed25519")` is called on every request, producing a new key pair each time. Remote servers fetching the actor to verify an Object Integrity Proof get a different public key than the one used to sign — causing silent verification failures. The Fedify docs say: "generate key pairs for each actor when the actor is created" and store them persistently using `exportJwk()`/`importJwk()`. - -**Step 1: Add `exportJwk` and `importJwk` to imports** - -At the top of `federation-setup.js`, add to the existing import: - -```javascript -import { - Endpoints, - Image, - InProcessMessageQueue, - Person, - PropertyValue, - createFederation, - exportJwk, // ADD - generateCryptoKeyPair, - importJwk, // ADD - importSpki, -} from "@fedify/fedify"; -``` - -**Step 2: Rewrite `setKeyPairsDispatcher` to persist Ed25519** - -Replace the entire `.setKeyPairsDispatcher(async (ctx, identifier) => { ... })` block (lines 139-167) with: - -```javascript - .setKeyPairsDispatcher(async (ctx, identifier) => { - if (identifier !== handle) return []; - - const keyPairs = []; - - // --- Legacy RSA key pair (HTTP Signatures) --- - const legacyKey = await collections.ap_keys.findOne({ type: "rsa" }); - // Fall back to old schema (no type field) for backward compat - const rsaDoc = legacyKey || await collections.ap_keys.findOne({ - publicKeyPem: { $exists: true }, - }); - - if (rsaDoc?.publicKeyPem && rsaDoc?.privateKeyPem) { - try { - const publicKey = await importSpki(rsaDoc.publicKeyPem); - const privateKey = await importPkcs8Pem(rsaDoc.privateKeyPem); - keyPairs.push({ publicKey, privateKey }); - } catch { - console.warn("[ActivityPub] Could not import legacy RSA keys"); - } - } - - // --- Ed25519 key pair (Object Integrity Proofs) --- - // Load from DB or generate + persist on first use - let ed25519Doc = await collections.ap_keys.findOne({ type: "ed25519" }); - - if (ed25519Doc?.publicKeyJwk && ed25519Doc?.privateKeyJwk) { - try { - const publicKey = await importJwk(ed25519Doc.publicKeyJwk, "public"); - const privateKey = await importJwk(ed25519Doc.privateKeyJwk, "private"); - keyPairs.push({ publicKey, privateKey }); - } catch (error) { - console.warn( - "[ActivityPub] Could not import Ed25519 keys, regenerating:", - error.message, - ); - ed25519Doc = null; // Force regeneration below - } - } - - if (!ed25519Doc) { - try { - const ed25519 = await generateCryptoKeyPair("Ed25519"); - await collections.ap_keys.insertOne({ - type: "ed25519", - publicKeyJwk: await exportJwk(ed25519.publicKey), - privateKeyJwk: await exportJwk(ed25519.privateKey), - createdAt: new Date().toISOString(), - }); - keyPairs.push(ed25519); - console.info("[ActivityPub] Generated and persisted Ed25519 key pair"); - } catch (error) { - console.warn( - "[ActivityPub] Could not generate Ed25519 key pair:", - error.message, - ); - } - } - - return keyPairs; - }); -``` - -**Step 3: Verify** - -Run: `fedify lookup https://rmendes.net/activitypub/users/rick` - -Expected: Actor still resolves with `publicKey` and `assertionMethod` visible. Restart the app and re-run — the same key should be returned (verify by comparing the key IDs between requests). - -**Step 4: Commit** - -``` -feat(keys): persist Ed25519 key pair to ap_keys collection - -Previously generated a new Ed25519 key pair on every request, -causing Object Integrity Proof verification failures on remote -servers. Now generates once and stores as JWK in MongoDB. -``` - ---- - -### Task 2: Fix `assertionMethods` (plural) on actor - -**Files:** -- Modify: `lib/federation-setup.js` (lines 113-117, actor dispatcher) - -**Context:** The actor currently sets `assertionMethod` (singular) with only the first key's multikey. The Fedify docs specify `assertionMethods` (plural array) containing ALL multikey instances — typically one per key pair (RSA + Ed25519). - -**Step 1: Replace singular with plural** - -In the actor dispatcher, find: - -```javascript -if (keyPairs.length > 0) { - personOptions.publicKey = keyPairs[0].cryptographicKey; - personOptions.assertionMethod = keyPairs[0].multikey; -} -``` - -Replace with: - -```javascript -if (keyPairs.length > 0) { - personOptions.publicKey = keyPairs[0].cryptographicKey; - personOptions.assertionMethods = keyPairs.map((k) => k.multikey); -} -``` - -**Step 2: Verify** - -Run: `fedify lookup https://rmendes.net/activitypub/users/rick` - -Expected: Actor output shows `assertionMethods` (plural) with entries for both RSA and Ed25519 keys. - -**Step 3: Commit** - -``` -fix(actor): use assertionMethods (plural) per Fedify spec - -Exposes all key pair multikeys (RSA + Ed25519) instead of only -the first. Required for proper Object Integrity Proof verification. -``` - ---- - -### Task 3: Add `@fedify/redis` dependency and `redisUrl` config option - -**Files:** -- Modify: `package.json` -- Modify: `index.js` (constructor, init method) - -**Context:** The plugin needs to accept an optional `redisUrl` from Indiekit config and pass it through to federation setup. When not provided, behavior remains unchanged (InProcessMessageQueue). - -**Step 1: Add dependencies** - -```bash -cd /home/rick/code/indiekit-dev/indiekit-endpoint-activitypub -npm install @fedify/redis ioredis -``` - -This adds both packages to `package.json` `dependencies`. - -**Step 2: Add `redisUrl` to defaults in `index.js`** - -Find the `defaults` object (line 32): - -```javascript -const defaults = { - mountPath: "/activitypub", - actor: { - handle: "rick", - name: "", - summary: "", - icon: "", - }, - checked: true, - alsoKnownAs: "", - activityRetentionDays: 90, - storeRawActivities: false, -}; -``` - -Add `redisUrl`: - -```javascript -const defaults = { - mountPath: "/activitypub", - actor: { - handle: "rick", - name: "", - summary: "", - icon: "", - }, - checked: true, - alsoKnownAs: "", - activityRetentionDays: 90, - storeRawActivities: false, - redisUrl: "", -}; -``` - -**Step 3: Pass `redisUrl` to `setupFederation` in `init()`** - -In the `init(Indiekit)` method, find the `setupFederation` call (around line 626): - -```javascript -const { federation } = setupFederation({ - collections: this._collections, - mountPath: this.options.mountPath, - handle: this.options.actor.handle, - storeRawActivities: this.options.storeRawActivities, -}); -``` - -Add `redisUrl`: - -```javascript -const { federation } = setupFederation({ - collections: this._collections, - mountPath: this.options.mountPath, - handle: this.options.actor.handle, - storeRawActivities: this.options.storeRawActivities, - redisUrl: this.options.redisUrl, -}); -``` - -**Step 4: Commit** - -``` -feat(redis): add @fedify/redis dependency and redisUrl config option - -Plugin now accepts optional redisUrl from Indiekit config. -Plumbing only — actual Redis usage is wired in the next commit. -``` - ---- - -### Task 4: Use `RedisMessageQueue` when `redisUrl` is provided - -**Files:** -- Modify: `lib/federation-setup.js` (imports and `createFederation` call) - -**Context:** Replace `InProcessMessageQueue` with `RedisMessageQueue` when Redis is available. The `RedisMessageQueue` constructor takes a factory function `() => new Redis(url)` so Fedify can create connections as needed. - -**Step 1: Update imports and federation creation** - -At the top of `federation-setup.js`, keep `InProcessMessageQueue` in the import (used as fallback) and add a conditional import approach. Replace the `createFederation` block: - -Find: - -```javascript -const federation = createFederation({ - kv: new MongoKvStore(collections.ap_kv), - queue: new InProcessMessageQueue(), -}); -``` - -Replace with: - -```javascript -let queue; -if (redisUrl) { - const { RedisMessageQueue } = await import("@fedify/redis"); - const Redis = (await import("ioredis")).default; - queue = new RedisMessageQueue(() => new Redis(redisUrl)); - console.info("[ActivityPub] Using Redis message queue"); -} else { - queue = new InProcessMessageQueue(); - console.warn( - "[ActivityPub] Using in-process message queue (not recommended for production)", - ); -} - -const federation = createFederation({ - kv: new MongoKvStore(collections.ap_kv), - queue, -}); -``` - -**Step 2: Add `redisUrl` to the destructured options** - -Find: - -```javascript -const { - collections, - mountPath, - handle, - storeRawActivities = false, -} = options; -``` - -Replace with: - -```javascript -const { - collections, - mountPath, - handle, - storeRawActivities = false, - redisUrl = "", -} = options; -``` - -**Step 3: Verify locally** - -Without Redis: Plugin should log the "in-process" warning and work as before. -With Redis: Plugin should log "Using Redis message queue". - -**Step 4: Commit** - -``` -feat(redis): use RedisMessageQueue when redisUrl is configured - -Falls back to InProcessMessageQueue when Redis is not available. -Redis provides persistent, retry-capable delivery that survives -process restarts — critical for reliable federation. -``` - ---- - -### Task 5: Add `Reject` inbox listener - -**Files:** -- Modify: `lib/inbox-listeners.js` - -**Context:** When a remote server rejects our Follow request, the `ap_following` entry stays as `refollow:sent` forever. A `Reject` listener should mark it as rejected and clean up. - -**Step 1: Add `Reject` to imports** - -Find: - -```javascript -import { - Accept, - Add, - Announce, - Block, - Create, - Delete, - Follow, - Like, - Move, - Note, - Remove, - Undo, - Update, -} from "@fedify/fedify"; -``` - -Add `Reject`: - -```javascript -import { - Accept, - Add, - Announce, - Block, - Create, - Delete, - Follow, - Like, - Move, - Note, - Reject, - Remove, - Undo, - Update, -} from "@fedify/fedify"; -``` - -**Step 2: Add the listener after the `Accept` handler** - -After the `.on(Accept, ...)` block (around line 162), add: - -```javascript - .on(Reject, async (ctx, reject) => { - const actorObj = await reject.getActor(); - const actorUrl = actorObj?.id?.href || ""; - if (!actorUrl) return; - - // Mark rejected follow in ap_following - const result = await collections.ap_following.findOneAndUpdate( - { - actorUrl, - source: { $in: ["refollow:sent", "microsub-reader"] }, - }, - { - $set: { - source: "rejected", - rejectedAt: new Date().toISOString(), - }, - }, - { returnDocument: "after" }, - ); - - if (result) { - const actorName = result.name || result.handle || actorUrl; - await logActivity(collections, storeRawActivities, { - direction: "inbound", - type: "Reject(Follow)", - actorUrl, - actorName, - summary: `${actorName} rejected our Follow`, - }); - } - }) -``` - -**Step 3: Verify** - -Check the plugin loads without errors. The Reject handler will activate when a remote server sends a Reject activity in response to a Follow. - -**Step 4: Commit** - -``` -feat(inbox): add Reject listener for rejected Follow requests - -Marks ap_following entries as "rejected" instead of leaving them -stuck in "refollow:sent" state indefinitely. -``` - ---- - -### Task 6: Add database indexes for query performance - -**Files:** -- Modify: `index.js` (in the `init()` method, after the TTL index creation) - -**Context:** With 830+ followers and 2500+ following, unindexed queries on `actorUrl`, `source`, and `objectUrl` are doing full collection scans. MongoDB uses these fields for lookups in every inbox activity handler. - -**Step 1: Add indexes after the existing TTL index block** - -Find the TTL index block in `init()` (around line 612-618): - -```javascript -if (retentionDays > 0) { - this._collections.ap_activities.createIndex( - { receivedAt: 1 }, - { expireAfterSeconds: retentionDays * 86_400 }, - ); -} -``` - -Add immediately after: - -```javascript -// Performance indexes for inbox handlers and batch refollow -this._collections.ap_followers.createIndex( - { actorUrl: 1 }, - { unique: true, background: true }, -); -this._collections.ap_following.createIndex( - { actorUrl: 1 }, - { unique: true, background: true }, -); -this._collections.ap_following.createIndex( - { source: 1 }, - { background: true }, -); -this._collections.ap_activities.createIndex( - { objectUrl: 1 }, - { background: true }, -); -this._collections.ap_activities.createIndex( - { type: 1, actorUrl: 1, objectUrl: 1 }, - { background: true }, -); -``` - -**Step 2: Verify** - -These are idempotent — `createIndex` on an existing index is a no-op. After deploying, verify with: - -```bash -cloudron exec --app rmendes.net -- bash -c 'mongosh "$CLOUDRON_MONGODB_URL" --quiet --eval " - db.ap_followers.getIndexes().forEach(i => print(JSON.stringify(i.key))); - db.ap_following.getIndexes().forEach(i => print(JSON.stringify(i.key))); - db.ap_activities.getIndexes().forEach(i => print(JSON.stringify(i.key))); -"' -``` - -**Step 3: Commit** - -``` -perf(db): add indexes for ap_followers, ap_following, ap_activities - -Prevents collection scans on actorUrl lookups (every inbox -activity), source queries (batch refollow), and objectUrl -deletions (Delete handler). Critical at 830+ followers. -``` - ---- - -### Task 7: Wire up `storeRawActivities` flag in activity logging - -**Files:** -- Modify: `lib/activity-log.js` -- Modify: `lib/inbox-listeners.js` (the local `logActivity` wrapper) - -**Context:** The `storeRawActivities` config option is accepted and threaded through to inbox listeners, but the local `logActivity` wrapper silently ignores it. The `logActivityShared` function in `activity-log.js` doesn't accept a raw JSON parameter either. - -**Step 1: Read the current `activity-log.js`** - -Current file: - -```javascript -export async function logActivity(collection, record) { - await collection.insertOne({ - ...record, - receivedAt: new Date().toISOString(), - }); -} -``` - -**Step 2: Add optional `rawJson` field support** - -Replace: - -```javascript -export async function logActivity(collection, record) { - await collection.insertOne({ - ...record, - receivedAt: new Date().toISOString(), - }); -} -``` - -With: - -```javascript -/** - * Log an activity to the ap_activities collection. - * - * @param {import("mongodb").Collection} collection - * @param {object} record - Activity fields (direction, type, actorUrl, etc.) - * @param {object} [options] - * @param {object} [options.rawJson] - Full raw JSON to store (when storeRawActivities is on) - */ -export async function logActivity(collection, record, options = {}) { - const doc = { - ...record, - receivedAt: new Date().toISOString(), - }; - if (options.rawJson) { - doc.rawJson = options.rawJson; - } - await collection.insertOne(doc); -} -``` - -**Step 3: Update the wrapper in `inbox-listeners.js`** - -Find: - -```javascript -async function logActivity(collections, storeRaw, record) { - await logActivityShared(collections.ap_activities, record); -} -``` - -Replace with: - -```javascript -async function logActivity(collections, storeRaw, record, rawJson) { - await logActivityShared( - collections.ap_activities, - record, - storeRaw && rawJson ? { rawJson } : {}, - ); -} -``` - -**Step 4: Commit** - -``` -fix(log): wire storeRawActivities flag through to activity log - -The config option was accepted but silently ignored. Now passes -raw JSON to the activity log when enabled. -``` - ---- - -### Task 8: Update Cloudron deployment config to pass `redisUrl` - -**Files:** -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/indiekit.config.js.rmendes` (ActivityPub section) -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/indiekit.config.js.template` (ActivityPub section) -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/Dockerfile` (bump plugin version) - -**Context:** The Cloudron container already has `CLOUDRON_REDIS_URL` available. The plugin needs it passed through as `redisUrl` in its config section. - -**Step 1: Update `.rmendes` config** - -In the ActivityPub config block, add `redisUrl`: - -```javascript - "@rmdes/indiekit-endpoint-activitypub": { - mountPath: "/activitypub", - actor: { - handle: "rick", - name: "Ricardo Mendes", - summary: "Personal website of Ricardo Mendes", - icon: "https://rmendes.net/images/user/avatar.jpg", - }, - checked: true, - alsoKnownAs: "", - activityRetentionDays: 90, - storeRawActivities: false, - redisUrl: process.env.CLOUDRON_REDIS_URL || "", - }, -``` - -**Step 2: Update `.template` config** - -Add the same `redisUrl` line to the template's ActivityPub section (if present), or add a full ActivityPub config block. - -**Step 3: Bump plugin version in Dockerfile** - -Update the `npm install` line in the Dockerfile to reference the new version once published. - -**Step 4: Commit (in indiekit-cloudron repo)** - -``` -feat: pass Redis URL to ActivityPub endpoint for persistent queue - -Cloudron provides CLOUDRON_REDIS_URL from the Redis addon. -The ActivityPub plugin uses it for RedisMessageQueue, which -survives process restarts unlike InProcessMessageQueue. -``` - ---- - -### Task 9: Bump plugin version and publish - -**Files:** -- Modify: `package.json` (version field) - -**Step 1: Bump version** - -```bash -cd /home/rick/code/indiekit-dev/indiekit-endpoint-activitypub -# Bump from 1.0.20 to 1.0.21 -``` - -Update `"version": "1.0.21"` in package.json. - -**Step 2: Commit all changes** - -``` -chore: bump version to 1.0.21 -``` - -**Step 3: Push and publish** - -```bash -git push origin main -``` - -Then **STOP — user must run `npm publish`** (requires OTP). - ---- - -### Task 10: Deploy and run federation test suite - -**Step 1: After user confirms publish, update Dockerfile and deploy** - -```bash -cd /home/rick/code/indiekit-dev/indiekit-cloudron -cloudron build --no-cache && cloudron update --app rmendes.net --no-backup -``` - -**Step 2: Verify Redis is connected** - -```bash -cloudron logs -f --app rmendes.net | grep -i "redis\|message queue" -``` - -Expected: `[ActivityPub] Using Redis message queue` - -**Step 3: Run the test suite** - -```bash -cd /home/rick/code/indiekit-dev/activitypub-tests -./run-all.sh -``` - -Expected: 12/12 passing. - -**Step 4: Verify Ed25519 key persistence** - -```bash -cloudron exec --app rmendes.net -- bash -c 'mongosh "$CLOUDRON_MONGODB_URL" --quiet --eval " - db.ap_keys.find({ type: \"ed25519\" }).toArray() -"' -``` - -Expected: One document with `publicKeyJwk` and `privateKeyJwk` fields. - -**Step 5: Verify indexes** - -```bash -cloudron exec --app rmendes.net -- bash -c 'mongosh "$CLOUDRON_MONGODB_URL" --quiet --eval " - print(\"ap_followers indexes:\"); - db.ap_followers.getIndexes().forEach(i => print(\" \" + JSON.stringify(i.key))); - print(\"ap_following indexes:\"); - db.ap_following.getIndexes().forEach(i => print(\" \" + JSON.stringify(i.key))); - print(\"ap_activities indexes:\"); - db.ap_activities.getIndexes().forEach(i => print(\" \" + JSON.stringify(i.key))); -"' -``` - -Expected: Indexes on `actorUrl`, `source`, `objectUrl`, and the compound `{ type, actorUrl, objectUrl }`. diff --git a/docs/plans/2026-02-20-fedify-feature-completeness.md b/docs/plans/2026-02-20-fedify-feature-completeness.md deleted file mode 100644 index 8846149..0000000 --- a/docs/plans/2026-02-20-fedify-feature-completeness.md +++ /dev/null @@ -1,650 +0,0 @@ -# Fedify Feature Completeness Implementation Plan - -Created: 2026-02-20 -Status: COMPLETE -Approved: Yes -Iterations: 0 -Worktree: No - -> **Status Lifecycle:** PENDING → COMPLETE → VERIFIED -> **Iterations:** Tracks implement→verify cycles (incremented by verify phase) -> -> - PENDING: Initial state, awaiting implementation -> - COMPLETE: All tasks implemented -> - VERIFIED: All checks passed -> -> **Approval Gate:** Implementation CANNOT proceed until `Approved: Yes` -> **Worktree:** Set at plan creation (from dispatcher). `Yes` uses git worktree isolation; `No` works directly on current branch (default) - -## Summary - -**Goal:** Implement all missing Fedify features in the `@rmdes/indiekit-endpoint-activitypub` plugin to achieve near-complete Fedify API coverage — delivery reliability (permanent failure handling, ordering keys, collection sync), content resolution (object dispatcher), standard collections (liked, featured, featured tags), access control (authorized fetch, instance actor), and quality-of-life improvements (dynamic NodeInfo, parallel queue, handle aliases). - -**Architecture:** All changes are additive to the existing `federation-setup.js` architecture. New dispatchers/handlers are registered on the same `federation` instance returned by `createFederation()`. New MongoDB collections are added via `Indiekit.addCollection()` in `index.js`. Admin UI views follow the existing Nunjucks template pattern in `views/`. The plugin's syndicator, inbox listeners, and existing collection dispatchers remain unchanged. - -**Tech Stack:** Fedify ^1.10.0, @fedify/redis ^1.10.3, MongoDB, Express 5, Nunjucks templates, ioredis - -## Scope - -### In Scope - -- Permanent failure handler for dead inboxes -- Followers collection sync (FEP-8fcf) on sendActivity -- Ordering keys on all sendActivity calls -- Object dispatcher for Note/Article resolution -- Liked collection dispatcher -- Featured (pinned) collection dispatcher + admin UI -- Featured tags collection dispatcher + admin UI -- Instance actor (Application type) for domain-level federation -- Authorized fetch with admin toggle -- ParallelMessageQueue wrapper -- Dynamic NodeInfo version from package.json -- Handle aliases via mapAlias -- Configurable actor type (Person/Service/Application) -- Context data propagation (handle + publicationUrl) -- Investigate and fix syndication delivery issues - -### Out of Scope - -- Custom collections API (extensible plugin system for registering arbitrary collections) -- Relay support (FEP-ae0c) -- Key rotation admin UI (deferred — keys are already dual RSA+Ed25519) -- Custom WebFinger links -- Separate inbox/outbox queue configuration - -## Prerequisites - -- Plugin repo at `/home/rick/code/indiekit-dev/indiekit-endpoint-activitypub` -- Cloudron deployment at `/home/rick/code/indiekit-dev/indiekit-cloudron` -- Fedify docs at `/home/rick/code/fedify/docs/manual/` -- Test suite at `/home/rick/code/indiekit-dev/activitypub-tests/` -- Node.js 22, MongoDB, Redis available on Cloudron - -## Context for Implementer - -> This section is critical for cross-session continuity. Write it for an implementer who has never seen the codebase. - -- **Patterns to follow:** All Fedify dispatchers are registered in `lib/federation-setup.js` using the pattern `federation.setXxxDispatcher(urlPattern, callback)` with `.setCounter()` and `.setFirstCursor()` chained (see `setupFollowers` at line 285, `setupOutbox` at line 345). -- **Conventions:** ESM modules (`"type": "module"`), no build step. MongoDB collections registered via `Indiekit.addCollection("name")` in `index.js:init()`. Dates stored as ISO strings. Admin views are Nunjucks templates in `views/` rendered by controllers in `lib/controllers/`. -- **Key files:** - - `index.js` — Plugin entry point, constructor defaults, `init()`, syndicator, routes - - `lib/federation-setup.js` — All Fedify configuration (actor, inbox, collections, NodeInfo) - - `lib/inbox-listeners.js` — Inbox activity handlers (Follow, Undo, Like, etc.) - - `lib/jf2-to-as2.js` — Converts Indiekit JF2 posts to ActivityStreams objects - - `lib/activity-log.js` — Logs activities to `ap_activities` collection - - `lib/kv-store.js` — MongoDB-backed KvStore for Fedify - - `package.json` — Dependencies, version (currently 1.0.21) -- **Gotchas:** - - `init()` is synchronous — cannot use `await`. All async work (profile seeding, batch refollow) uses `.catch()` or `setTimeout()`. - - The syndicator's `syndicate(properties)` returns the post URL on success or `undefined` on skip/failure. - - `createFederation()` options like `onOutboxError` and `permanentFailureStatusCodes` are set at creation time, not after. - - The `setOutboxPermanentFailureHandler()` is called on the federation instance AFTER creation. - - `ctx.sendActivity()` options: `{ preferSharedInbox, syncCollection, orderingKey }`. -- **Domain context:** Indiekit stores posts as JF2 objects in MongoDB `posts` collection with `properties.url`, `properties.published`, `properties["post-type"]`, etc. The `jf2ToAS2Activity()` function converts these to Fedify Activity objects. - -## Runtime Environment - -- **Start command:** `node --import @indiekit/indiekit/register index.js` (via Cloudron start.sh) -- **Port:** 8080 (Indiekit), 3000 (nginx) -- **Deploy path:** Cloudron at rmendes.net -- **Health check:** `curl -s https://rmendes.net/.well-known/webfinger?resource=acct:rick@rmendes.net` -- **Restart procedure:** `cloudron build --no-cache && cloudron update --app rmendes.net --no-backup` - -## Progress Tracking - -**MANDATORY: Update this checklist as tasks complete. Change `[ ]` to `[x]`.** - -- [x] Task 1: Investigate and fix syndication delivery -- [x] Task 2: Permanent failure handler (SKIPPED — requires Fedify 2.0+) -- [x] Task 3: Ordering keys on sendActivity -- [x] Task 4: Followers collection sync (FEP-8fcf) -- [x] Task 5: Dynamic NodeInfo version -- [x] Task 6: Context data propagation -- [x] Task 7: Object dispatcher for Note/Article -- [x] Task 8: Liked collection -- [x] Task 9: Featured (pinned) collection + admin UI -- [x] Task 10: Featured tags collection + admin UI -- [x] Task 11: Instance actor -- [x] Task 12: Authorized fetch with admin toggle -- [x] Task 13: ParallelMessageQueue -- [x] Task 14: Handle aliases (mapAlias) -- [x] Task 15: Configurable actor type -- [x] Task 16: New test scripts -- [x] Task 17: Version bump + Cloudron config update - -**Total Tasks:** 17 | **Completed:** 17 | **Remaining:** 0 - -## Implementation Tasks - -### Task 1: Investigate and fix syndication delivery - -**Objective:** The user reports new posts are not appearing on their fediverse profile despite syndication logs showing "Sent Create to 830 followers". Investigate whether the issue is delivery (queue not processing), content resolution (remote servers can't fetch the post), or something else entirely. - -**Dependencies:** None - -**Files:** -- Investigate: `lib/federation-setup.js` (queue startup, sendActivity) -- Investigate: `index.js` (syndicator.syndicate method) -- Investigate: `lib/jf2-to-as2.js` (activity structure) - -**Key Decisions / Notes:** -- Check if `federation.startQueue()` is working correctly -- Check if the activities in the Redis queue are actually being processed (not just enqueued) -- Test whether a remote Mastodon instance can fetch the content negotiation response and display it -- Check if the `Create` activity wrapping is correct (Mastodon requires `Create(Note)`, not bare `Note`) -- Use `fedify lookup` CLI to test if the actor and posts resolve correctly from external perspective - -**Definition of Done:** -- [ ] Root cause of syndication failure identified -- [ ] Fix implemented (if code change needed) -- [ ] Verified a new post appears on fediverse profile after syndication - -**Verify:** -- `fedify lookup acct:rick@rmendes.net` — actor resolves with outbox -- `curl -s -H "Accept: application/activity+json" https://rmendes.net/notes/YYYY/MM/DD/slug` — returns valid AS2 Note -- Check Mastodon instance shows the post - ---- - -### Task 2: Permanent failure handler - -**Objective:** Register `setOutboxPermanentFailureHandler()` on the federation instance to automatically remove followers whose inboxes return 404/410. Log these events to `ap_activities`. - -**Dependencies:** None - -**Files:** -- Modify: `lib/federation-setup.js` — add handler after `createFederation()` -- Modify: `lib/federation-setup.js` — add `permanentFailureStatusCodes: [404, 410, 451]` to createFederation options - -**Key Decisions / Notes:** -- The handler receives `{ inbox, activity, error, statusCode, actorIds }` via `values` parameter -- `values.actorIds` are URL objects — match against `ap_followers.actorUrl` (string comparison with `.href`) -- When `preferSharedInbox: true`, one inbox may represent multiple followers -- Log each removal as an activity with type "PermanentFailure" -- Don't throw errors in the handler (Fedify catches and ignores them anyway) - -**Definition of Done:** -- [ ] `setOutboxPermanentFailureHandler` registered on federation instance -- [ ] Dead followers removed from `ap_followers` when inbox returns 404/410/451 -- [ ] Permanent failures logged to `ap_activities` with direction "system" -- [ ] `permanentFailureStatusCodes` set to `[404, 410, 451]` - -**Verify:** -- Check that `permanentFailureStatusCodes` appears in `createFederation()` call -- Check that `setOutboxPermanentFailureHandler` is called on the federation instance -- `node -e "import('./lib/federation-setup.js')"` — no import errors - ---- - -### Task 3: Ordering keys on sendActivity - -**Objective:** Add `orderingKey` to all `ctx.sendActivity()` calls so that related activities (Create→Update→Delete for the same post) are delivered in order per recipient server. - -**Dependencies:** None - -**Files:** -- Modify: `index.js` — syndicator.syndicate method (line ~331 and ~341) -- Modify: `index.js` — followActor method (line ~426) -- Modify: `index.js` — unfollowActor method (line ~534) - -**Key Decisions / Notes:** -- For post syndication: `orderingKey: properties.url` — ensures Create/Update/Delete for same post are ordered -- For follow/unfollow: `orderingKey: actorUrl` — ensures Follow then Undo(Follow) arrive in order -- Don't add ordering keys for unrelated activities (reduces parallelism) -- The ordering key is per-recipient-server — two servers can receive in parallel - -**Definition of Done:** -- [ ] `orderingKey` added to sendActivity in syndicator.syndicate (both followers and reply-to-author calls) -- [ ] `orderingKey` added to sendActivity in followActor -- [ ] `orderingKey` added to sendActivity in unfollowActor - -**Verify:** -- Grep for `sendActivity` — every call should have `orderingKey` in its options -- `node -e "import('./index.js')"` — no import errors - ---- - -### Task 4: Followers collection sync (FEP-8fcf) - -**Objective:** Add `preferSharedInbox: true` and `syncCollection: true` to the `sendActivity` call that sends to `"followers"` in the syndicator. - -**Dependencies:** None - -**Files:** -- Modify: `index.js` — syndicator.syndicate method, the `ctx.sendActivity({ identifier: handle }, "followers", activity)` call (~line 331) - -**Key Decisions / Notes:** -- `syncCollection: true` is ONLY valid when recipients is `"followers"` (string) -- `preferSharedInbox: true` consolidates delivery to shared inboxes (more efficient) -- Fedify automatically includes a followers collection digest in the delivery payload -- This implements FEP-8fcf for Mastodon-compatible servers - -**Definition of Done:** -- [ ] `sendActivity` to `"followers"` includes `{ preferSharedInbox: true, syncCollection: true }` -- [ ] Other `sendActivity` calls (to specific actors) do NOT include `syncCollection` - -**Verify:** -- Read the syndicate method and confirm the options are present on the followers call only - ---- - -### Task 5: Dynamic NodeInfo version - -**Objective:** Read the actual Indiekit version from `@indiekit/indiekit` package.json instead of hardcoding `{ major: 1, minor: 0, patch: 0 }`. - -**Dependencies:** None - -**Files:** -- Modify: `lib/federation-setup.js` — NodeInfo dispatcher (~line 253) - -**Key Decisions / Notes:** -- Use `import { createRequire } from "node:module"` and `createRequire(import.meta.url)` to resolve `@indiekit/indiekit/package.json` -- Or use a simpler approach: read the version from `@indiekit/indiekit` package.json at module load time -- Parse semver string "1.0.0-beta.25" → `{ major: 1, minor: 0, patch: 0 }` (ignore prerelease) -- Fallback to `{ major: 1, minor: 0, patch: 0 }` if resolution fails - -**Definition of Done:** -- [ ] NodeInfo dispatcher reads version from @indiekit/indiekit package.json -- [ ] Version parsed correctly into `{ major, minor, patch }` format -- [ ] Fallback to 1.0.0 if package.json can't be found - -**Verify:** -- `curl -s https://rmendes.net/nodeinfo/2.1 | jq .software.version` — should return actual version -- Test script: `tests/02-nodeinfo.sh` passes - ---- - -### Task 6: Context data propagation - -**Objective:** Pass `handle` and `publicationUrl` in the Fedify context data instead of using closures, so dispatchers and handlers have cleaner access to these values. - -**Dependencies:** None - -**Files:** -- Modify: `lib/federation-setup.js` — `createFederation()` and all `createContext()` calls -- Modify: `index.js` — all `this._federation.createContext()` calls in syndicator, followActor, unfollowActor - -**Key Decisions / Notes:** -- Currently `createContext(new URL(publicationUrl), {})` passes empty context -- Change to `createContext(new URL(publicationUrl), { handle, publicationUrl })` -- This doesn't change any behavior — it just makes the data available via `ctx.data` in dispatchers -- Future tasks (object dispatcher, collections) will use `ctx.data.handle` and `ctx.data.publicationUrl` - -**Definition of Done:** -- [ ] `createContext()` calls pass `{ handle, publicationUrl }` as context data -- [ ] No behavioral changes — existing functionality preserved - -**Verify:** -- `node -e "import('./index.js')"` — no import errors -- Existing test suite passes - ---- - -### Task 7: Object dispatcher for Note/Article - -**Objective:** Register `setObjectDispatcher()` for `Note` and `Article` types so individual posts are dereferenceable at proper Fedify-managed URIs. This is critical for remote servers to properly display shared content. - -**Dependencies:** Task 6 (context data propagation) - -**Files:** -- Modify: `lib/federation-setup.js` — add `setupObjectDispatchers()` function and call it -- Read: `lib/jf2-to-as2.js` — understand how posts are converted to AS2 - -**Key Decisions / Notes:** -- Register `federation.setObjectDispatcher(Note, ...)` and `federation.setObjectDispatcher(Article, ...)` -- URL pattern: `${mountPath}/objects/note/{id}` and `${mountPath}/objects/article/{id}` -- The `{id}` maps to the post's URL slug (e.g., `notes/2026/02/20/52ef4`) -- Dispatcher looks up the post in MongoDB `posts` collection by URL and converts to AS2 -- Use `{+id}` (reserved expansion) since IDs contain slashes -- This complements the existing content-negotiation route — Fedify handles proper ActivityPub discovery while content negotiation handles direct URL requests - -**Definition of Done:** -- [ ] `setObjectDispatcher(Note, ...)` registered for note/reply/like/repost/bookmark/jam/rsvp posts -- [ ] `setObjectDispatcher(Article, ...)` registered for article posts -- [ ] Dispatcher looks up post in MongoDB and returns proper Fedify Note/Article object -- [ ] Actor dispatcher references `liked` and `featured` URIs (prepared for later tasks) - -**Verify:** -- `curl -s -H "Accept: application/activity+json" "https://rmendes.net/activitypub/objects/note/notes%2F2026%2F02%2F20%2F52ef4"` — returns AS2 Note -- Existing content negotiation still works - ---- - -### Task 8: Liked collection - -**Objective:** Expose a `liked` collection showing objects the actor has liked. Query the MongoDB `posts` collection for `post-type: "like"` posts and return their `like-of` URLs. - -**Dependencies:** Task 6 - -**Files:** -- Modify: `lib/federation-setup.js` — add `setupLiked()` function -- Modify: `lib/federation-setup.js` — call `setupLiked()` in `setupFederation()` -- Modify: actor dispatcher to include `liked: ctx.getLikedUri(identifier)` - -**Key Decisions / Notes:** -- Pattern: `${mountPath}/users/{identifier}/liked` -- Query: `collections.posts.find({ "properties.post-type": "like" })` sorted by published desc -- Return items as URLs: `new URL(post.properties["like-of"])` for each like post -- Include `.setCounter()` and `.setFirstCursor()` for pagination (same pattern as followers) -- Add `liked` property to the Person actor options - -**Definition of Done:** -- [ ] `setLikedDispatcher` registered with pagination -- [ ] Actor includes `liked` URI in Person properties -- [ ] Collection returns liked object URLs - -**Verify:** -- `curl -s -H "Accept: application/activity+json" "https://rmendes.net/activitypub/users/rick/liked"` — returns OrderedCollection -- New test script added - ---- - -### Task 9: Featured (pinned) collection + admin UI - -**Objective:** Expose a `featured` collection for pinned posts and add an admin UI to manage them. Store pinned post URLs in a new `ap_featured` MongoDB collection. - -**Dependencies:** Task 6 - -**Files:** -- Modify: `lib/federation-setup.js` — add `setupFeatured()` function -- Modify: `index.js` — register `ap_featured` collection, add admin routes -- Create: `lib/controllers/featured.js` — admin controller for pin/unpin -- Create: `views/featured.njk` — admin template for managing pinned posts -- Modify: actor dispatcher to include `featured: ctx.getFeaturedUri(identifier)` - -**Key Decisions / Notes:** -- New collection `ap_featured` stores `{ postUrl, pinnedAt }` documents -- Pattern: `${mountPath}/users/{identifier}/featured` -- Dispatcher returns the full Note/Article objects (not just URLs) — Mastodon expects objects -- Admin UI at `/activitypub/admin/featured` — list pinned posts, pin/unpin buttons -- Pin limit: 5 posts max (Mastodon convention) -- On pin: look up post in `posts` collection, convert to AS2 Note/Article, store URL in `ap_featured` -- On unpin: remove from `ap_featured` - -**Definition of Done:** -- [ ] `setFeaturedDispatcher` registered, returns AS2 objects for pinned posts -- [ ] `ap_featured` MongoDB collection registered -- [ ] Admin UI at `/activitypub/admin/featured` to manage pins -- [ ] Actor includes `featured` URI in Person properties -- [ ] Pin limit of 5 enforced - -**Verify:** -- `curl -s -H "Accept: application/activity+json" "https://rmendes.net/activitypub/users/rick/featured"` — returns OrderedCollection -- Admin UI renders at `/activitypub/admin/featured` - ---- - -### Task 10: Featured tags collection + admin UI - -**Objective:** Expose a `featured tags` collection for hashtags the actor wants to highlight. Store them in a new `ap_featured_tags` MongoDB collection with an admin UI. - -**Dependencies:** Task 6 - -**Files:** -- Modify: `lib/federation-setup.js` — add `setupFeaturedTags()` function -- Modify: `index.js` — register `ap_featured_tags` collection, add admin routes -- Create: `lib/controllers/featured-tags.js` — admin controller -- Create: `views/featured-tags.njk` — admin template -- Modify: actor dispatcher to include `featuredTags: ctx.getFeaturedTagsUri(identifier)` - -**Key Decisions / Notes:** -- New collection `ap_featured_tags` stores `{ tag, addedAt }` documents -- Pattern: `${mountPath}/users/{identifier}/tags` -- Dispatcher returns `Hashtag` objects with `name` (`#tag`) and `href` (link to tag page) -- Import `Hashtag` from `@fedify/fedify` -- Admin UI at `/activitypub/admin/tags` — add/remove featured tags -- Tag href: `${publicationUrl}categories/${encodeURIComponent(tag)}` - -**Definition of Done:** -- [ ] `setFeaturedTagsDispatcher` registered, returns Hashtag objects -- [ ] `ap_featured_tags` MongoDB collection registered -- [ ] Admin UI at `/activitypub/admin/tags` to manage featured tags -- [ ] Actor includes `featuredTags` URI - -**Verify:** -- `curl -s -H "Accept: application/activity+json" "https://rmendes.net/activitypub/users/rick/tags"` — returns Collection of Hashtags -- Admin UI renders - ---- - -### Task 11: Instance actor - -**Objective:** Create an Application-type instance actor (`rmendes.net@rmendes.net`) that represents the domain itself. This is required for authorized fetch to work without infinite loops. - -**Dependencies:** Task 6 - -**Files:** -- Modify: `lib/federation-setup.js` — extend actor dispatcher to handle instance actor identifier -- Modify: `lib/federation-setup.js` — extend key pairs dispatcher for instance actor -- Modify: `lib/federation-setup.js` — extend `mapHandle` to accept hostname - -**Key Decisions / Notes:** -- When `identifier === ctx.hostname` (e.g., "rmendes.net"), return an `Application` actor -- Import `Application` from `@fedify/fedify` -- Instance actor uses the same RSA+Ed25519 key pairs as the main actor (simplicity) -- Instance actor properties: `id`, `preferredUsername: hostname`, `inbox`, `outbox` (empty) -- `mapHandle` returns hostname when username matches hostname -- The instance actor does NOT need followers/following/liked/featured collections - -**Definition of Done:** -- [ ] Actor dispatcher returns `Application` when identifier is hostname -- [ ] Key pairs dispatcher returns keys for hostname identifier -- [ ] `mapHandle` accepts hostname as valid username -- [ ] Instance actor resolves via WebFinger: `acct:rmendes.net@rmendes.net` - -**Verify:** -- `fedify lookup acct:rmendes.net@rmendes.net` — returns Application actor -- `curl -s -H "Accept: application/activity+json" "https://rmendes.net/activitypub/users/rmendes.net"` — returns Application - ---- - -### Task 12: Authorized fetch with admin toggle - -**Objective:** Add optional authorized fetch support via `.authorize()` predicates on the actor and collection dispatchers. Controlled by a config option and admin toggle. - -**Dependencies:** Task 11 (instance actor needed to prevent infinite loops) - -**Files:** -- Modify: `lib/federation-setup.js` — add `.authorize()` to actor, collections -- Modify: `index.js` — add `authorizedFetch: false` to defaults, pass to setupFederation -- Modify: `views/dashboard.njk` or `views/profile.njk` — add toggle (or use ap_profile) - -**Key Decisions / Notes:** -- When `authorizedFetch` is enabled: - - Actor dispatcher: `.authorize()` returns true for instance actor, checks signed key for others - - Collections: `.authorize()` same logic -- When `authorizedFetch` is disabled (default): don't chain `.authorize()` at all -- Store setting in `ap_profile.authorizedFetch` (boolean) -- The `authorize` predicate: `ctx.getSignedKeyOwner({ documentLoader: await ctx.getDocumentLoader({ identifier: ctx.hostname }) })` -- Instance actor is always accessible without auth (prevents infinite loops) - -**Definition of Done:** -- [ ] `.authorize()` chained on actor and collection dispatchers when enabled -- [ ] Instance actor always returns true (no auth required) -- [ ] Config option `authorizedFetch` defaults to false -- [ ] Setting stored in ap_profile for runtime toggle - -**Verify:** -- With `authorizedFetch: false` — unsigned GET requests work normally -- Check that `.authorize()` is conditionally applied - ---- - -### Task 13: ParallelMessageQueue - -**Objective:** Wrap the Redis message queue with `ParallelMessageQueue` for concurrent activity processing. Add a config option for the number of workers. - -**Dependencies:** None - -**Files:** -- Modify: `lib/federation-setup.js` — wrap queue with ParallelMessageQueue -- Modify: `index.js` — add `parallelWorkers: 5` to defaults - -**Key Decisions / Notes:** -- Import `ParallelMessageQueue` from `@fedify/fedify` -- When `redisUrl` is set AND `parallelWorkers > 1`: wrap `RedisMessageQueue` with `ParallelMessageQueue` -- When `parallelWorkers <= 1` or no Redis: don't wrap (single worker) -- Default: 5 workers (good balance for ~800 followers) -- `ParallelMessageQueue` inherits `nativeRetrial` from the wrapped queue - -**Definition of Done:** -- [ ] `ParallelMessageQueue` wraps Redis queue when parallelWorkers > 1 -- [ ] Config option `parallelWorkers` with default 5 -- [ ] InProcessMessageQueue is NOT wrapped (only used in dev) - -**Verify:** -- Console log shows "Using Redis message queue with 5 parallel workers" -- `node -e "import('./lib/federation-setup.js')"` — no import errors - ---- - -### Task 14: Handle aliases (mapAlias) - -**Objective:** Register `mapAlias()` so that the actor's profile URL and common alias patterns resolve via WebFinger. - -**Dependencies:** None - -**Files:** -- Modify: `lib/federation-setup.js` — chain `.mapAlias()` on actor dispatcher - -**Key Decisions / Notes:** -- When someone queries WebFinger for the profile URL (e.g., `https://rmendes.net/`), resolve to the actor -- Pattern: check if `resource.hostname` matches and `resource.pathname` is `/` or `/@handle` -- Return `{ identifier: handle }` for matching URLs -- This allows `https://rmendes.net/` to be discoverable via WebFinger alongside `acct:rick@rmendes.net` - -**Definition of Done:** -- [ ] `mapAlias()` registered on actor dispatcher -- [ ] Profile URL resolves via WebFinger -- [ ] `/@handle` pattern resolves via WebFinger - -**Verify:** -- `curl -s "https://rmendes.net/.well-known/webfinger?resource=https://rmendes.net/"` — returns actor link -- `curl -s "https://rmendes.net/.well-known/webfinger?resource=https://rmendes.net/@rick"` — returns actor link - ---- - -### Task 15: Configurable actor type - -**Objective:** Add a config option to choose the actor type (Person, Service, Application) instead of hardcoding Person. - -**Dependencies:** Task 11 (instance actor uses Application) - -**Files:** -- Modify: `lib/federation-setup.js` — use config-based actor class -- Modify: `index.js` — add `actorType: "Person"` to defaults - -**Key Decisions / Notes:** -- Import `Person`, `Service`, `Application`, `Organization`, `Group` from `@fedify/fedify` -- Map string config to class: `{ Person, Service, Application, Organization, Group }` -- Instance actor always uses `Application` regardless of config -- Default: "Person" (most common for individual blogs) - -**Definition of Done:** -- [ ] Config option `actorType` with default "Person" -- [ ] Actor dispatcher uses configured type class -- [ ] Instance actor always uses Application - -**Verify:** -- Actor endpoint returns `type: "Person"` by default -- Config change to "Service" would change the type field - ---- - -### Task 16: New test scripts - -**Objective:** Add test scripts for the new features: liked collection, featured collection, featured tags, instance actor, and handle aliases. - -**Dependencies:** Tasks 7-14 - -**Files:** -- Create: `activitypub-tests/tests/13-liked.sh` -- Create: `activitypub-tests/tests/14-featured.sh` -- Create: `activitypub-tests/tests/15-featured-tags.sh` -- Create: `activitypub-tests/tests/16-instance-actor.sh` -- Create: `activitypub-tests/tests/17-object-dispatcher.sh` -- Create: `activitypub-tests/tests/18-webfinger-alias.sh` - -**Key Decisions / Notes:** -- Follow existing test pattern in `tests/common.sh` (BASE_URL, curl, jq assertions) -- Each test verifies the HTTP response and JSON structure -- Liked: GET liked collection, verify OrderedCollection type -- Featured: GET featured collection, verify OrderedCollection type -- Featured tags: GET tags collection, verify items are Hashtag type -- Instance actor: WebFinger + actor endpoint for hostname identifier -- Object dispatcher: GET object URI, verify Note/Article type -- WebFinger alias: query WebFinger with profile URL - -**Definition of Done:** -- [ ] All 6 test scripts created and passing -- [ ] `run-all.sh` updated to include new tests - -**Verify:** -- `cd activitypub-tests && bash run-all.sh` — all tests pass - ---- - -### Task 17: Version bump + Cloudron config update - -**Objective:** Bump plugin version, update Cloudron Dockerfile and config files, prepare for deployment. - -**Dependencies:** All previous tasks - -**Files:** -- Modify: `package.json` — bump version to 1.0.22 -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/Dockerfile` — update version -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/indiekit.config.js.template` — add new config options -- Modify: `/home/rick/code/indiekit-dev/indiekit-cloudron/indiekit.config.js.rmendes` — add new config options - -**Key Decisions / Notes:** -- New config options to add: `authorizedFetch`, `parallelWorkers`, `actorType` -- Register new collections `ap_featured` and `ap_featured_tags` (automatic via plugin init) -- Bump CACHE_BUST in Dockerfile - -**Definition of Done:** -- [ ] Version bumped to 1.0.22 -- [ ] Dockerfile references @1.0.22 -- [ ] Config templates updated with new options -- [ ] CACHE_BUST incremented - -**Verify:** -- `jq .version package.json` — returns "1.0.22" -- `grep "1.0.22" /home/rick/code/indiekit-dev/indiekit-cloudron/Dockerfile` — found - ---- - -## Testing Strategy - -- **Shell tests:** Extend existing `activitypub-tests/` suite with 6 new tests (Task 16) -- **Integration testing:** Deploy to Cloudron and verify: - - All existing 12 tests still pass - - New 6 tests pass - - `fedify lookup` resolves actor, instance actor, and post objects - - Posts syndicated after deploy appear on fediverse (Mastodon search) -- **Manual verification:** - - Admin UI pages render for featured posts, featured tags - - Pin/unpin a post and verify it appears in featured collection - - Add/remove a tag and verify featured tags collection - -## Risks and Mitigations - -| Risk | Likelihood | Impact | Mitigation | -|------|-----------|--------|------------| -| Fedify API version incompatibility (features require newer Fedify) | Low | High | Check Fedify version in package.json (^1.10.0) — most features available since 0.7-1.0. ParallelMessageQueue since 1.0. Permanent failure handler since 2.0 — verify this is available | -| Breaking existing federation by changing actor properties | Medium | High | Test with `fedify lookup` before and after. Ensure `publicKey` and `assertionMethods` unchanged. Instance actor uses separate identifier | -| MongoDB index conflicts with existing collections | Low | Medium | Use `createIndex` with `background: true` and catch errors | -| Redis queue wrapper breaking delivery | Low | High | Only wrap when `parallelWorkers > 1` and Redis is configured. Fallback to unwrapped queue if ParallelMessageQueue import fails | -| Authorized fetch blocking legitimate unsigned requests | Medium | Medium | Default to `authorizedFetch: false`. Only enable when explicitly configured. Instance actor always allows unsigned access | - -## Open Questions - -- Is `setOutboxPermanentFailureHandler` available in Fedify ^1.10.0? The docs say "since 2.0.0" — if the installed version is <2.0, we'll need to skip this feature or use `onOutboxError` as a fallback. -- The `ParallelMessageQueue` docs say "since 1.0.0" which should be fine with ^1.10.0. - -### Deferred Ideas - -- Custom collections API for other plugins to register arbitrary collections -- Relay support (FEP-ae0c) for large-scale content distribution -- Key rotation admin UI with grace period -- Activity transformers for custom pre-send validation -- Separate inbox/outbox queue configuration for different processing priorities diff --git a/docs/plans/2026-02-21-activitypub-reader.md b/docs/plans/2026-02-21-activitypub-reader.md deleted file mode 100644 index 8bde8eb..0000000 --- a/docs/plans/2026-02-21-activitypub-reader.md +++ /dev/null @@ -1,882 +0,0 @@ -# ActivityPub Reader Implementation Plan - -Created: 2026-02-21 -Status: VERIFIED -Approved: Yes -Iterations: 0 -Worktree: No - -> **Status Lifecycle:** PENDING → COMPLETE → VERIFIED -> **Iterations:** Tracks implement→verify cycles (incremented by verify phase) -> -> - PENDING: Initial state, awaiting implementation -> - COMPLETE: All tasks implemented -> - VERIFIED: All checks passed -> -> **Approval Gate:** Implementation CANNOT proceed until `Approved: Yes` -> **Worktree:** Set at plan creation (from dispatcher). `Yes` uses git worktree isolation; `No` works directly on current branch - -## Summary - -**Goal:** Build a dedicated ActivityPub reader within the `@rmdes/indiekit-endpoint-activitypub` plugin, providing a timeline view of followed accounts' posts, a notifications stream, native AP interactions (like, boost, reply, follow/unfollow), and Micropub-based content creation — then remove the Microsub bridge dependency. - -**Architecture:** The reader adds new MongoDB collections (`ap_timeline`, `ap_notifications`, `ap_muted`, `ap_blocked`) alongside new controllers, views, and a CSS stylesheet. Inbox listeners are refactored to store items natively instead of bridging to Microsub. Alpine.js provides client-side reactivity for interactions. Content creation uses two paths: direct Fedify `ctx.sendActivity()` for quick likes/boosts, and Micropub POST for replies that become blog posts (user chooses per-reply). - -**Tech Stack:** Node.js/Express, MongoDB, Nunjucks templates, Alpine.js, Fedify SDK (`ctx.sendActivity()`, `ctx.lookupObject()`), Indiekit frontend components, CSS custom properties. - -## Scope - -### In Scope - -- Timeline view showing posts from followed accounts with threading, content warnings, boosts, and rich media (images, video, audio, polls) -- Tab-based filtering (All, Notes, Articles, Replies, Boosts, Media) -- Notifications stream (likes, boosts, follows, mentions, replies received) -- Native AP interactions: like, boost, reply (with choice of direct AP or Micropub), follow/unfollow -- Mute/unmute (accounts and keywords), block/unblock -- Profile view for remote actors (view posts, follow/unfollow, mute, block) -- Compose form that submits via Micropub endpoint (for blog-worthy replies) -- Custom CSS stylesheet with card-based layout inspired by Phanpy/Elk -- Content warning spoiler toggle (Alpine.js) -- Image gallery grid for multi-image posts -- Video/audio embed rendering -- Removal of Microsub bridge (`storeTimelineItem`, `getApChannelId`, lazy `microsub_items`/`microsub_channels` accessors) - -### Out of Scope - -- Mastodon REST API compatibility (no mobile client support — would be a separate project) -- Lists (organizing follows into named groups) — deferred to future plan -- Local/Federated timeline distinction (single timeline of followed accounts only) -- Full-text search within timeline items -- Polls (rendering existing polls is in scope; creating polls is not) -- Direct messages / conversations -- Push notifications (browser notifications) -- Infinite scroll (standard pagination is used) -- Video/audio upload in compose form - -## Prerequisites - -- Plugin is at v1.0.29+ with all federation hardening features complete -- Fedify SDK available via `this._federation` on the plugin instance -- MongoDB collections infrastructure in `index.js` -- Indiekit frontend components available (`@indiekit/frontend`) -- Alpine.js: **NOT loaded by Indiekit core**. The reader layout must explicitly load Alpine.js via a ``). The existing AP dashboard views use `x-data` directives — they work because the Cloudron deployment's CSP allows `cdn.jsdelivr.net` (see `nginx.conf`). The reader layout template must include Alpine.js in its `` block. -- `sanitize-html` package (add to `package.json` dependencies — used by Microsub plugin already, needed here for XSS prevention on remote content) - -## Context for Implementer - -> This section is critical for cross-session continuity. Write it for an implementer who has never seen the codebase. - -- **Patterns to follow:** - - Route registration: See `index.js:143-169` — admin routes go in `get routes()` method, registered at `/admin/activitypub/*` - - Controller pattern: Each controller exports async functions taking `(request, response)`. See `lib/controllers/dashboard.js` as example - - View pattern: Views are `activitypub-*.njk` files in `views/`. They extend `document.njk` and use Indiekit frontend component macros (`card`, `button`, `badge`, `pagination`, etc.) - - Collection registration: See `index.js:614-621` — register via `Indiekit.addCollection("name")` calls in `init()`, then store references via `this._collections.name = indiekitCollections.get("name")` - - i18n: All user-visible strings go in `locales/en.json` under the `activitypub` namespace, referenced via `__("activitypub.reader.xxx")` - - Asset serving: Place CSS/JS in `assets/` directory. Indiekit core serves at `/assets/@rmdes-indiekit-endpoint-activitypub/`. Reference from views with `` tag. - -- **Conventions:** - - ESM modules throughout (`import`/`export`) - - ISO 8601 strings for dates in MongoDB (except `published` in timeline items which uses `Date` for sorting queries) - - Nunjucks templates use `{% from "xxx.njk" import component %}` for Indiekit frontend components - - Alpine.js `x-data`, `x-show`, `x-on:click` for client-side interactivity (loaded explicitly in reader layout, NOT by Indiekit core) - - CSRF protection: Indiekit core has no CSRF middleware. POST endpoints that trigger ActivityPub activities must validate a CSRF token. Use a simple pattern: generate a token per-session and embed as a hidden field in forms / include in `fetch()` headers. Validate on the server side before processing. - -- **Key files:** - - `index.js` — Plugin entry point, routes, collections, syndicator, follow/unfollow methods - - `lib/inbox-listeners.js` — All inbox activity handlers (Follow, Like, Announce, Create, Delete, etc.) - - `lib/federation-setup.js` — Fedify federation object configuration (dispatchers, queue, etc.) - - `locales/en.json` — English translations - - `views/activitypub-dashboard.njk` — Dashboard view (reference for card-grid patterns) - - `views/activitypub-following.njk` — Following view (reference for list+pagination) - -- **Gotchas:** - - Fedify returns `Temporal.Instant` for dates, not JS `Date`. Convert with `new Date(Number(obj.published.epochMilliseconds))` - - Fedify object properties are often async getters — `await actorObj.icon` not `actorObj.icon` - - `ctx.sendActivity()` first argument is `{ identifier: handle }` where `handle` comes from plugin options - - The plugin stores `this._federation` and creates context via `this._federation.createContext(new URL(this._publicationUrl), { handle, publicationUrl })` - - Remote actor lookup uses `ctx.lookupObject("@handle@instance")` or `ctx.lookupObject("https://url")` - - The AP plugin's asset directory is `assets/` at the package root, served at `/assets/@rmdes-indiekit-endpoint-activitypub/` - -- **Domain context:** - - ActivityPub activities: `Like` (favorite), `Announce` (boost/repost), `Create` (new post), `Follow`/`Undo(Follow)`, `Accept`, `Reject`, `Delete`, `Update`, `Block`, `Move` - - Content warnings use the `summary` field on AP objects (Mastodon convention) - - Boosts are `Announce` activities wrapping the original post — the reader must render the original post with boost attribution - - Replies use `inReplyTo` linking to the parent post URL - - Sensitive content uses the `sensitive` boolean on AP objects - -## Runtime Environment - -- **Start command:** `cloudron exec --app rmendes.net` or locally `npm start` in the Cloudron container -- **Port:** Indiekit on 8080 (behind nginx on 3000) -- **Health check:** `curl https://rmendes.net/.well-known/webfinger?resource=acct:rick@rmendes.net` -- **Deploy:** Build via `cloudron build --no-cache && cloudron update --app rmendes.net --no-backup` - -## Feature Inventory — Microsub Bridge Being Replaced - -### Files Being Modified (Bridge Removal) - -| Old Code | Functions | Mapped to Task | -|----------|-----------|----------------| -| `lib/inbox-listeners.js` — function `storeTimelineItem()` (~line 468) | Timeline item storage from AP activities | Task 2 (store natively), Task 12 (remove bridge) | -| `lib/inbox-listeners.js` — function `getApChannelId()` (~line 413) | Auto-creates Microsub "Fediverse" channel | Task 12 (remove) | -| `index.js` — lazy accessors in `init()` (~line 638) | `microsub_items`, `microsub_channels` collection refs | Task 12 (remove) | -| `lib/inbox-listeners.js` — Create handler (~line 262, calls `storeTimelineItem` at ~line 310) | Stores incoming posts via bridge | Task 2 (redirect to native storage) | - -### Feature Mapping Verification - -- [x] `storeTimelineItem()` → Task 2 (native `ap_timeline` storage) -- [x] `getApChannelId()` → Task 12 (removed; no longer needed) -- [x] Lazy Microsub collection accessors → Task 12 (removed) -- [x] Inbox Create handler → Task 2 (rewired to native storage) -- [x] Like/Announce inbox storage → Task 3 (notification storage) - -## Progress Tracking - -**MANDATORY: Update this checklist as tasks complete. Change `[ ]` to `[x]`.** - -- [x] Task 1: MongoDB collections and data models -- [x] Task 2: Inbox listener refactor — native timeline storage (includes Delete/Update handling) -- [x] Task 3: Inbox listener refactor — notification storage -- [x] Task 4: Timeline controller and view -- [x] Task 5: Reader CSS stylesheet -- [x] Task 6: Notifications controller and view -- [x] Task 7a: Interaction API — Like and Boost endpoints (with CSRF) -- [x] Task 7b: Interaction UI — Like and Boost buttons (Alpine.js) -- [x] Task 8: Compose form — Micropub reply path -- [x] Task 9: Content warning toggles and rich media rendering -- [x] Task 10: Mute, block, and tab filtering -- [x] Task 11: Remote profile view -- [x] Task 12: Remove Microsub bridge -- [x] Task 13: Timeline retention cleanup - -**Total Tasks:** 14 | **Completed:** 14 | **Remaining:** 0 - -## Implementation Tasks - -### Task 1: MongoDB Collections and Data Models - -**Objective:** Register new MongoDB collections (`ap_timeline`, `ap_notifications`, `ap_muted`, `ap_blocked`, `ap_interactions`) and create indexes for efficient querying. - -**Dependencies:** None - -**Files:** - -- Modify: `index.js` — Register collections via `Indiekit.addCollection()` in `init()`, store references in `this._collections`, create indexes -- Create: `lib/storage/timeline.js` — Timeline CRUD functions -- Create: `lib/storage/notifications.js` — Notification CRUD functions -- Create: `lib/storage/moderation.js` — Mute/block CRUD functions - -**Key Decisions / Notes:** - -- `ap_timeline` schema: - ```js - { - uid: "https://remote.example/posts/123", // canonical AP object URL (dedup key) - type: "note" | "article" | "boost", // boost = Announce wrapper - url: "https://remote.example/posts/123", - name: "Post Title" | null, // Articles only - content: { text: "...", html: "..." }, - summary: "Content warning text" | null, // CW / spoiler - sensitive: false, // Mastodon sensitive flag - published: Date, // Date object for sort queries - author: { name, url, photo, handle }, // handle = "@user@instance" - category: ["tag1", "tag2"], - photo: ["url1", "url2"], - video: ["url1"], - audio: ["url1"], - inReplyTo: "https://parent-post-url" | null, - boostedBy: { name, url, photo, handle } | null, // For Announce activities - boostedAt: Date | null, // When the boost happened - originalUrl: "https://original-post-url" | null, // For boosts: the wrapped object URL - readBy: [], - createdAt: "ISO string" - } - ``` -- `ap_notifications` schema: - ```js - { - uid: "activity-id", // dedup key - type: "like" | "boost" | "follow" | "mention" | "reply", - actorUrl: "https://remote.example/@user", - actorName: "Display Name", - actorPhoto: "https://...", - actorHandle: "@user@instance", - targetUrl: "https://my-post-url" | null, // The post they liked/boosted/replied to - targetName: "My Post Title" | null, - content: { text: "...", html: "..." } | null, // For mentions/replies - published: Date, - read: false, - createdAt: "ISO string" - } - ``` -- `ap_muted`: `{ url: "actor-url", keyword: null, mutedAt: "ISO" }` — url OR keyword, not both -- `ap_blocked`: `{ url: "actor-url", blockedAt: "ISO" }` -- `ap_interactions`: `{ type: "like"|"boost", objectUrl: "https://...", activityId: "urn:uuid:...", createdAt: "ISO" }` — tracks outgoing interactions for undo support and UI state -- Indexes: - - `ap_timeline`: `{ uid: 1 }` unique, `{ published: -1 }` for timeline sort, `{ "author.url": 1 }` for profile view, `{ type: 1, published: -1 }` for tab filtering - - `ap_notifications`: `{ uid: 1 }` unique, `{ published: -1 }` for sort, `{ read: 1 }` for unread count - - `ap_muted`: `{ url: 1 }` unique (sparse), `{ keyword: 1 }` unique (sparse) - - `ap_blocked`: `{ url: 1 }` unique - - `ap_interactions`: `{ objectUrl: 1, type: 1 }` compound unique (one like/boost per object), `{ type: 1 }` for listing -- Storage functions follow the pattern in Microsub's `lib/storage/items.js` — export pure functions that take `(collections, ...)` parameters -- `addTimelineItem(collections, item)` uses atomic upsert: `updateOne({ uid }, { $setOnInsert: item }, { upsert: true })` -- `getTimelineItems(collections, { before, after, limit, type, authorUrl })` returns cursor-paginated results -- `addNotification(collections, notification)` uses atomic upsert -- `getNotifications(collections, { before, limit })` returns paginated, newest-first -- `getUnreadNotificationCount(collections)` returns count of `{ read: false }` - -**Definition of Done:** - -- [ ] All five collections registered via `Indiekit.addCollection()` in `init()` (ap_timeline, ap_notifications, ap_muted, ap_blocked, ap_interactions) -- [ ] Indexes created in `init()` method -- [ ] `addTimelineItem` stores item and deduplicates by uid -- [ ] `getTimelineItems` returns paginated results with before/after cursors -- [ ] `addNotification` stores notification and deduplicates -- [ ] `getNotifications` returns paginated newest-first -- [ ] `getUnreadNotificationCount` returns correct count -- [ ] Mute/block CRUD operations work (add, remove, list, check) -- [ ] All storage functions have unit tests - -**Verify:** - -- `cd /home/rick/code/indiekit-dev/indiekit-endpoint-activitypub && node -e "import('./lib/storage/timeline.js').then(m => console.log(Object.keys(m)))"` — exports exist -- `cd /home/rick/code/indiekit-dev/indiekit-endpoint-activitypub && node -e "import('./lib/storage/notifications.js').then(m => console.log(Object.keys(m)))"` — exports exist -- `cd /home/rick/code/indiekit-dev/indiekit-endpoint-activitypub && node -e "import('./lib/storage/moderation.js').then(m => console.log(Object.keys(m)))"` — exports exist - ---- - -### Task 2: Inbox Listener Refactor — Native Timeline Storage - -**Objective:** Modify the inbox Create handler to store posts in `ap_timeline` instead of bridging to Microsub. Also handle Announce (boost) activities by storing the wrapped object with boost attribution. - -**Dependencies:** Task 1 - -**Files:** - -- Modify: `lib/inbox-listeners.js` — Refactor Create handler (~line 262) and Announce handler (~line 233) to store in `ap_timeline`, plus Delete/Update handlers for timeline cleanup -- Modify: `package.json` — Add `sanitize-html` to dependencies -- Create: `lib/timeline-store.js` — Helper that extracts data from Fedify objects and calls storage functions - -**Key Decisions / Notes:** - -- The existing Create handler at `inbox-listeners.js` (function `registerInboxListeners`, Create section ~line 262) currently calls `storeTimelineItem()`. Replace that call with the new native storage -- **CRITICAL — Announce handler bifurcation required:** The current Announce handler (line ~237) has an early return that ONLY processes boosts of our own content: `if (!objectId || (pubUrl && !objectId.startsWith(pubUrl))) return;`. This filter MUST be modified to create two code paths: - 1. **Boost of our content** (objectId starts with pubUrl) → store as notification (Task 3) - 2. **Boost from a followed account** (announcing actor is in our followers/following) → store in `ap_timeline` with `type: "boost"` - 3. **Both conditions true** (a followed account boosts our post) → store BOTH notification AND timeline item -- For timeline boosts: fetch the wrapped object via `await announce.getObject()` (the current handler only reads `announce.objectId` URL, NOT the full object), extract its data, then store with `type: "boost"` and `boostedBy` populated from the announcing actor -- To check if the announcing actor is followed: query `ap_followers` or `ap_following` collection for the actor URL -- Keep the same Fedify object→data extraction logic from `storeTimelineItem` (content, photos, videos, tags, etc.) but move it to a reusable `extractObjectData(object, actorObj)` function in `lib/timeline-store.js` -- **CRITICAL: HTML sanitization** — Remote content HTML MUST be sanitized before storage using `sanitize-html` (same library used in Microsub's `lib/webmention/verifier.js`). Allow safe tags: `a`, `p`, `br`, `em`, `strong`, `blockquote`, `ul`, `ol`, `li`, `code`, `pre`, `span`, `h1`-`h6`, `img`. Allow `href` on `a`, `src`/`alt` on `img`, `class` on `span` (for Mastodon custom emoji). Strip all other HTML including `