feat: integrate YouTube likes sync via forked endpoint
Switch @rmdes/indiekit-endpoint-youtube to forked repo with OAuth 2.0 liked-videos sync. Add OAuth client config and likes sync settings. Also document outgoing webmentions architecture in README. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -197,6 +197,120 @@ The body buffering patch must preserve raw bytes in `req._rawBody`. If `JSON.str
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Outgoing webmentions
|
||||||
|
|
||||||
|
The blog sends [webmentions](https://www.w3.org/TR/webmention/) to every external URL found in a published post. This is handled by the `@rmdes/indiekit-endpoint-webmention-sender` plugin, extended by several patches and a shell-based poller.
|
||||||
|
|
||||||
|
### How it works
|
||||||
|
|
||||||
|
```
|
||||||
|
Post created via Micropub → saved to MongoDB
|
||||||
|
↓
|
||||||
|
Shell poller (every 300s) POSTs to /webmention-sender?token=JWT
|
||||||
|
↓
|
||||||
|
Plugin queries MongoDB for posts with webmention-sent != true
|
||||||
|
↓
|
||||||
|
For each unsent post:
|
||||||
|
1. Fetch the live HTML page (not stored content)
|
||||||
|
2. Parse with microformats — scope to .h-entry
|
||||||
|
3. Extract all <a href="…"> links
|
||||||
|
4. Filter to external links only
|
||||||
|
5. For each link: discover webmention endpoint via <link> / HTTP header
|
||||||
|
6. Send webmention (source=postUrl, target=linkUrl)
|
||||||
|
7. Mark post as webmention-sent with results {sent, failed, skipped}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why live-fetch instead of stored content
|
||||||
|
|
||||||
|
Post content stored in MongoDB (`post.properties.content.html`) is just the post body text. It does **not** contain the microformat links rendered by the Eleventy templates:
|
||||||
|
|
||||||
|
- `u-in-reply-to` — rendered by `reply-context.njk` inside the `.h-entry` wrapper
|
||||||
|
- `u-like-of` — same template
|
||||||
|
- `u-repost-of` — same template
|
||||||
|
- `u-bookmark-of` — same template
|
||||||
|
|
||||||
|
These links only exist in the live HTML page, so the webmention sender must always fetch the rendered page to discover them. This is what `patch-webmention-sender-livefetch.mjs` does.
|
||||||
|
|
||||||
|
### Poller architecture (start.sh)
|
||||||
|
|
||||||
|
The webmention sender plugin does not have its own scheduling — it exposes an HTTP endpoint that triggers a scan when POSTed to. The `start.sh` script runs a background shell loop:
|
||||||
|
|
||||||
|
1. **Readiness check** — polls `GET /webmention-sender/api/status` every 2s until it returns 200 (up to 3 minutes). This ensures MongoDB collections and plugin routes are fully initialised before the first scan.
|
||||||
|
2. **JWT generation** — mints a short-lived token (`{ me, scope: "update" }`, 5-minute expiry) signed with `SECRET`.
|
||||||
|
3. **POST trigger** — `curl -X POST /webmention-sender?token=JWT` triggers one scan cycle.
|
||||||
|
4. **Sleep** — waits `WEBMENTION_SENDER_POLL_INTERVAL` seconds (default 300 = 5 minutes), then repeats.
|
||||||
|
|
||||||
|
The poller routes through nginx (`INTERNAL_FETCH_URL`) rather than hitting Indiekit directly, so the request arrives with correct `Host` and `X-Forwarded-Proto` headers.
|
||||||
|
|
||||||
|
### Internal URL rewriting
|
||||||
|
|
||||||
|
When the livefetch patch fetches a post's live page, it rewrites the URL from the public domain to the internal nginx address:
|
||||||
|
|
||||||
|
```
|
||||||
|
https://blog.giersig.eu/replies/693e6/
|
||||||
|
↓ rewrite via INTERNAL_FETCH_URL
|
||||||
|
http://10.100.0.10/replies/693e6/
|
||||||
|
↓ nginx proxies to Indiekit
|
||||||
|
http://10.100.0.20:3000/replies/693e6/
|
||||||
|
```
|
||||||
|
|
||||||
|
Without this, the node jail cannot reach its own public HTTPS URL (TLS terminates on the web jail). The fallback chain is:
|
||||||
|
|
||||||
|
1. `INTERNAL_FETCH_URL` environment variable (production: `http://10.100.0.10`)
|
||||||
|
2. `http://localhost:${PORT}` (development)
|
||||||
|
|
||||||
|
### Retry behaviour
|
||||||
|
|
||||||
|
If the live page fetch fails (e.g. deploy still in progress, 502 from nginx), the post is **not** marked as sent. It stays in the "unsent" queue and is retried on the next poll cycle. This prevents the original upstream bug where a failed fetch would permanently mark the post as sent with zero webmentions.
|
||||||
|
|
||||||
|
### Patches
|
||||||
|
|
||||||
|
| Patch | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `patch-webmention-sender-livefetch.mjs` | Always fetch live HTML instead of stored content; rewrite URL for jailed setups; skip (don't mark sent) on fetch failure |
|
||||||
|
| `patch-webmention-sender-retry.mjs` | Predecessor to livefetch — only fixed the fetch-failure path. Now a no-op because livefetch runs first and is a superset. Kept for safety in case livefetch fails to apply. |
|
||||||
|
| `patch-webmention-sender-reset-stale.mjs` | One-time MongoDB migration: resets posts incorrectly marked as sent with 0/0/0 results. Guarded by `migrations` collection (`webmention-sender-reset-stale-v8`). |
|
||||||
|
| `patch-webmention-sender-empty-details.mjs` | UI patch: shows "No external links discovered" in the dashboard when a post was processed but had no outbound links (instead of a blank row). |
|
||||||
|
|
||||||
|
### Patch ordering
|
||||||
|
|
||||||
|
Patches run alphabetically via `for patch in scripts/patch-*.mjs`. For webmention patches:
|
||||||
|
|
||||||
|
1. `patch-webmention-sender-empty-details.mjs` — targets the `.njk` template (independent)
|
||||||
|
2. `patch-webmention-sender-livefetch.mjs` — replaces the fetch block in `webmention-sender.js`
|
||||||
|
3. `patch-webmention-sender-reset-stale.mjs` — MongoDB migration (independent)
|
||||||
|
4. `patch-webmention-sender-retry.mjs` — targets the same fetch block, but it's already gone (livefetch replaced it), so it reports "already applied" and skips
|
||||||
|
|
||||||
|
### Environment variables
|
||||||
|
|
||||||
|
| Variable | Default | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `WEBMENTION_SENDER_POLL_INTERVAL` | `300` | Seconds between poll cycles |
|
||||||
|
| `WEBMENTION_SENDER_MOUNT_PATH` | `/webmention-sender` | Plugin mount path in Express |
|
||||||
|
| `WEBMENTION_SENDER_TIMEOUT` | `10000` | Per-endpoint send timeout (ms) |
|
||||||
|
| `WEBMENTION_SENDER_USER_AGENT` | `"Indiekit Webmention Sender"` | User-Agent for outgoing requests |
|
||||||
|
| `INTERNAL_FETCH_URL` | — | Internal nginx URL for self-fetches (e.g. `http://10.100.0.10`) |
|
||||||
|
| `SECRET` | _(required)_ | JWT signing secret for poller authentication |
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
**"No external links discovered in this post"**
|
||||||
|
The live page was fetched successfully but no `<a href>` tags with external URLs were found inside the `.h-entry`. Check that the post's Eleventy template renders the microformat links (`u-like-of`, etc.) correctly.
|
||||||
|
|
||||||
|
**502 Bad Gateway on first poll**
|
||||||
|
The readiness check (`/webmention-sender/api/status`) should prevent this. If it still happens, the plugin may have registered its routes but MongoDB isn't ready yet. Increase the readiness timeout or check MongoDB connectivity.
|
||||||
|
|
||||||
|
**Posts stuck as "not sent" / retrying every cycle**
|
||||||
|
The live page fetch is failing every time. Check:
|
||||||
|
1. `INTERNAL_FETCH_URL` is set and nginx port 80 is reachable from the node jail
|
||||||
|
2. nginx port 80 has `proxy_set_header X-Forwarded-Proto https` (prevents redirect loop)
|
||||||
|
3. The post URL actually resolves to a page (not a 404)
|
||||||
|
|
||||||
|
**Previously failed posts not retrying**
|
||||||
|
Run the stale-reset migration: `node scripts/patch-webmention-sender-reset-stale.mjs`. It resets all posts marked as sent with 0/0/0 results. It's idempotent (guarded by a migration ID in MongoDB).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Patch scripts
|
## Patch scripts
|
||||||
|
|
||||||
Patches are Node.js `.mjs` scripts in `scripts/` that surgically modify files in `node_modules` after install. They are idempotent (check for a marker string before applying) and run automatically via `postinstall` and at the start of `serve`.
|
Patches are Node.js `.mjs` scripts in `scripts/` that surgically modify files in `node_modules` after install. They are idempotent (check for a marker string before applying) and run automatically via `postinstall` and at the start of `serve`.
|
||||||
|
|||||||
@@ -444,6 +444,15 @@ export default {
|
|||||||
limits: {
|
limits: {
|
||||||
videos: 10,
|
videos: 10,
|
||||||
},
|
},
|
||||||
|
oauth: {
|
||||||
|
clientId: process.env.YOUTUBE_OAUTH_CLIENT_ID || "",
|
||||||
|
clientSecret: process.env.YOUTUBE_OAUTH_CLIENT_SECRET || "",
|
||||||
|
},
|
||||||
|
likes: {
|
||||||
|
syncInterval: 3_600_000, // 1 hour
|
||||||
|
maxPages: 3, // up to 150 likes per sync
|
||||||
|
autoSync: true,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
"@rmdes/indiekit-syndicator-indienews": {
|
"@rmdes/indiekit-syndicator-indienews": {
|
||||||
languages: ["en", "de"],
|
languages: ["en", "de"],
|
||||||
|
|||||||
+1
-1
@@ -44,7 +44,7 @@
|
|||||||
"@rmdes/indiekit-endpoint-webmention-io": "^1.0.7",
|
"@rmdes/indiekit-endpoint-webmention-io": "^1.0.7",
|
||||||
"@rmdes/indiekit-endpoint-webmention-sender": "^1.0.8",
|
"@rmdes/indiekit-endpoint-webmention-sender": "^1.0.8",
|
||||||
"@rmdes/indiekit-endpoint-webmentions-proxy": "^1.0.3",
|
"@rmdes/indiekit-endpoint-webmentions-proxy": "^1.0.3",
|
||||||
"@rmdes/indiekit-endpoint-youtube": "^1.2.3",
|
"@rmdes/indiekit-endpoint-youtube": "github:svemagie/indiekit-endpoint-youtube",
|
||||||
"@rmdes/indiekit-post-type-page": "^1.0.4",
|
"@rmdes/indiekit-post-type-page": "^1.0.4",
|
||||||
"@rmdes/indiekit-preset-eleventy": "^1.0.0-beta.38",
|
"@rmdes/indiekit-preset-eleventy": "^1.0.0-beta.38",
|
||||||
"@rmdes/indiekit-syndicator-bluesky": "^1.0.19",
|
"@rmdes/indiekit-syndicator-bluesky": "^1.0.19",
|
||||||
|
|||||||
Reference in New Issue
Block a user