SkillAgentSearch skills...

wiki-dedup

Detect and merge wiki pages representing the same concept under different names. Use for page-identity deduplication and consolidation. Destructive merges require confirmation; not for link repair or general linting.

Install / Use

npx skills add Ar9av/obsidian-wiki --skill wiki-dedup

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

95/100

Category

Automation

Supported Platforms

Universal

Tags

Our assessment of wiki-dedup

wiki-dedup scores 95/100 on our quality scale, 323rd of 2,790 Automation skills we index (top 12%).

Its SKILL.md is 15 KB long, well organised into 27 sections with 9 code examples: a thorough specification that gives an agent plenty to work with.

With 3,485 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
20/20
Description
15/15
Adoption
15/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so wiki-dedup is actively maintained.
  • Our last check on 2026-11-16 could not reach the source (HTTP 404); it may have been moved or deleted.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

wiki-dedup compared with similar skills

All 4 of these similar skills score higher than wiki-dedup; compare them before choosing.

SkillScoreStarsUpdatedFormat
wiki-dedup (this skill)by Ar9av953.5k54d agoSKILL.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
rufloby ruvnet10081.9ktodayCLAUDE.md
Claude-Code-Game-Studiosby Donchitos10039.7k7d agoCLAUDE.md
Scraplingby D4Vinci100103.6k2d agoMCP Server

Frequently asked questions

How do I install wiki-dedup?
Run npx skills add Ar9av/obsidian-wiki --skill wiki-dedup. The install tabs above show the steps for each supported agent.
Which AI agents does wiki-dedup work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is wiki-dedup safe to use?
It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is wiki-dedup still maintained?
The repository was last updated about 2 months ago, so wiki-dedup is actively maintained.

name: wiki-dedup description: > Detect and merge wiki pages representing the same concept under different names. Use for page-identity deduplication and consolidation. Destructive merges require confirmation; not for link repair or general linting.

Wiki Dedup — Identity Resolution and Page-Level Deduplication

You are finding and merging wiki pages that cover the same concept under different names. This is a write-heavy, potentially destructive skill — page merges cannot be automatically undone. Work carefully and confirm before acting in merge mode.

Follow the Retrieval Primitives table in llm-wiki/SKILL.md. The candidate-detection pass uses only frontmatter and titles (cheap). Only open full page bodies for confirmed candidate pairs.

Before You Start

Writing profile: Before drafting or rewriting natural-language Markdown, read and apply the Writing Profile Resolution section in llm-wiki/SKILL.md. Framework schema, provenance, safety, and operation-specific requirements take precedence. WRITING.md preferences apply only to newly drafted or rewritten natural-language Markdown; preserve source content and structured records.

  1. Resolve config — follow the Config Resolution Protocol in llm-wiki/SKILL.md (inline @name override → walk up CWD for .env → global config → prompt setup). This gives OBSIDIAN_VAULT_PATH and OBSIDIAN_LINK_FORMAT.
  2. Read index.md to get the full page inventory with one-line descriptions and tags.
  3. Read log.md briefly — if a dedup run just happened, note what was already merged.

Modes

| Mode | Flag | Behavior | |---|---|---| | Audit | (default) | Report candidates only — no writes | | Merge | --merge | Show each confirmed pair, ask for confirmation before merging | | Auto-merge | --auto | Merge all high-confidence pairs (score ≥ 0.90) non-interactively |

If the user doesn't specify, run in Audit mode and present findings before asking whether to proceed.

Step 1: Build the Page Registry

Glob all .md files in the vault (excluding _archives/, _raw/, .obsidian/, index.md, log.md, hot.md, _insights.md, and any file that contains redirects_to: in its frontmatter — those are already merged redirect stubs).

For each remaining page, extract from frontmatter:

  • node_id — relative path from vault root, without .md
  • title — frontmatter title field
  • aliases — frontmatter aliases list (may be absent)
  • tags — frontmatter tags list
  • category — directory prefix

Build a lookup table: node_id → {title, aliases, tags, category, summary}.

Step 2: Detect Candidate Pairs

For every pair of pages in the registry, compute a similarity score using these signals:

2a. Title similarity signals

| Signal | How to assess | Max contribution | |---|---|---| | Token overlap | Jaccard similarity of lowercased title word-tokens (split on spaces, hyphens, underscores, punctuation) | 0.65 | | Edit distance | Normalized edit distance on lowercased titles: 1 - (edits / max(len_a, len_b)) | 0.40 | | Substring containment | One title is a substring of the other (e.g. "RSC" ⊂ "React Server Components") | 0.50 | | Alias cross-match | Page A's title appears in page B's aliases, or vice versa | 0.65 |

Composite title score = min(max(token_overlap, edit_distance, substring), 0.65) + alias_cross_bonus.

You don't need exact arithmetic — make a confident judgement about degree of similarity.

Title extraction note: Some pages use YAML block scalars (title: >- or title: |). When the title: value is >-, >, |, or |-, the actual title is on the next indented line — read it from there. Never compare the literal string >- as a title.

2b. Semantic signals (cheap pass)

| Signal | Points | |---|---| | Same category directory | +0.10 | | Tag overlap ≥ 3 shared tags | +0.15 | | Tag overlap ≥ 2 shared tags | +0.05 | | Same first tag (dominant tag) | +0.05 |

2c. Threshold

Flag pairs with composite score ≥ 0.75 as candidates. Pairs scoring 0.90+ are high-confidence.

Score ranges → confidence labels:

| Score | Label | |---|---| | ≥ 0.90 | HIGH — almost certainly the same concept | | 0.75–0.89 | MEDIUM — likely the same, verify | | 0.60–0.74 | LOW — possible abbreviation or specialisation; skip unless user asks |

Only carry HIGH and MEDIUM candidates into Step 3.

2d. Quick exit rule

If the vault has fewer than 10 pages, skip the pair loop and report "vault too small to have meaningful duplicates". If the vault has more than 500 pages, process candidates in batches of 50 pairs — pause and report progress between batches.

Step 3: Semantic Verdict

For each candidate pair (sorted by score descending):

  1. Read both pages in full (full page read — justified because candidate pool is small).
  2. Ask: are these pages covering the same concept, or are they distinct?

Assign one of three verdicts:

| Verdict | Meaning | |---|---| | merge | Same concept — different name, abbreviation, alias, or accidental duplicate. Safe to merge. | | keep-separate | Related but distinct — e.g. "Server Actions" vs "Server Components" are related React features, not duplicates. | | needs-review | Ambiguous — substantial overlap but also meaningful differences. Flag for the user to decide. |

Attach a short reason to each verdict (one sentence). This appears in the report and the log.

Step 4: Audit Report

Always produce this report, even in merge/auto-merge mode (so the user sees what will happen):

## Wiki Dedup Report

### High-Confidence Candidates (score ≥ 0.90): N pairs

| Score | Page A | Page B | Verdict | Reason |
|---|---|---|---|---|
| 0.95 | `concepts/rsc.md` | `concepts/react-server-components.md` | merge | "RSC" is the abbreviation; both pages cover identical material |
| 0.91 | `entities/vaswani-2017.md` | `references/attention-is-all-you-need.md` | keep-separate | One is a person stub, one is a paper reference |

### Medium-Confidence Candidates (score 0.75–0.89): N pairs

| Score | Page A | Page B | Verdict | Reason |
|---|---|---|---|---|
| 0.82 | `concepts/fine-tuning.md` | `concepts/finetuning.md` | merge | Same concept, hyphenation variant |

### Needs Human Review: N pairs

| Score | Page A | Page B | Reason |
|---|---|---|---|
| 0.78 | `concepts/agents.md` | `concepts/autonomous-agents.md` | Substantial overlap but "agents" may intentionally be broader |

### Summary
- Pages scanned: N
- Candidate pairs found: M
- Recommended merges: X
- Keep separate: Y
- Needs review: Z

In Audit mode, stop here and ask: "Run --merge to interactively merge the recommended pairs, or --auto to merge all high-confidence ones automatically?"

Step 5: Merge

Pre-write snapshot — before the first file write, check whether the vault itself is the root of a Git repository. Merely being a subdirectory of a larger repository does not qualify: running git add -A there could capture unrelated files. If the vault is not a standalone Git repository, skip this step silently — no nagging, no suggesting git init.

VAULT_REAL_PATH=$(cd "$OBSIDIAN_VAULT_PATH" && pwd -P)
VAULT_GIT_ROOT=$(git -C "$OBSIDIAN_VAULT_PATH" rev-parse --show-toplevel 2>/dev/null || true)
SNAPSHOT_SHA=""

if [ -n "$VAULT_GIT_ROOT" ] && [ "$VAULT_GIT_ROOT" = "$VAULT_REAL_PATH" ]; then
  if git -C "$OBSIDIAN_VAULT_PATH" diff --quiet \
    && git -C "$OBSIDIAN_VAULT_PATH" diff --cached --quiet \
    && [ -z "$(git -C "$OBSIDIAN_VAULT_PATH" ls-files --others --exclude-standard)" ]; then
    SNAPSHOT_SHA=$(git -C "$OBSIDIAN_VAULT_PATH" rev-parse HEAD)
  else
    if ! git -C "$OBSIDIAN_VAULT_PATH" add -A; then
      echo "Pre-write snapshot failed; abort the skill without writing any vault files." >&2
      exit 1
    fi
    if ! git -C "$OBSIDIAN_VAULT_PATH" commit -m "pre-wiki-dedup snapshot" --quiet; then
      echo "Pre-write snapshot failed; abort the skill without writing any vault files." >&2
      exit 1
    fi
    SNAPSHOT_SHA=$(git -C "$OBSIDIAN_VAULT_PATH" rev-parse HEAD)
  fi
fi

The clean-repository branch deliberately avoids calling git commit, so "nothing to commit" is not treated as an error. If git add or git commit fails, stop before editing the vault; never continue without the promised snapshot.

If SNAPSHOT_SHA is non-empty and the skill writes files, include the SHA in the final report. To discard the entire run, after confirming there are no later changes worth keeping, the user can run:

git -C "$OBSIDIAN_VAULT_PATH" reset --hard "$SNAPSHOT_SHA"
git -C "$OBSIDIAN_VAULT_PATH" clean -fd

For each merge verdict pair (in merge or auto-merge mode):

In merge mode: show the pair and verdict, then ask: "Merge [Page A] into [Page B]? (yes/skip/review)". Skip on anything other than yes.

In auto-merge mode: only process HIGH-confidence (score ≥ 0.90) merges without prompting.

5a: Pick the canonical page

Apply these tiebreakers in order until one wins:

  1. More incoming wikilinks — grep the vault for [[node_id]] references; higher count wins
  2. Richer content — longer page body (more lines) wins
  3. More sources — larger sources: list wins
  4. Title length — longer, more descriptive title wins (e.g. "React Server Components" beats "RSC")
  5. Alphabetical — earlier title wins

The canonical page is the survivor. The other page becomes the secondary (to be merged in, then replaced with a redirect stub).

5b: Merge content into the canonical page

Read both pages. Update the canonical page:

  • aliases: — add secondary page's title and all its aliases (no duplicates)
  • tags: — merge both tag lists (deduplicate, cap at 5 domain tags + system tags)
  • sources: — merge both source lists (deduplicate)
  • relationships: — merge both relationship lists (deduplicate by target, prefer typed entries over untyped)
  • base_confidence — recompute using the union of sources and the formula from llm-wiki/SKILL.md
  • updated — set to now
  • summary: — rewrite to cover the merged scope if the secondary page added new ground
  • Body content — merge unique sections and bullets from the secondary page. Do not blindly append — integrate the content. Avoid duplicating claims already present in the canonical page. Use ^[inferred] markers where synthesis is needed.
  • provenance: — recompute after merging

5c: Write a redirect stub at the secondary page path

---
title: <secondary page title>
redirects_to: "[[<canonical node_id>]]"
aliases: [<secondary aliases>]
category: <secondary category>
tags: []
created: <secondary original created>
updated: <obsidian-wiki memory now>
---

This page has been merged into [[<canonical page title>]].

The redirects_to: field tells any skill reading this page to follow the redirect rather than treat it as content.

5d: Rewrite wikilinks vault-wide

Grep the entire vault for any link pointing at the secondary slug:

  • [[secondary-slug]] → [[canonical-slug]]
  • [[secondary-slug|display text]] → [[canonical-slug|display text]]
  • If OBSIDIAN_LINK_FORMAT=markdown: [text](../path/to/secondary.md) → [text](../path/to/canonical.md)

Safety rules:

  • Never rewrite inside code blocks (``` fences or inline code)
  • Never rewrite inside the redirect stub itself (that's the one place the old slug should remain legible)
  • Never use rm or destructive shell ops — only Edit/Write tools
  • Rewrite one file at a time, verifying each before moving on
  • If a file has zero occurrences, skip it

5e: Update tracking files

index.md and hot.md are regenerated by the memory sync call in Step 6 — the redirect stub is skipped by the page walker, so the secondary's entry disappears and the canonical entry picks up the merged summary on its own.

.manifest.json — For the secondary page's source entries: add `"merged_i

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars3.5k
CategoryAutomation
Updated1mo ago
Forks345

Languages

Python

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions