Hash-Based Validation: Prove Your Migration Is Complete

TL;DR: Don’t trust the migration — prove it. A hash-based completeness check confirms every source file is accounted for, even after renaming or conversion.

Going AI-native means moving thousands of files to plain text. The question isn’t if you’ll do it — it’s how you’ll prove nothing got lost.

The naive answer — compare filenames — lies. Client Leads.xls becomes Client Leads.md during conversion, and a name-based check flags it “missing.” Rename one folder, change one extension, and your validation reports a catastrophe that never happened.

Trust, but be ha(r)sh.

Hash-based, not name-based

The migration-toolkit checks contents, not names. Each source file gets a unique fingerprint — a hash. Find that hash in the target tree and the file is accounted for, regardless of renaming.

But here’s the nuance most “hash everything” posts miss: converted files change their hash. leads.xls → leads.md is a different byte stream and will never match by content. The check isn’t pretending hashes never change — it’s proving nothing falls through the cracks unaccounted for. Two categories cover every source file:

• Copied as-is — matched by hash in the target. Byte-for-byte identical.
• Converted — bytes differ, but the copy log shows the source was processed and the folder mapping records where it landed.

Either way, the file is accounted for. The only real failure mode is silence — a file that was neither copied, nor converted, nor excluded on purpose. That’s the one the check catches.

The three scripts

One workflow, three tools. Each uses uv run — if you followed Convert Office to Markdown, everything is already set up.

1. Snapshot before you start — make_file_inventory.py

Run this on the source tree before touching a file. It’s your ground truth.

uv run make_file_inventory.py \
  --root /vault/_Archive \
  --hash \
  --output-dir "Migration Logs"

You get a CSV (a table stored as plain text): each file’s path (where it lives in the folder structure), size, last-modified time, and a content hash. Run it again on the target after migration and compare the two inventories. This is the “before” picture nobody argues with.

2. Copy safely, log every move — safe_copy_with_log.py

Never move (mv) a file. Never blind-overwrite. This script copies and writes one CSV row per file: source, destination, size, hashes, status.

uv run safe_copy_with_log.py \
  "Source/Legacy Deck.pptx" \
  "01-Main Vault/03-Investors/Pitch Materials/" \
  --hash

Flags worth knowing:

• --dry-run — see the plan before committing.
• --overwrite — opt-in; off by default. Existing files are skipped, never clobbered.
• --hash — verifies every copy. Hashes source and destination after transfer. Match → copied. Mismatch → hash-mismatch. Proves the bytes made it whole.

The copy log closes the loop on converted files. If leads.xls shows up as “converted → leads.md,” it’s accounted for even though its hash changed.

3. Prove completeness — check_migration_completeness.py

The gate. Hashes every source file, hashes every target file, reports what’s missing.

uv run check_migration_completeness.py \
  --original /vault/_Archive \
  --target  /vault/Reorganization \
  --exclude-source-dir "Reorganization" \
  --mode hash

It excludes the reorg folder from the source scan (so copied files don’t get counted twice), then writes four reports:

• Missing-files CSV — every source key with no target match.
• Source duplicates CSV — exact-duplicate groups in the source.
• Target duplicates CSV — same, in the target.
• Migration summary — counts, keys represented, PASS/FAIL.

Exit code is 0 (PASS) or 1 (FAIL) — wire it into your CI (automated checks that run on every change), your justfile, or your approval checklist.

Exact duplicates only

The duplicate reports group by hash, so only byte-identical files get flagged. A near-duplicate, a draft, a v2 — different hash, kept. The discipline is surgical: collapse true copies, preserve every real version. No automated “similar file” deletion, ever — that’s a one-way ticket to losing the only copy of the version you actually wanted.

The validation gate

Copy-first only buys you safety if you don’t delete the source prematurely. The rule, written into the plan:

No old-tree deletion until the completeness report is clean AND a human approves.

A failed chunk costs nothing but a re-run — the source is still there. A clean report costs nothing — approve and move on. The expensive mistake is deleting the source before the check passes, and this gate makes that mistake impossible to do by accident.

Why it works

Hash-based validation closes the loop on an AI-native migration. Plain text is the ultimate insurance policy — but only if you can prove the move was complete. A hash doesn’t trust the migration. It proves it.

What’s next

Safety checks in place. Now move your files: 14-Step AI-Native Migration Process — the step-by-step guide from first folder to final vault.

Markdown your next move. By Charles Henri Gayot.