Dedupe — Data Deduplication Reference

Quick-reference skill for deduplication strategies, algorithms, and data quality patterns.

When to Use

• - Removing duplicate rows from datasets or databases
• Deduplicating files in storage systems
• Implementing fuzzy matching for near-duplicate detection
• Choosing between exact and probabilistic dedup methods
• Building ETL pipelines with deduplication stages

Commands

intro

CODEBLOCK0

Overview of deduplication — types, strategies, and tradeoffs.

exact

CODEBLOCK1

Exact deduplication — hash-based, key-based, and sorting approaches.

fuzzy

CODEBLOCK2

Fuzzy deduplication — similarity measures, blocking, and record linkage.

files

CODEBLOCK3

File-level deduplication — fdupes, jdupes, rdfind, and storage dedup.

algorithms

CODEBLOCK4

Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash.

sql

CODEBLOCK5

SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies.

cli

CODEBLOCK6

Command-line dedup tools — sort, uniq, awk, and stream processing.

checklist

CODEBLOCK7

Deduplication quality checklist and validation steps.

help

CODEBLOCK8

version

CODEBLOCK9

Configuration