TBToolbox
Developer Tools

CSV Duplicate Remover

CSV Duplicate Remover keys every data-row cell array by JSON.stringify and keeps the first occurrence. Matching is exact and case-sensitive after parsing: whitespace matters, numeric-looking text is not normalized, and no selected-column, fuzzy, locale, or business-key logic exists.

Runs locally — files stay on your device
Quick answer

Which duplicate is retained?

The first parsed occurrence.

Exact equality compares every cell

Rows duplicate only when ordered cell strings agree at every position.

How it works

A Map deduplicates JSON-stringified row arrays and output is CSV-escaped.

Key concepts

All columns participate

One differing cell makes a unique row.

First order retained

Earliest copy stays.

When to use it

  • Remove repeated identical exports.
  • Clean a small table.
  • Keep one exact record copy.

When not to use it

  • Do not expect case-insensitive matching.
  • Do not dedupe by one key.
  • Do not expect near-duplicate detection.

Worked examples

Exact

Input
a\n1\n1
Expected output
a\n1

Second removed.

Case

Input
Ana\nana
Expected output
Both

Strings differ.

Space

Input
x and x-space
Expected output
Both

Whitespace matters.

Common mistakes

Expecting key dedupe

Every cell compares. Correction: Choose keys explicitly.

Ignoring normalization

Case/spaces remain. Correction: Normalize by rules.

Assuming fuzzy match

No similarity. Correction: Use entity resolution.

Best practices

  • Define duplicate semantics.
  • Normalize deliberately.
  • Audit removals and retain source.

Related operations and differences

CSV Column Extractor

Choose CSV Column Extractor when you need to extract one CSV column by header name.; use CSV Duplicate Remover specifically for csv remove duplicates.

CSV Column Remover

Choose CSV Column Remover when you need to remove one CSV column by header name.; use CSV Duplicate Remover specifically for csv remove duplicates.

CSV Row Filter

Choose CSV Row Filter when you need to keep rows whose selected column contains a value.; use CSV Duplicate Remover specifically for csv remove duplicates.

CSV Sorter

Choose CSV Sorter when you need to sort CSV rows by a named column.; use CSV Duplicate Remover specifically for csv remove duplicates.

Reverse CSV Rows

Choose Reverse CSV Rows when you need to reverse data-row order while keeping the header.; use CSV Duplicate Remover specifically for csv remove duplicates.

Compare JSON

Choose Compare JSON when you need to show value differences between two JSON documents.; use CSV Duplicate Remover specifically for csv remove duplicates.

Limitations and privacy

  • Exact full-row dedupe only; no key selection, trimming, case folding, fuzzy matching, removal report, schema, or streaming.

The pasted data is parsed and transformed locally in this browser.

Frequently asked questions

Is matching case-sensitive?

Yes.

Does whitespace matter?

Yes.

Is header deduplicated?

No.

Which copy stays?

The first.

References