Remove Duplicate Lines
Remove Duplicate Lines converts CRLF to LF, splits on LF, and uses a Set to retain the first exact line string. Matching is case-sensitive and includes leading/trailing spaces, so visually similar lines can remain distinct. Blank lines are also values and only the first identical blank line survives. Lone carriage returns are not explicitly normalized.
Runs locally — files stay on your deviceIs duplicate matching case-insensitive?
No.
Exact line identity includes whitespace and case
Two lines duplicate only when their JavaScript strings are identical after CRLF normalization.
How it works
CRLF becomes LF, split creates lines, Set removes later exact repeats, and LF rejoins output.
Key concepts
First wins
Original first occurrence order remains.
Whitespace matters
x and x-space differ.
When to use it
- Deduplicate exact exported lines.
- Keep first-seen ordering.
- Remove repeated IDs already normalized.
When not to use it
- Do not expect trimming or case folding.
- Do not dedupe semantically equivalent records.
- Do not expect counts of removed lines.
Worked examples
Exact
- Input
a\nb\na- Expected output
a\nb
Later a removed.
Case
- Input
A\na- Expected output
Both
Case differs.
Spaces
- Input
x\nx- Expected output
Both
Trailing space differs.
Common mistakes
Skipping normalization
Near-duplicates remain. Correction: Trim/fold first when domain rules allow.
Expecting sorting
Order is first-seen. Correction: Sort separately.
Using for CSV rows
Text equality ignores CSV semantics. Correction: Use a CSV deduplicator.
Best practices
- Define duplicate semantics.
- Normalize deliberately before dedupe.
- Keep a source and count results.
Related operations and differences
Remove Duplicate Words
Choose Remove Duplicate Words when you need to keep only the first occurrence of each word.; use Remove Duplicate Lines specifically for remove duplicate lines.
Add Prefix to Every Line
Choose Add Prefix to Every Line when you need to add the same prefix before every line.; use Remove Duplicate Lines specifically for remove duplicate lines.
Add Suffix to Every Line
Choose Add Suffix to Every Line when you need to add the same suffix after every line.; use Remove Duplicate Lines specifically for remove duplicate lines.
Capitalize Every Word
Choose Capitalize Every Word when you need to capitalize the first character of every word.; use Remove Duplicate Lines specifically for remove duplicate lines.
Character Counter
Choose Character Counter when you need to count Unicode characters in text.; use Remove Duplicate Lines specifically for remove duplicate lines.
Extract Email Addresses
Choose Extract Email Addresses when you need to find unique email addresses in text.; use Remove Duplicate Lines specifically for remove duplicate lines.
Limitations and privacy
- Exact case-sensitive whole-line Set only; no trimming, case folding, Unicode normalization, key selection, fuzzy match, counts, sort, or lone-CR normalization.
Text is transformed locally in this browser and is not uploaded by the tool.
Frequently asked questions
Is order preserved?
First occurrence order is.
Are blank lines deduplicated?
Yes.
Does whitespace matter?
Yes.
Are removed lines reported?
No.