<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Outfit:wght@600;700;800&family=Plus+Jakarta+Sans:wght@400;500;600;700&display=swap">
⚡ Productivity guide

How to Clean Up Messy Text: Extra Spaces, Duplicates, Hidden Characters and Case

Clean text pasted from PDFs, Excel and email: non-breaking spaces, tabs, line endings, invisible characters, duplicate lines, sorting and title case rules.

📅 Published 2026-11-15⏱️ 5 min read🛠️ 6 free tools

Text rarely arrives clean. A list copied from a PDF has a line break in the middle of every sentence, a column pasted from Excel carries tabs and trailing spaces, and a block of copy from a web page hides non-breaking spaces that look identical to normal ones. None of it is visible, but all of it breaks the next step: a duplicate check misses repeats, a sort puts items in the wrong place, and a lookup fails because "London " is not "London".

This guide explains where each kind of mess comes from, how to spot it, and the order in which to clean it so that one fix does not undo another. The same principles apply whether you are tidying a mailing list, a keyword export or a paragraph headed for a CMS.

Know what "whitespace" actually contains

A space is not always a space. Unicode defines more than twenty whitespace and space-like characters, and the ones you meet most often in pasted text behave differently from the plain space (U+0020) on your keyboard. Tools that only look for a normal space will leave the others behind, which is why a cleaned list can still fail to match.

  • Non-breaking space (U+00A0): inserted by Word, Google Docs, many CMS editors and HTML entities like &nbsp;. It looks like a space but is a different character, so "New York" with a non-breaking space will not equal "New York" typed by hand.
  • Tab (U+0009): appears whenever you copy cells from a spreadsheet. Each cell boundary becomes a tab, which is useful if you want to paste back into a sheet and harmful if you want a single line of text.
  • Line endings: Windows uses CR+LF (\r\n), macOS and Linux use LF (\n). Mixing them produces lines that look identical but differ by an invisible carriage return, which defeats exact-match deduplication.
  • Zero-width characters (U+200B zero-width space, U+200C, U+200D, U+2060): take up no room on screen but count as characters. They come from web pages, chat apps and some AI-generated text.
  • Byte order mark (U+FEFF): sits at the very start of files saved as "UTF-8 with BOM", usually by older Windows tools. It makes the first line of a list different from the same text further down.
  • Soft hyphen (U+00AD): an invisible hint for where a word may break, common in text copied from PDFs and typeset documents.

Text copied from PDFs, spreadsheets and email

Each source damages text in its own way, so it pays to recognise the pattern before choosing a fix.

PDFs store text as positioned fragments, not as paragraphs. When you copy, every visual line becomes a real line break, hyphenated words at line ends stay split ("informa- tion"), and multi-column layouts can interleave the columns. Collapse the extra spaces first, then join lines that belong together. A useful trick is to replace a double line break with a placeholder, replace the remaining single line breaks with spaces, then turn the placeholder back into a paragraph break.

Spreadsheets give you tabs between cells, trailing spaces that someone typed years ago, and sometimes numbers formatted with thousands separators or non-breaking spaces (some locales use U+00A0 or U+202F as the thousands separator). If you want plain values, replace tabs with the separator you need before you collapse spaces.

Email clients quote replies with ">" at the start of each line, wrap plain-text messages at around 72 to 78 characters, and convert straight quotes into curly ones. Strip the quote markers with a find-and-replace on "> " before rejoining wrapped lines.

Removing duplicates without missing near-matches

Exact deduplication compares lines character by character. That is predictable, but it means "Apple", "apple" and "apple " (with a trailing space) are three different lines. Before removing duplicates, decide what should count as the same item and normalise the text to match that rule.

  • Trim leading and trailing spaces on every line, so stray whitespace does not create false "unique" lines.
  • Decide on case sensitivity. Email addresses, keywords and tags usually should be compared in lowercase. Product codes and passwords should not.
  • Normalise line endings so CRLF and LF lines compare equal.
  • Remove blank lines if you do not need them, because many tools treat every empty line after the first as a duplicate and delete it, which merges paragraphs.
  • Keep the original order unless you have a reason to sort. Order-preserving deduplication keeps the first occurrence, which matters when the list is ranked.

Sorting: alphabetical, natural and locale-aware

Computers sort strings by comparing characters, and the result depends on which comparison is used. A plain code-point sort puts all capital letters before lowercase ones ("Zebra" before "apple") and places accented letters after "z". A locale-aware sort, which is what browsers use by default for human-facing lists, interleaves case and keeps "é" next to "e".

Numbers are the other trap. Sorted as text, "item10" comes before "item2", because the comparison stops at the first differing character and "1" is less than "2". A natural sort treats runs of digits as numbers and gives the order people expect. If your tool only sorts alphabetically, pad numbers with leading zeros (item02, item10) before sorting.

Text sort        Natural sort
file1.txt        file1.txt
file10.txt       file2.txt
file2.txt        file10.txt
File3.txt        File3.txt   (position depends on case handling)

Case conversion and title case rules

Uppercase, lowercase and sentence case are mechanical, but title case is a style decision. A simple converter capitalises every word, which is fine for many headings but does not follow any major style guide.

AP style capitalises words of four or more letters and lowercases short articles, conjunctions and prepositions ("a", "an", "the", "and", "but", "or", "for", "in", "of", "on", "to") unless they are the first or last word. Chicago style lowercases articles, coordinating conjunctions and all prepositions regardless of length, so "Through" is capitalised in AP but "through" stays lowercase in Chicago. Both always capitalise the first and last word.

Whichever style you follow, check acronyms and brand names by hand. "NASA", "iPhone" and "McDonald" should keep their capitals, and sentence case should not lowercase proper nouns in the middle of a sentence.

Clean in the right order

The order of operations matters because each step changes what the next one sees. Sorting before trimming puts lines with leading spaces at the top; deduplicating before lowercasing keeps case variants; replacing words before fixing spaces misses matches that span a double space. This sequence works for almost any list:

  • 1. Fix structure: rejoin broken lines and strip quote markers using Find and Replace.
  • 2. Normalise whitespace with the Extra Space Remover so every line is trimmed and runs of spaces collapse to one.
  • 3. Normalise case with the Case Converter if matching should ignore capitals.
  • 4. Remove repeats with the Duplicate Line Remover.
  • 5. Order the result with the Text Sorter.
  • 6. Use Reverse Text only for special jobs such as checking palindromes or building test strings, never as a privacy measure.

More guides you might like

Ready to try it?

Open the free Extra Space Remover and apply what you just read. No sign-up, nothing uploaded.

Open Extra Space Remover →

← All guides