Fix UnicodeEncodeError ASCII Codec Errors in Python BeautifulSoup

You will learn why web scraping can produce Unicode text, why converting that text with str() causes an ASCII encoding error in Python 2, and how to keep text as Unicode until it must be encoded for output, storage, or a network request. You will also see how to clean non-breaking spaces such as \u00A0 safely.

A UnicodeEncodeError happens when Python tries to convert text characters into bytes using an encoding that cannot represent one of those characters.

In the original error, u'\xa0' is a Unicode non-breaking space (\u00A0). It looks similar to a normal space, but it is a different character. ASCII supports only character values 0 through 127, while a non-breaking space has value 160, so ASCII cannot encode it.

This issue is especially common in Python 2 code:

• unicode represents text characters.
• str represents bytes.
• Calling str(unicode_value) implicitly tries to encode the Unicode text as ASCII.

This line is therefore risky in Python 2:

str(agent_contact + ' ' + agent_telno)

If the combined value contains a non-ASCII character, str() attempts ASCII encoding and fails.

Web pages do not need to be written in another language to contain Unicode. English pages routinely include non-breaking spaces, curly quotation marks, en dashes, pound signs (£), accented names, and copied text from other systems.

The reliable rule is: decode incoming bytes once, work with Unicode text internally, and encode only at an external boundary using an explicit encoding such as UTF-8.

Think of Unicode text as a message containing ideas and characters, and an encoding as the shipping format used to send that message as bytes.

• Unicode is the message: “Call us 0207 123 4567”.
• UTF-8 is a shipping box that can carry nearly every character in the message.
• ASCII is a small box that only fits basic English letters, digits, and punctuation.

A non-breaking space does not fit in the ASCII box. Calling str() on Unicode in Python 2 can make Python choose that small ASCII box automatically. Instead, keep the message as Unicode while processing it, then explicitly choose UTF-8 when you need bytes.

In Python 2, use Unicode string literals (u'...') while combining scraped text, and do not call str() on Unicode text.

agent_telno_tag = agent.find('div', 'agent_contact_number')
agent_telno = u'' if agent_telno_tag is None else agent_telno_tag.get_text()

agent_info = (agent_contact + u' ' + agent_telno).strip()
p.agent_info = agent_info

If agent_contact might be a byte string, decode it first using the encoding that produced it:

agent_contact = agent_contact.decode('utf-8')

When writing Unicode text to a byte-oriented destination, encode it explicitly:

with open('agents.txt', 'wb') as output:
    output.write(p.agent_info.encode('utf-8'))

To replace non-breaking spaces with ordinary spaces before trimming or displaying the value:

clean_info = p.agent_info.replace(u'\u00a0', u' ').strip()

In Python 3, already represents Unicode text. The equivalent approach is:

Consider this Python 2 example:

agent_contact = u'Phone:'
agent_telno = u'0207\u00a0123\u00a04567'
info = agent_contact + u' ' + agent_telno

1. agent_contact is Unicode text.
2. agent_telno is Unicode text and contains \u00A0, a non-breaking space.
3. u' ' is also Unicode, so concatenation stays in Unicode.
4. info now contains valid Unicode text. No encoding has happened, so no error occurs.

This fails in Python 2:

byte_value = str(info)

1. str(info) asks Python to turn Unicode into bytes.
2. Without an explicitly supplied encoding, Python uses ASCII.
3. ASCII cannot represent \u00A0, so Python raises UnicodeEncodeError.

This succeeds:

utf8_bytes = info.encode('utf-8')

1. UTF-8 can represent the non-breaking spaces, so Python produces valid bytes.

Unicode-safe handling is needed whenever text can come from outside your program:

• Web scraping: HTML may contain non-breaking spaces, symbols, or names with accents.
• API clients: JSON responses are usually UTF-8 and may include user-entered text.
• CSV imports: Supplier or customer files may contain pound signs, smart quotes, or international addresses.
• Databases: Store and retrieve text as Unicode so customer names and product descriptions are preserved.
• Logging and command-line tools: Encode output appropriately when writing to a terminal, file, or remote logging service.
• Text cleanup: Normalize visually similar whitespace before parsing phone numbers, prices, or addresses.

In production scrapers, developers usually separate text handling into stages:

1. Fetch bytes from HTTP.
2. Determine or declare the page encoding and decode bytes into text. BeautifulSoup can often detect an HTML document's declared encoding, but a scraper should not assume every site uses the same encoding.
3. Extract Unicode text with methods such as get_text().
4. Normalize content needed for matching or parsing, such as converting non-breaking spaces to normal spaces.
5. Store Unicode text in a Unicode-capable database column and driver configuration.
6. Encode explicitly only when writing bytes to files, sockets, or legacy interfaces.

A small normalization helper keeps this behavior consistent:

# Python 2
import re

def clean_text(value):
    if value is None:
        return u''
    if not isinstance(value, unicode):
        value = value.decode('utf-8')
    value = value.replace(u'\u00a0', u' ')
    return re.sub(ur'\s+', u' ', value).strip()

Use a helper only when you know the byte encoding. If a value is bytes from an HTTP response, decode it using the response or document encoding rather than blindly assuming UTF-8.

For BeautifulSoup extraction, is often clearer than indexing , because a tag may contain nested elements or multiple text nodes.

Calling str() to "fix" Unicode in Python 2

# Broken when value contains non-ASCII text
result = str(unicode_value)

str() encodes Unicode using ASCII by default. Keep it as Unicode, or use unicode_value.encode('utf-8') only where bytes are required.

Mixing byte strings and Unicode strings

# Risky in Python 2: ' ' is bytes, not Unicode
info = agent_contact + ' ' + agent_telno

Use u' ' when working with Unicode in Python 2:

info = agent_contact + u' ' + agent_telno

Ignoring invisible characters

A normal space and a non-breaking space look almost identical:

u' '       # normal space, U+0020
u'\u00a0'  # non-breaking space, U+00A0

Replace or normalize whitespace before parsing structured values such as phone numbers.

Using contents[0] for all HTML text

# Python 2: use Unicode internally
text = u'Hello\u00a0world'
text = text.replace(u'\u00a0', u' ').strip()

# Decode bytes when they enter the program
text = raw_bytes.decode('utf-8')

# Encode only when a destination requires bytes
raw_bytes = text.encode('utf-8')

# Combine Unicode values safely in Python 2
combined = first + u' ' + second

# Safely extract text from a BeautifulSoup tag
value = u'' if tag is None else tag.get_text()

• \u00A0 is a non-breaking space.
• Do not use str(unicode_text) in Python 2 unless the text is guaranteed ASCII.
• Do not guess the encoding of arbitrary bytes; use HTTP headers, HTML metadata, or source documentation where available.
• Prefer UTF-8 for new files, APIs, and output.
• Python 3 str is Unicode; use .encode('utf-8') only to create bytes.

Why does an English web page contain Unicode characters?

Unicode includes much more than non-English alphabets. English pages commonly use non-breaking spaces, curly quotes, pound signs, dashes, and copied content containing accented names.

What is u'\xa0' in the error message?

It is a Unicode non-breaking space, also written as \u00A0. It prevents a line break at that location and is often used in HTML around phone numbers, prices, and labels.

Why does str() cause UnicodeEncodeError in Python 2?

In Python 2, str is bytes. Converting Unicode to str requires an encoding. If none is specified, Python tries ASCII, which cannot encode characters such as \u00A0.

Should I call .encode('utf-8') immediately after scraping?

Usually no. Keep scraped text as Unicode while cleaning, combining, and storing it. Encode only when a specific external API, file, or network destination requires bytes.

Is .strip() enough to remove a non-breaking space?

Unicode-aware strip() can handle many whitespace characters, but replacing \u00A0 with a normal space is useful when you need consistent spacing before parsing or comparing text.

Does BeautifulSoup automatically solve all encoding problems?

• Character encodings — Learn how UTF-8, ASCII, and other encodings represent text as bytes.
• Python strings and bytes — Understand the important type distinction, especially when maintaining Python 2 code or integrating with byte-based APIs.
• Unicode normalization — Helps make visually similar characters consistent for searching and comparisons.
• BeautifulSoup text extraction — Covers get_text(), tag navigation, and reliable HTML text cleanup.
• HTTP response encoding — Explains how headers and HTML metadata identify the encoding used by a web page.
• Regular expressions and whitespace — Useful for normalizing repeated or unusual whitespace in scraped content.
• File I/O encoding — Covers reading and writing UTF-8 text files safely.