Unit 4 From ASCII to Unicode: How Computers Represent Text

Instructions

As you read this lesson, pay attention to how computers use binary code to represent text characters. Take note of the differences between ASCII and Unicode, and how each system converts letters, numbers, and symbols into binary values that computers can process. Try to understand why Unicode was developed and how it expanded on ASCII's capabilities.

Introduction: The Language Barrier Between Humans and Computers

Have you ever wondered how computers understand the letters, numbers, and symbols that we type? While we communicate using alphabets and writing systems, computers can only process binary data: sequences of 0s and 1s. This creates a fundamental communication challenge: how do we translate human language into something a computer can understand?

The solution to this challenge lies in character encoding systems: standardized methods for converting text characters into binary code. In this lesson, we'll explore the two most important character encoding systems: ASCII and Unicode. We'll learn how these systems work, how they differ, and why they're essential for computers to process and store text.

Understanding character encoding is a fundamental concept in computer science. It reveals how computers bridge the gap between human language and machine processing, allowing us to communicate with technology using our natural languages.

The Challenge of Representing Text

Numbers can be directly represented in binary, but what about text? Letters, punctuation marks, and other symbols don't have inherent numeric values. To solve this problem, computer scientists developed character encoding systems that assign a unique number to each character. These numbers can then be converted to binary for storage and processing.

ASCII: The Original Character Encoding Standard

What is ASCII?

ASCII (American Standard Code for Information Interchange) was developed in the early 1960s as a standardized way to represent text in computers. The name is pronounced "ask-ee."

ASCII is a character encoding standard that assigns a unique decimal number (between 0 and 127) to each character commonly used in American English, including:

Uppercase letters (A-Z)
Lowercase letters (a-z)
Numbers (0-9)
Punctuation marks (!, ?, ., etc.)
Control characters (like tab, carriage return, etc.)

For example:

Character	ASCII Code
A	65
a	97
1	49
!	33

How ASCII Works

To understand how computers store text, we first need to know how data is stored.

A bit is the smallest unit of data. It can be either a 0 or a 1.
A byte is a group of 8 bits. One byte can store a single character or small number.

When you type a character on your keyboard, like the letter ‘A’, the computer follows these steps:

It looks up the ASCII code for ‘A’, which is 65.
It converts 65 to binary: 01000001.
It stores that binary number (which is 1 byte, or 8 bits) in memory.

Later, when the computer needs to use that character again:

It reads the binary value 01000001 from memory.
It converts it back to the number 65.
It recognizes 65 as the ASCII code for ‘A’ and displays or processes the letter.

This process happens for every character you type. Each one is turned into a number, then into binary, and stored as a byte in memory.

ASCII Patterns

ASCII codes follow logical patterns that make them easier to remember and work with:

Uppercase letters (A-Z) have consecutive codes from 65 to 90
A = 65, B = 66, C = 67, and so on
Lowercase letters (a-z) have consecutive codes from 97 to 122
a = 97, b = 98, c = 99, and so on
Digits (0-9) have consecutive codes from 48 to 57
0 = 48, 1 = 49, 2 = 50, and so on
The difference between an uppercase letter and its lowercase equivalent is always 32
A = 65, a = 97 (difference of 32)
B = 66, b = 98 (difference of 32)

These patterns make it easy for programmers to manipulate text. For example, to convert an uppercase letter to lowercase, you can simply add 32 to its ASCII code.

ASCII in Binary

Since computers store all data in binary, ASCII codes must be converted to binary for storage. Here are some examples:

'A' (ASCII 65) = 01000001 in binary
'B' (ASCII 66) = 01000010 in binary
'a' (ASCII 97) = 01100001 in binary
'1' (ASCII 49) = 00110001 in binary
'!' (ASCII 33) = 00100001 in binary

Notice that each binary representation uses 7 bits (the leftmost bit is always 0 in standard ASCII). However, computers typically store data in 8-bit bytes, so an extra 0 is added at the beginning.

Limitations of ASCII

While ASCII was revolutionary for its time, it has significant limitations:

Limited character set: ASCII only defines 128 characters, which is sufficient for English but inadequate for other languages with different alphabets or writing systems.

No support for international characters: ASCII doesn't include characters with accents (like é, ñ, or ü), non-Latin alphabets (like Cyrillic, Greek, or Arabic), or logographic writing systems (like Chinese or Japanese).

No support for modern symbols: ASCII was developed before emojis, mathematical symbols, and many other special characters that are commonly used today.

These limitations became increasingly problematic as computers started to spread globally and needed to support multiple languages and writing systems.

Unicode: The Global Character Encoding Standard

What is Unicode?

Unicode is a character encoding standard developed in the early 1990s to address the limitations of ASCII. Unlike ASCII, which was designed primarily for English, Unicode aims to include every character from all human languages, past and present.

Unicode assigns a unique number (called a "code point") to each character, regardless of platform, program, or language. The Unicode standard is maintained by the Unicode Consortium, which continues to add new characters with each release.

Unicode's Scope

Unicode is vastly more comprehensive than ASCII:

ASCII defines 128 characters
Unicode (as of version 15.0) includes nearly 150,000 characters

Unicode includes:

Characters from all modern languages
Historical scripts and symbols
Mathematical and technical symbols
Punctuation marks
Emojis
Various other symbols and notations

Unicode and ASCII Compatibility

Unicode was designed to be backward compatible with ASCII. This means:

The first 128 Unicode code points (0-127) are identical to ASCII
Any text that can be represented in ASCII can be represented the same way in Unicode
Systems that can process Unicode can also process ASCII

This compatibility was crucial for the adoption of Unicode, as it allowed existing ASCII-based systems to transition smoothly to Unicode.

Unicode Encoding Formats

While Unicode defines which number corresponds to which character, it doesn't specify how those numbers should be stored in computer memory. Several encoding formats have been developed to implement Unicode:

UTF-8: A variable-width encoding that uses 1 to 4 bytes per character. ASCII characters use just 1 byte, making it efficient for English text. UTF-8 is the most common encoding on the web.

UTF-16: Uses 2 or 4 bytes per character. It's more efficient than UTF-8 for languages with characters outside the ASCII range but within the Basic Multilingual Plane.

UTF-32: Uses a fixed 4 bytes per character, making it simple but less space-efficient.

UTF-8 has become the dominant encoding format because:

It's backward compatible with ASCII
It's space-efficient for English and European languages
It can represent any Unicode character
It avoids issues with byte order that affect other encodings

How Unicode Works

Let's see how Unicode works with a few examples:

English letter: The uppercase 'A' has the Unicode code point U+0041 (hexadecimal for 65, the same as ASCII). In UTF-8, this is stored as the single byte 01000001.

Spanish character: The letter 'ñ' has the Unicode code point U+00F1. In UTF-8, this is stored as the two bytes 11000011 10110001.

Chinese character: The character '中' (meaning "middle") has the Unicode code point U+4E2D. In UTF-8, this is stored as the three bytes 11100100 10111000 10101101.

Emoji: The smiling face emoji 😊 has the Unicode code point U+1F60A. In UTF-8, this is stored as the four bytes 11110000 10011111 10011000 10001010.

As you can see, Unicode can represent characters from any language or symbol system, making it truly universal.

Comparing ASCII and Unicode

Now that we understand both ASCII and Unicode, let's compare them directly:

Scope and Coverage

ASCII: 128 characters, primarily for English
Unicode: Nearly 150,000 characters from all world languages and symbol systems

Storage Requirements

ASCII: 7 bits per character (typically stored in 8-bit bytes)
Unicode: Variable, depending on the encoding format:
UTF-8: 1 to 4 bytes per character
UTF-16: 2 or 4 bytes per character
UTF-32: 4 bytes per character

Compatibility

ASCII: Limited to English and basic computing needs
Unicode: Includes ASCII as a subset, ensuring backward compatibility

Development and Evolution

ASCII: Developed in the 1960s, essentially fixed since the 1980s
Unicode: Developed in the 1990s, continuously evolving with new characters added regularly

Global Support

ASCII: Supports only English and basic computing
Unicode: Supports all written languages and symbol systems

Why Character Encoding Matters

Understanding character encoding is important for several reasons:

1. Internationalization
Character encoding enables software to support multiple languages and writing systems, making technology accessible to people worldwide. Without Unicode, computers and the internet would be much less inclusive and global.
1. Data Integrity
Using the wrong character encoding can corrupt text data, leading to garbled characters (often called "mojibake"). Understanding encoding helps prevent these issues.
1. Web Development
Web developers need to specify character encoding in their HTML documents to ensure text displays correctly across different browsers and devices.
1. Programming
Many programming tasks involve manipulating text, and understanding character encoding helps programmers handle text data correctly.
1. Data Exchange
When systems exchange text data, they need to agree on a character encoding to ensure the data is interpreted correctly.

Character Encoding in Practice

Let's look at some practical examples of how character encoding works in real-world scenarios:

Example 1: Storing a Name

Imagine you want to store the name "José" in a computer system:

In ASCII:

'J' = 74 = 01001010 in binary
'o' = 111 = 01101111 in binary
's' = 115 = 01110011 in binary
'e' = 101 = 01100101 in binary
'é' = Not available in ASCII!

ASCII can't represent "José" correctly because it doesn't include the character 'é'.

In Unicode (UTF-8):

'J' = U+004A = 01001010 in binary (1 byte)
'o' = U+006F = 01101111 in binary (1 byte)
's' = U+0073 = 01110011 in binary (1 byte)
'e' = U+0065 = 01100101 in binary (1 byte)
'é' = U+00E9 = 11000011 10101001 in binary (2 bytes)

Unicode can represent "José" correctly because it includes the character 'é'.

Example 2: Multilingual Text

Consider the phrase "Hello, 你好, Привет!" which combines English, Chinese, and Russian:

ASCII can only represent the "Hello, " part. The Chinese characters "你好" and the Russian word "Привет" cannot be represented in ASCII.

Unicode can represent the entire phrase:

"Hello, " in ASCII-compatible UTF-8
"你好" using UTF-8 encoding for Chinese characters
"Привет!" using UTF-8 encoding for Cyrillic characters

This multilingual capability is why Unicode has become the standard for text representation in modern computing.

The Future of Character Encoding

Unicode continues to evolve, with new characters and symbols added regularly. Recent additions include:

Historical scripts and rare characters for linguistic and cultural preservation
New emoji and symbols reflecting contemporary communication needs
Technical symbols for specialized fields
Characters for newly documented or revitalized languages

As computing becomes increasingly global and diverse, the importance of comprehensive character encoding standards like Unicode will only grow.

Conclusion: Bridging Human Language and Computer Processing

Character encoding systems like ASCII and Unicode serve as bridges between human language and computer processing. They allow us to communicate with computers using our natural languages, while computers process and store this information in their native binary format.

ASCII played a crucial role in the early development of computing, standardizing how text was represented in computer systems. However, its limitations became apparent as computing spread globally.

Unicode represents a more inclusive approach to character encoding, embracing the diversity of human languages and symbol systems. By providing a universal standard for representing text, Unicode has helped make computing and the internet truly global.

Understanding how computers represent text through character encoding gives us insight into a fundamental aspect of how computers work. It reminds us that even the most basic elements of human communication—letters, numbers, and symbols—must be translated into a language that computers can understand.

Key Concepts to Remember

Bits and Bytes: A bit is the smallest unit of data (0 or 1). A byte is 8 bits and can store a single character or small number.
ASCII: The original character encoding standard that defines 128 characters, primarily for English text.
Unicode: A comprehensive character encoding standard that includes nearly 150,000 characters from all world languages and symbol systems.
Encoding Formats: Different ways to implement Unicode, with UTF-8 being the most widely used due to its efficiency and compatibility with ASCII.
Character Representation: All text characters must be converted to numeric codes and then to binary for a computer to process them.

Select Next to Continue