Tokenization in NLP

 

Tokenization 

Tokenization is the process of splitting text into smaller units called tokens (words, phrases, or subwords) for analysis.

Types of Tokenization

(a) Word Tokenization

  • Splits text into words based on spaces or punctuation.
  • Example:
    from nltk.tokenize import word_tokenize
text = "I love Natural Language Processing!"
print(word_tokenize(text))
    Output: ['I', 'love', 'Natural', 'Language', 'Processing', '!']

(b) Sentence Tokenization

  • Splits text into sentences based on punctuation like . or !.
  • Example:
    from nltk.tokenize import sent_tokenize
text = "NLP is amazing. It helps machines understand language."
print(sent_tokenize(text))
    Output: ['NLP is amazing.', 'It helps machines understand language.']

(c) Subword Tokenization

  • Breaks words into smaller meaningful units, used in deep learning models (e.g., BERT, WordPiece).
  • Example:
    • "unhappiness" → "un", "happiness"
    • "playing" → "play", "##ing"

(d) Character Tokenization

  • Splits text into individual characters, useful for languages without spaces (like Chinese).
  • Example: "hello" → ['h', 'e', 'l', 'l', 'o']

Tokenization Libraries in Python

Using NLTK (Natural Language Toolkit)

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize

text = "Tokenization is an important step in NLP. It helps process text."
print(word_tokenize(text))  # Word tokenization
print(sent_tokenize(text))  # Sentence tokenization

Using SpaCy

import spacy
nlp = spacy.load("en_core_web_sm")

text = "Tokenization is important in NLP."
doc = nlp(text)

tokens = [token.text for token in doc]
print(tokens)  # ['Tokenization', 'is', 'important', 'in', 'NLP', '.']

Using Hugging Face Tokenizers (For Transformer Models)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("Tokenization in NLP is crucial!")
print(tokens)  # ['tokenization', 'in', 'nlp', 'is', 'crucial', '!']

Challenges in Tokenization

  1. Handling Different Languages – Some languages (e.g., Chinese) don’t have spaces.
  2. Ambiguity – "New York" should be one token, not two.
  3. Slang & Abbreviations – "gonna" vs. "going to".
  4. Handling Hyphenated Words – "mother-in-law" should be one token.

Applications of Tokenization

  • Search Engines (Indexing words for faster search).
  • Chatbots & Assistants (Understanding user queries).
  • Text Classification (Tokenized words as input for ML models).
  • Sentiment Analysis (Breaking text for polarity detection).

Comments

Post a Comment

Popular posts from this blog

Dependency Parsing in NLP

  1. What is Dependency Parsing? Dependency Parsing is a process in Natural Language Processing (NLP) that establishes grammatical relationships between words in a sentence. Instead of breaking a sentence into hierarchical phrases (like Constituency Parsing), it represents relationships using a Dependency Tree , where: Each word is connected to another word through directed edges (arcs) . The main verb of the sentence is often the root of the tree. Words are connected by dependencies like subject, object, modifier, etc. 2. Why is Dependency Parsing Important? Dependency parsing is widely used because it captures the syntactic structure of a sentence concisely, making it useful for: Machine Translation (e.g., Google Translate) Chatbots & Virtual Assistants (e.g., Siri, Alexa) Text Summarization Relation Extraction (e.g., extracting "Apple acquired Beats" from a sentence) Sentiment Analysis 3. Dependency Structure Example Consider the sentence: "...
Read more

Challenges in NLP

  Challenges in NLP Natural Language Processing (NLP) faces several challenges due to the complexity, ambiguity, and diversity of human language.  1. Ambiguity in Language Lexical Ambiguity : Words with multiple meanings (e.g., bank can mean a financial institution or a riverbank). Syntactic Ambiguity : Different possible sentence structures (e.g., "The chicken is ready to eat" - is the chicken eating or being eaten?). Semantic Ambiguity : Different meanings depending on context (e.g., "I saw the man with the telescope." - who has the telescope?). Pragmatic Ambiguity : Understanding implied meanings (e.g., "Can you pass the salt?" is a request, not a yes/no question). 2. Understanding Context and Sarcasm NLP struggles to interpret context-specific meaning. Sarcasm and irony are difficult to detect because they often contradict literal meanings. Example: "Oh great, another rainy day!" (Positive words but negative meaning). 3. D...
Read more

Syntax Analysis (Parsing) in NLP

1. Introduction to Syntax Analysis Syntax analysis, or parsing , is the process of analyzing the grammatical structure of a sentence to determine its syntactic relationships. It ensures that a given input follows the grammatical rules of a language. Syntax analysis is an essential step in NLP tasks such as machine translation, information extraction, and question-answering systems. 2. Steps in Syntax Analysis Tokenization – The sentence is broken down into individual words or tokens. Part-of-Speech (POS) Tagging – Assigns POS tags (noun, verb, adjective, etc.) to words. Parsing – Constructs a parse tree or dependency graph to analyze sentence structure. 3. Types of Parsing There are two major types of parsing techniques: A. Constituency Parsing (Phrase Structure Parsing) Represents a sentence using a Parse Tree based on grammar rules (Context-Free Grammar, CFG). Breaks the sentence into hierarchical phrases like noun phrases (NP) and verb phrases (VP). Exam...
Read more