Stemming vs. Lemmatization in NLP

 

Stemming vs. Lemmatization 

Stemming and Lemmatization are text normalization techniques in Natural Language Processing (NLP). Both methods reduce words to their base or root form, but they differ in how they achieve this.

1. What is Stemming?

Stemming is the process of reducing a word to its root form by removing prefixes and suffixes (affixes). It applies heuristic rules (not dictionary-based), which may sometimes produce non-meaningful words.

Example of Stemming:

Original Word           Stemmed Word
Running                  run
Studies                                studi
Happily                  happi
Better                  better (incorrect as "bet" is expected)

Stemming does not guarantee valid words (e.g., "happily" → "happi").
It is faster and less accurate than lemmatization.

2. What is Lemmatization?

Lemmatization reduces words to their dictionary form (lemma) based on linguistic rules. Unlike stemming, it considers context and part of speech (POS), producing meaningful root words.

Example of Lemmatization:

Original Word     Lemmatized Word
Running     run
Studies     study
Happily     happy
Better     good

Lemmatization ensures valid words (e.g., "happily" → "happy").
It is slower but more accurate than stemming because it uses a dictionary lookup.

3. Differences Between Stemming and Lemmatization

Feature Stemming Lemmatization
Definition Reduces words by chopping off prefixes/suffixes Converts words to their dictionary form
Speed Faster (rule-based) Slower (dictionary-based)
Accuracy Less accurate More accurate
Output May produce non-meaningful words Always produces valid words
Context-aware? No (blindly removes suffixes) Yes (considers POS & meaning)
Use Case Simple text pre-processing (e.g., search engines) NLP tasks requiring high accuracy (e.g., chatbots, machine translation)

4. Stemming in Python (Using NLTK)

NLTK provides different stemming algorithms, with Porter Stemmer and Lancaster Stemmer being the most common.

4.1 Using Porter Stemmer

from nltk.stem import PorterStemmer

ps = PorterStemmer()
words = ["running", "studies", "happily", "better"]

stemmed_words = [ps.stem(word) for word in words]
print(stemmed_words)

Output

['run', 'studi', 'happili', 'better']

"studies" → "studi" and "happily" → "happili" (incorrect stems).

4.2 Using Lancaster Stemmer

The Lancaster Stemmer is more aggressive than Porter Stemmer.

from nltk.stem import LancasterStemmer

ls = LancasterStemmer()
words = ["running", "studies", "happily", "better"]

stemmed_words = [ls.stem(word) for word in words]
print(stemmed_words)

Output

['run', 'study', 'happy', 'bet']

"better" → "bet" (incorrect).

5. Lemmatization in Python (Using NLTK & SpaCy)

5.1 Lemmatization Using WordNetLemmatizer (NLTK)

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()
words = ["running", "studies", "happily", "better"]

lemmatized_words = [lemmatizer.lemmatize(word, pos="v") for word in words]  # "v" means verb
print(lemmatized_words)

Output

['run', 'study', 'happy', 'good']

"better" → "good" (correct transformation).

5.2 Lemmatization Using SpaCy

SpaCy provides part-of-speech (POS) tagging for more accurate lemmatization.

import spacy

nlp = spacy.load("en_core_web_sm")
text = "running studies happily better"
doc = nlp(text)

lemmatized_words = [token.lemma_ for token in doc]
print(lemmatized_words)

Output

['run', 'study', 'happily', 'good']

"happily" remains unchanged because it is an adverb.


6. When to Use Stemming vs. Lemmatization?

Scenario Use Stemming? Use Lemmatization?
Search Engines (Fast but approximate matching)        Yes    No
Chatbots & NLP Apps (Accuracy needed)        No    Yes
Sentiment Analysis        No    Yes
Machine Translation        No    Yes
Keyword Extraction (General text processing)        Yes     No

7. Challenges in Stemming and Lemmatization

7.1 Challenges in Stemming

  1. May produce non-dictionary words ("running" → "runn")
  2. Different stemmers give different results
  3. Struggles with irregular words ("better" → "bet")

7.2 Challenges in Lemmatization

  1. Requires POS tagging for better results
  2. Slower due to dictionary lookup
  3.  Requires external libraries like WordNet or SpaCy


Comments

Post a Comment

Popular posts from this blog

Dependency Parsing in NLP

  1. What is Dependency Parsing? Dependency Parsing is a process in Natural Language Processing (NLP) that establishes grammatical relationships between words in a sentence. Instead of breaking a sentence into hierarchical phrases (like Constituency Parsing), it represents relationships using a Dependency Tree , where: Each word is connected to another word through directed edges (arcs) . The main verb of the sentence is often the root of the tree. Words are connected by dependencies like subject, object, modifier, etc. 2. Why is Dependency Parsing Important? Dependency parsing is widely used because it captures the syntactic structure of a sentence concisely, making it useful for: Machine Translation (e.g., Google Translate) Chatbots & Virtual Assistants (e.g., Siri, Alexa) Text Summarization Relation Extraction (e.g., extracting "Apple acquired Beats" from a sentence) Sentiment Analysis 3. Dependency Structure Example Consider the sentence: "...
Read more

Challenges in NLP

  Challenges in NLP Natural Language Processing (NLP) faces several challenges due to the complexity, ambiguity, and diversity of human language.  1. Ambiguity in Language Lexical Ambiguity : Words with multiple meanings (e.g., bank can mean a financial institution or a riverbank). Syntactic Ambiguity : Different possible sentence structures (e.g., "The chicken is ready to eat" - is the chicken eating or being eaten?). Semantic Ambiguity : Different meanings depending on context (e.g., "I saw the man with the telescope." - who has the telescope?). Pragmatic Ambiguity : Understanding implied meanings (e.g., "Can you pass the salt?" is a request, not a yes/no question). 2. Understanding Context and Sarcasm NLP struggles to interpret context-specific meaning. Sarcasm and irony are difficult to detect because they often contradict literal meanings. Example: "Oh great, another rainy day!" (Positive words but negative meaning). 3. D...
Read more

Syntax Analysis (Parsing) in NLP

1. Introduction to Syntax Analysis Syntax analysis, or parsing , is the process of analyzing the grammatical structure of a sentence to determine its syntactic relationships. It ensures that a given input follows the grammatical rules of a language. Syntax analysis is an essential step in NLP tasks such as machine translation, information extraction, and question-answering systems. 2. Steps in Syntax Analysis Tokenization – The sentence is broken down into individual words or tokens. Part-of-Speech (POS) Tagging – Assigns POS tags (noun, verb, adjective, etc.) to words. Parsing – Constructs a parse tree or dependency graph to analyze sentence structure. 3. Types of Parsing There are two major types of parsing techniques: A. Constituency Parsing (Phrase Structure Parsing) Represents a sentence using a Parse Tree based on grammar rules (Context-Free Grammar, CFG). Breaks the sentence into hierarchical phrases like noun phrases (NP) and verb phrases (VP). Exam...
Read more