:

$\text{NLP 101: Hands On with NLTK}$

$\text{Natural language processing (NLP) is the ability of a computer program to understand human language}$

What all tools we need for NLP ?

NLP 101: Text Preprocessing 1 - Cleaning the input

• Tokenization
• Stemming
• Lemmatization
• Stopwords
• POS Tagging
• Name-Entity Recognition

NLP 201: Text Prepocessing 2 - Basic(Input Text -> Vector)

• One hot Encoding
• BOW
• TF/IDF
• Unigram, Bigram N-grams

NLP 202: Text Prepocessing 3 - Advanced(Input Text -> Vector)

• Word-Embeddings
• Word2Vec, Average Word2Vec
• Glove

NLP 301: Deep Learning - Basic(Modelling)

• RNN
• LSTM
• GRU
  

NLP 302: Deep Learning - Advanced(Modelling)

• Encoder-Decoder
• Transformers
• Bert

$\text{Note: Here, We will be looking at NLP 101 tools only.}$

$\text{Terminology in NLP}$

• Corpus: A structured collection of texts used for NLP tasks. It is collection of Documents. for ex. Paragraph
• Documents: Individual units of text within a corpus. It represent a fact or an entity for ex. Sentences
• Vocabulary: The set of unique words in a corpus. for ex. Unique Words

Tokenization

$\text{Tokenization refers to the process of converting a sequence of text into smaller parts known as tokens}$

We tokenize corpus into

• Sentences
• Words

Installation:

• NLTK, which stands for Natural Language Toolkit
• It comes with easy-to-use tools to access lots of different types of language data, like WordNet.
• NLTK also has helpful tools for working with text, like splitting it into words, figuring out what type of word it is, and understanding its meaning.

1. Sentence Tokenizer

$\text{(Corpus -> Document)}$

Note:

• First-time nltk tokenize use or fresh environment requires punkt download.
• sent_tokenize needs punkt tokenizer models; NLTK prompts for download if absent.

2. Word Tokenizer

$\textbf{documents -> Word or corpus -> Word}$

Famous word tokenizers in nltk:

2.1 Word Tokenizer

2.2 Wordpunct Tokenizer

2.3 Tree Bank Tokenizer

2.1 Word Tokenizer:

It helps break down text into individual words, which is useful for understanding and analyzing language in various ways.

2.2 WordPunct Tokenizer:

'wordpunct_tokenize' tokenizer splits text into words and punctuation marks, treating punctuation marks also as separate tokens.

Note: Word tokenizer splits text into words, while wordpunct_tokenizer additionally tokenizes punctuation marks as separate tokens.

2.3 TreebankWord Tokenizer

• The TreebankWord Tokenizer in NLTK is good at breaking down text in a way that's commonly seen in English writing.
• It knows how to handle contractions like "don't" and punctuation. Its tokenization is based on Penn Treebank corpus

For example:

Input Text: "Don't hesitate to ask questions."

Tokenized Output: ['Do', "n't", 'hesitate', 'to', 'ask', 'questions', '.']

In this example, it correctly splits "don't" into "do" and "n't", and treats punctuation like "." as separate tokens. This tool is often used for understanding English text in natural language processing.

Stemming

$\text{Stemming is the process of reducing words to their base or root form to normalize text for analysis.}$

eg:

• [going, gone, goes] -> go
• [eating, eaten, eats] -> eat
  

1. Porterstemmer
2. Snowball Stemmer
3. RegexpStemmer
   

1. Porterstemmer

Source: vijinimallawaarachchi.com

Note1:

• We first find tokenize the document.
• Then, apply predefined rules of porter stemming algorithm (1a-1e) to accurately strip common suffixes in order to find the stem of the word.

Note2:

• It gives incorrect output for some words
  
  

$\text{Disadvantage of Porter Stemming}$

1. Over-stemming: Porter stemming can excessively strip suffixes, leading to stems that aren't linguistically valid, termed over-stemming.
2. Under-stemming: Conversely, it may fail to strip all suffixes when necessary, resulting in related words not sharing the same stem, known as under-stemming.
3. Language Specificity: While effective for English, Porter stemming's rules may not apply well to other languages, limiting its use in multilingual settings.
4. Lack of Semantic Understanding: Operating purely on string manipulation, the algorithm may miss nuances in word variations due to a lack of semantic comprehension.
5. Performance Trade-offs: Porter stemming prioritizes efficiency over linguistic precision, necessitating careful consideration of trade-offs between accuracy and speed for specific tasks.

2. Snowball Stemmer

$\text{It is also known as the Porter2 stemming algorithm as it is a better version of the Porter Stemmer since some issues of it were fixed in this stemmer.}$

Note: Snowball supports multiple languages and not only english.

$\text{Improvement over Porter Stemmer}$

• The Snowball Stemmer tends to be more aggressive in its stemming approach compared to the Porter Stemmer.
• Snowball Stemmer addresses some known issues present in the Porter Stemmer, offering improvements and fixes.
• In Snowball Stemmer, words like 'fairly' and 'sportingly' are stemmed to 'fair' and 'sport', whereas in Porter Stemmer, they are stemmed to 'fairli' and 'sportingli'.
  

3. RegexpStemmer

$\text{A stemmer that uses regular expressions to identify morphological affixes. Any substrings that match the regular expressions will be removed.}$
It combines the power of regular expressions(re) with stemming. It would use regular expressions to identify patterns in words and then apply stemming rules to reduce those words to their base forms.

• RegExpStemmers involve complex regex patterns which can be challenging to handle.
• RegExpStemmers lack context awareness in stemming.
• They are prone to both overstemming and understemming.
• RegExpStemmers can be computationally inefficient, especially with large datasets.
• They might not generalize well across different languages.

Lemmatization

Wordnet Lemmatizer

$\text{The output we get after lemmatization is called 'lemma', which is a root word rather than root stem.}$

How Lemmatization differs from stemming?

• Stemming takes a word down to its root form by removing its prefixes and suffixes.
• Lemmatization considers the context and meaning of a word and tries to convert it to a more meaningful and easier-to-work format.

For example:

• The words was, is, and will be can all be lemmatized to the word be.
• Similarly, the words better and best can be lemmatized to the word good.

Note: Generally, lemmatization is more sophisticated and accurate than stemming but can also be more computationally expensive.

Note:

• First-time NLTK use or fresh environment requires wordnet download.
• WordNetLemmatizer needs wordnet model; NLTK prompts for download if absent.

It is giving you result with respect to noun by default, but since POS for eating is verb we need to pass it manually to lemmatize.

pos

• noun - n
• verb = v
• adjective - a
• adverb - r

Stopwords

$\text{Stopwords are the words in any language which does not add much meaning to a sentence and can be removed without sacrificing the meaning of sentence.}$
Note1: "stop words" usually refers to the most common words in a language.

Note2: stopwords from different language is available in nltk library.

$\textbf{Pros and Cons of Stopwords}$

Pros:

• Efficiency: Removing stop words reduces dataset size and training time.
• Improved Performance: Eliminating stop words enhances token significance and classification accuracy.

Cons:

• Semantic Alteration: Improper stop word selection can change text meaning.

Note: $\textbf{Often you will need to do modification i.e. add or remove stopword suitable to your use case}$ $\text{For example: You my sometime want to 'not' outside the stopwords to for negative sentiments.}$

Let's use it in corpus

Note: In Snowball, all words are in lowercase.

Part of Speech Tagging

$\text{It involves assigning a part-of-speech tag (such as noun, verb, adjective, etc.) to each word in a given text.}$

Source: ByteIota

The purpose of POS tagging is:

• to analyze the grammatical structure of sentences
• identify the syntactic roles of individual words within sentences.

Note: POS tagging can be performed using various techniques, including rule-based approaches, statistical models, and deep learning methods.

Some important encoding we get will nltk pos-tagging with what each represesnt.

Note:

• First-time NLTK use or fresh environment requires you to download tagger
• NLTK prompts for download if absent.

Note:

• First-time nltk pos_tagger use or fresh environment requires punkt download.

• pos_tagging on tokenized words.

• pos_tagging on under unprocessed words in sentences

Name-Entity Recoginition

Source: MonkeyLearn

Note 1: NER enables the extraction of structured data from documents, improving search, analytics, and integration across systems.

Note 2: NER can be performed using various techniques, including rule-based approaches, ML method like HMM and deep learning methods.

Note:

• First-time NLTK use or fresh environment requires maxent_ne_chunker and word download for nltk NER.
• NLTK prompts for download if absent.

$\textbf{You may need to install:}$

Now, let's do NER