5 NLP techniques every Data Scientist should know
Data Scientists deal with huge data that can be natural language text. Hence, comes the Natural Language Processing (NLP) that one needs to have an idea to perform analysis on textual data.
Companies are getting deeper into data and need to work on different data formats likes documents, emails, JSON, spreadsheets etc which needs to be processed and analyzed with easy to use containers for storing data which si why NLP becomes a crucial necessity.
Here are the NLP techniques that every data scientist needs to be aware of:
1. Tokenization: Tokenization is a way of separating a piece of text into smaller units called tokens. Here, tokens can be either words, characters, or subwords. Hence, tokenization can be broadly classified into 3 types — word, character, and subword (n-gram characters) tokenization.
For example, consider the sentence: “Never give up”.
Assuming space as a delimiter, Word Tokenization– Never-give-up.
Character tokens: s-m-a-r-t-e-r
Subword tokens: smart-er
2. Stemming and Lemmatization: Stemming is the process of reducing the words (generally modified or derived) to their word stem or root form. The objective of stemming is to reduce related words to the same stem even if the stem is not a dictionary word.
For example, in the English language- beautiful and beautifully are stemmed to beauti
good, better and best are stemmed to good, better and best respectively.
Lemmatisation is the process of reducing a group of words into their lemma or dictionary form. It takes into account things like POS (Parts of Speech), the meaning of the word in the sentence, the meaning of the word in the nearby sentences etc. before reducing the word to its lemma.
For example, in the English Language-
beautiful and beautifully are lemmatised to beautiful and beautifullyrespectively.
good, better and best are lemmatised to good, good and good respectively.
3. Word Embeddings: They are useful because of computers’ inability to process Natural Language. Word Embeddings capture the essence and relationship between words in a Natural Language using real numbers.
For Example: A word “man” might be represented in a 5-dimension vector as
3.2 , 4.6 , 5 3, 2 , 0.6
4. Term frequency: Inverse Document Frequency: TF-IDF is used as a weighting factor including text mining and information retrieval. TF helps to measure the document string that are frequently coming up. IDF helps in measuring the string importance in data.
TF = (Number of times term t appears in a document)/(Number of terms in the document)
IDF = log(N/n), where, N is the number of documents and n is the number of documents a term t has appeared in.
TF-IDF = TF*IDF
5. Sentiment Analysis:Sentiment Analysis is a broad range of subjective analysis which uses Natural Language processing techniques to perform tasks such as identifying the sentiment of a customer review, positive or negative feeling in a sentence, judging mood via voice analysis or written text analysis etc.
For example-
“I did not like the chocolate ice-cream” — is a negative experience of ice-cream
“I did not hate the chocolate ice-cream” — may be considered as a neutral experience
Hope this article gave insights to few of the important NLP techniques.
