Natural Language Processing

Question 1

Pipeline of natural language processing

Answer 1

1) Text processing
2) Feature extraction
3) Modeling

Question 2

Text processing

Answer 2

• When reading html eliminate tags
• Put all letters in lowercase
• Sometimes it is a good idea to remove punctuation
• Sometimes it is a good idea to remove words like “are, for, a, the”

Question 3

Feature extraction

Answer 3

• Letters in Unicode are represented by numbers, but they if they are treated as numbers that can affect the models
• The are many ways to represent text info
• If you want a graph based model to discover insights you want to represent words as node with relations
• If you want to recognize spam or text sentiment use bag of words
• Text generation or translation use word-to-vec

Question 4

Modeling

Answer 4

• Create a statistical or machine learning model

Question 5

How to read a file in python

Answer 5

with open("hola.txt", "r") as f:
text = f.read()

Question 6

How to read tabular data or csv

Answer 6

• You can use panda

- df = pd.read_csv(“hola.csv”)

Question 7

How to get a website or a file in the web?

Answer 7

import requests
# Fetch a web page
r = requests.get("https://www.udacity.com/courses/all")

Question 8

How to clean the text from a website?

Answer 8

• Use a library
  from bs4 import BeautifulSoup

# Remove HTML tags using Beautiful Soup library
soup = BeautifulSoup(r.text, "html5lib")
print(soup.get_text())

Question 9

Tips for text cleaning

Answer 9

• All letters to lower case

- In document classification or clustering eliminate punctuation

Question 10

How to eliminate punctuation

Answer 10

• Use the regular expressions library “re”
• import re
• Replace punctuation with a space

Question 11

Useful libraries

Answer 11

• NLTK
• BeautifulSoup
• re

Question 12

What is a token

Answer 12

• Is something that represents a unique concept like a dog

Question 13

Tokenization with NLTK

Answer 13

• Token words with word_tokenize. It is smarter than split
• Tokenize sentences with sent_tokenize
• To tokenize twitters

Question 14

Stop word removal

Answer 14

• Eliminate words like “are”,”the” that don’t give extra info
• nltk has a list of stop words
• [word for word in querywords if word.lower() not in stopwords]

Question 15

Part-of-speech tagging

Answer 15

• It is helpful on some applications to classify words by verbs, nouns, etc
• Use NLTK pos_tag

Question 16

Named entity recognition

Answer 16

• Classify a noun by the type of entity: person, organization, government, etc
• Use NLTK ne_chunk

Question 17

Stemming and Lemmatization

Answer 17

• Methods to turn variations of a word to a stem or root word. Example: Started»_space; Start
• In stemming the result is not always a real word, but it is more efficient
• NLTK has a stemming method PorterStemmer.stem()
• lemmatization is more computationally consuming because it uses a dictionary, but the result word is a real word
• NLTK has a lemmatization method WordNetLemmatizer.lemmatize() and it is setup by default to nouns

Question 18

Lesson summary

Answer 18

1) Normalize
2) Tokenize
3) Remove stop words
4) Stem / Lemmatize

Question 19

Bag of words

Answer 19

• Interpret each document as a group of words without order
• To compare how similar are two bag of words use dot product and cosine similarity
• The document is a vector with the frequency of each word
• To compare documents make a matrix, columns are word frequency and rows are the documents
• Each word has the same importance

Question 20

TF-IDF

Answer 20

• Highlight words that are more unique to a document

- tdidf = td*idf = count(d,t)/|d| * log(|D|/|dED:tEd|)

Question 21

Word2Vec

Answer 21

• The idea is a model able to predict a word given neighboring words (Continuous Bag of Words) or given word predict the neighbors (Continuous Skipgram-model)

Question 22

GloVe

Answer 22

• Use a co-occurrence probability matrix of words of a document.
• P(Water | Ice) = 0.2

Question 23

language model

Answer 23

A language model captures the distributional statistics of words. In its most basic form, we take each unique word in a corpus, i, and count how many times it occurs.

Question 24

Bigram Model

Answer 24

• Matrix of a corpus that tells the probability of a word to occurs given another previous word
• It is used for generating text

Question 25

Regex eliminate punctuation

Answer 25

- "\W" eliminate non letter characters