NLP Foundations : Text Preprocessing & Vectorization

Business Scenario

Welcome!

In Lab 2 you built the network’s training loop step-by-step in Keras/TensorFlow without .fit(), separated the forward pass, loss, backpropagation, and gradient descent using tf.GradientTape

SmartCart's search bar receives raw, messy customer text -- search queries like "CHEAP wireless earbuds under 1000!!" and short reviews like "delivery was late but the laptop quality is good." Before any model (including the RNN in the next lab) can use this text, it has to be cleaned, tokenized, normalized, and turned into numbers.

Git Pull

git pull origin branchName

Pre-Lab Preparation

Topic : NLP Foundation

1) Introduction to NLP
2) Text Processing - (Cleaning, tokenization, lemmatization)
3) Vectorization - Bag of words(BoW) / TF-IDF
4) Word Embeddings

Natural Language Processing

NLP is the field concerned with getting computers to work with human language. Neural networks only understand numbers, so every NLP pipeline has to answer the same question: "how do we turn a messy sentence into numbers without throwing away its meaning?"

Setup

1

Task 1: Loading Dataset

import re
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.tokenize import word_tokenize

import tensorflow as tf
from tensorflow import keras
from keras.layers import TextVectorization, Embedding, GlobalAveragePooling1D, Dense
from keras import Sequential

from sklearn.preprocessing import LabelEncoder

np.random.seed(42)
tf.random.set_seed(42)

Load Dataset

2

Dataset : 

# One-time downloads for nltk's tokenizer, stopword list, and lemmatizer dictionary

nltk.download('punkt')
nltk.download('punkt_tab')
nltk.download('stopwords')
nltk.download('wordnet')
nltk.download('omw-1.4')
df = pd.read_csv('smartcart_customer_queries.csv')
print(df.shape)
df.head(10)
df['category'].value_counts()

Task 2 : Text Cleaning

def clean_text(text):
    text = str(text).lower()
    text = re.sub(r'[^a-z0-9\s]', ' ', text)
    text = re.sub(r'\s+', ' ', text).strip()
    return text

df['clean_text'] = df['query_text'].apply(clean_text)
df[['query_text', 'clean_text']].sample(5)

Lowercase everything, strip punctuation/special characters and collapse extra whitespace. Numbers are kept, since prices ("under 1000") are meaningful for SmartCart search.

Output

Task 3 : Tokenization

df['tokens'] = df['clean_text'].apply(word_tokenize)
df[['clean_text', 'tokens']].sample(5, random_state=1)

Split each cleaned string into individual word tokens with nltk's word_tokenize.

Output

Task 3 : Stopword Removal 

stop_words = set(stopwords.words('english'))

def remove_stopwords(tokens):
    return [t for t in tokens if t not in stop_words]

df['tokens_no_stop'] = df['tokens'].apply(remove_stopwords)
df[['tokens', 'tokens_no_stop']].sample(5, random_state=1)

Very common words ("the", "is", "a") carry little distinguishing meaning for search or classification, so they're usually dropped before vectorizing.

Output

Back Propagation

3

def backward(model, X, y_true):
    with tf.GradientTape() as tape:
        y_pred = forward(model, X)
        loss = compute_loss(y_true, y_pred)

    gradients = tape.gradient(loss, model.trainable_variables)
    return loss, gradients
optimizer = keras.optimizers.SGD(learning_rate=0.1)

def update(model, gradients, optimizer):
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

Weight Update

4

Loss Calculation

2

loss_fn = keras.losses.SparseCategoricalCrossentropy()
def compute_loss(y_true, y_pred):
    return loss_fn(y_true, y_pred)

Training Loop and Loss Tracking

5

loss_history = []
for epoch in range(500):
    loss, gradients = backward(model, X_train_tf, y_train_tf)
    update(model, gradients, optimizer)

    loss_history.append(float(loss))

    if epoch % 50 == 0:
        print(f"Epoch {epoch:4d} │ Loss: {float(loss):.4f}")
plt.plot(loss_history)
plt.xlabel("Epoch")
plt.ylabel("Sparse categorical cross-entropy loss")
plt.title("SmartCart ANN (Keras) — manual GradientTape training loop loss")
plt.show()

Output

Model Evaluation

6

test_probs = model(X_test_tf, training=False)       # no training here

y_pred = np.argmax(test_probs.numpy(), axis=1)      # highest-probability class

print("Test accuracy:", round(accuracy_score(y_test, y_pred), 4))

Output

 

Great job!

You have successfully rebuilt the Lab 1 network’s training loop step-by-step in Keras/TensorFlow without .fit(), separated the forward pass, loss, backpropagation, and gradient descent using tf.GradientTape, tracked loss reduction over 500 epochs, and verified comparable test accuracy on the same SmartCart dataset.

Checkpoint

   Git Push

git push origin branchName

Next-Lab Preparation

Topic : NLP Foundation

1) Introduction to NLP
2) Text Processing - (Cleaning, tokenization, lemmatization)
3) Vectorization - Bag of words(BoW) / TF-IDF
4) Word Embeddings

GenAI - 3

By Content ITV

GenAI - 3

  • 17

More from Content ITV