🏢 The situation
The head of support comes to Lena:
"5,000 tickets a day. Agents spend the first 40 seconds of every ticket figuring out where it belongs: payments, delivery, returns, account, fraud. That's 55 person-hours a day just on sorting!"
Lena:
"Auto-classification of tickets — the perfect first NLP task. We have 200,000 historical tickets with categories assigned by agents. Start with TF-IDF + a linear model, then compare with a pre-trained transformer. Real language, typos, slang — welcome to real-world NLP."
🎯 Your task
- Understand how text becomes numbers: from bag-of-words to embeddings.
- Build the TF-IDF + logreg baseline.
- Compare with a fine-tuned transformer and decide what goes to production.
📚 Theory
Text → numbers: the evolution of approaches
- Bag of Words: text = a vector of word counts. Loses word order.
- TF-IDF: counts weighted by corpus rarity. A word frequent in a document but rare in the corpus ("refund") matters more than one frequent everywhere ("hello"). Still the strongest classification baseline.
- Word embeddings (word2vec, fastText): word → dense vector; semantically similar words are close. fastText learns vectors from character n-grams — robust to typos.
- Transformers (BERT and heirs): a word's vector depends on context ("the card is blocked" vs "a card of the city"). Pre-trained on huge corpora, fine-tuned for the task.
The classic NLP pipeline
- Tokenization — split into words/tokens.
- Normalization: lowercase; lemmatization (critical for morphologically rich languages: "payments", "payment's", "paid" → the lemma).
- N-grams: word pairs catch "not working", "money not received" — negations change the meaning.
When to use what
| Approach | Pros | Cons |
|---|---|---|
| TF-IDF + linear model | fast, cheap, interpretable, CPU | doesn't understand context or synonyms |
| Fine-tuned transformer | best quality, robust to phrasing | GPU, latency, harder maintenance |
The decision, as always, comes from the requirements: latency, budget, size of the win.