Text mining: TF-IDF, search and labels
Bag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.
Create a free account Try part of it here
Why it matters
Support tickets, reviews, survey answers, logs and chat transcripts pile up faster than anyone can read them. Before you reach for a large model, it's worth knowing the classic tools: they run in milliseconds on a laptop, cost nothing per document, and explain themselves. They're also the baseline every LLM approach should beat, and the right choice more often than people expect. This lesson works through 24 support tickets from the course's gear shop.
What you'll be able to do
- Turn text into numbers with word counts and TF-IDF, and say why rare words weigh more
- Find similar documents with cosine similarity on TF-IDF vectors, and see where keywords fail
- Train a fast, explainable text classifier, and group unlabeled text into rough topics
- Decide between TF-IDF methods and LLM labels on cost, speed and accuracy, and measure agreement
What it covers
- Words into numbers
- Find the bug: one word counted three ways
- TF-IDF: weight the words that tell documents apart
- Inverse document frequency
- Search by similarity, and where keywords fail
- Classifying, and finding topics without labels
- Topics without labels
- Using LLM labels well
Show the answer
refund
"the" and "my" appear in nearly every ticket, and "order" in many. "refund" shows up in only a few, and those few are nearly all about returns. Words that are common everywhere say little; that's the idea behind TF-IDF.
Words into numbers
A model needs numbers, so the first step is to turn each document into a vector. The simplest way is a bag of words: build a vocabulary of every word seen, then represent a document by how many times it uses each one. Word order is thrown away, which sounds drastic but keeps enough to tell a delivery problem from a sizing question.
from sklearn.feature_extraction.text import CountVectorizer tickets = [ "My order still hasn't arrived after ten days", "Tracking says delivered but there's no parcel", "When will my tent ship? It says processing", ] counts = CountVectorizer().fit(tickets) print(len(counts.vocabulary_), "words") print(counts.transform(["my parcel says delivered"]).toarray()[0].sum(), "of them in the new ticket")
CountVectorizer lowercases and splits on anything that isn't a letter or digit. Real data needs the same care by hand when you do it yourself: "Tent," and "tent" must be the same word, or the counts split.
Find the bug: one word counted three ways
word_counts(texts) should count each word across all the texts, treating "Tent,", "tent" and "TENT!" as the same word. Right now they're counted separately. Fix it.
import string from collections import Counter def word_counts(texts): counts = Counter() for text in texts: for word in text.split(): counts[word] += 1 return counts print(word_counts(["Tent, tent!", "TENT and poles"]).most_common(2))
Write a sentence or two, then compare it with a good answer in the lesson.
TF-IDF: weight the words that tell documents apart
Raw counts let common words dominate. TF-IDF fixes that by multiplying two parts. Term frequency is how often a word appears in this document. Inverse document frequency is how rare it is across all documents:
In plain words: the fewer documents a word appears in, the more it tells you about the ones it does appear in.
Here is the number of documents and how many of them contain the word. A word in every document gets : no weight at all. A word in 2 of 24 documents gets . scikit-learn adds 1 to both counts and 1 to the result, so no word ever gets exactly zero, and it scales each vector to length 1 so long tickets don't outweigh short ones. Common words like "the" and "my" can also be dropped outright as stop words.
import math print(round(math.log(24 / 4), 2))
Show the answer
1.79
log(24/4) = log 6 ≈ 1.79, a middling weight: "tent" is in a sixth of the tickets. "refund", in 2 of 24, gets log 12 ≈ 2.48.
Inverse document frequency
Write idf(term, docs): the natural log of the number of documents divided by how many documents contain term as a word (compare lowercased words from split()). Return 0.0 when no document contains it.
import math def idf(term, docs): return 1.0 docs = ["When will my tent ship", "Which tent size fits two", "Refund for my stove", "Mug arrived cracked"] print(round(idf("tent", docs), 3))
Search by similarity, and where keywords fail
With every ticket as a TF-IDF vector, "find tickets like this one" is cosine similarity, as in the embeddings lesson:
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity tickets = ["My order still hasn't arrived after ten days", "Refund for my returned stove hasn't come through", "Still waiting on my refund after the return", "Mug arrived cracked", "Do the boots run small"] tfidf = TfidfVectorizer(stop_words="english").fit(tickets) scores = cosine_similarity(tfidf.transform(["my refund never arrived"]), tfidf.transform(tickets))[0] for score, text in sorted(zip(scores, tickets), reverse=True)[:3]: print(round(score, 2), text)
The refund ticket that says "refund" ties for first with "Mug arrived cracked", and the late-order ticket comes third, both purely because they share the word "arrived". The other refund ticket, which says "Refund for my returned stove", misses the top three: it shares only one word and has many others. Keyword vectors don't know that "parcel" and "package" mean the same, or that "never arrived" is about delivery, not damage. That's what embeddings add, at the cost of a model call per document. TF-IDF is still excellent for exact terms such as product codes, error messages and names, which is why search systems often combine both.
Show the answer
"my parcel is late" and "the package hasn't come"
The first pair shares no words at all, so its TF-IDF cosine is 0, yet it means the same thing. The others share a key word, so even TF-IDF links them.
Write a sentence or two, then compare it with a good answer in the lesson.
Classifying, and finding topics without labels
With labeled tickets, TF-IDF plus logistic regression makes a strong classifier that trains in milliseconds. Its weights are readable: the words with the largest weights for "returns" are literally the words that say returns.
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression train = [("My order still hasn't arrived", "shipping"), ("The parcel is stuck at the depot", "shipping"), ("Where is my package, tracking stopped", "shipping"), ("I want to return the fleece", "returns"), ("Refund for my returned stove", "returns"), ("How do I send back a mug", "returns"), ("The tent pole snapped", "damaged"), ("Mug arrived cracked", "damaged"), ("Stove leaks at the valve", "damaged")] tfidf = TfidfVectorizer(stop_words="english") model = LogisticRegression(max_iter=1000).fit(tfidf.fit_transform([t for t, _ in train]), [y for _, y in train]) print(list(model.predict(tfidf.transform(["the lamp arrived broken", "please refund my order", "parcel never came"]))))
Look at the middle answer. "please refund my order" should be returns, but "order" appears only in a shipping ticket here, and nine examples aren't enough to outvote it. Small labeled sets make confident mistakes like this; a few hundred labels per class usually fix it, and reading the errors tells you which labels to add.
Single words also lose phrases: "not working" and "working" share their only content word. Adding two-word phrases with ngram_range=(1, 2) lets the model weigh "not working" on its own, at the cost of a much bigger vocabulary.
Show the answer
"order" appeared only in a shipping ticket
With nine examples, one word seen only in shipping tickets can outweigh "refund". More labeled tickets, especially returns that mention orders, fix it.
Topics without labels
Without labels, you can still cluster the TF-IDF vectors (k-means works) and read each cluster's top words as a topic. On a few dozen short tickets the clusters come out rough; on thousands they surface themes nobody had a category for yet. Naming and checking the topics is human work.
Try a few cluster counts and read a sample of tickets from each cluster, not only its top words: a cluster whose top words are "week", "parcel" and "broke" might mix delivery delays with items that broke within a week. Topics are a way to find questions worth asking, not answers to report as they stand.
Think about volume, cost per document, and whether the job needs understanding or just keywords.
Using LLM labels well
An LLM can label text with no training data: give it the categories with short definitions and a few examples, and ask for one label per ticket. That makes it ideal for getting started, and for small or subtle jobs. At large volume it's slower and costs money per document, and its mistakes are harder to explain.
A common pattern combines the two. Have the LLM label a few thousand documents, check a sample against a person's labels, then train a TF-IDF (or embedding) classifier on the LLM's labels and run that on everything. The cheap model does the volume; the LLM did the expensive part once.
Whichever labels the data, measure agreement on a sample that a careful person also labeled: the share of tickets where both agree, per category, and a look at every disagreement. Disagreements often reveal fuzzy category definitions rather than a bad model.
Two practical costs come with any API model. Tickets contain names, addresses and order numbers, so check what your provider keeps and strip what the labels don't need. And price the volume: a fraction of a cent per ticket is nothing for a sample and real money across millions a month, so price it before you build on it.
Show the answer
Whether those two category definitions overlap
Disagreements concentrated between two categories usually mean the categories themselves are ambiguous, such as a broken item someone wants to send back. Sharpen the definitions, then relabel.
agreed, total = 22, 24 print(round(agreed / total, 3))
Show the answer
0.917
22 of 24 is 91.7% agreement. Before trusting it, read the two disagreements: they often point at a category whose definition needs sharpening.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Why does TF-IDF give "the" almost no weight?