I trained a model on 729 of my own ratings. It predicts them well. It is also worse than sorting by release date at the job I actually wanted it for.
I keep a small database of everything I watch, with a 1 to 5 rating. The obvious thing to build on top of that is a model that learns my taste and recommends things. So I built one, mostly to have a low stakes place to practice.
Two things came out of it that I did not expect. The first happened before I trained anything.
Every watched title in the database has vibe tags on it, written by an LLM.
Things like mind-bending or background-watchable.
They looked like the strongest signal available, and the plan was to train on
them. They cannot be used, for two reasons that are independent enough that
either one alone would have killed them.
Vibes are on 729 of 729 watched titles and 0 of the 1,371 unwatched ones. The entire job is scoring things I have not seen. The feature is absent exactly where the model has to operate. Train on it and you get a model that works in testing and has nothing to read at serving time.
The prompt that generates the tags is handed my rating. So the tags were
written by something that already knew the score. Mean rating runs from 3.04
for background-watchable up to 4.22 for mind-bending,
against a global mean of 3.516. Some of that spread is real. Some of it is a
model restating the answer back to me in adjectives.
The fix is not to drop the idea. It is to regenerate the tags without showing the model the rating, over all 2,400 titles, into a separate column. Then the feature exists at training and serving time both, and comparing it against raw metadata becomes a real experiment: does an LLM's blind read of a title beat the numbers TMDB already has? That is worth doing. It is just a different project, with a schema change and 2,400 model calls attached.
Meanwhile the acclaim data that arrived in the same migration, vote average and vote count, is legitimate in a way vibes are not. It is the world's opinion, formed before I watch anything, and available for everything. Adding it moved AUC from 0.712 to 0.760.
Target is liked = rating >= 4, which splits 52/48 and
sidesteps the fact that 79% of my ratings are a 3 or a 4. Repeated stratified
five-fold, five repeats, because single splits on 729 rows move several points
on the random seed alone.
| model | accuracy | ROC AUC |
|---|---|---|
| random forest | 0.694 ± .035 | 0.760 ± .039 |
| hist gradient boosting | 0.683 ± .029 | 0.744 ± .033 |
| logistic regression | 0.675 ± .032 | 0.732 ± .035 |
| majority class | 0.520 ± .001 | 0.500 |
That is +0.174 accuracy over always guessing the majority class, which is 4.9 times the best model's own fold standard deviation. Genres, age, and acclaim genuinely carry signal about whether I will like something. The direction of the fit matches what I already knew about myself: drama and documentary positive, action and family and romance negative.
So far this is an ordinary, slightly boring success.
Rating prediction is a proxy. What I wanted was a queue: sort the 1,371 unseen titles so the good ones float up. There is a natural ground truth sitting right there, because 34 of those unseen titles are already on my watchlist. A good ranker should surface them.
| ranker | top 50 | top 100 | top 200 |
|---|---|---|---|
| the model | 3 | 6 | 10 |
| sort by popularity | 3 | 7 | |
| sort by oldest | 3 | 8 | |
| shrunk vote average | 9 | 13 | 15 |
The model ties with sorting by release date. A three line heuristic beats it three to one in the top 50. Not one thing I actually want to watch appears in the model's top 30.
What it surfaces instead is obscure British crime drama. Which is a fair read of my taste. It is just not a list of things I have any intention of watching, because I have never heard of any of them.
Predicting a rating and predicting a want are different problems. The model learned what I enjoy. A watchlist is built from what I enjoy and have heard of, and awareness is not in the training data at all.
Which is why acclaim wins. Vote count is a proxy for cultural footprint, so shrinking a title's average rating toward the global mean in proportion to how few votes it has approximates "good and also famous enough that I would know about it." I tuned the shrinkage prior against that same 34 item ground truth: 0 gives 3 hits, 500 gives 7, 2000 gives 9, and it is flat above that. So 2000 shipped.
The useful output of this project was not the model. It was a constant in
a config file, ACCLAIM_PRIOR = 2000, and the knowledge that the
recommender should not be driving the queue on its own.
The honest limits are worth stating, because they are large. The corpus is retrospective recall, seeded in about nine days from memory, so it oversamples titles that were memorable and liked. There is no usable time ordering, which is why the splits are random rather than temporal. And 729 rows against 22 features is small enough that gradient boosting overfits quickly, which is the reason the harness reports spread across repeated cross-validation instead of a single number.
None of that changes the finding. If anything it makes the case stronger, because the heuristic that won does not care about any of those weaknesses.
The thing I would tell myself at the start: decide what question you are actually asking before you pick the target variable. I spent the effort on predicting ratings because ratings were the column I had. The job was ranking a queue, and those turned out to be different enough that doing the first one well got me almost nothing on the second.