Why jobs are not like movies

November 30, 2009 | (No Comments)

The internet is such an amazing phenomenon and has created such uniquely interesting products for we humans that we often ascribe to it, and the developers behind those cool products and features, an almost magical power to solve any data-related conundrum we may face.

The problem of recommending jobs to job-seekers, and recommending specific job-seekers to recruiters looking to fill specific jobs, is one of those conundra.

The gold medal for predicting what will titillate, please or amuse humans in their quest for accurate machine-generated recommendations must surely go to Netflix. Netflix launched CineMatch in 2000 and famously offered, in 2006, a $1,000,000 prize to any computer scientists who could improve it by 10%. The story that followed – three years of toil on the most tedious problems in computer science, the collaboration of and eventual merging of championship teams in order to mount the winner’s podium – inspired the Web community and illuminated the power of the crowd-sourcing model in a highly entertaining way. Just the characteristics one would hope for from a publicity stunt! (Interestingly, Reed Hastings, CEO of Netflix has mentioned to me the twenty other projects they’ve tried on a similar basis but never got “legs”. I suppose that’s how it goes in igniting the public’s imagination. Lots of dross to produce the one filament of gold.)

A persistent hope for our industry – online recruiting – is that we will also invent the algorithm that makes sourcing candidates for jobs or suggesting jobs for job-seekers as easy as the Netflix Queue.

Alas, not yet.
(I’m reminded of my doctor’s always-amusing comment to when he prefaces his diagnoses with “in our present state of ignorance, we believe….” And I think for any scientifically-minded person, that’s the right approach. So let me re-state: Alas, in our present state of ignorance, not yet.)
The movie to movie-watcher problem is a stable relationship between two static entities: users and movies.
Users’ preferences stay roughly the same over their lifetimes – the list of favorite movies they enjoyed in their thirties are similar to the movies they’ll enjoy in their fifties.

And movies stay the same over the years. “The Godfather” that we watched in the movie theaters is exactly the same as “The Godfather” we watch today – the same actors, plot, cinematography, etc. It seems like an obvious thing to say, but when we’re talking about expressing human preferences for things, it becomes important to define its characteristics accurately, as we shall see.

The result of movies staying the same is that movies have a consistent ISBN number over the years. “The Godfather” has been identified by the same numeric code and its successors since it was released.

So improving Netflix’s Cinematch is about improving the predictive capability of a system that compares static preference sets over a (growing) number of static entities. “Collaborative filtering” enables us to predict a particular user’s preference for a movie based on all the other users that have similar preferences.

It might help to think about it visually — if you imagine the movies themselves as points in space and the user’s preference set as the shape of the line connecting those points. Over the years, as you collect more and more and more shapes to compare to each other, you improve your ability to predict the shape of one preference set based on the shapes of all the others.

By contrast, the job search is a bi-lateral search between two human agents each with a depreciating, perishable, information entity.

The job-seeker’s resume changes regularly. As a human works and takes on new projects, accomplishes new achievements, and earns new responsibilities, the document outlining a description of that human changes. In theory, it’s a continuous change, but for practical purposes, let’s say it changes annually.

Jobs are perishable. For the moment, let’s focus on the jobs I work on day-to-day at TheLadders, professional jobs at the $100,000 per year or more compensation level. From the time the paperwork is approved to the time the job is filled may be as long as six to eight months or more. And the actual time that a job exists on the web as an identifiable entity with a unique number varies quite a bit, but let’s use 3 months as the typical length of time it is published and available on the internet.

So improving the job matching system is about improving our predictive capability in comparing a group of annually changing entities (resumes) to quarter-annually disappearing entities (jobs).

Compared to our ever-growing and stable “movie” shapes above, our “job” shapes keep disappearing! The lines disappear with too much rapidity to be of any value. It’s as if a user’s preferences for movies they watch rapidly evolve each year, so that after ten years, there is no resemblance between their current and prior preferences. And it’s as if the entire catalog of movies available for watching was thrown out every three months to be replaced with a new set of movies, slightly different, slightly updated, but that after five years bear almost no resemblance to their earlier form. (Think about the change in duties for a ‘Product Manager’, a ‘Developer’, or a ‘VP, Marketing’ since 1999.)

And to understand the scale of the problems, consider what the Netflix Prize actually accomplished. With the stable universe of un-changing users and un-changing movies, a $1 mm prize and plenty of notoriety attracting the world’s best computer scientists to the problem, over three years they improved the predictive capability of the system… by 10%? As others have mentioned, that makes the silver bullet of an improved recommendation engine a very rare case indeed.

So the collaborative filtering problem is very hard in the case of movies, and very, very, very hard in the case of jobs.

That is why “recommended” features can so often miss the mark in online recruiting.

And, of course, I reserve the right to update my present state of ignorance immediately upon your enlightening me.