The invisible salesman — An introduction to recommendation engines
With the onset of the internet, we have moved from an era of scarcity to an era of abundance. No longer do we have to choose from a limited amount of shirts or shoes at a tiny store. No longer do we have to scan through a tiny menu at a restaurant with limited cuisines. On the internet, we can now choose from a wide range of apparel and from thousands of dishes across restaurants. But with so many choices, how do we scan through them all and arrive at the one product that we really want? Is having too many choices a problem for retailers?
Let’s look at this famous study on the sale of jam bottles.
On a particular day an upscale grocery store offered 24 varieties of jams at a tasting counter. On another counter the same store offered only 6 varieties of jam.
The assumption was that the counter with 24 jams would end up with the highest sales, however a strange behaviour was noticed with this experiment.
Although the counter with 24 varieties of jam got the highest interest and footfall, the conversions to sale at that counter was almost 10 times lower than the counter with 6 jams.
This goes to show that having too many options may not always be the best. Although e-commerce platforms may have access to thousands of items, without recommending the most relevant ones to their users, they may be doing more harm than good.
The paradox of choice is that if a person is presented with too many choices, they are actually less likely to buy.
When I wanted to purchase a car, a few years ago I walked into a store. The salesman there was keen on understanding my preferences, and was then able to recommend to me the best vehicle options based on my choice of color, my requirement of speed and comfort and my budget. His recommendations were relevant and converted my visit to a successful sale.
In digital platforms, a recommendation engine is a lot like that car salesman, with one major difference. The salesman is invisible!
While not getting too technical, I’m going to introduce you to a few methods for recommending items to users. I will dive into the depth of these methods in future posts. Let’s start with the easiest and move on to the more complex methods.
Popularity based recommendations
As the name suggests, this technique works on the principles of popularity, or anything that is trending. This is one of the easiest ways to recommend, as the system only needs to track the most popular or trending product and recommend them to all users.
For example, a footwear website may recommend their list of most sold footwear upfront.
Advantages
It does not need a user’s historical data to recommend items.
It does not suffer from a cold-start problem, which means that a product can get on the trending list even on the first day of launch.
Disadvantages
It is not personalised.
All users will see the same recommendations, whether or not they like the recommended items.
Content based filtering
Content-based filtering uses item features to recommend other items to users based on the users interaction or explicit feedback.
For example, let’s say a Netflix user watches a lot of action movies. This is an implicit indication that the user prefers action movies. They have also explicitly given a 4/5 rating to a few comedy movies. This hints that the user prefers to watch action and comedy movies and the recommendation engine will suggest movies based on these item-features.
In this example, other item-features may include the name of the actors, director, movie-language etc.
Advantages
The recommendation engine does not need data from other users, since it is a user-specific recommendation. This makes it scalable to many users.
The engine can learn very specific insights of a user, and recommend items that may not be very popular, but may fit the users choice or preference.
Disadvantages
The engine can only recommend based on the user’s current preference. In other words, it will not be able to predict what new items the user may like.
Collaborative filtering
Collaborative filtering uses similarities between users and items simultaneously to provide recommendations. It means that it may recommend items to UserA based on the interests of a similar UserB.
Collaborative filtering addresses the limitations of Content based filtering.
For example, if UserA and User B have been watching similar comedy and action movies and UserA suddenly watches a suspense-thriller movie, this new movie will be recommended to UserB.
The similarity between these users is usually measured based on a method called Centered Cosine Similarity also known as Pearson Correlation coefficient.
Advantages
The model is continuously learning and can give relevant recommendations to users that may be beyond their interests, because similar users are interested in the item.
The recommendations are more diverse.
Disadvantages
It cannot handle new items. This is known as the cold-start problem.
For eg. In the case of movies, a collaborative filtering model would recommend a movie to UserA if a similar UserB has interacted with it. However when a new movie is launched, only a small set of people interact with it initially and hence it ranks lower in the recommendations.
There are techniques to counter this cold start problem including WALS projections, which we will look at in future posts.
Final notes
All of the above recommendation methods come with their own set of pros and cons. Real world platforms usually tailor their recommendation methods extensively, as the effectiveness of recommendation methods vary from business to business.
Most digital platforms use a hybrid model, which encompasses multiple recommendation methods, thus reaping the benefits of all and where most of the disadvantages are cancelled out.
In the next post we will have a look at examples of a few digital platforms that use recommendations, and the best practices in a real-world scenario.
Thank you for reading. Please share this if you’ve learnt something new today.
