N11.com is a leading Turkish marketplace for the 7th year in a row. Currently, it has 250 million monthly visits, with more than 20 million available items.
The marketplace generates revenue from the number of items purchased. In this model, personalization can add value by serving relevant offers and recommendations to the visitors, increasing the conversion rate, average cart size, and creating a better user experience, leading to more loyal and frequent customers.
N11 already had a personalization solution in place once they decided to test it against another solution in the market.
The marketplace decided to compare Yusp to its previous vendor during an A/B test and in an attempt to increase their performance and receive day-to-day help with their support requests.
We summarize the test and the results below.
The critical requirement of the A/B-test is to guarantee the same conditions for the solutions compared before and during the evaluation period.
The test also has to meet the following conditions:
‘Have you seen these products?’ placement on the homepage
The Proof of Concept (POC) period was running for two months, during which Yusp managed to outperform its competitor significantly in all KPIs. Personalized placements were available on all N11 platforms (web, mobile web, iOS, and Android applications) measured in 8 different placements. In the case of each placement, Yusp proved to be the more efficient solution.
During the Proof of Concept period, Yusp managed to increase the recommendation driven GMV compared to the previous vendor. In terms of CTR, Yusp managed to perform 33.8% better, increasing the baseline from 1.96% to an overall average of 2.65%. Yusp performed even better on mobile platforms lifting the CTR benchmark from 2.43% to 3.28%.
Working together on marketplace-specific challenges, increasing the overall revenue through recommendations
Ever since the successful POC period, Yusp and N11 continuously work together to increase the performance of recommendations further and find new ways to optimize the service.
Both teams are entirely on board with the latest results and developments thanks to our daily interactions, which allows us to react to any unforeseen events quickly.
Recently, we aimed to make better use of the data to tackle some marketplace-specific challenges. These improvements touched on the data connected to product identification and the rating of sellers.
Improving product identification was necessary because of the high number of similar or identical products offered by various sellers, each using different names and attributes. N11 developed a three-level product identification based on three parameters:
In the case of the seller ratings, N11 created a tool for its users to rate sellers based on their shopping experience – from choosing the product on the website until the final delivery. The explicit user feedback is now being taken into account when calculating prediction scores to recommend a specific product type. The better overall user experience a seller provides, the better their products rank.
Deep learning is a subclass of machine learning that uses complex models of a cascade of non-linear processing layers – also called deep neural networks – to learn different data representations.
Starting from the early 2010s, deep learning has achieved significant improvements in complex domains such as image recognition, speech recognition, natural language processing, and more.
The uptake of deep learning applications for recommendation engines began in late 2015, fostered by a handful of pioneering research groups. Yusp has been one of these pioneers and is considered one of the field’s leaders.
Results of applying a deep learning-based personalization algorithm:
The Yusp algorithms provide recommendations with session-based user behavior in mind. The engine recommends items that are suitable for continuing the current session of the user.
The transition between item-to-item and personalized recommendations provided the following results during a recent internal A/B test:
Deep learning at scale:
Deep learning models are computationally expensive and often require specific hardware to run efficiently (GPUs instead of CPUs). Yusp developed a unique way to serve recommendations by our deep learning models using CPUs only (training still uses GPU) without affecting the quality of recommendations. This development allows Yusp to cost-efficiently use deep learning for more use cases.
We placed 25 recommendation scenarios on the N11 web, mobile, and app platforms. These scenarios created 62 placements that contain the most relevant available items to all users.
The recommendation engine defines the products displayed and their order in the boxes. They appear on all important screens, such as the homepage, the category page, the product detail page, the basket page, and the successful order page.
Personalized recommendations on the main page – Personalized recommendations based on recently viewed products, from any category, recommended products should not overlap with other recommendation boxes on the main page.
Personalized recommendations on the product page (web) – Recommending similar products to a selected product based on user behavior (collaborative similarity), recommended products should not overlap with other recommendation boxes.
Personalized recommendations on the product page (mobile web)
Personalized recommendations on the category page (web) – Personalized recommendations based on recently viewed products
filtered for the actual main category
N11 Turkey’s largest marketplace aimed to improve its personalization solution’s performance and decided to compare their previous vendor with Yusp.
During the Proof of Concept period, Yusp managed to increase the recommendation-driven Gross Merchandising Value compared to the previous provider.
Today, N11 leverages the personalization potential of Yusp on its web and mobile platforms in over 60 placements. They create a significant part of their total revenue through personalization solutions.