Search Engines Amsterdam: Sponsored Search and Online Evaluation

After a summer break SEA is returning with two great talks followed by drinks.

First the academic talk is given by Harrie Oosterhuis, who is a PhD candidate at the University of Amsterdam under the supervision of Prof. Maarten de Rijke.

Then our industrial talk is given by Or Levi from Marktplaats.nl/eBay.

This edition of SEA will be held in at Science Park 904 in Room C0.110.

Program:

16:00 – 16:30 Harrie Oosterhuis

16:30 – 17:00 Or Levi

17:00 – 18:00 Drinks & Snacks

Details of the talks:

—————————————————————————————-

Harrie Oosterhuis — Sensitive and Scalable Online Evaluation with Theoretical Guarantees

The research and development of search engines is becoming increasingly rapid. Applying a change to your algorithms is only half the work, figuring out whether you actually made an improvement is just as crucial. Traditionally A/B testing has been used for comparisons between ranking systems, while reliable these types of test are very costly. As an alternative Interleaving was introduced, it combines the rankings of two systems into a single result list. From interactions with these lists preferences can be inferred, as a result much fewer interactions are required for comparisons. Recently this has been extended further to Multileaving, which can handle comparisons containing any number of rankers. This reduces the cost of evaluation even further.

During this talk I will go over the existing Multileaving methods and show that although some methods guarantee a level of correctness, and others guarantee to maintain the user experience during evaluation, none can do both. Then I will introduce a novel Multileaving method for which we have proved it has fidelity (a level of correctness) and that it will always be considerate to the user experience. Finally, our experimental results confirm that it is also more efficient than any previous method.

Harrie Oosterhuis is a PhD candidate under the supervision of Prof. Maarten de Rijke at the ILPS research group at the University of Amsterdam. His work focusses on Online Learning to Rank: how to optimize from user interactions alone; and Online Evaluation: ranking comparisons based only on users interactions. During his PhD and Msc. in Artificial Intelligence Harrie has also completed several research internships at Google and Google Brain.

—————————————————————————————-

Or Levi — Admarkt’s Next Gen Sponsored Search: Counterfactual Inference in Learning-to-Rank for Revenue Optimization

Admarkt is the leading sponsored search platform for the ebay classifieds group. In this talk we will present Admarkt’s next gen ranking system, representing a significant progress over the existing ranking in both algorithms and architecture.

Focusing on algorithms, Learning-to-Rank has been successful for optimizing search relevance in commercial applications, such as Google Brain and Bing’s RankNet; Learn how it could be employed in sponsored search where the main objective is revenue optimization.

Additionally, Counterfactual Inference has been gaining strong interest recently. It aims to answer questions such as, can we optimize our models for some online metrics without subjecting users to online learning? We will share interesting findings from our attempts to address Counterfactual Inference challenges, which could serve participants to address similar challenges in their own domains.

Or Levi is a Data Scientist at Marktplaats.nl/ebay, specializing in Machine Learning for Computational Advertising. He holds a M.Sc. in Information Retrieval from the Technion, the Israel Institute of Technology. In his spare time, he enjoys playing in the Marktplaats soccer team and teaching recurrent neural networks to generate humor.