How Machine Learning Delivers Personalized Experiences by Expedia Group Product Manager

Please download to get full document.

View again

of 34
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Similar Documents
Information Report



Views: 0 | Pages: 34

Extension: PDF | Download: 0

Product Management Event at #ProductCon London about Machine Learning and how it Delivers Personalized Experiences by Product Manager at Expedia Group, Ammar Jawad.
  • 1. How Machine Learning Delivers Personalized Experiences by Expedia Group Product Manager
  • 2. FREE INVITE Join 35,000+ Product Managers on
  • 3. COURSES Product Management Learn the skills you need to land a product manager job
  • 4. COURSES Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  • 5. COURSES Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  • 6. COURSES Digital Marketing for Managers Learn how to acquire more users and convert them into clients
  • 7. COURSES UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to-end experience
  • 8. COURSES Product Leadership for Managers For experienced Product Managers looking to gain strategic skills needed for top leadership roles
  • 9. COURSES Corporate Training Level up your team’s product management skills
  • 10. Speaker Ammar Jawad Expedia Group Product Manager
  • 11. How Machine Learning Delivers Personalized Experiences Ammar Jawad Product Manager, Personalisation & ML Platform Expedia Group 11
  • 12. Agenda - Understanding AI, ML and Deep Learning - AI and Product Management - Degree of Personalisation - Feedback loops and Collective intelligence - AI Product Manager 12
  • 13. Difference between software engineering and ML In traditional software engineering... 13 In (most) ML applications... A human analyses a problem, writes code, turns into a program which then translates inputs to outputs. A computer figures out the best program to write using statistics by looking at a set of examples and their desired output.
  • 14. AI, Machine Learning and Deep Learning 14 AI is... A discipline that has to do with the theory and methods to build machines to resemble humans. ML is... A toolset which can be used to solve certain kinds of AI problems. Machines do not start out intelligent but become so after being trained. DL is... A subfield of ML which learn representations of data by layers of increasingly meaningful representations. Layered representations are learned through neural networks. Machine Learning Deep Learning Artificial Intelligence
  • 15. Impact of ML on Products 15 No more one-size-fits-all solutions Increased commercial value ● Adapt to users ● Anticipate users’ needs ● Fulfill user intent ● De-averaging ● Upsell, cross sell and the right complimentary products ● Human-level expertise at scale
  • 16. When should a consumer-facing feature be powered by ML? 16 Would some users find the feature more relevant than others? Machine Learning Heuristic rules Yes No
  • 17. Types of ML features 17 Feature-level ● Recommended destinations custom-tailored to each user ● Ordering images based on what the user is most likely interested to see ● Chatbots to answer simple customer service tickets Delivery-level ● Multi-Armed Bandits experimenting to identify the right widget size per user segment ● Contextual Bandits testing different string translations based on behaviour/technological differences of users in a particular region, e.g. youth/adults.
  • 18. Degree of Personalization - Considerations for Product 18 Non-Personalized Targeted Highly Targeted Semi-Personalized Personalized Source: BBC Source: Katar Investments Source: Copenhagen Airport Source: Manning Publications Source: Spotify Low High
  • 19. Personalization: Linearity & Nonlinearity 19 How most product people think... How we should be thinking...
  • 20. Targeted products: The case for Multi-Armed Bandits 20 Variant AControl Variant B Variant C 25% 25% 25% 25% A/B Testing Multi-Armed Bandits 75% of all traffic is sent to suboptimal variants. Control Variant A Variant B Variant C 25% 25% 25% 25% Day 1 Control Variant A Variant B Variant C 13% 28% 5% 54% Day 2 Variant A Variant C 19% 81% Day n
  • 21. Targeted products: Multi-Armed Bandits (Part II) Online decision making - Should I exploit? (make the best decision given current info) - Should I explore? (gather more info) Best long-term strategy may involve short-term sacrifices to maximise long-term gain Other examples may include: 21
  • 22. Targeted products: Multi-Armed Bandits (Part III) 22 Three main approaches to exploration: 1. Random exploration Explore based on a probability to take a random action, e.g. explore 20% of the time. 1. Optimism in the face of uncertainty When faced with options for which we know the value of each except one action which value is unknown then there is a bias towards the action with an unknown outcome. 1. Information state space Consider agent’s information as part of its state Look ahead to see how information helps reward
  • 23. Reinforcement Learning in Online decision making 23 Action Reward Action RewardContext Multi-Armed Bandits Contextual Bandits
  • 24. Highly Targeted Products: Contextual Bandits 24 Region Time of Day Variants Latin America Morning Control EMEA Night Variant A APAC Morning Variant B Contextual Bandits are Multi-Armed Bandits for each segment
  • 25. Optimising the right KPI in Reinforcement Learning 25 The AI agent is optimising for “score earned” which has unintended consequences.
  • 26. Personalized products: Recommender Systems (RS) Recommender systems are based on calculating similarity and distance between a set of users or items 26 UserId MovieId Rating 0 0 4 0 1 5 0 2 4 2 0 4 2 1 4 2 2
  • 27. Personalized products: Humans-in-the-loop in RS 27 User1 User2 Dominican Republic Dominican Republic Thailand Thailand Cambodia Cambodia The Netherlands The Netherlands Spain Spain Brazil Brazil Philippines Hypothetical example of destinations travelled by two users
  • 28. Feedback Loops: Personalization is data-hungry 28 MODEL TUNING PROCESS INSIGHT GENERATION INPUTS OUTPUTS Adapted from “How Googles does Machine Learning”, Coursera
  • 29. Collective Intelligence 29 Source: “Collective Intelligence in Action”, Satnam Alag As users interact on the web and express their opinions, they influence others.
  • 30. Rise of the AI Product Manager ● 20% of jobs in the UK expected to be displaced by AI over the next 20 years 1 ○ Approximately equal to the additional jobs created by AI in the same period. ● Companies increasingly leverage ML to gain competitive advantages ● Users increasingly demand better experiences only delivered via AI ● Personalisation only possible through ML ● Tech companies are looking for product leaders with AI expertise to help navigate 30 1. PWC, UK Economic Outlook July 2018
  • 31. Roadmap to become a great AI Product Manager ● Excel (formulas, pivot tables, vlookups) ● Statistics (descriptive, inferential) ● Maths (algebra, linear algebra, calculus) ● Coding in Python (variables, functions, objects) ● SQL (joins, order by, group by, data aggregation, basic subqueries) ● Machine Learning (supervised, unsupervised) ● Deep Learning (ANNs, CNNs, NLP) 31
  • 32. Summary ● Understanding AI, ML and DL ● Product and ML ● Degree of Personalization ● Feedback loops in features ● Collective intelligence in products ● AI Product Managers 32
  • 33. Contact and links - LinkedIn: - Email: - Quora: - Presentation available here: © Expedia, Inc. All rights reserved. 33
  • 34. Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design and Product Leadership courses in San Francisco, Silicon Valley, New York, Santa Monica, Los Angeles, Austin, Boston, Boulder, Chicago, Denver, Orange County, Seattle, Bellevue, Washington DC, Toronto, London and Online
  • We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!