In his talk, Ed will highlight three key features of Memrise's experimental product development process: their use of prototyping and user-testing in product exploration; their commitment to radical diversity in team and technique; and the exploration of bold ideas to develop fresh content and ideas, such as their tour round Europe in the double-decker 'Membus'. Ed will explain how these e orts tie together in helping Memrise invent and quickly deploy unorthodox learning experiences at scale.

1. Avant-garde product
management in the age of data
Ed Cooke
@tedcooke

2. Memrise is 25 million people’s favourite
Language learning app
● 200 language pairs
● Billions of things learned monthly
● Team of 60 in London
● iOS, Android, Web devs / PMs /
Designers, content creators,
language specialists

3. Holistic language learning with a multitude
of learning modes

4. Help anyone to fall in love with
and learn a language in a few months
OUR PRODUCT VISION

5. Help anyone to fall in love with
and learn a language in a few months
OUR PRODUCT VISION
Enchantment

6. Help anyone to fall in love with
and learn a language in a few months
OUR PRODUCT VISION
Enchantment Effectiveness

7. Help anyone to fall in love with
and learn a language in a few months
OUR PRODUCT VISION
Enchantment Broad appealEffectiveness

8. The adventures in product management:
1) Enchantment
3) Broad appeal
2) Effectiveness

9. Enchantment

10. Building the magic of travel and adventure
into the product-development process

17. ● 20k beautiful videos
● Top premium feature
● Good expression of product
vision

18. Reflections and learnings on this kind of
experiential product development
● It is is deliberately upside-down
● Challenging to get buy-in from more technically minded team-members
● By betting big, you create an energy that’s difficult creatively to resist

19. Effectiveness

20. The challenges of enhancing learning
effectiveness
● Cognitive science largely hasn’t studied learning effectiveness
● Difficult to generalise from (sometimes conflicting) lab findings
● Optimisation only relevant to tuning what you have

21. a) External efforts
Effectiveness

22. “The people outside
your organisation are
always going to be
infinitely smarter than
those within.”
Matt Mullenweg

23. Memrise Prize: an applied science competition
● Create the best learning method
to teach 100 foreign words in an
hour, tested a week later
● Compete by providing data from
an empirical trial
● Final round is a test on 10s of
1000s of Memrise users

24. ● 15 different labs tested experiments
● 5 made it through to the final round
● Finalists entries tested on 10s of
1000s of Memrise users

25. ● 4 showed >2x of flashcard control
● Methodologies varied widely

27. Reflections and learnings
● State of expert art + where we stand
● Litany of actionable micro-improvements + techniques
● New way of thinking about modelling complicated mental capacities
REFLECTIONS AND LEARNING:
● It’s a big long term bet, collaborations take years not quarters.
● Danger creating a fourth code-base:

28. b) Internal efforts
Effectiveness

29. Learning Research Team

30. Prototype > user test > analyse
(RINSE AND REPEAT)

31. User testing
● 3 days per week
● Full experiments
● Weekly iteration cycle
● Empowers language creators

32. Example experiment
Training
Evaluation
Results 21% 75% 86% 62%
Condition A Condition B
B-tests A-tests
B-testsA-tests

33. Prototyping as a tool for research (and product
development)
Dynamics of product experiment
● 1 experiment
● 3 developers
● 2-3 product / data
● 1 month
● 1-2 iterations
Dynamics of prototype-experiment
● 12 experiments
● 1 developer
● 4 learning researchers
● 1 user tester
● 1 month
Cost per experiment:
$50k
Cost per experiment:
$4k

34. Hybrid approach to cognitive engineering
Effectiveness
EXTERNALLY
● Stimulate original scientific research
● Integrate the discoveries
● Collaborate with that community
Challenge: SLOW !!!
INTERNALLY
● Prototype and user test for
accelerated discovery
● Build tools to increase accessibility
and speed
Challenge: 4th CODE BASE

35. Broad appeal

36. Build diversity into team

37. Opening up ideation as widely as possible
Prototyping Hackathons

38. Opening up ideation as widely as possible

41. Hackathons
● Every 6 weeks
● Engineers set free
● Aim to have working prototype in
2 days

43. Challenges and reflections
● Ideas that you don’t personally like can thrive
● You kill ideas you build (which non-battle hardened product managers can
find upsetting)
● Can over-complicate the product

44. Help anyone to fall in love with
and learn a language in a few months
OUR PRODUCT VISION
Enchantment Broad appealEffectiveness

45. memrise.com/jobs