Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch

620 visualizaciones

Publicado el

Slides for the presentation at the Seminar Series on Advanced Techniques & Tools for Software Evolution (SATToSE), which took place in Madrid (Spain) on 7–9 June 2017.

Publicado en: Educación
  • Sé el primero en comentar

Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch

  1. 1. 1/35 SATToSE 17, Madrid Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch Jes´us Moreno-Le´on, Gregorio Robles, Marcos Rom´an-Gonz´alez jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es GSyC/Libresoft, Universidad Rey Juan Carlos SATToSE, Madrid, June 7th, 2017 Jes´us Moreno-Le´on Present and Future of Dr. Scratch
  2. 2. 2/35 SATToSE 17, Madrid (cc) 2017 Jes´us Moreno Le´on, Gregorio Robles and Marcos Rom´an Gonz´alez Some rights reserved. This work licensed under Creative Commons Attribution-ShareAlike License. To view a copy of full license, see http://creativecommons.org/licenses/by-sa/3.0/ or write to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. Some of the figures have been taken from the Internet Source, and author and licence if known, is specified. For those images, fair use applies. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  3. 3. 3/35 SATToSE 17, Madrid What is Dr. Scratch? J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  4. 4. 4/35 SATToSE 17, Madrid Why automatic analysis? (Teacher perspective) Enjoying while marking students’ projects J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  5. 5. 5/35 SATToSE 17, Madrid Why automatic analysis? (Learner perspective) Analyzing a Python program with Pylint J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  6. 6. 6/35 SATToSE 17, Madrid Remixing other researchers’ ideas Analysis of Scratch projects Scrape: visual representation of the blocks used (and not used). Hairball: static analyzer of Scratch projects to detect errors. Brennan & Resnick: New frameworks for studying and assessing the development of CT. Seiter & Foreman: Progression of Early CT Model. Wilson, Hainey & Connolly: Evaluation of games to gauge understanding of programming concepts. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  7. 7. 7/35 SATToSE 17, Madrid Assessment of CT development CT dimension Basic Intermediate Proficient Logical Thinking if if else logic operations Data representation modifiers of object properties variables lists User interactivity green flag keyboard, mouse, ask and wait webcam, input sound Control flow sequence of blocks repeat, forever repeat until Abstraction and problem decomposi- tion more than one script use of custom blocks use of ’clones’ (in- stances of sprites) Parallelism two scripts on green flag two scripts on key pressed or sprite clicked two scripts on receive message, video/audio input, backdrop change Synchronization wait message broad- cast, stop all, stop program wait until, when backdrop changes to, broadcast and wait Table: Level of development for each CT dimension J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  8. 8. 8/35 SATToSE 17, Madrid Assessment of CT development: Logical Thinking Different levels of development of logical thinking: basic (top), developing (center) and proficient (bottom). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  9. 9. 9/35 SATToSE 17, Madrid Assessment of CT development: Data Representation Different levels of development of data representation: basic (top), developing (center) and proficient (bottom). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  10. 10. 10/35 SATToSE 17, Madrid Code smells (I) Errors or bad programming habits Dead code Attribute initialization Default names Repeated scripts J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  11. 11. 11/35 SATToSE 17, Madrid Code smells (II) Bad/default naming of sprites J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  12. 12. 12/35 SATToSE 17, Madrid Code smells (and III) Example of repeated code Solution to avoid repeated code Blocks should be created to avoid repetition of code J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  13. 13. 13/35 SATToSE 17, Madrid Goals Review the current state of the validation process of the tool Show examples to illustrate the kind of software evolution investigations that could be performed using the tool Background picture: rebel-performance.com J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  14. 14. 14/35 SATToSE 17, Madrid Validation Process: Ecological Validity (I) Ecological validity Are young learners able to analyze their projects and independently learn from the tips that the tool provides? Workshops with over 100 students (10 to 14 years) in 8 schools J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  15. 15. 15/35 SATToSE 17, Madrid Validation Process: Ecological Validity (II) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  16. 16. 16/35 SATToSE 17, Madrid Validation Process: Ecological Validity (and III) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  17. 17. 17/35 SATToSE 17, Madrid Validation Process: Convergent Validity (I) Convergent validity Comparison of the evaluations provided by Dr. Scratch with other measurements of similar constructs (Human) expert evaluators Software engineering complexity metrics CT-test Background picture: Joanna Bourne J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  18. 18. 18/35 SATToSE 17, Madrid Validation Process: Convergent Validity (II) Comparison with human experts - Dr. Scratch contest award ceremony at Google Campus, Madrid (Spain) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  19. 19. 19/35 SATToSE 17, Madrid Validation Process: Convergent Validity (III) Scatter plot for experts evaluation (x-axis) and Dr. Scratch assessment (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  20. 20. 20/35 SATToSE 17, Madrid Validation Process: Convergent Validity (IV) Scatter plot for Dr. Scratch assessment (x-axis) and Cyclomatic Complexity (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  21. 21. 21/35 SATToSE 17, Madrid Validation Process: Convergent Validity (V) Scatter plot for Dr. Scratch assessment (x-axis) and Halstead’s vocabulary (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  22. 22. 22/35 SATToSE 17, Madrid Validation Process: Convergent Validity (VI) One of the CT-test items J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  23. 23. 23/35 SATToSE 17, Madrid Validation Process: Convergent Validity (and VII) Scatter plot for Dr. Scratch assessment (x-axis) and CT-test (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  24. 24. 24/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (I) Discriminant validity Projects shared in the Scratch repository are categorized under one or more project types: games, animations, music, art and stories. Is Dr. Scratch able to detect differences in the CT dimensions developed when programming different types of Scratch projects? Background picture: Bodie Pyndus J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  25. 25. 25/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (II) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  26. 26. 26/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (and III) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  27. 27. 27/35 SATToSE 17, Madrid Validation Process: Face Validity (I) Face validity Educators won’t use the tool unless they feel that Dr. Scratch measures what it promises (i.e. assessing CT) Over 400 teachers who participate in a 40-hours Scratch coding training course have been surveyed Background picture: janmennens J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  28. 28. 28/35 SATToSE 17, Madrid Validation Process: Face Validity (and II) The CT score provided by Dr. Scratch is accurate J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  29. 29. 29/35 SATToSE 17, Madrid Validation Process: Factorial Validity Factorial validity Study potential relationships between the CT dimensions assessed by Dr. Scratch Simplification of the feedback report? Background picture: jennyrotten J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  30. 30. 30/35 SATToSE 17, Madrid Limitations Teachers should not rely exclusively on Dr. Scratch Fundamental CT skills not assessed: debugging and remixing. Functionality or creativity not evaluated. Portfolio analysis would be more accurate. Background picture: Robert Couse-Baker J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  31. 31. 31/35 SATToSE 17, Madrid Future Work 1 User accounts 2 Teacher dashboard 3 Organization dashboard Background picture: Simon Cunningham J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  32. 32. 32/35 SATToSE 17, Madrid Links with Software Evolution (I) Tools like Dr. Scratch offer the possibility of easily tracking learners progression and projects evolution, both in terms of software complexity and presence of bad smells J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  33. 33. 33/35 SATToSE 17, Madrid Links with Software Evolution (II) Examining the Relationship between Socialization and Improved Software Development Skills in the Scratch Code Learning Environment. Moreno-Le´on J., Robles G. & Rom´an-Gonz´alez M. Journal of Universal Computer Science 22 (12), pages 1533-1557, 2016. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  34. 34. 34/35 SATToSE 17, Madrid Links with Software Evolution (and III) Software clones in scratch projects: on the presence of copy-and-paste in computational thinking learning. Robles, G., Moreno-Le´on, J., Aivaloglou, E., & Hermans, F. In 2017 IEEE 11th International Workshop on Software Clones (IWSC), pages 1-7, 2017. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  35. 35. 35/35 SATToSE 17, Madrid Learn more Dr. Scratch references Moreno, J., & Robles, G. (2014). Automatic detection of bad programming habits in scratch: A preliminary study. In Frontiers in Education Conference (FIE), 2014 IEEE (pp. 1-4). IEEE. Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2015). Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking. RED. Revista de Educaci´on a Distancia, 15(46). Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2016). Comparing computational thinking development assessment scores with software complexity metrics. In Global Engineering Education Conference (EDUCON), 2016 IEEE (pp. 1040-1045). IEEE. Moreno-Le´on, J., Roman-Gonz´alez, M., Harteveld, C. & Robles, G. (2017). On the Automatic Assessment of Computational Thinking Skills: A Comparison with Human Experts. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. (pp. 2788-2795). ACM. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  36. 36. 36/35 SATToSE 17, Madrid Present and Future of Dr. Scratch J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es GSyC/Libresoft, Universidad Rey Juan Carlos SATToSE, Madrid, June 7th, 2017 J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

×