Towards Incremental Separation of Surmountable and Insurmountable Programming Difficulties

Conference talk of paper: Jason Carter, Prasun Dewan, and Mauro Pichiliani, ITowards Incremental Separation of Surmountable and Insurmountable Programming Difficulties, Proc. SIGCSE 2015, ACM. See http://youtu.be/h5q8U7x2DKg for YouTube version

1.0x

Towards Incremental Separation of Surmountable and Insurmountable Programming Difficulties

Created 2 years ago

Duration 0:22:18
lesson view count 15
Conference talk of paper: Jason Carter, Prasun Dewan, and Mauro Pichiliani, ITowards Incremental Separation of Surmountable and Insurmountable Programming Difficulties, Proc. SIGCSE 2015, ACM. See http://youtu.be/h5q8U7x2DKg for YouTube version
Select the file type you wish to download
Slide Content
  1. Towards Incremental Separation of Surmountable and Insurmountable Programming Difficulties

    Slide 1 - Towards Incremental Separation of Surmountable and Insurmountable Programming Difficulties

    • Jason Carter
    • Cisco Systems Inc., Research Triangle Park, NC, USA
    • Prasun Dewan
    • University of North Carolina, Chapel Hill, NC, USA
    • Mauro Pichiliani
    • Instituto Tecnologico de Aeronautica, San Jose, Brazil
  2. Slide 2

    • Vision and Scope
    • Dayton Ellwanger, Nick Dillon, Tim Wu, Jason Carter, and Prasun Dewan, Scalable Mixed-Focus Collaborative Difficulty Resolution: A Demonstration, CSCW Companion Proc., 2015 (http://youtu.be/1-AqMCidx48)
    • 2
    • Need help?
    • No difficulty
    • Surmountable
    • Insurmountable
    • Co-location does not scale or allow remote help
    • Incremental Difficulty Level Detection Algorithm
    • Programmer Actions
    • Collaboration Environment
    • Do students want incremental help?
    • Help vs. final grade?
    • Auto difficulty level detection?
    • Cognitive State
    • Difficulty Level
    • Help Requests
  3. Research Questions and Studies

    Slide 3 - Research Questions and Studies

    • 3
    • Do students want incremental help?
    • Help vs. final grade?
    • Automatic difficulty-level detection?
    • Field Study
  4. CS1 Class Taught as Part of Jason’s Ph.D. Requirements

    Slide 4 - CS1 Class Taught as Part of Jason’s Ph.D. Requirements

    • 4
    • 35 Students
  5. Research Questions and Studies

    Slide 5 - Research Questions and Studies

    • 5
    • Do students want incremental help?
    • Help vs. final grade?
    • Automatic difficulty level detection?
  6. Do Students want Incremental Help?

    Slide 6 - Do Students want Incremental Help?

    • 6
    • Asynchronous help through Email
    • Synchronous office hours
    • Co-located help sessions (2 weeks)
    • Help offered in the act of programming
    • What fraction of the class would come to help sessions?
  7. Do students want incremental help?

    Slide 7 - Do students want incremental help?

    • 7
    • 17/35 students used help sessions
    • Instantaneous help
    • One on one help
    • Some students did not feel comfortable asking questions in crowded office hours
  8. Research Questions and Studies

    Slide 8 - Research Questions and Studies

    • 8
    • Do students want incremental help?
    • Difficulty level detection?
    • Help vs. final grade?
  9. Slide 9

    • Help vs. Final Grade?
    • 9
    • Asynchronous help through Email
    • Synchronous office hours
    • Co-located help sessions (2 weeks)
    • For six weeks, determined for each student how many times help was requested through email and office hours
    • Monitored how many times each student asked for help
    • #times help was offered vs. student grade?
  10. Help vs. Final Grade?

    Slide 10 - Help vs. Final Grade?

    • 10
    • Better students ask for more help
    • More help improves grades
    • Giving help does not hurt students
    • Median help of each cluster
    • Grade = F (Help, Ability, Time, …)
  11. Research Questions and Studies

    Slide 11 - Research Questions and Studies

    • 11
    • Do students want incremental help?
    • Difficulty level detection?
    • Help vs. final grade?
  12. Slide 12

    • Difficulty Level Detection?
    • 12
    • Need help?
    • Incremental Difficulty Level Detection Algorithm
    • Programmer Actions
    • Collaboration Environment
    • Help Requests
    • No Difficulty
    • Surmountable
    • Insurmountable
    • No previous work with this specific goal
    • First-cut research
    • Difficulty Level
  13. Slide 13

    • Previous Work on Difficulty Detection
    • 13
    • Need help?
    • No Difficulty
    • Facing difficulty
    • Carter and Dewan 2010 Difficulty Detection Algorithm
    • Collaboration Environment
    • Help Requests
    • Jason used this algorithm in his class for last 3 assignments
    • Programmer Actions
    • Incremental Difficulty Status
    • But found students asked for help with surmountable difficulties
    • Motivated work on difficulty classification after the course
    • Carter, J. and P. Dewan (2010) Design, Implementation, and Evaluation of an Approach for Determining When Programmers are Having Difficulty. in Proc. Group 2010. ACM.
  14. Slide 14

    • From detection to Classification?
    • 14
    • No difficulty
    • Surmountable
    • Difficulty Level
    • Facing difficulty
    • No Difficulty
    • Facing difficulty
    • Detection
    • Classification
    • Incremental Difficulty Status
  15. Integrated Detection and Classification

    Slide 15 - Integrated Detection and Classification

    • 15
    • Incremental Difficulty Detection
    • and Classification Algorithm
    • Programmer Actions
    • No Difficulty
    • Surmountable
    • Insurmountable
    • Could not make this approach work
  16. Composed Detection and Classification

    Slide 16 - Composed Detection and Classification

    • 16
    • Incremental Difficulty Detection Algorithm
    • Incremental Difficulty Classification Algorithm
    • Programmer actions or some other mechanism
    • No Difficulty
    • Surmountable
    • Insurmountable
    • Facing Difficulty
    • Help Requests
    • Programmer Actions
    • Much better success with this approach
  17. Classification Algorithm

    Slide 17 - Classification Algorithm

    • 17
    • Feature Extractor
    • Surmountable
    • Insurmountable
    • Facing Difficulty
    • Programmer Actions
    • Machine Learning Algorithm
    • Features
    • Training Data
  18. Data Needed

    Slide 18 - Data Needed

    • 18
    • Needed input data (help requests, difficulty predictions, programmer actions) and ground truth (difficulty level) for training and evaluating machine learning algorithm
    • Did a 10-student lab study with advanced undergrad and grad students in which screens were recorded
    • Needed recordings of programmer screens
    • to identify features
    • determine if a difficulty was eventually surmounted
    • Recording field work raises privacy issues
  19. Swing/AWT Lab Task

    Slide 19 - Swing/AWT Lab Task

  20. Post Task Steps

    Slide 20 - Post Task Steps

    • 20
    • Jason Carter and another coder used the recordings to
    • do the classification
    • Jason Carter used recordings to also determine features
  21. Observations and Features

    Slide 21 - Observations and Features

    • 21
    • Insurmountable
    • Time between commands increased and programmers spent large amount of time outside programming environment
    • Surmountable
    • Certain sequences of actions were repeated when surmounting difficulties
    • Lots of other reasons for this behavior
    • Progress
    • Difficulty
    • Browser
    • Action sequences are represented as strings
  22. Algorithm Insight

    Slide 22 - Algorithm Insight

    • 22
    • Detect sequences that occur during surmountable difficulties
    • Other sequences are insurmountable difficulties
    • Approach does not work in integrated algorithm because non surmountable sequence can be non difficulty or insurmountable sequence
    • How to match sequences?
  23. K Nearest Neighbor Algorithm

    Slide 23 - K Nearest Neighbor Algorithm

    • 23
    • Insurmountable
    • Insurmountable
    • Surmountable
    • Surmountable
    • Surmountable
    • ACBDJ
    • DEHJL
    • ADFGHI
    • DGHIJ
    • AGHJ
    • Training Sequences
    • Surmountable
    • ABDHI
    • Test Sequence
    • (K = 3) Nearest Neighbors
    • Prediction: Most frequent label in the K neighbors
    • Levenshtein Distance
    • Similarity Metric
    • Log Splitting Scheme
    • Split by Save
    • Logical piece of work
    • Can compare sequences of different lengths
    • 295 Surmountable, 121 Insurmountable Sequences
  24. Effect of K

    Slide 24 - Effect of K

    • 24
    • K >= 25, results stabilize
    • Fraction of surmountable predictions that were correct
    • Fraction of insurmountable predictions that were correct
    • Algorithm biased towards surmountable
  25. Split by save, K = 25 Nearest Neighbor

    Slide 25 - Split by save, K = 25 Nearest Neighbor

    • 25
    • 96% of surmountable difficulties correctly identified
    • 63% of insurmountable difficulties correctly identified
    • When a difficulty is labelled as surmountable, 86% chance it is correct
    • When a difficulty is labelled as insurmountable, 84% chance it is correct
    • Only insurmountable predictions (29% of difficulties) can be investigated to find 63% of genuine cases
    • How many values with actual label T were assigned label P
    • Other forms of help still matter!
  26. Conclusions

    Slide 26 - Conclusions

    • 26
    • About half the class wanted incremental help
    • The vast majority of students were not hurt by help and probably benefited from it
    • Instructors could scrutinize a small number of total difficulties (29%) to get the majority (63%) number of insurmountable difficulties
    • Sequence matching detected surmountable difficulties in a composed algorithm
    • Traditional forms of help still important
  27. Future Work

    Slide 27 - Future Work

    • 27
    • Different (non API) studies
    • Implement algorithm by trapping programming environment and browser events
    • How surmountable/insurmountable classification is actually used
    • Different features and algorithms
    • Numeric difficulty degree
    • Sharing solutions to similar “difficulties”
  28. Complementary Work from Jason’s Thesis (Ph.d. Defense Talk on YouTube)

    Slide 28 - Complementary Work from Jason’s Thesis (Ph.d. Defense Talk on YouTube)

    • 28
    • Programming Barriers
    • Programmer Actions vs. Programmer Postures
    • Context needed to offer Help
    • Carter, J. (2014), Automatic Difficulty Detection, in Ph.D. Thesis, Department of Computer Science, University of North Carolina (http://youtu.be/HURoaO5f2FA): Chapel Hill.
    • Non Education Applications
    • Helpee vs. Helper Effort