The first requirement for an online mathematics homework engine is to encourage students to practice and reinforce their mathematics skills in ways that are as good or better than traditional paper homework. The use of the computer and the internet should not limit the kind or quality of the mathematics that we teach and if possible it should expand it. Now that much of the homework practice takes place online we have the potential of a new and much better window into how students learn mathematics but we must continue to ensure that students are studying the mathematics we want to have learned and not just mathematics that is easily gradable. Several of the open source mathematics engines that do this well are represented at this conference. The WeBWorK mathematics rendering engine started twenty years ago as a stand alone application. Since then homework questions contributed by many, many mathematicians to the OpenProblemLibrary (OPL) have created a collection of over 30,000 Creative Commons licensed problems primarily directed toward calculus but ranging from basic algebra through matrix linear algebra. I’ll present one of the adaptations of WeBWorK which allows it to render mathematics questions for a standard Moodle quiz in much the same way that STACK functions. Both STACK and WeBWorK vastly increase Moodle’s ability to handle mathematics. Using the Moodle quiz format will make the OPL available to many more educators and allows utilization of Moodle’s facility at collecting student data. If there is time I’ll show a second adaptation which allows WeBWorK to serve as an assignment type within Moodle. These same mechanisms allow active WeBWorK questions to be embedded in other learning management systems, in interactive textbooks and even HTML pages. This capability fits well with an emerging trend to use smaller, more specialized, inter-operating components for online education.

1. WeBWorK within
Moodle and WEPs
April 2016
Shape of Educational Data
Fairfax, VA
Michael Gage
University of Rochester
https://hosted2.webwork.rochester.edu/
gage/SOED_urls.pdf

2. WeBWorK
WeBWorK
• The WeBWorK rendering engine, like STACK, does an
excellent job of presenting mathematical homework
questions and checking the students’ answers in a
robust way — one which closely mimics paper-and-
pencil homework.
• WeBWorK was originally developed with its own basic
LearningManagementSystem (LMS).
• WeBWorK also has a webservice which allows it to plug
into other LMS’s in order to share WW’s math smarts and
the 30,000+ math questions (precalculus through linear
algebra) that have been developed by the WW
community.

3. WeBWorK
Outline
The WW webservice can be used in several modes:
• 1. The Moodle quiz plugin which roughly mimics STACK.
• 2. The wwassignment plugin which supplies a homework
assignment within Moodle. Provides single-sign-on and passes
grades back to Moodle gradebook.
• 3. A recent LTI plugin, which may replace wwassignment, and
allows plugins to Canvas, Blackboard, Moodle and other LMS
that implement LTI 2.x with grade transfer. (w/ Geoff Goehle)
• 4. An HTML plugin which allows active WW problems within a
webpage to be used as ungraded exercises. These also
interoperate with MathBookXML — a write-once-print-in-all-
formats authoring tool by Rob Beezer

4. WeBWorK
Moodle quiz plugin
https://devel3.webwork.rochester.edu/moodle
login:visitor password: visitor
login:practice1 password: practice1
login:practice2 password: practice2
…

5. WeBWorK
Moodle quiz plugin
https://devel3.webwork.rochester.edu/moodle
login:visitor password: visitor
login:practice1 password: practice1
login:practice2 password: practice2
…

6. WeBWorK
Moodle quiz plugin
Moodle handles the presentation and quiz navigation.
WeBWorK only renders the questions and evaluates the answers.

7. WeBWorK
Click “grade and finish”: the answer is graded and the correct
answers are reported immediately.

8. WeBWorK
Problem 2. We’ll enter answers
and click “submit attempt”.

9. WeBWorK
Some answers are wrong.
I can resubmit and try again.

10. WeBWorK
I click “grade and finish”.
My answers are recorded,
the correct answers are shown.
and I can no longer change the answers.

11. WeBWorK
Moodle quiz plugin
• The data collected by the Moodle quiz plugin is the
same as the data collected by STACK
• The WW questions function similarly to STACK and
you can mix STACK and WW questions in a quiz.
• The user interface appearance for the two
question types is not yet identical — some more
work could be done here.
• I could use some help from experienced Moodle
users on this.

12. WeBWorK
Assignment plugins
• The wwassignment and the LTI type are similar
• There is a script which allows a WeBWorK
homework set of questions to be imported to the
Moodle question bank.
• The linear algebra and ODE course on the next
slide has homework sets in both formats.

13. WeBWorK
Linear Algebra and ODE: Spring 2009
2009 Linear Algebra and ODE course demos both
regular webwork homework AND Moodle quiz versions

14. WeBWorK
Standard WW homework set
Single-sign-on (SSO) brings you to a standard webwork
homework set (set 1) — grades are returned to Moodle

15. WeBWorK
Data collected by WW
1. Problem usage available in the library
2. Student progress and real time statistics
3. Past answers list (preliminary analysis by Roth,
Record, et. al.)
4. “email the professor” conversations.

16. WeBWorK
Problem usage statistics in OPL
• Usage: Number of times problem used
• Attempts: Average number of attempts for success
• Status: Success rate

17. WeBWorK
Student progress
• Final score on problem
• Number of attempts used
• There is also a percentage statistics page

18. Past Answer data
An answer

19. Hand analysis of past answer data
• Vicki Roth, V. Ivanchenko, and N. Record, “Evaluating student
response to WeBWorK, a web-based homework delivery and
grading system”, Computers & Education, vol. 50, no. 4,
pp. 1462-1482, 2008.
• Results:
• Analysis of answer data can distinguish between productive
progress and random guessing.
• Guessing is extinguished for longer answers (or expressions).
• Feedback from this study has been used to improve user
interface.
• Analysis revealed an unexpected pattern: strong students
resubmitted wrong answers far more than other students
WW past answer data

20. WeBWorK
Categorizing past answer data
• Student Response Model categories:
1. Reworking
2. Fixing an Entry Error
3. Re-submission (literal or mathematically equivalent)
4. Nonstrategic guessing
5. Nonsense
• https://math.webwork.rochester.edu/assessment/RaterTrainingManual.htm
• Can this analysis be automated and improved? BigData opportunity?

21. Email instructor data

22. Email instructor data
On Wed, Oct 15, 2014 at 7:04 PM, XXXXXXX <xxxxxx@u.rochester.edu> wrote:
This message was automatically generated by the WeBWorK system at
https://math.webwork.rochester.edu/webwork2/, in response to a request from
Click this link to see the page from which the user sent feedback:
https://math.webwork.rochester.edu/webwork2/fall14mth141/6/24/?
effectiveUser=xxxxx&showSolutions=1&showOldAnswers=1&displayMode=MathJax&showCorrectAnswer
s=0&showHints=1
***** The feedback message: *****
For Problem 24, Set 6, why is the answer +inf when the limit goes to -inf
of sqrt(x^2-8x+1)-x
I think I am forgetting an important rule...
***** Data about the problem processor: *****
Display Mode: MathJax
Show Old Answers: yes
Show Correct Answers: no
Show Hints: yes
Show Solutions: yes
An “email the instructor” message

23. Email instructor data
***** Data about the user: *****
User ID: XXXXX
Name:
Email:
Student ID:
Status: Enrolled ('C')
Section:
Recitation:
Comment:
***** Data about the problem: *****
Problem ID: 24
Source file: Library/Rochester/setLimitsRates3Infinite/ur_lr_3_13.pg
Value: 1
Max attempts unlimited
Random seed: 3749
Status: 1
Attempted: yes
Last answer:
AnSwEr0001: -4
AnSwEr0002: INF
Number of correct attempts: 1
Number of incorrect attempts: 0

24. Email instructor data
***** Data about the homework set: *****
Set ID: 6
Set header file: screenSetHeader.pg
Hardcopy header file: paperSetHeader.pg
Open date: 09/29/2014 at 07:00am EDT
Due date: 10/10/2014 at 11:59pm EDT
Answer date: 10/14/2014 at 12:59am EDT
Visible: yes
Assignment type: default
------------------------------------------------------------------------------------------------
Instructor response:
------------------------------------------------------------------------------------------------
Hi,
If you look conceptually at the question:
sqrt(x^2-8x+1) when x is negative will be x^2 (positive) -8x (positive) so it will be
constantly increasing and thus approaching infinity.
Let me know if that doesn't help and I will do my best to explain it better.
Good luck tomorrow!

25. WeBWorK
Email instructor data
• Another opportunity for BigData analysis?
• Natural language processing?

26. WeBWorK
Advantages of interoperation
• Advantages of WW
• Many math questions in the OPL
• More than 770 institutions used it this semester.
• Potential for collecting data from many sources.
• Many authors contributing questions
• Advantages of Moodle
• Even more institutions use Moodle and adding WW
in addition to STACK increases its math power.

27. WeBWorK
Institutions using WW

28. WeBWorK
OPL with 33K questions

29. WeBWorK
Embedding WW in HTML
• https://hosted2.webwork.rochester.edu/gage/
2016JMM/mth162_overview.html
<p><iframe width="800" height="400"
src="https://hosted2.webwork.rochester.edu/webwork2/html2xml?
&answersSubmitted=0&
&sourceFilePath=Library/Union/setIntFTC/sc5_4_13.pg&
&problemSeed=123567&
&courseID=daemon_course&
&userID=daemon&
&course_password=daemon&
&showSummary=1&
&displayMode=MathJax&
&problemIdentifierPrefix=102&
&language=en&
&outputformat=sticky"><br/>
</iframe></p>

30. Thank you
https://hosted2.webwork.rochester.edu/gage/
2016WEPS.pdf
for this presentation
LTI plugins and data collection are joint work with
Goeff Goehle at Western Carolina University

31. WeBWorK
Canvas LTI demonstration
• https://canvas.instructure.com/courses/1000526
• (You’ll need to sign up for an account.)

32. WeBWorK
Additional data available from WW
• The WeBWorK assignment mode collects
additional data on the server for each problem.
• Past answer data
• Email the instructor data
• How can this data be analyzed effectively?

33. WeBWorK-Blackboard
Uses LTI standard. What is done here in Blackboard can in principal be done in
other LMS such as D2L or Canvas. It’s not as flexible as the connection to
Moodle.

34. WeBWorK
Construct links with single sign on (SSO) to WeBWorK
site or a Moodle site using LTI.

35. WeBWorK
Display a webwork course within the Blackboard page

36. WeBWorK
Display homework set for that week.

37. WeBWorK
Or display a single WeBWorK problem