My presentation for Open Help 2011.

1. Starting an Open Source
Certification Program
Dru Lavigne
Chair, BSD Certification Group Inc.
Open Help 2011

2. This presentation will discuss:
Brief introduction (why certification?)
Questions to answer before starting
Details (10 step program)
Summary (is it worth the effort?)

3. Introduction
What does this have to do with
documentation?
Doc team members often have the knowledge
and writing skills required by a cert program's
SMEs (subject matter experts) and authors of
training materials

4. Introduction
Why consider a certification program?
Increased project exposure and enterprise
adoption of that open source software
Increased business credibility; certification is a
requirement in some environments
Possible revenue stream

5. Introduction
Is it a good fit for our project?
How mature is the project?
Is there a corporate sponsor or foundation?
What is the project's “product”?
Are the “skills” for that product assessable and of
interest to employers?
Is there a strong documentation focus?

6. Before Starting
Determine the organizational structure:
For profit? non-profit? incorporated?
Part of the project or separate organization?
Paid, volunteer, mixture of both?
Create exam only or also provide training
materials?

7. Before Starting
Determine the exam structure:
Who is the target audience?
Which skills are to be assessed?
Are you aiming for psychometric validity?
accreditation?
How many exams must be passed to be certified?
What value do certificants receive?

8. Before Starting
Psychometrics:
The science of assessment
Not cheap: $25k-100k+ to get started, at least $5k
/yr to maintain each exam
Provides process and expertise--useful when you
don't know what you're doing
Worth every penny

9. Before Starting
Accreditation:
Not cheap: expect $25k to get started, $2-5k/yr to
maintain
Time consuming, tedious, boring work (i.e. you
will document all processes to ISO standards)
Exam program must already be psychometrically
valid and recertification process is mandatory
https://www.ansica.org/wwwversion2/outside/PER
general.asp?menuID=2

10. Before Starting
Exam delivery:
Paper or computer based?
Multiple choice, virtual lab?
Proctored environment?
Who provides scoring services?
Issues: cost, proprietary solutions

11. Step 1
JTA:
Job Task Analysis: a survey to determine the
importance and frequency of tasks--results are
used to create exam objectives
Requires knowledge of which tasks are possible
for that product
Survey should be completed by employers,
employees, and hiring staff; aim for a large and
balanced sampling

12. Step 2
Certification Requirements (Blueprint):
Typically preceded by a report on the JTA results;
results are analyzed to determine what belongs
on the exam
Certification Requirements document contains:
description of the exam audience, study domains
and their %, and detailed exam objectives. Each
exam objective includes its importance and
frequency rating, bloom's taxonomy classification,
and a clear explanation of what is required of the
exam candidate to successfully pass that
objective

15. Step 3
Question Writing:
SMEs (subject matter experts) write exam
questions to match exam blueprint
Aim for a large pool of questions, preferably
several per objective
There are only so many ways you can ask a
question
Original draft of question typically bears little
resemblance to the final form of the question

16. Step 3
The Art of Question Writing:
Items must be written to the skill level of the
audience defined in the Certification
Requirements
Items must be clearly written and understandable,
even to non-native English speakers
Each item assesses only one objective
Items should be consistent in formatting style,
punctuation, and grammar

17. Step 3
The Art of Question Writing:
Avoid items that require the examinee to select
multiple answers or that include an answer of "all
of the above" or "none of the above"
If instructional materials exist, do not cut/paste
information to create items
Do not place enemies on the same exam form;
enemies are items that contain information that
can be used to answer another question on the
exam

18. Step 3
The Art of Question Writing:
Distractors should be of similar length and
plausibility as the correct answer; someone who
does not understand the objective should not be
able to answer correctly from contextual cues
Avoid stems containing negatives or double
negatives
Have a standard for determining the order of the
options and ensure that the correct answer is not
always in the same position

19. Step 4
Review Period:
Once a pool of questions is created, it undergoes
scrutiny to make sure questions meet the criteria
identified in Step 3
SMEs can help review technical accuracy of exam
questions
Review should include non-SMEs to look for
contextual cues, formatting/grammatical issues

20. Step 5
Beta Period:
Once the question review is complete,
psychometrician will determine which questions
go on which form: determination must match
exam blueprint and multiple forms must provide
equal assessments
Beta period is used to identify anomalies in how
candidates respond to questions
Beta participants should closely match the exam's
audience and beta should be available in
countries where the exam will be offered

21. Step 5
Beta Period:
Beta participants can not "pass" or "fail" the beta
exam as the passing score can not be determined
before the Angoff session
Any questions identified by the beta as being
problematic (e.g. nearly everyone answers
correctly or incorrectly, English speakers answer
correctly but non-native speakers do not) need to
be reviewed to determine if belong on exam or
need to be rewritten

22. Step 6
Angoff session:
Used to finalize questions and set the cut
(passing) score by determining the number of
items a minimally competent examinee would
answer correctly; anyone who correctly answers
more items passes the exam as they will have
demonstrated that they are more than
minimally competent
SMEs and psychometrician should attend

23. Step 6
Angoff session:
Each SME reviews every exam item and rates it
on a scale of 1-10 on the likelihood that a
minimally competent person would answer that
question correctly
Once all the items are individually rated, the
ratings are compared, any discrepancies are
discussed, and a concensus is reached on the
rating for every item
The psychometrician uses the list of ratings to
determine the passing score for the exam

24. Step 7
Launch Exam:
Prior to launch, certification organization needs to
publish their policies: cheating, recertification,
handling disputes, etc. as well as finalize any
contracts regarding delivery and scoring
Need to write marketing materials, press releases,
update website information, etc.
If project will be providing training materials, are
they created in parallel with the exam creation
process or serially after the exam becomes
available?

25. Step 8
Ongoing Metrics:
Psychometrician collects ongoing metrics on how
every exam candidate responds to each exam
question
Psychometrician will advise of any anomalies that
may indicate cheating is occuring at a testing
center or with a particular proctor
Certification body determines frequency of review
process (annually or after so many exams)

26. Step 9
Periodic Review:
Psychometrician will identify which items should
be reviewed by the SMEs
These will include items that are
disproportionately answered correctly or
incorrectly and any items that are statistically
strange (e.g. items which minimally competent
examinees tend to answer correctly while very
competent examinees tend to answer incorrectly)

27. Step 9
Periodic Review:
SMEs may rewrite these items or replace them
with different items
Psychometrician may decide to introduce "pilot"
items which don't affect the examinee's score and
therefore don't require a beta or angoff period
Once the review is complete, the certifying
organization will publish the new version of the
exam (typically not announced)

28. Step 10
Rinse and Repeat:
In other words, an exam may be published but it
is never “done”
Organization needs to decide how often the JTA
needs to be repeated and the Certification
Requirements document is updated to reflect
changes in the product
Brave organizations may even decide to undergo
the whole process again with another certification
program

29. Summary
Creation and maintenance of certification exams
is a lot of work and can cost a lot of money
Know your goals beforehand
Aim to have a few individuals who are committed
to the long haul
Expect the first exam to take at least twice as long
to launch as you think it should
Don't reinvent the wheel--make relationships with
other certification organizations

30. Questions & Discussion
Slides:
http://www.slideshare.net/dlavigne/openhelp11
BSD Certification website:
http://www.bsdcertification.org
Contact:
dru@freebsd.org