1. DEMYSTIFYING REFLECTIVE PRACTICE:
USING THE DATA MODEL TO ENHANCE
EVALUATORS’ PROFESSIONAL ACTIVITIES
“Insights and innovation await us only if we are capable of stepping
outside the frenzied worlds of data and distraction that wash over us…
time for reflection is an open invitation to discover what awaits us…”
(Forrester, 2011, pp. 216-217)
Tiffany L. Smith
John M. Peters
Gary J. Skolits
Patrick B. Barlow
Expert Lecture
American Evaluation Association 2013
Friday, October 18th, 1:45-2:30pm

2. INTRODUCTION: WHO ARE WE?
TIFFANY SMITH
JOHN PETERS
GARY SKOLITS
PATRICK BARLOW
A doctoral candidate in
the Evaluation, Statistics
and Measurement
program at the
University of Tennessee
Knoxville. She has taken
two courses on
Reflective Practice and
has interned with the
Institute for Reflective
Practice at the University
of Tennessee with John
Peters. Her primary
research interests
include stakeholder
communication and
involvement as well as
reflective practice in
evaluation.
The Director of the
Tennessee Teaching and
Learning Center’s
Institute for Reflective
Practice. The primary
mission of the Institute is
to promote reflective
practice by individuals
and organizations served
by the University of
Tennessee, Knoxville. Its
service is based on the
belief that addressing
and managing change in
individual lives and
organizations demands
flexible thinking
processes by persons
capable of proactive
interpersonal
communication.
The Director of the
Institute for Assessment
and Evaluation at the
University of Tennessee’s
in the College of
Education, Health, and
Human Sciences. He is a
tenured associate
professor in the
Evaluation, Statistics, an
d Measurement Ph.D.
program. His research
interests include
evaluation methods, the
evaluation of
educational
interventions, P-16 and
college access program
evaluation as well as
higher education
accountability.
A doctoral candidate in
the
Evaluation, Statistics, an
d Measurement program
at the University of
Tennessee. He works as a
Statistical and Research
Design Consultant for
the University of
Tennessee Graduate
School of Medicine. His
primary research areas
are in higher education
assessment and teaching
statistics and research
methods in the social
sciences, epidemiology, a
nd clinical medical
fields.

3. WHY REFLECT?
“Reflective practice is more than spending ten
minutes at the end of an evaluation
congratulating oneself on getting the damn
thing done” (Patton, 2012, p. 401).

4. REFLECTIVE PRACTICE IN EVALUATION
• One of six Essential Competencies for program
evaluators, defined as:
– “being acutely aware of personal evaluation
preferences, strengths, and limitations; selfmonitoring the results of actions intended to
facilitate effective evaluation studies; and planning
how to enhance future endeavors”
(Stevhan, King, Ghere, & Minnema, 2005, p. 46)

5. HOW DOES REFLECTION IMPROVE
EVALUATION PRACTICE?
• Can be used as a means for selfawareness, professional growth and
development, improved ethical practice, dialogue
and stakeholder communication, and for learning
in organizations
• The reflection process can happen while engaging
in daily practice as well as after the fact, either
alone or with others.
• Reflection involves being a student of the actions
that you make, and the study of those actions is

6. Patton (2012) reports that “in speeches and
workshops at professional evaluation
association meetings, I like to ask for a show
of hands of those who systematically reflect
on evaluations they have conducted for
learning and further professional
development. Few hands go up; typically, in
fact, no one raises a hand” (p. 400).

7. A SHOW OF HANDS?
Who systematically reflects on evaluations they
have conducted for learning and further
professional development?
Why Not?
Why is reflective practice not a part of regular
evaluation practice?

8. WHY IS RP NOT A PART OF REGULAR
EVALUATION PRACTICE?
• Perhaps there is a lack of awareness of the
purpose of reflection in evaluation?
• Unclear what reflective practice is in the first
place?

9. WHAT IS RP, ANYWAY?
According to Peters (1991), “reflective practice
involves more than simply thinking about what
one is doing and what one should do next. It
involves identifying one’s assumptions and
feelings associated with practice, theorizing
about how these assumptions and feelings are
functionally or dysfunctionally associated with
practice, and acting on the basis of the resulting
theory of practice” (p. 89).

10. THE DATA MODEL FOR REFLECTION
•
•
•
•
DESCRIBE
ANALYZE
THEORIZE
ACT

11. HOW HAS DATA BEEN USED AS A REFLECTIVE
PRACTICE EFFORT?
• Mainly to guide individual and group
reflection on a variety of work-related
issues, train teams of workers, and for
supervisory training.
• Also as major component of action research
projects in community education
programs, business applications, and in higher
education settings

12. DESCRIBE
• A detailed account of the situation, task, or
incident that has happened in one’s practice.
• Concerned with identifying the specifics of the
situation at hand.
– Context of the situation, the setting in which it
occurred, who was involved, etc.
• Use as much detail as possible, so as to reflect on
the whole of the problem and its setting.
• Everything that you know about the situation and
context are key to this step of the reflective
process.

13. DESCRIBE
• This might prove to be harder than one might
expect
– A description only involves accounts of what actually
occurred
• Description is devoid of any why inclinations or
explanations in order that a clear picture of the
problem is painted (Peters, 2009).
• This portion of the DATA model could prove to be
beneficial if one includes other colleagues, be it
evaluators or stakeholders.

14. ANALYZE
• The why that it was necessary to avoid during
the Describe phase gets its turn.
• Identification of factors contributing to the
situation.
• Examination of one’s assumptions, biases, and
preconceived notions

15. ANALYZE
• Beliefs, rules, motives, and facts should be things that
the practitioner is aware of during this stage of
reflection.
• Through the first two steps of DATA, problem is at least
tentatively outlined
– The practitioner should have a very clear and holistic
understanding of the problem, its context, and why it has
come up in their practice.
• This will in turn produce a practical question of how to
solve the problem.
– How, specifically, can I change my practice to produce a
better outcome?

16. THEORIZE
• The focus is on answering the practical question
introduced in the analysis.
• Refers to deriving a practical theory from the
description and analysis in order to improve one’s
practice.
– A practical theory refers to understanding the structure
and implications of a theory’s use in professional practice.
• This does not need to be a scholarly theory, derived
from the literature, although previous literature may
influence this section
• This theory comes from the practice in which one
works.

17. THEORIZE
• At the end of Theorize, the practitioner should be able
to ask What am I going to do, and Why is this solution
better than other potential solutions?
• How many options do you have?
• Is there relevant literature or research to indicate that
this option is better over others?
• In practice, what has worked before, and how well?
• The Theorize portion of DATA is intended to provide
practitioners with a theoretical solution to their
concern.

18. ACT
• Taking action.
• What are you going to do about it, specifically?
• Moving forward based on a practical theory which is
derived from reflection on the situation, including the
what and the why.
• When one takes action, it is in the context and for the
betterment of their practice.
• This action tests the practical theory that was
developed via this reflective process.
• This can lead to further reflection on one’s practice and
further professional development.

19. A CASE FOR REFLECTION

20. Michael is a seasoned evaluator who has
worked many different evaluation projects over
the past 20 years. The following is a case of
Michael utilizing the DATA model of reflective
practice in order to understand and solve a
dilemma in his evaluation work.

21. DESCRIBE
At the beginning of the evaluation project:
• Contract awarded for 3 years of funding
– Serving a true community need
• Stakeholders and evaluators come together to define the
evaluation design, data collection, schedules, procedures
• Client not familiar with evaluation or working with
evaluators and very reluctant and cautious
– Displaying signs of confusing evaluator with auditor
• Client staffs project and project data collection and
reporting based on evaluation design

22. DESCRIBE
• About a year in, change in requirements from funding agency
–
–
–
–
Reporting is vastly increased
Increase in the amount of data to be collected by tenfold
Complications for collecting the data are greatly increased
Project itself is being redefined by new data collection requirements
• Changes mandated at the policy level resulting from inner turmoil
in funding agency
• This issue requires a total renegotiation of the client’s data
collection process as well as the evaluation design, breaking all of
the prior agreements.
• Data collection is complex, training implemented that they never
counted on

23. Due to this issue, the evaluation is struggling.
Evaluative progress is not being made.

24. ANALYZE
• Why is this situation occurring?
– Evaluator’s assumption that no major policy change
would occur was totally inaccurate; after 20 years of
practice, never has such a change been encountered
– Examination of how evaluator’s behavior could have
helped to create the situation
– Examine angers and frustrations
– Evaluator and client, and now multiple stakeholders
come to new understandings about what is expected
and required
• Sense of a weak relationship, a weak tie, and a
mistrust in the stakeholders

25. Practical Question: How can we work to
reestablish trust in the evaluation and program
environment?

26. THEORIZE
• Utilization of Program Evaluation Standards (as well as ethical
guidelines, Essential Competencies, Utilization Focused evaluation)
to decide on the most effective means of establishing trust.
– Some specific Standards: F1, F2, F4, U1, U2, U3, U4, U8, P2, P4, P7
• Discusses options with trusted colleagues
• After thinking about other options, including termination of the
project, Michael decides that the best strategy would be:
– Face to face discussion with primary stakeholders
• Review the status of the relationship prior to the change (contracting, working
relationship)
• Review the relationship after the change
• Gauge the perspectives of stakeholders
• Is there a basis for continuing? Want?
• Both the evaluation staff and the client must be willing to put the cards on the
table and understand that this was an imposed change.

27. ACT
• Michael acts on the basis of a practical theory.
• Renegotiation of agreement
• Re-establishment of trust through critical
conversation with stakeholders

28. USING DATA IN EVALUATION
• How did reflection on the situation help
Michael to come up with a plan of action?
– How could it have improved his professional
practice? Way of thinking?
• When is DATA practical for evaluators?
• Should RP be an alone thing? Or is this best
done collaboratively?

29. USING DATA IN ACTION RESEARCH: THE
DATA-DATA APPROACH
• DATA-DATA has been used to plan and conduct more
than 125 projects in higher education, community
education, and business.
–
–
–
–
Design
Analyze
Theorize
Act
• Using the second DATA when you want to have more
confidence in the result of your thinking and action
that results from the first DATA.
• The systematic design work and careful analysis in the
second DATA accomplishes this.

30. WHAT DOES REFLECTION PROVIDE FOR
EVALUATION PRACTITIONERS?
• A way to step back from daily practice and to
create an encompassing understanding of the
issues and situations that we face
• Reflecting helps us to learn more about context
• Reflection provides us an opportunity to
investigate our own assumptions and how they
influence our actions.
• How can RP be integrated into evaluation work?

31. Can anyone think of a way that the DATA
model can be used in their evaluation work?

32. QUESTIONS?
CONTACT INFORMATION:
Tiffany Smith: tsmith92@utk.edu
John Peters: jpeters@utk.edu
Gary Skolits: gskolits@utk.edu
Patrick Barlow: pbarlow1@utk.edu
Institute for Reflective Practice: irp@utk.edu