This document discusses quality criteria, research ethics, and other research issues. It covers several topics: - Quality criteria for quantitative, qualitative, and mixed methods research including reliability, validity from measurement and research perspectives. - Research ethics including legal context, researcher integrity, protection from harm, privacy/confidentiality, informed consent, and issues around deception. - Other essentials for research including developing a research topic, purpose, and questions; conducting a pilot study; maintaining a research log; and managing data.

1. Chapter 3
Quality criteria, research ethics, and other research issues

2. 3.I Quality criteria for research
As we have seen, the basic definition of scientific research is that
it is a 'disciplined' inquiry, and therefore one thing research cannot
afford is to be haphazard or lacking rigour. Unfortunately, general
agreement about research quality in scholarly circles stops at the
recognition of its importance; when it comes to specifying the
concrete 'quality criteria' to be applied, the literature is
characterized by a host of parallel or alternative views and very
little consensus.

3. 3. I. I Quality criteria in quantitative research
The concept of 'reliability‘ is fairly straightforward, but when we
look at 'validity' we find two parallel systems in the quantitative
literature-one centered around 'construct validity' and its
components, the other around the 'internal/external validity’
dichotomy-and scholars tend to be surprisingly vague about the
relationship between these two systems: the usual practice is
that a work either covers one or the other. This dualistic
approach is due to the fact that within the quantitative paradigm
meaningfulness' has been conceptualized from two
perspectives: research design and measurement.

4. The discussion of quantitative quality standards is best
dividedinto three parts: (a) reliability (b) measurement
validity, and(c) research validity.
Reliability
The term reliability comes from measurement theory and refers
to the 'consistencies of data, scores or observations obtained
using elicitation instruments, which can include a range of tools
from standardized tests administered in educational settings to
tasks completed by participants in a research
study.

5. Measurement validity
the concept of validity from a measurement perspective
has traditionally been summarized by the simple phrase: a test is valid
if it measures what it is supposed to measure. Lynch (2003: 149)
summarizes the new conception clearly: 'When examining the validity
of assessment, it is important to remember that validity is a property
of the conclusions, interpretations or inferences that we draw from the
assessment instruments and procedures, not the instruments and
procedures themselves.
Research validity
The second type of validity, 'research validity' is broader than
measurement validity as it concerns the overall quality of the whole
research project and more specifically (a) the meaningfulness of the
interpretations that researchers make on the basis of their
observations, and (b) the extent to which these interpretations
generalize beyond the research study (Bachman 2004a).

6. 3.1.2 Quality criteria in qualitative research
The usual statement we find in the literature about quality in
QUAL research is that it is less straightforward to define than
quality in QUAN research. We saw in chapter 2 that a qualitative
study is inherently subjective, interpretive as well as time and
context bound; that is in a qualitative inquiry ‘truth’ is relative
and ‘facts’ depend upon individual perceptions(Morse and
Richards 2002); for this reason several researchers have argued
that qualitative research requires its own procedures for
attaining validity that are different from those used in
quantitative approaches.

7. Three basic quality concerns in qualitative research
1- Insipid data focusing on 'individual meaning' does not offer any
procedures for deciding whether the particular meaning is interesting
enough.
2- Quality of the researcher Morse and Richards (2002) are-right when
they warn us that any study is only as good as the researcher, and in a
qualitative study this issue is particularly prominent because in a way
the researcher is the instrument.
3- Anecdotalism and the lack of quality safeguards.

8. Strategies to ensure validity in qualitative research
Building up an image of researcher integrity
the most important strategy to ensure the trustworthiness of a project
is to create in the audience an image of the researcher as a scholar
with principled standards and integrity. This image of integrity is made
up of several small components but there are certain strategies that
are particularly helpful in showing up the researcher's high standards
(provided, of course, those exist):
• Leaving an audit trail By offering a detailed and reflective account of
the steps taken to achieve the results.
• Contextualization and thick description Presenting the findings in rich
contextualized detail helps the reader to identify with the project and
thus come on board.

9. • Identifying potential researcher bias Given the important role of the
researcher in every stage of a qualitative study.
• Examining outliers extreme or negative cases and alternative
explanations No research study is perfect and the readers know this.
Therefore, explicitly pointing out and discussing aspects of the study
that run counter to the final conclusion is usually not seen as a
weakness but adds to the credibility of the researcher.

10. 3. I . 3 Quality criteria in mixed methods research
The rationale for mixing methods
The important point to emphasize is that a mixed methods inquiry
offers a potentially more comprehensive means of legitimizing findings
than do either QUAL and QUAN methods alone by allowing
investigators to assess information from both data types.
The 'design validity' of the study
the term 'design validity‘ refers to the extent to which the QUAL
and QUAN components of a mixed methods study are combined or
integrated in a way that the overall design displays complementary
strengths and nonoverlapping weaknesses of the constituent methods.

11. 3.2 Research ethics
Social research-including research in education-concerns people's lives
in the social world and therefore it inevitably involves ethical issues. As
Punch (2005) points out, such issues are more acute in QUAL than in
QUAN approaches because qualitative research often intrudes more
into the human private sphere: it is inherently interested in peoples
personal views and often targets sensitive or intimate-matters.
3.2.2 Legal context
In many countries, observing ethical principles is enforced by legal and
institutional requirements. In the US, for example, researchers have to
submit a detailed research plan for approval to an Institutional Review
Board prior to starting their investigations in order to comply with
federal regulations that provide protection against human rights
violations.

12. 3. 2. 3 Researcher integrity
The Ethical standards of the American Educational Research
Association (AERA 2002), starts out with a set 'of 'guiding standards'
describing the researchers' general responsibilities to the field. These
include the following points:
- Educational researchers must not fabricate, falsify, or misrepresent
authorship, evidence, data, findings, or conclusions.
- Educational researchers must not knowingly or negligently use their
professional roles for fraudulent purposes.
- Educational researchers should attempt to report their findings to all
relevant stakeholders, and should refrain from keeping secret or
selectively communicating their findings.

13. 3.2.4 Protection from harm
The primary principle of research ethics is that no mental or physical
harm should come to the respondents as a result of their participation
in the investigation.
3.2.5 Privacy, confidentiality and anonymity
It is a basic ethical principle that the respondent's right to privacy
should always be respected and that respondents are within their
rights to refuse to answer questions or to withdraw from the study
completely without offering any explanation. t is also the participants'
right to remain anonymous, and if the participant's identity is known
to the research group, it is the researcher's moral and professional
(and in some contexts legal) obligation to maintain the level of
confidentiality that was promised at the onset.

14. 3.2.6 Informed consent and the issue of deception
The most salient and most often discussed aspect of research ethics is
the issue of informed consent. In the US, for example, federal
regulations not only require written consent from the
participants but also require informed consent before a
researcher can use an individual's existing records for
researchpurposes ( Johnson and
Christensen 2004).
Deception
It does not require much justification that sometimes
researchers cannot provide full disclosure of the nature and
purpose of the study without causing participant bias or even
invalidating the study, and in some (rare) cases the researcher
needs not only to withhold some information but to actively
mislead the participants.

15. Research topic, research purpose and research questions
Every investigation has a starting point and unless we adopt someone
else's (for example, the supervisor's) design idea or join an ongoing
investigation as a co-researcher, this starting point is a broad 'research
topic'.
The research purpose is relatively short statement that describes the
objective of the planned study, explaining why the investigation is
undertaken and what its potential significance is.
what is a good research question like? When we have thought of some
possible questions, we have to ask ourselves: are they worth asking
and, more importantly, answering? I believe that one thing we must
try and avoid at all cost is to 'run into a "so what“ response to our
research' (Mackey and Gass 2005: 17).

16. Other essentials for launching a study: pilot study,
research log, and data management
piloting the research
Piloting is more important in quantitative studies than in qualitative
ones, because quantitative studies rely on the psychometric
propertiesof the research
instruments.
Research log
As with any real logbook, all the entries should be properly dated and
the consecutive pages of the logbook should be numbered and kept
together in a folder. Alternatively, we can keep an electronic log file in
our computer that we regularly update.
Techniques to manage and store data records
Researchers need to make a detailed list of all the data sources in a
research log (see previous section), organized according to key
parameters (for example, type of data, place of data-gathering) and then