The document discusses validation of electronic source (eSource) data in clinical trials that use electronic health record (EHR) data. It describes conducting a mock clinical trial using synthetic data to validate the computer systems used for data collection and management. This validation process tests that the systems perform as intended and produce reliable results in a consistent manner. The validation documents all aspects of the computerized systems to provide evidence they are properly qualified before being used in actual clinical trials.

1. Wolfgang Kuchinke
Joint Semantic Group and CRI Group Meeting
University Duesseldorf, Duesseldorf, Germany
18.1.2016 Duesseldorf
Clinical Trials in the Learning
Health System: Computer System
Validation of eSource and EHR
Data

2. Basic question
How to make a clinical trial data management
system that uses EHR data, Patient Reported
Outcome (PRO) and eSource data as part of
the Learning Health System compliant with
regulations and with Good Clinical Practice
(GCP)?

3. Introduction

4. What is the Learning Health
System (LHS)
• Connecting health care with translational and
clinical research
• Generation of new medical knowledge as a by-
product of the care process
• Aim is to improve health and safety of patients
• Each participant in the LHS (clinician, patient,
researcher, database owner) can be both, a
data consumer and a data producer
• The LHS generates and applies knowledge
W. Kuchinke (2016)

5. What Are Clinical Trials?
• Clinical research is research involving humans
• Two types:
– Observational studies
• Observation of people in normal settings, where
researchers gather information, group volunteers according
to broad characteristics, and compare changes over time
– Clinical trials
• Research studies performed with people that are aimed at
evaluating a medical, surgical, or behavioral intervention
• A method to find out if a new treatment, like a new drug or
medical device is safe and effective
W. Kuchinke (2016)

6. Clinical Data to drive the LHS
• Results from randomized controlled trials are seen as
the “gold standard” for medical evidence
• But such trials are often performed outside the usual
system of care and recruit highly selected populations
• Thus, they not truly represent the populations of
patients with particular diseases
• For these reasons, the concept of using data gathered
directly from the patient care environment has
enormous potential for accelerating the rate at which
useful knowledge is generated

7. Randomized controlled trials (RCTs)
• Randomized controlled trials (RCTs) are rigorously
performed, permanently controlled and
transparently reported in peer-reviewed medical
journals
• They have the capacity to balance confounders (both
measured and unmeasured) between groups,
thereby strengthening the validity of treatment
effect estimation
• Therefore, although disadvantages exists, RCTs must
become a part of the LHS to generate medical
knowledge

8. Patient-Reported Outcome
• Patient-reported outcome (PRO) provides real-time,
actionable information that affects clinical care and can
address patient-centric research questions
• Potential methods for electronic data collection include
devices such as computer tablets or smart phones, EHRs,
online submission forms, and wearable devices
• Safeguards are necessary to ensure data security,
availability of an audit trail, data backup, and
appropriate access to investigators
• Patient-reported outcome devices are subject to
computer system validation
W. Kuchinke (2016)

9. Research as part of the LHS
• As part of the LHS, research is conducted in a real-
world setting
• Researchers are embedded in the health system and
both patients and clinicians are engaged in the
conduct of the clinical trial
• RCTs in the LHS typically have few patient selection
criteria to produce results that are valid for the large
majority of ill patients
• This type of research produces practical information
that can become input in decision-making processes
W. Kuchinke (2016)

10. The border between clinical
practice and research
• A new approach is needed for the regulatory and
ethical models that currently govern the conduct
of patient care and clinical research
• The line between clinical practice and research
becomes blurred creating challenges within
regulatory frameworks
• The joint involvement of care and research can
easily be described by a zone model, which
graphically depicts privacy data protection zones
involved in the data flow between care and
research

11. Ethical and Regulatory Challenges
and Data Privacy Zones
• Traditionally thr activities of care and research are kept separate to avoid
“therapeutic misconceptions”
– Patients enrolled in clinical research may not appreciate the difference between
research and care treatment
• Our zone model of data privacy is being built upon the concept of three
privacy zones (Care Zone, Non-care Zone and Research Zone) containing
graphic elements for databases, and data transformation operators, such
as data linkers and privacy filters
• Using these model components, a risk gradient for moving data from a
zone of high risk for patient identification to a zone of low risk can be
described
• The model allows analysis of data privacy and confidentiality issues that
emerge during research with patient data
– It provides a framework to specify a privacy compliant data flow, to communicate
privacy requirements and to identify weak points of data privacy

12. The Clinical Trials infrastructure of
the TRANDFoRm project

13. Learning Health System
Infrastructure for Europe
• It binds routine healthcare systems into research
translation and knowledge translation
• Extension of the evidence-based medicine
paradigm, employing electronic health records
(EHR)
– EHR data is used for research purposes
– Research results are displayed in the EHR
• TRANSFoRm is an EU FP7 project that seeks to
develop an infrastructure for the LHS in European
primary care
W. Kuchinke (2016)

14. What is an Electronic Health
Record (EHR)
• A digital version of a patient’s paper chart
• Real-time, patient-centered records that make information
instantly available
• Records
– patient’s medical history, diagnoses, medications, treatment plans,
immunization dates, allergies, radiology images, and laboratory and
test results
• EHRs are built to share information with other health care
providers and organizations – such as laboratories,
pharmacies, emergency facilities
• TRANSFoRm will use the EHR in its infrastructure to collect
data for clinical trials
W. Kuchinke (2016)

15. TRANSFoRm project
• Developes three clinical use cases to test its infrastructure
– Genotype-phenotype study in diabetes
– Randomised controlled trial with gastroesophageal reflux disease
– Diagnostic decision support system for chest pain, abdominal pain, and
shortness of breath
• Four models: clinical research, clinical data, provenance, and
diagnosis
– These models are maintained as ontologies with binding of terms to
define data elements
– CDISC ODM and SDM standards are extended using an archetype
approach to enable a two-level model of individual data elements
• Separate configurations for each use case
• Heterogeneity of data sources in the LHS makes it necessary to
employ ontologies and archetypes together

16. Clinical trial software in
TRANSFoRm
• Used in GORD use case
• Include components for trial design, deployment, and
management and collection of clinical trials data
• Backed by a framework for data provenance, privacy
protection and secure user authentication
• Clinical trial data is collected by
– Web browsers for electronic Case Report Forms (eCRFs)
– Web or mobile devices for Patient Reported Outcome
Measures (PROMs)
• Processes are monitored by a system for orchestration
the data collection of multiple clinical sites
W. Kuchinke (2016)

17. Computer System Validation
Part: Legal Framework

18. Legal framework for TRANSFoRm
• Development of a privacy framework
confronted us with the need of an in-depth
discussion of the meaning of data privacy
• Privacy is still a controversial concept, with
different possibilities for interpretation
• Privacy analysis of the data flow in two
research use cases
– Epidemiological cohort study on Diabetes
– Randomised clinical trial about different GORD
treatment regimes

19. The idea of privacy zones
• A standardized graphic model to describe data
privacy frameworks in primary care research
• Privacy protection methods lack formal modelling of
inferences from linked data sets
• Application of privacy principles and privacy
requirements for research activities
• Zones combine context characteristics in a graphic
way
• Zones are areas controlled by similar policies and
applicable regulations

20. database
actor
privacy
filter
Zone
Subzone
database
Actors: treating physician in the
Care Zone, researcher in None
Care Zone
Research question, query
Data linker, one-way coding
Data linker, two-way coding
Data flow, data transfer
Zone and Subzone
Database in the
Care Zone
Database in the
None Care Zone
database
Database in the
Research Zone
Data protection filter
The building blocks

21. Care Zone Non-Care Zone
GP practice
Research Zone
ECRIN
repository
researcher
GCP Subzone
physician
Study DB
filter
filter
EHR
investigator
eCRF
filter
Clinical trial with eCRF/EHR

22. Care Zone
Subzone GP practice
Non-Care Zone
Subzone NIVEL
Research Zone
Research
B1
researcher
Subzone TTP
DB C
patient
TTP
physician
invitation
IC
GP DB
DB A
Research
B2
Research
B3
Research
B4
privacy
filter
privacy
filter
privacy
filter
privacy
filter
privacy
filter
privacy
filter
data transfer >
implied IC
Patient questionnaires and IC >
Example for database research
TTP=Trusted Third Party

23. GP practices
SAFE
HAVEN
TTP
filter
privacy
Primary
research DB
Query workbench
Researcher
Custodix
EHR
EHR
EHR
filter
privacy
Genetic
data DB
filter
privacy
filter
privacy
filter
privacy
Researcher
GPs
Care zone
Diabetes study use case
Research zone
Non-care zone

24. Computer System Validation
Part: Validation Process and Documentation

25. Computer System Validation (CSV)
• Computerized system validation (CSV) is a
requirement for all computer systems involved in
clinical trials for drug submission
• Documented process to produce evidence that a
computerized system does exactly what it is
designed to do in a consistent and reproducible
manner
• Begins with the system requirements definition
and continues until system retirement
• System requirements are required to test the
intended use of a computer system
W. Kuchinke (2016)

26. Validating electronic source data in
clinical trials
• FDA and EMA require validation of all clinical data
from each trial and provide guidance documents
for electronic systems capturing clinical data
• This includes validation for clinical data that is
either captured from the subject directly or from
the subject’s medical records
• The problem is the correct and appropriate system
validation of electronic source data
• The investigator has to conduct direct data
verification
W. Kuchinke (2016)

27. System
requirements
specification
System
requirements
specification
Functional
specification
Functional
specification
Design
specification
Design
specification
System
building /
programming
System
building /
programming
Performance
qualification
(PQ)
Performance
qualification
(PQ)
Operational
qualification
(OQ)
Operational
qualification
(OQ)
Installation
qualification (IQ)
Installation
qualification (IQ)
Validation
Verification
Verification
Detailed Design
Specification
Component testing
Unit / integration
testing
Code verification
System integration
Software code
User / GCP
requirements
User / GCP
requirements
Validated
system
Validated
system
Performed together by using
a mock trial with synthetic
data
The complete Computer System Validation process
V-model

28. The main componenets of CSV
• Validation Master Plan
– outlines the principles involved in the qualification of a facility,
defining the areas and systems to be validated, and provides a
written program for achieving and maintaining a qualified
facility
• User Requirements Specification
• Hardware Requirements Specification
• Design qualification
• Installation qualification
• Operational qualification
• Performance qualification

29. Specification of eCRFs integrated
with the EHR
• The eCRF specification of the TRANSFoRm project supports the collection of
semantically controlled data
• Enables the collection of EHR data coded in different coding schemes from
various EHR systems in different countries
• The eCRF model is part of the framework established by Clinical Research
Information Model (CRIM)
• Enables semantic interoperability between the study system and the EHR system
• Supporst automated pre-population of EHR data into study eCRFs
• Support of a variety of technologies: web interface, mobile application, as well as
data collection embedded in an EHR system
• The eCRF model covers the whole lifecycle of study data collection, including
support for eligible patient identification and recruitment, informed consent,
study id allocation, randomisation, adverse event monitoring, etc
• Aligned to CDISC standards: Case report forms are modelled by CDISC ODM, ODM
is extended to embed data extraction query requests
W. Kuchinke (2016)

30. Validation Documents
• Requirements and Design Specifications
– User Requirement Specification
– Functional Requirements
– Design Specification
• Test Plans/Test Protocols
– Test Plans/Test Protocols
– Installation Qualification
– Operational Qualification
– Performance Qualification
• Additional Validation Document
– Validation Plans
– Traceability Matrix
– Validation Summary Report
– Computer System Validation
– Part 11 Training
– Auditing and Assessments

31. Traceability Matrix
• The Traceability Matrix (TM) links all requirements
to the tests throughout the validation process
• It ensures that all requirements defined for a
system are tested in the test protocols
• It is developed in concurrence with the User
Requirements Specification or Functional
Requirements Specification
• Requirements are traced to the specific test step
in the testing protocol in which they are tested

32. Validation Plans (VP)
• Validation Plans define the scope and goals of
a validation project
• It is usually specific to a single validation
project or infrastructure to be validated

33. GORD Use Case
• GORD = Gastro-Oesophageal Reflux Disease
• A common health condition
– Acid from the patient’s stomach leaks into the gullet or
oesophagus
• Different complications
– Most of the patients deal with the problem of
Oesophageal ulcers
– Acid of the stomach leaking into the oesophagus may
cause damage to the oesophagitis or oesophagus linings
– This may result in ulcers and bleeding creating pain

34. Components of clinical trials
framework
• Patient eligibility checks and enrolment
• Pre-population of eCRFs with data from EHRs
• PROM data collection by patients
• Storing of a copy of study data in the EHR
• The Study System coordinates all study and data collection events
• Included is an extension of CDISC SDM / ODM standard
• Whenever an interaction is required between Study System and
EHR, a query is transmitted to the EHR via the data node connector
– The DNC acts as a web server that displays eCRF forms for the clinician to
fill with information not present in the EHR
– The form is submitted to both the study database and the EHR for storage
considering requirements for eSource data use in clinical trials

35. Clinical trial system architecture

36. Components of epidemiological
study framework
• Genotypic-phenotypic T2D diabetes study use case
• Tools ensure the secure, provenance-enabled design and execution of
eligibility queries and data extractions from heterogeneous data
sources
• Eligibility queries are formulated by researchers in the query
workbench
– Enter clinical terms presents the user with a list of corresponding concepts
from standard terminologies
• Queries are dispatched to the data sources via the middleware to the
local data node connector (DNC)
– DNC sits at the data source and translates the generic CDIM-based query into
a local representation using the semantic mediator component
– It presents the locally interpretable query either to the data source directly or
to a human agent for final approval, before returning the result
W. Kuchinke (2016)

37. Epidemiological study architecture

38. Computer System Validation
Part: Validation of eSource data

39. What is Source Data and GCP
• GCP guide defines source data as any information
in original records and certified copies of original
records of clinical findings, observations or other
activities in a clinical trial necessary for the
reconstruction and evaluation of the trial
• A source document is a document in which data
collected for a clinical trial is first recorded
• This data is later entered in the case report form
(CRF)
W. Kuchinke (2016)

40. eSource guidance documents

41. Requirements for eSource data in
clinical trials
• Any instrument used to capture source data
should ensure that the data are captured as
specified within the protocol
• Source data should be accurate, legible,
contemporaneous, original, attributable,
complete and consistent
• An audit trail should be maintained as part of
the source documents for the original creation
and subsequent modification of all source data
W. Kuchinke (2016)

42. eSource direct data entry in clinical
trials and GCP requirements
• Declaration of Helsinki
– it is the duty of physicians who are involved in medical research to protect
the privacy and confidentiality of personal information of research
subjects
• The responsibility for the protection of research subjects always
rest with the involved physicians
• Any eSource system should be fully compliant with the provisions
of applicable data protection legislation
• eSource data capture concept implies that source data is primarily
no longer captured in the document management system of the
investigator’s site
– This creates the need to develop and implement processes that ensure the
continuous control of the investigators over these data
W. Kuchinke (2016)

43. The validation test study
Using a realistic mock study for validation

44. GORD Valuation Study
• Visit 1
– The patient attends the primary care practice, either already taking PPI, or as
a new reflux case with a diagnosis of GORD or a symptom of GORD
– The general practitioner checks eligibility criteria and gives study information
– If the patient consents to take part in the study, the patient is randomised
• Clinical data collection
– CROMs are completed by the GP in the eCRF
– PROMs are completed by the patient via their smartphone or computer (via
web browser)
• Visit 2: After 8 weeks, the patient returns for a follow-up visit
– CROMs and ROMs are completed by the GP and patient respectively
• When the patient visits the practice between Visit 1 and Visit 2,
CROMs for event visits are completed by the GP
W. Kuchinke (2016)

45. Time line of GORD study
If all inclusion criteria are fulfilled, the patient is enrolled and answers the Web questionnaire (WebQ) at study start
(T=0), and at 8 weeks (end of follow-up). The eCRF is filled out at study start (T=0) and study finish (8 weeks), and
whenever there is an event (visit to PHC).

46. Time line of test GOVOS study
Four test patients are enrolled and answer the eCRF and Web questionnaire (WebQ) at study start (T=0) and event
driven at t=1. WebQ is initiated by a reminder (t=1).

47. Time line of test GOVOS study
Overview of data collection events and roles. Yellow arrows represent movement of data; green represents data
extraction and collection processes. Reminders are generated automatically

48. Data collection with EHR
Data
collection
with EHR
Data collection
with eCRF

49. The validation test study
Generation of necessary documents to
document successful validation

50. System Validation Documentation

51. System Validation Master Plan

52. System Documentation and User
Manual

53. Validation plan

54. System Validation Plan for the
GORD Test Study

55. User Requirements Specification for the GORD
study and mHealth system for the data collection

56. System Documentation

57. Installation Qualification (IQ)

58. Installation Qualification
Testing Scripts
GORD Mobile Applications

59. Operational Qualification (OQ)

60. Performance Qualification (PQ)

61. OQ documentation includes
• Acceptance of successful completion
• Installation of computer system: information about
the environment in which the software has been
installed
• Tester: the persons conducting the test scripts
• Procedures: descriptions on how to use test scripts
for the qualification
• Test Scripts: lists the functions formulated as
instructions; including a signature areas for the
testers
W. Kuchinke (2016)

62. Description of the Validation test
Study

63. Test data for input into eCRF
The test data are realistic
fake data. Shown are the
data of the first 6 patients

64. Validation Summary Report
The Summary Report
summarizes the results
obtained during the
execution of the Installation/
Operational (IOQ)
Qualification, for the
validation of the
TRANSFoRm Study
System
It documents that the tested
system performed in
accordance with its
intended use as indicated in
the Functional
Requirements Specification
and the Software Design
Specification.

65. Test Scripts
• The scripts are numbered and are executed in a defined
order
• The scripts contain a list of functions or attributes to be
tested
• Tester inputs the date and signature (Tester’s Signature)
• Tester may enter a comment into “Comments”
• In case the result of the test does not match the expected
result, the actual result is documented in “Comment”.
• Tester indicates “p” for passed or “f” for failed
• If the test failed, the tester writes a deviation description
and may proceeds with deviation resolution
W. Kuchinke (2016)

66. Change Control Document
Change Control describes the
process of managing how
changes are introduced into a
controlled System.
Change control demonstrates to
regulatory authorities that
validated systems remain under
control during and after the
system has been changed.
Organizations need to explicitly
define their processes for
evaluating changes to validated
systems.

67. Training plan

68. References
• Institute of Medicine (US) Roundtable on Value & Science-Driven Health
Care. Clinical Data as the Basic Staple of Health Learning: Creating and
Protecting a Public Good: Workshop Summary. Washington (DC): National
Academies Press (US); 2010. Available from:
https://www.ncbi.nlm.nih.gov/books/NBK54302/ doi: 10.17226/12212
• Ruth R. Faden Nancy E. Kass Steven N. Goodman Peter Pronovost Sean
Tunis Tom L. Beauchamp. An Ethics Framework for a Learning Health Care
System: A Departure from Traditional Research Ethics and Clinical Ethics.
Hastings Center Report Volume 43, Issue s1: S16-S27. First Published: 11
January 2013
• Kuchinke W, Ohmann C, Verheij RA, van Veen EB, Arvanitis TN, Taweel A,
Delaney BC. A standardised graphic method for describing data privacy
frameworks in primary care research using a flexible zone model. Int J Med
Inform. 2014 Dec;83(12):941-57. doi: 10.1016/j.ijmedinf.2014.08.009. Epub
2014 Sep 3. PMID: 25241154.

69. Contact
Wolfgang Kuchinke
Heinrich-Heine University Duesseldorf, Duesseldorf, Germany
wolfgang.kuchinke@uni-duesseldorf.de
More information:
www.learninghealthcareproject.org/publication/5/104/the-transform-project
TRANSFoRm
This project has received funding from the European Union’s Seventh
Framework Programme for research, technological development and
demonstration under grant agreement no 247787 [TRANSFoRm].