This is my presentation at HICSS 2014, on evaluation models for Open Data sites and portals, based on the work we are doing at the ENGAGE project

1. Charalabidis,Y., Loukis, E., Alexopoulos, H.
University of the Aegean, Greece
University of the Aegean – Department of Information and Communication Systems Engineering

2. INTRODUCTION: THE OPEN /BIG DATA
MOVEMENT IN THE BACKGROUND
Governments are increasingly opening to the society
important data they possess, in order to be used for
scientific, commercial and political purposes.
Initially a first generation of Internet-based open
government data (OGD) infrastructures has been
developed in many countries, influenced by the Web
1.0 paradigm, in which there is a clear distinction
between content producers and content users.
2

3. A SECOND GENERATION OF OGD
INFRASTRUCTURES
Recently a second generation of more advanced OGD
infrastructures is under development, which is
influenced by the principles of the new Web 2.0
paradigm:
elimination of the clear distinction between ‘passive’
content users/consumers and ‘active‘ content producers
They aim to support highly active users,
who assess the quality of the data they consume and
mention weanesses of them and new needs they have
and often become data pro-sumers‘ = both consumers
and providers of data
3

4. THE NEED FOR AN EVALUATION METHOD
The big investments in this area necessitate a systematic
evaluation of these OGD infrastructures, in order to gain a
better understanding and assessment of the multidimensional value they generate
However, a structured and comprehensive evaluation
methodology is missing.
This method contributes to filling this gap.
It presents and validates a methodology for evaluating these
advanced second generation of ODG infrastructures,
based on a ‘value model approach’,
i.e. on the estimation of value models of these infrastructures
from users’ ratings.
4

5. INTRODUCTION
In particular: it assesses various measures of generated
value by OGD infrastructures,
structured in three layers (associated with efficiency,
effectiveness and users’ future behavior),
and also the relations among them,
leading finally to the formation of a value model of the
OGD infrastructure, which enables:
a deeper understanding of the whole value generation
mechanism of it
and also a rational definition of IS improvement
priorities
5

6. BACKGROUND / SYNTHESIS
Literature Review
IS Evaluation
TAM
IS Success Models
E-Services
Scoping eInfrastructures
Stakeholders
Data Acquisition
Data Provision
Communication
6

7. Research Streams Insights
IS Evaluation
IS’s offer various types of benefits, both financial and
non-financial, and also tangible and intangible ones,
which differ among the different types of IS
it is not possible to formulate one generic IS evaluation
method, which is applicable to all IS
a comprehensive methodology for evaluating a
particular type of IS should include evaluation of both
its efficiency and its effectiveness, taking into
account its particular characteristics, capabilities and
objectives
7

8. Research Streams Insights
TAM (Technology Acceptance Model)
identify the characteristics and factors affecting the attitude
towards using an IS, the intention to use it and finally the
extent of its actual usage
perceived usefulness and perceived ease of use determine an
individual's intention to use a system with intention to use
serving as a mediator of actual system use
IS Success Models
IS evaluation should adopt a layered approach based on the
above interrelated IS success measures (information quality,
system quality, service quality, user satisfaction, actual use,
perceived usefulness, individual impact and organizational
impact) and on the relations among them
8

9. Research Streams Insights
e-Services Evaluation
frameworks that assess the quality of the capabilities
that the e-service provides to its users
frameworks that assess the support it provides to users
for performing various tasks and achieving various
objectives, or users’ overall satisfaction
the above frameworks do not include advanced ways of
processing the evaluation data collected from the
users, in order to maximize the extraction of valuerelated knowledge from them
9

10. Our Evaluation Model
Approach
(a) Efficiency layer: it includes ‘efficiency’ measures,
which assess the quality of the basic capabilities
offered by the e-service to its users.
(b) Effectiveness layer: it includes ‘effectiveness’
measures, which assess to what extent the e-service
assists the users for completing their tasks and
achieving their objectives.
(c) Future behaviour layer: it includes measures
assessing to what extent the e-service influences the
future behaviour of its users (e.g. to what extent they
intend to use the e-service again in the future, or
recommend it to friends and colleagues).
10

11. Value Model Definition
Data Provision Capabilities
Data Search & Download Capabilities
User-level Feedback Capabilities
Support for
Achieving User
Objectives
Ease of Use
Future
Behaviour
Performance
Data Processing Capabilities
Data Upload Capabilities
Support for
Achieving
Provider Objecti.
Provid-level Feedback Capabilities
Efficiency Level
Effectiveness
Level
Fut. Behavior
Level
11

12. Value Measures
The total of 41 value measures (all layers) were
defined where 35 for the 1st layer
14 common value measures
15 value measures for users
06 value measures for providers
These value measures was then converted to a
question to be included in questionnaires to be
distributed to stakeholders
A five point Likert scale is used to measure
agreement or disagreement
2 Questionnaires have been formulated
12

13. Indicative Value Dimension – 1st Level
Ease of Use
1.1
Friendliness
The platform provides a user friendly and easy to use
environment.
1.2
Learning Easiness
It was easy to learn how to use the platform.
1.3
Aesthetics
The web pages look attractive.
1.4
Ease of performing
tasks
It is easy to perform the tasks I want in a small number
of steps.
1.5
Multilingual aspects
The platform allows me to work in my own language.
1.6
Personalization
The platform supports user account creation in order
to personalize views and information shown.
1.7
Support & Training
The platform provides high quality of documentation
and online help.
13

14. Indicative Value Dimension – 1st Level
Data Processing Capabilities
7.1 Data Enrichment
The platform provides good capabilities for data
enrichment (i.e. adding new elements - fields)
7.2 Data Cleansing
The platform provides good capabilities for data
cleansing (i.e. detecting and correcting ubiquities
in a dataset)
7.3 Linking
The platform provides good capabilities for linking
datasets.
7.4 Visualisation
The platform provides good capabilities for
visualization of datasets
14

15. Indicative Value Dimension – 2nd Level
Support for Achieving User Objectives
8.1 ACC1
I think that using this platform enables me to do better
research/inquiry and accomplish it more quickly
8.2 ACC2
This platform allows me to draw interesting conclusions on
past government activity
8.3 ACC3
This platform enables me to create successful added-value
electronic services
8.4 ACC4
I am in general highly satisfied with this platform
15

16. Application : The ENGAGE project
OGD system to evaluated: ENGAGE - A new multicountry, multi-lingual open data infrastructure for
researchers, available at www.engagedata.eu
Target user group: post-graduate students from TU
Delft and Uaegean, trained in the platfom
Method of user input: electronic questionnaires
Number of valid questionnaire responses processed: 42
(when the paper was submitted, now more than 100)
16

17. The ENGAGE System
Social
sciences
ICT
Natural
Sciences and
Engineering
Governance
Policy
Modelling
Law
Providing PSI to research
communities and citizens in
a personalised manner
Single point of
Access
User groups
Tailored data
services
Data Service
Provision
Infrastructure
Citizens
Research and Industry
Governance and
policy making
Search and
Navigation tools
Knowledge /
Data Mining
Collaboration /
Communities
Visualisation
- Analytics
Data
analytics
Citizens and
education
Personalisation
Directory services
and direct linking to
data archives
Curating, Annotating,
Harmonising , Visualising
Data Quality
Data Curation
Infrastructure
Gathering data from
governmental
organisations and systems
(the Gov Cloud)
Data Linking
Knowledge Mapping
Semantic Annotation
Automatic curation
algorithms
Anonymisation
Public Sector Information Sources
Public Organisations, Repositories, Databases
Harmonisation

18. Value Model Estimation Algorithm
Value Dimensions
Internal
Consistency
Examination
Value
Dimensions
Variables
Calculation
Average Ratings
Calculation
Value Models’
Construction
Correlations
Estimation
Regression
Models
Estimation
Improvement
Priorities
Identification
18

19. Data Provision
Capabilities
3.03
Data Search & Download
Capabilities
3.03
User-level Feedback
Capabilities
2.97
Ease of Use
3.35
Estimated Value Model
0.639
0.760
Support for Achieving
User Object.
3.17
0.651
0.624
0.730
Future Behaviour
3.19
0.379
0.735
Performance
2.15
Data Processing
Capabilities
3.27
Data Upload Capabilities
2.93
0.489
0.479
0.135
0.632
Support for Achieving
Provider Obj.
3.12
0.680
0.307
Provider-level Feedback
Capabilities
3.44
19

20. R2 coefficients of second and third layer
value dimensions’ regression models
Regression Models
SUO model (8 indep. variables)
0.776
SPO model (8 indep. variables)
0.599
FBE model (2 indep. variables)
0.412
FBE model (10 indep. variables)
6-9/01/2014
R2
0.647
HICSS 47 - University of the Aegean
20

21. Improvement Priorities
Identification
Such an OGD infrastructure value model,
Enables the identification of improvement
priorities,
which are the first layer OGD systems
capabilities that receive low evaluation by the
users,
and at the same time have high impact on
higher layers’ value generation

22. Mapping for decision support
Lower Ratings
Group
data provision
capabilities
Higher Ratings
Group
provider-level
feedback cap.
Lower Impact
Group
data provision
capabilities
Higher Impact
Group
data processing
capabilities
data searchdownload cap.
ease of use
user-level feedback
capab.
ease of use
data upload
capabilities
performance
6-9/01/2014
data processing
capabilities
performance
data searchdownload cap.
user-level
feedback capabil.
provider-level
feedback cap.
data upload
capabilities
HICSS 47 - University of the Aegean
22

23. Conclusions 1/2
This paper has presented a methodology for determining the value
generation mechanism and the improvement priorities of advanced
2nd generation open government data systems,
which are characterized by the elimination of the distinction
between providers and consumers of such data.
The proposed methodology assesses a wide range of types of value
generated by such OGD infrastructures for data ‘pro-sumers’,
and at the same time exploits the relations between the above
types of value (which are usually not exploited and ignored by IS
evaluation methodologies in general),
leading to additional useful value-related information and more
insights into these advanced ODG systems,
providing valuable support for making important ODG systems
investment, management and improvement decisions.
23

24. Conclusions 2/2
An algorithm for advanced processing of users’ evaluation
data has been proposed,
which leads to the estimation of the value model of the
OGD infrastructure,
enabling a better understanding of the whole value
generation mechanism of its,
and the identification of improvement priorities,
which are the first layer OGD systems capabilities that
receive low evaluation by the users, and at the same time
have high impact on higher layers’ value generated.
A first application-validation of the proposed methodology
provided interesting conclusions for the OGD systems
developed in ENGAGE infrastructure
24