Compared Assessment of Foreign Assistance Websites

Compared Assessment of
Foreign Assistance Websites
Julie Biau Jbiau@brookings.edu
Christine Zhang CYZhang@brookings.edu
Global Economy & Development
Development Assistance and Governance Initiative

2
• Notes on terminology and user profiles
• Evaluation Criteria
• Walkthrough of different interfaces
» Interfaces that are sources of data:
– AidData
– OECD Creditor Reporting System
– IATI Registry
» Interfaces that are visualization platforms:
– US Foreign Assistance Dashboard
– DFID Development Tracker
– IATI apps (OpenAidNL, d-portal, AidView)
Outline

3
• The following terms are used in this analysis to evaluate the
selected websites:
» Clarity: content is easy to locate
» ‘Recently updated’: data is not more than 1-2 years old
» Accuracy : degree to which the data compare to official sources
» Quality : degree to which the data are well organized and reliable (few
missing values, no double counting, etc.)
» Comprehensive: data cover most donor and recipient countries (DAC
donors; 100-200 recipients; all regions)
» Accessible: portal allows free bulk export of data; portal contains
disaggregated data; data is published under an open license
A note on terminology

4
• The usability and relevance of a website varies by who is
using it, and what they are using it for. We distinguish by:
A note on user profiles
Technical
Expertise
Technical users: Someone with strong
data manipulation skills who needs to
download the underlying data in order to
perform detailed analysis (e.g.
Researchers, academics, students)
Everyday users: Someone looking for
summary figures of aid flows from a
given donor to a given country and/or
sector, with no need for further analysis
(e.g. Development practitioners,
government officials, general public)
Geographic Scope
‘Macro’ users: Someone needing
information on aid flows by donor,
country and sector, but not necessarily
interested in project-level information
(e.g. Capitol Hill staffer, USAID regional
office, recipient-country finance ministry)
‘Micro’ users: Someone interested in
project-level information with sub-sector
and sub-national detail, perhaps less
concerned with ‘big picture’ statistics
(e.g. USAID country office, local NGO,
recipient-country line ministry)

5
Criterion 1: Ease of using
interface
Applicable
to
Good Moderate Bad
All users Interface is clear
(making it easy to
find and understand
content), fast, and
has useful data
visualization / query
customization options
Interface lacks
clarity or has
uninformative or
unwieldy data
visualization and
query
customization
options
Interface is very
unclear
(making it very
hard to find
and/or
understand
content) and/or
very slow
Note: the usability of image heavy websites is likely to decline with a limited internet
connection. Simplified versions for slow connections would be useful.

6
Criterion 2: Relevance of ContentApplicable to Good Moderate Bad
Micro users (e.g. donor
agency field office,
recipient country finance
ministry, local NGO)
Content (i) has been recently updated
(1-2 years), (ii) is comprehensive in
terms of country coverage, and (iii) covers
all aspects relevant to the audience, .e.g.:
- Resource flows from all donor
governments to the recipient country
by sector
- Project-level information on activity
within a country by region
- Historical and forward-looking
information
Content covers most aspects
relevant to the audience but has not
been recently updated (3+ years)
and/or is not comprehensive in
terms of country coverage
Content covers few or no
aspects relevant to the
audience
Macro users (e.g. donor
government
congressperson, donor
agency regional office)
Content (i) has been recently updated
(1-2 years), (ii) is comprehensive in
terms of country coverage, and (iii) covers
all aspects relevant to the audience, .e.g.:
- Resource flows from donor
government by source
(ministry/agency)
- Resource flows from donor
government by world region, country
and sector
- Historical and forward-looking
information
Content covers most aspects
relevant to the audience but has not
been recently updated (3+ years)
and/or is not comprehensive in
terms of country coverage
Content covers few or no
aspects relevant to the
audience
Note: this analysis intends to apply equally to donor and recipient-country users;
however poor availability of information in other languages is a limiting factor.

7
Criterion 3: Availability of Data
Applicable to Good Moderate Bad
Technical users
(e.g. researcher,
academic,
economist)
Raw data is available for
download in a readily
usable format (Excel,
SPSS, STATA…) with
minimal user effort (low # of
clicks); file is clearly
organized ; there is
sufficient guidance on the
data (frequency of update,
source, meaning)
Raw data is available
for download in a
readily usable format
with considerable user
effort and/or file is not
clearly organized; there
is some guidance on
the data
Raw data is not
available for
download in a
readily usable
format; there is no
guidance on the
data
Non-technical
users
Data is clearly presented ;
specific statistics can be
accessed with minimal
user effort; there is
sufficient guidance on the
data (frequency of update,
source, meaning)
Data is presented in an
unclear or misleading
fashion; there is some
guidance on the data
Data is difficult to
access; there is no
guidance on the
data
Note: large data files may be difficult to download with a limited internet connection.
Simplified versions for slow connections would be useful.

INTERFACES THAT ARE
SOURCES OF DATA
AidData, CRS, IATI Registry:

9
AidData
Clear homepage
allows to navigate
between data
visualization and use
Also allows to obtain
‘quick stats’ by donor,
country and sector,
with historical
information; however
these are very
aggregated and do
not allow to answer a
specific donor-
recipient query
Research tab allows
access to core dataset,
research datasets from
various academic articles,
replication datasets and
geocoded data
Ease of
use: Good

10
Coverage: Good - 100+ recipients for 2010,
80+ donors (fewer in certain years)
Timeliness: Poor - latest update 2012 but
most countries only go to 2011; no forward-
looking information
Relevance of content: Moderate
AidData
Micro: project
level data with
geographic
information is
available, with
factsheets for
each project of
1,322,824.
However these
are hard to filter
or aggregate
across projects.
Macro: Aid visualization option
allows access to several
dashboards, one at the very
macro level (aggregate flows by
donor, recipient, sector, year)…
but these are hard to narrow
down to a specific donor-
recipient-sector query
Relevance of
macro
content:
Moderate
The data needed to answer specific queries for both
micro and macro users (# of donors active in a
given sector of a country, total aid flows from a
donor for a given sector by recipient, etc.) is
available but cannot be easily generated without
downloading the full dataset and manipulating it.
Relevance of
micro
content:
Moderate

11
AidData
Main dataset can be downloaded in csv or in
STATA, making this very useful for technical
users; full dataset is so large that it loads
incompletely in STATA or Excel, and a
statistical transfer software is required;
however the ‘thin’ version and
aggregates versions allow for easier use.
Guidance: Numerous non-
typical datasets are
available for download,
including on Chinese aid
and underreported
financial flows, with full
description of methodology
A variety of datasets in
Excel and STATA can
be downloaded from
recent academic papers
Data for
technical
users: Good

12
AidData
… And downloading the data
from a specific query is
onerous (requires entering
personal information and waiting
for a link to be sent via email)
Detailed search options
exist by whether one is
looking for aggregate or
detailed data, allowing to
select recipient, donor,
sector, and year – but
some of the options are
presented in a confusing
manner and it is unclear
whether certain filters
are mutually exclusive,
or whether they are
working at all ..Search results
are not
presented in a
very useable or
informative
manner (graphs
but no table)
Data for
everyday
users: Bad

13
OECD Creditor Reporting
System
The website not very
user friendly to the
extent that there is no
introduction page and
users are launched
directly into the
dataset, which can be
intimidating with no
overall explanation of
what it is or can do
Clicking on the
information icon next to
a dataset name
generates a useful
explanation of its
contents, however
There is no
explanation of
how the CRS
differs from the
other OECD DAC
datasets also on
this page
Ease of
use:
Moderate
Filter function works well with
many options to customize the
query but it may be unclear to
users that a filter is currently
being applied; queries are
slow to load and website
sometimes freezes

14
System
Macro: Queries are easily customized with a
detailed menu of drop-down options by sector,
recipient, flow, channel, amount type, flow
type, and type of aid. This function allows to
select only some donors and recipients to
arrive at a very specific query (e.g. US
education assistance to Kenya in 2012)
without downloading the full dataset, and also
enables aggregation (e.g. how much are all
donors providing to Kenya in education). Note
that this kind of query is answered in fewer
steps in CRS than on other websites.
Micro: Customization options allow for
more in-depth selection of flows by sector,
recipient and region and enables to
compare the funding that several donors
provide to a given sector in a given
country. However, the information is not
at the project-level or at the sub-
national level so would be of limited
interest to local users. Access to
project-level data requires
downloading the full dataset and
cannot be done through web queries.
Relevance
of macro
content:
GoodRelevance of
micro
content:
Moderate
Coverage: Good – 60+ donors and 180+ recipients
Timeliness: Moderate– includes some information for 2013; more data for 2012
Relevance of content: Good (macro); Moderate (micro)

15
System
Customization and table layout options allow to
display data exactly as it will be most useful for
statistical analysis; but loading is very slow
and selection time consuming / confusing
A bulk download of the full dataset,
including historically, is available to Excel
(however this bulk download option is
hard to locate on the website and the
file is so large that it loads incompletely
in Excel, requiring a statistical transfer
software to download it correctly)
Data for
technical
users: Good
Drop-down filters allow an
at-a-glance check of what
certain queries look like
before the data is
customized and
downloaded.

16
System
Customization options and categories
can be confusing for lay users
Charts and graphs cannot be generated
for data queries that are too large
Data for
everyday
users: Bad
Unless one looks for a specific query,
the website does not provide
aggregate statistics and there are no
data visualizations
Need for more guidance: non-technical
users commonly find CRS data in
summary reports published by the OECD,
rather than by browsing the database.
Website should be clear that it targets
technical users, and provide links to these
summary reports for others.

17
QWIDS (Query Wizard for
International Development
Statistics)
More user friendly display of CRS
data and filters, faster to navigate
and less confusing; ‘popular
queries’ button allows access to
useful aggregated tables without
needing to download the data.
Hosts data from all the main OECD aid
datasets (DAC aggregate tables and CRS)
Data is displayed for the specified query
and can be easily downloaded into Excel;
however this interface remains
complicated and difficult to use (hard
to get numbers to add up).
Improves
ease of use,
but needs
further work

18
IATI Registry
The IATI Registry
consists mainly of a list
of datasets provided by
different aid practitioners
that can be sorted
through with several
filters (source, publisher,
organization type,
recipient country, flow
type). However there is
no way on the Registry
website to aggregate
these datasets or even to
filter them by sector or
type, and no overall
statistics are provided
for lay users.
Some interfaces have been developed by third parties to facilitate use
of the Registry data (particularly d-portal, OpenaidNL and Aidview),
which and we will look at these in turn.
“The data is published in the international IATI
standard which is easy for computers to read,
but very hard for humans”(OpenAid NL).
However, the IATI Registry was not intended for use
as a research tool, but rather as a data repository,
with the expectation that tools for its navigation
would be developed separately.

INTERFACES THAT ARE
VISUALIZATION PLATFORMS
IATI applications, US Foreign Assistance Dashboard, DFID
Development Tracker:

20
OpenAidNL
Each project has a fact sheet with project
documents, transactions, and other basic
information on budget and implementation.
However, the sector links and ‘export’
button do not work.
Presents the Registry’s project-level information in a
format that would be very useful to local actors seeking to
know which sectors are being funded in which parts of the
country by the Netherlands Projects can be filtered by
country. However these cannot be readily aggregated
in a single dataset and have limited search options.
Filters at the top of the homepage
allows to filter by country, region,
sector, and budget, with a detailed
drop-down menu for each
dimension. Filtering produces a list
of projects in the selected fields.
Ease of
use: Good

21
OpenAidNL
Each project has a fact sheet
with project documents,
transactions, and other basic
information on budget and
implementation, which is quite
useful for project-level queries.
However, the sector links and
‘export’ button do not work,
and projects cannot be
aggregated.
Relevance of
micro
content:
Moderate
Presents the Registry’s project-level information in a
format that would be very useful to local actors seeking to
know which sectors are being funded in which parts of the
country by the Netherlands Projects can be filtered by
country. However these cannot be readily aggregated
in a single dataset or into country totals, making the
information of limited use for macro queries.
Coverage: Poor – covers >2000 activities but just for Netherlands
Timeliness: Good – updated every 3 months
Relevance of content: Moderate/Bad
Relevance
of macro
content:
Bad
Data cannot be aggregated
beyond the project level or
downloaded into excel (links are
broken)
No overall statistics for everyday
users to refer to
Data for
everyday
users: Bad
Data for
technical
users: Bad

22
d-portal.org
“click & scroll”
Ease of
use: Good
HOMEPAGE
RECIPIENT PROFILE PAGE:
CAMBODIA
d-portal presents IATI data by recipient
country
“click&scroll” model makes it easy to click
and scroll through project-level info

23
d-portal.org
Cambodia example
VERY MACRO LEVEL
WHERE DOES THE MONEY COME FROM?
WHERE DOES THE MONEY GO?
ACTIVE PROJECTS+
RECENTLY ENDED PROJECTS
Macro: able to attain high-level & detail
information on projects in Cambodia.
Can filter by exact location (if provided),
donor, and sector. Cannot filter by
project size or year. 2012 = most recent
year (not forward-looking)
Both:
d-portal identifies US funding to Cambodia
as 88mn according to CRS but 0 according
to IATI. This is because of insufficient
reporting by donor into IATI.
Micro: clicking on an
individual donor is confusing
because it lists ALL projects
(ranked by size), regardless
of year
INTERMEDIATE LEVEL
MICRO LEVEL
Coverage: moderate – only allows searches by recipient country
Timeliness: moderate – data until 2012
Relevance of content: moderate/bad – confusing due to data gaps Relevance
of micro
content:
Bad
Relevance of
macro
content:
Moderate

24
d-portal.org
Cambodia example
Data for
technical
users: Bad
Data for
everyday
users:
ModerateBoth: One important note is that the info is
organized by recipient country. So there is
no way to see all recipients on one page.
Micro: data is only “downloadable” by
clicking on a given project, then “view
source,” which generates an (unhelpful
for manipulation) xml file that requires a
CSV conversion tool to transfer into Excel.
This tool should be listed on the site.
Macro: layout is user-friendly but not
always clearly organized (e.g., donors
are by agency rather than country)

25
AidView
Can filter projects by recipient country, sector or
organization/donor. Filter by sector leads to
projects grouped by sector and then by
subsector. Buttons allow to sort projects
alphabetically or by budget size
Two things are misleading here: once at the
subsector level, each bubble is an individual
activity, not a collection of projects in that
subsector; and each screen only lists a handful
of projects, rather than a comprehensive
mapping for that sector, so that the user has to
navigate through many screens (in the case of
education, 126) to see the full list of projects.
The use of bubbles and apparent grouping is
therefore actually not a map, and conveys no
more informative than using a simple listing of
the projects by size (not an aggregation).
Ease of
use: Bad
Results can also be viewed on map or as list but navigation
between layouts and filters is very slow, sometimes
resulting in an error message (with no way back!)

26
AidView
A useful feature for micro and macro users is that
the searches can be overlaid: here the screen is
showing all projects financed by the World Bank
to Bangladesh in education. However this
requires applying each filter successively on
different webpages, which is more time
consuming than selecting the desired
combination from a drop down menu.
Relevance
of micro
content:
Good
Relevance of
macro
content:
Moderate
Coverage: Good: allows to search full IATI database covering 3000
activities, over 160 donors, and a large range of recipient countries.
Timeliness: Good – includes forward looking information (projects still in
planning stage); but some visualizations may need updating
Relevance of content: Good/Moderate
Project factsheets are very
detailed, with project
description, transactions,
budget, implementing
agency, partner agencies,
contact details, and link to
main project documents.
When more than one
dimension
(country/sector/organization)
is selected, the ‘datafile’
option becomes active and
presents information within
those filters (e.g. all basic
education projects in
Argentina) with some level
of aggregation. This is
especially useful for micro
users.

27
AidView
IATI information at the project level can be
downloaded into Excel from the project factsheet.
However the downloading option is
not available at any level other than
the project level (data from factsheets
at a country-sector level, for example,
cannot be downloaded into excel).
Hence this is not a large improvement
over downloading project information
straight from the IATI registry, where it is
also available on a project by project
basis.
Data for
technical
users: Bad
Data for
everyday
users:
Moderate
When more than
one dimension
(country/sector/orga
nization) is selected,
the ‘datafiles’
present some
summary statistics
that can be useful to
everyday users.
The bubble view gives overall
volumes per sector, country or donor.

28
US Foreign Assistance
Dashboard
The website is user
friendly and well laid out,
allowing users through
its ribbon and tabs to
understand quite easily
what kind of information
it can provide in its
different sections
(foreign assistance by
agency, by country
office, by initiative; data
at aggregated or
transaction level; notes
on methodology…).
Information is available
on maps and charts as
well as tables.
Ease of
use:
Moderate
However, the multiplicity of places
to look for information – with
different cuts of the data presented
on different pages rather than all in
one place – can make it hard to
understand exactly where to go to
answer a specific query. A lot of
navigation required.

29
Dashboard
The dashboard allows to
explore data, including
aggregate planned spending,
by country office, agency and
fiscal year.
Coverage: Poor – only one donor
Timeliness: Good – includes information for 2013 (spent) and 2014/5 (planned)
Relevance of content: Good
The country office and sector pages also give
project/transaction-level information for that country,
by sector, year and agency (downloadable in Excel)
Clicking on a transaction accesses a detailed
project page. However, data could also be linked
to other relevant sources, e.g. MCC transaction-
level data, geocoded data from USAID or
information from FBO on tenders.
This includes data
on planned spending
(2014/5), which is
not usually available
on aid websites
Relevance of
micro
content:
Moderate
Relevance of
macro
content:
Moderate

30
Dashboard
Note: data is not yet available for every
agency but this is work in progress. The
fact that State Department spending
is not yet on the website means that
the information is a very incomplete
picture of US aid flows.
Full dataset is easily available for download in
Excel at transaction-level and at aggregate level.
Data is available until 2015 with information on
agency, operating unit, sector, and amount.
Data for
technical
users: Good

31
Dashboard
Some useful aggregations are available under
the ‘where is the money going?’ tab, where
spending figures and charts can be obtained
by office, by recipient country or region, by
sector, and by agency.
Data for
everyday
users: Good
Sector and country specific breakdowns with their own
summary statistics allow to answer specific queries (e.g. how
much education assistance is the US providing to Kenya)
without downloading the full dataset).
Guidance: website provides some explanation on terminology
used; but further clarity on US-specific budget terminology
could make this clearer to external audience.

32
DFID Development Tracker
Clear homepage with
a search bar allowing
to search projects by
keyword
Top statistics and
achievements are
useful for those
looking for an
overview of UK aid
Ease of
use: Good
Quick access to
searching for projects
by location or by
sector, with some data
visualization

33
DFID Development TrackerMicro: For more detail, a country’s project list can
be filtered by status, sector, agency, budget, and
date, and directs the user to project factsheets.
Project factsheets also contain separate tabs with
information on transactions and project documents.
Relevance
of micro
content:
Good
Macro: Presents summary information
on the country before going to the
project-by-project detail: total budget
and total number of projects; also
allows to search for regional projects
Relevance
of macro
content:
Good
Timeliness: Good – information on planned projects up to start date of 2018
Each country has a fact sheet
with summary of ongoing
projects by budget and sector

34
Macro: Presents summary information
on the each sector (share of total
budget) before going to the sector-by-
sector detail: also allows to sort sectors
by name and budget %
Each sector is organized into detailed sub sectors
Relevance
of micro
content:
Good
Relevance
of macro
content:
Good
Micro: Each sub-sector has a project list that can be filtered by
status, sector, agency, budget, and date, and directs the user to
individual project factsheet. Project factsheets also contain separate
tabs with information on transactions and project documents.
Importantly, there is also information on implementing
organizations (i.e. subcontractors). This level of detail is unique
amongst the websites reviewed, and important for traceability.
Timeliness: Good – information on planned projects up to start date of 2018

35
However, this link leads to an
xml file that requires a CSV
conversion tool to transfer into
Excel. The website should make
clear to users how to use this
tool, or should provide data
directly in Excel (insufficient
guidance).
Data for
technical
users: Bad
Each project and
country factsheet has a
button allowing
download of IATI data
for that project
The only option to download the full dataset is to access the Annual Report
Excel files, which are poorly advertised on the main website. Main website
has no functionality to sort and filter data other than at the project level. For
example an everyday user looking for the budget going to a specific country and
sector would need to manually add the budgets of all individual projects in that
category.
Data for
everyday
users:
Moderate

37
EASE OF
USE
RELEVANCE OF
CONTENT –
MICRO
RELEVANCE OF
CONTENT –
MACRO
DATA FOR
TECHNICAL
USERS
DATA FOR
EVERYDAY
USERS
AidData
System
Dashboard
OpenAidNL
d-portal.org
AidView
Bad
Moderate
Good
Summary Table

38
Summarizing…
• Visualization ≠ Value
» Data visualizations must be useful in addition to aesthetically pleasing; excessively image-
heavy sites can be difficult to navigate with limited internet connection
• Aggregation is important
» It is often difficult to find totals (by donor/recipient country or by sector), unless one
downloads the entire dataset (if it is available on the site)
• Who is the audience?
» Everyday Users versus Tech-Savvy users
» Sometimes it can’t (or shouldn’t) be both… but this should be made clear at the outset
• Looking forward…
» Planned in addition to disbursed aid – so recipients can plan ahead (greater predictability)
» Outputs (project results) in addition to inputs (project budgets)
» Data can only be as robust as a donor presents it (must increase reporting)

Compared Assessment of Foreign Assistance Websites

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (15)

Similar a Compared Assessment of Foreign Assistance Websites

Similar a Compared Assessment of Foreign Assistance Websites (20)

Último

Último (20)

Compared Assessment of Foreign Assistance Websites