The Impact of Different Validation Datasets on Air Quality Modelling Performance, Transportation Research Board, Washington DC, 7–11 January 2018.

1. 1
Center for Advancing Research
in Transportation Emissions,
Energy and Health
U.S. Department of Transportation University
Transportation Centers Program
HaneenKhreis, PhD
www.carteeh.org

2. 2
“The Impact of
Different
Validation
Datasets on Air
Quality Modeling
Performance” –
paper # 18-
01950

3. 3
•Not possible to make sufficient air pollution exposure
measurements for epidemiological and health impact
assessments  143,472 children
•Many studies rely on air pollution modeling
•Commonly used models include:
• Land use regression (LUR) modeling
• Atmospheric dispersion (AD) modeling
Background

4. 4
LUR modeling
• Repeated measurements
(passive samplers) at N
sites
• Average measurements
over longer term (usually a
year) and adjust for
temporal variations
• Regression model to
combine measurements
with GIS-based predictor
data within certain buffers
• Apply model to non-
measured locations

5. 5
LUR modeling
• BUILDINGS  Local land use, Area/number of buildings, m2/N(umber)
• TRAFLOAD  Local road network, Total traffic load of all roads in a buffer (sum of
(traffic intensity*length of all segments)), Veh. Day-1 m
• NATURAL  Semi-natural and forested areas, m2
• HEAVYTRAFMAJOR  Heavy-duty traffic intensity on nearest major road, Veh. Day-1

6. 6
AD modeling

7. 7
•These models are only validated using one validation
dataset
• Their estimates at select receptor points are
sometimes generalized to larger areas
•This may lead to unsatisfactory validation and/or
inaccurate insights about the models’ performance
and suitability for large-scale application
Background

8. 8
•Objective 1  explore the effect of different
validation datasets on the validation results of two
commonly used air quality models
•Objective 2  explore the effect of the model
estimates’ spatial resolution on the models’ validity at
different locations
Objectives

9. 9
Study area

10. 10
Annual (2009) NO2 and NOx
LUR model
Validation against 4
different datasets
Spatial resolution
analysis
Estimate at exact location
of validation point
Estimates at centroid of
100x100m grid in which
validation point fell
AD model
Validation against 4
different datasets
Spatial resolution
analysis
Estimates at exact
locations of validation
point
Estimates at centroid of
100x100m grid in which
validation point fell
Methods

11. 11
Annual (2009) NO2 and NOx
LUR model
Validation against 4
different datasets
Spatial resolution
analysis
Estimate at exact location
of validation point
Estimates at centroid of
100x100m grid in which
validation point fell
AD model
Validation against 4
different datasets
Spatial resolution
analysis
Estimates at exact
locations of validation
point
Estimates at centroid of
100x100m grid in which
validation point fell
Methods

12. 12
Measurement
campaign and
dataset (n = 126)
Pollutants
measured Measurement device
Year and time
interval for final
dataset
Locations and purpose of
measurements
ESCAPE diffusion
tubes (n=41) NO2 and NOx Ogawa badges 2009
(annualized)
At the façade of homes of
study subjects as the primary
objective of the ESCAPE
project was to characterize
residential exposures and
associated health
CBMDC diffusion
tubes (n=29) NO2 “Diffusion tubes” 2009
(annualized)
Three sites were not close to
main road whilst the rest
were kerbside sites at 0.5-5m
from the nearest road,
monitoring undertaken to
review and assess air quality
progress
de Hoogh diffusion
tubes (n=48) NO2 Palmes tubes
Four 2-week
periods during
2007-2008
Close to the front door of 48
homes of study subjects from
the Born in Bradford cohort
to characterize their
residential exposures and
compare with future ESCAPE
work
CBMDC fixed-site
monitoring (n=8) NO2
Automatic urban network
chemiluminescence
2009
(annualized)
Two sites were classified as
urban background whilst the
rest were kerbside sites at
1.5-2 m from the nearest
road, monitoring undertaken
to review and assess air
quality progress
Methods

13. 13
Results: validation against different datasets
Models combination Validation dataset
ESCAPE
NOx
diffusion
tubes
(n=41)
ESCAPE
NO2
diffusion
tubes
(n=41)
CBMDC
NO2
diffusion
tubes
(n=29)
De Hoogh
NO2
diffusion
tubes
(n=48)
CBMDC NO2
fixed-site
monitoring
(n=8)
ADmodel
COPERT dispersion
model NOx at points
(varying background)
R2
= 0.30
COPERT dispersion
model NO2 at points
(varying background)
R2
= 0.33 R2
= 0.20 R2
= 0.59 R2
= 0.24
LURmodel
NOx LUR estimates
at points
R2
= 0.58
NO2 LUR estimates
at points
R2
= 0.54 R2
= 0.21 R2
= 0.61
R2
= 0.38
(r= 0.62)

14. 14
AD vs. LUR annual average
NO2/NOx Estimates (µg/m3) at
 46,452 specified output
points centering each 100m x
100m grid across 40 * 33 km
Results: spatial
resolution of
estimates

15. 15
AD vs. LUR Modeling
Model and statistic
COPERT-based dispersion model
(traffic)
LUR model
Minimum 10.03 (0.40) 0.00064
1st quartile 15.93 22.00
Median 17.80 (0.78) 24.93
Mean 19.47 (2.45) 24.91
3rd quartile 20.34 32.10
Maximum 110.15 (93.14) 95.18

16. 16
Results: spatial resolution of estimates
Validation dataset
ESCAPE NOx
diffusion tubes
(n=41)
ESCAPE NO2
diffusion tubes
(n=41)
CBMDC NO2
diffusion tubes
(n=29)
de Hoogh NO2
diffusion tubes
(n=48)
CBMDC
NO2
fixed-site
monitorin
g (n=8)
LURmodels
NOx LUR
estimates at
points
R2= 0.58
NOx LUR
estimates at
raster
R2= 0.35
NO2 LUR
estimates at
points
R2= 0.54 R2= 0.21 R2= 0.61
R2= 0.38
(r= 0.62)
NO2 LUR
estimates at
raster
R2= 0.31 R2= 0.06 R2= 0.32 R2= 0.38
(r=- 0.61)
-23%

17. 17
Results: spatial resolution of estimates
Validation dataset
ESCAPE NOx
diffusion tubes
(n=41)
ESCAPE NO2
diffusion tubes
(n=41)
CBMDC NO2
diffusion tubes
(n=29)
de Hoogh NO2
diffusion tubes
(n=48)
CBMDC
NO2
fixed-site
monitorin
g (n=8)
LURmodels
NOx LUR
estimates at
points
R2= 0.58
NOx LUR
estimates at
raster
R2= 0.35
NO2 LUR
estimates at
points
R2= 0.54 R2= 0.21 R2= 0.61
R2= 0.38
(r= 0.62)
NO2 LUR
estimates at
raster
R2= 0.31 R2= 0.06 R2= 0.32 R2= 0.38
(r=- 0.61)
-23% -15% -29%

18. 18
• LUR and AD model estimates were validated against four different
validation datasets
Summary and discussion

19. 19
• LUR and AD model estimates were validated against four different
validation datasets
• LUR and AD model estimates were made at different spatial
resolution and validated against four different validation datasets
Summary and discussion

20. 20
• LUR and AD model estimates were validated against four different
validation datasets
• LUR and AD model estimates were made at different spatial
resolution and validated against four different validation datasets
• The validation metrics varied substantially (R2 0.20 – 0.61) based on
• which model was used
• which validation dataset was used
• whether exposure estimates were made at exact validation point or at
centroid of containing grid
Summary and discussion

21. 21
• LUR and AD model estimates were validated against four different
validation datasets
• LUR and AD model estimates were made at different spatial
resolution and validated against four different validation datasets
• The validation metrics varied substantially (R2 0.20 – 0.61) based on
• which model was used
• which validation dataset was used
• whether exposure estimates were made at exact validation point or at
centroid of containing grid
• The validation results based on the actual points’ locations were
generally much better than at a grid level
Summary and discussion

22. 22
• There is a value of validating modeled air quality data against various
datasets
Conclusions

23. 23
• There is a value of validating modeled air quality data against various
datasets
• The spatial resolution of the models’ estimates has a significant
influence on the validity at the application point (even at 100m level)
Conclusions

24. 24
• There is a value of validating modeled air quality data against various
datasets
• The spatial resolution of the models’ estimates has a significant
influence on the validity at the application point (even at 100m level)
• Have implications for epidemiological studies disregarding time-activity
patterns or using location proxies
Conclusions

25. 25
• There is a value of validating modeled air quality data against various
datasets
• The spatial resolution of the models’ estimates has a significant
influence on the validity at the application point (even at 100m level)
• Have implications for epidemiological studies disregarding time-activity
patterns or using location proxies
• Have implications for health impact assessment studies where estimates of
air quality models at select receptor points are extrapolated and assumed to
apply to larger areas and populations
Conclusions

26. 26
• There is a value of validating modeled air quality data against various
datasets
• The spatial resolution of the models’ estimates has a significant
influence on the validity at the application point (even at 100m level)
• Have implications for epidemiological studies disregarding time-activity
patterns or using location proxies
• Have implications for health impact assessment studies where estimates of
air quality models at select receptor points are extrapolated and assumed to
apply to larger areas and populations
• Can improve understanding of the most influential
uncertainties/errors across full-chain health impact assessment
Conclusions

27. 27

28. 28
Thank you!
Haneen Khreis
H-khreis@tti.tamu.edu
Haneen.khreis@isglobal.org