The DSDI Task has been running at TRECVID to benchmark systems working on automatically detecting features after natural disaster events (e.g. earthquakes, flooding, etc) captured from civil air patrol and airborne video footage.

1. Disaster Scene Description and
Indexing (DSDI) Task Overview
TRECVID 2022
Asad Anwar Butt, National Institute of Standards and Technology; Johns Hopkins University
George Awad, National Institute of Standards and Technology
Jeffrey Liu, MIT Lincoln Laboratory
William Drew, Office of Homeland Security and Preparedness

2. • Video and imagery data can be extremely helpful for public
safety operations.
• Natural Disasters, e.g.,
• Wildfires
• Hurricanes
• Earthquakes
• Floods
• Man-made Disasters, e.g.,
• Hazardous material spills
• Mining accidents
• Explosions
Introduction - DSDI
* All images are under creative common licenses

3. • Prior knowledge about affected areas can be very useful for
the first responders.
• Oftentimes, the communication systems go down in major
disasters, which makes it very difficult to get any information
regarding the damage.
• Automated systems to gather information before rescue
workers enter the area can be very helpful.
Introduction - DSDI

4. • Computer vision capabilities have rapidly advanced recently
with the popularity of deep learning.
• Research groups have access to large image and video datasets for various tasks.
• However, the capabilities do not meet public safety needs.
• Lack of relevant training data.
• Most current image and video datasets have no public safety
hazard labels.
• State-of-the-art systems trained on such datasets fail to provide helpful labels.
Introduction - DSDI

5. • In response, the MIT Lincoln Lab developed a dataset of images
collected by the Civil Air Patrol of various natural disasters.
• The Low Altitude Disaster Imagery (LADI) dataset was developed as
part of a larger NIST Public Safety Innovator Accelerator Program
(PSIAP) grant.
• Two key properties of the dataset are:
• Low altitude
• Oblique perspective of the imagery and disaster-related features.
• The DSDI test data and ground truth from 2020 & 2021 are also
available for teams to use as training data.
Training Dataset

6. • LADI Dataset:
• Hosted as part of the AWS Public Dataset program.
• Consists of 20,000+ annotated images.
• The images are from locations with FEMA (Federal Emergency Management
Agency) major disaster declarations for a hurricane or flooding.
• Lower altitude criteria distinguish the LADI dataset from satellite datasets to support
the development of computer vision capabilities with small drones operating at low
altitudes.
• A minimum image size was selected to maximize the efficiency of the crowd source
workers; lower resolution images are harder to annotate.
• 2020 - 2021 DSDI Test Set:
• ~ 11 hours of video.
• Segmented into small video clips (shots) of maximum 20 sec.
• Videos are from earthquake, hurricane, and flood affected areas.
• Total number of shots: 4626
Training Dataset

7. • A test dataset of about 6 hours of video was distributed for this task.
• Collected by FEMA as individual images after disaster events.
• Individual images were stitched to form videos with reasonable
speed.
• The test dataset was segmented into small video clips (shots) of a
maximum of 16 sec, with a mean length of 10 sec.
• Subset was selected by NIST taking into account diversity
• In addition, a small set of videos was collected from Defense Visual
Information Distribution Service (DVIDS): https://www.dvidshub.net/
• Total number of shots: 2157
Test Dataset

8. Testing Data: Example Videos
* All videos are under public domain licenses

9. • Hierarchical labeling scheme: 5 coarse categories, each with 4 to 9
more specific annotations.
Testing Dataset - Categories
Damage Environment Infrastructure Vehicles Water
Misc. Damage Dirt Bridge Aircraft Flooding
Flooding/Water Damage Grass Building Boat Lake/Pond
Landslide Lava Dam/Levee Car Ocean
Road Washout Rocks Pipes Truck Puddle
Rubble/Debris Sand Utility Or Power Lines/Electric Towers River/Stream
Smoke/Fire Shrubs Railway
Snow/Ice
Wireless/Radio Communication
Towers
Trees Water Tower
Road

10. • We had 2 full time annotators (same annotators since 2020) instead
of crowdsourcing.
• For each category, a practice web page was created with multiple
examples and sample test videos.
• This allowed the annotators to become familiarized with the task and
labels before starting a category.
• The annotators worked independently on each category.
• For each coarse category, they marked all the specific labels that
were present in the video.
• To create the final ground truth, for each shot, the union of labels
was used.
Annotation

11. The annotators watch the video and mark the categories that are visible in the video.
Annotation Tool

12. • Systems are required to return a ranked list of up to 1000 shots for
each of the 32 features.
• Each submitted run specified its training type:
• LADI-based (L): The run only used the supplied LADI dataset for development of its system.
• Non-LADI (N): The run did not use the LADI dataset, but only trained using other dataset(s).
• LADI + Others (O): The run used the LADI dataset in addition to any other dataset(s) for
training purposes.
DSDI System Task

13. • The following evaluation metrics were used to compare the
submissions:
Evaluation Metrics
Metric Description
Speed Clock time per inference (reported by participants).
Mean Average
Precision (MAP)
Average precision is calculated for each feature, and the mean average
precision reported for a submission.
Recall True positive, true negative, false positive, and false negative rates.

14. Submissions
Run Type Run Id
O PKU_WICT_7
O PKU_WICT_2
O PKU_WICT_4
O PKU_WICT_5
L PKU_WICT_6
L PKU_WICT_3
L PKU_WICT_1
L PKU_WICT_8
L UMKC_1
O UMKC_1
PKU_WICT : Peking University
UMKC : University of Missouri-Kansas City

15. Frequency of Features
• Graph shows number of shots containing each feature.
• Some features (e.g. grass, trees, buildings, roads, etc.) occur much more frequently than others.
• 4 features were dropped due to the rare occurrence in ground truth (lava, snow/ice, landslide, road washout)
0
200
400
600
800
1000
1200
1400
1600
1800
2000
2200
d
a
m
a
g
e
(
m
i
s
c
)
f
l
o
o
d
i
n
g
r
u
b
b
l
e
/
d
e
b
r
i
s
s
m
o
k
e
/
f
i
r
e
d
i
r
t
g
r
a
s
s
r
o
c
k
s
s
a
n
d
s
h
r
u
b
s
t
r
e
e
s
b
r
i
d
g
e
b
u
i
l
d
i
n
g
d
a
m
/
l
e
v
e
e
p
i
p
e
s
u
t
i
l
i
t
y
/
p
o
w
e
r
l
i
n
e
s
r
a
i
l
w
a
y
w
i
r
e
l
e
s
s
c
o
m
m
u
n
i
c
a
t
i
o
n
t
o
w
e
r
s
w
a
t
e
r
t
o
w
e
r
a
i
r
c
r
a
f
t
b
o
a
t
c
a
r
t
r
u
c
k
f
l
o
o
d
i
n
g
l
a
k
e
/
p
o
n
d
o
c
e
a
n
p
u
d
d
l
e
r
i
v
e
r
/
s
t
r
e
a
m
r
o
a
d
Frequency
of
shots
True Positives per feature

16. Results by Teams
0.501 0.5 0.482
0.429
0.354
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PKU_WICT _7 PKU_WICT_2 PKU_WICT_4 PKU_WICT_5 UMKC_1
Mean
Average
Precision
LADI+Others-based Systems
Performance across all features
0.468 0.468 0.465
0.423
0.354
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PKU_WICT_6 PKU_WICT_3 PKU_WICT_1 PKU_WICT_8 UMKC_1
Mean
Average
Precision
LADI-based Systems
Performance across all features

17. Results by Categories
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
D
a
m
a
g
e
E
n
v
i
r
o
n
m
e
n
t
I
n
f
r
a
s
t
r
u
c
t
u
r
e
V
e
h
i
c
l
e
s
W
a
t
e
r
Mean
Average
Precision
Categories of tested features
Performance by category
LADI+Others
LADI

18. • Average precision values for each feature categorized by training type.
• 5 LADI-based runs ; 5 LADI+Others-based runs
Results by Features
0
0.2
0.4
0.6
0.8
1
1.2
damage
flooding
rubble
smoke
dirt
grass
rocks
sand
shrubs
trees
bridge
building
dam
pipes
utility
line
railway
wireless
tower
water
tower
aircraft
boat
car
truck
flooding
lake
ocean
puddle
river
road
Average
Precision
LADI-based runs - by feature
Min
Median
Max
0
0.2
0.4
0.6
0.8
1
1.2
damage
flooding
rubble
smoke
dirt
grass
rocks
sand
shrubs
trees
bridge
building
dam
pipes
utility
line
railway
wireless
tower
water
tower
aircraft
boat
car
truck
flooding
lake
ocean
puddle
river
road
Average
Precision
LADI+Others-based runs - by feature
Min
Median
Max

19. Efficiency
0
100
200
300
400
500
600
700
800
0 0.2 0.4 0.6 0.8 1 1.2
Time
(sec)
Average Precision
LADI-based runs
0
100
200
300
400
500
600
700
800
0 0.2 0.4 0.6 0.8 1 1.2
Time
(sec)
Average Precision
LADI+Others-based runs
• LADI-based systems reported less processing time
• Majority of systems consumed more time but without more gain in performance
• Lowest processing time was ~30 sec at max performance
O_UMKC_1
L_UMKC_1
L_PKU_WICT_8

20. F-Measure per feature

21. • A new test dataset from various event sources were employed
representing more diversity.
• Performance varies by feature.
• L+O runs performed higher than L-based runs.
• Few runs/features are good and efficient.
• Challenges include:
• Small dataset and limited resources for annotation.
• Training and testing dataset should be from the same distribution. Hard to do with
different nature of calamities.
• The task had little participation compared to the last 2 years.
• Question to teams about continuation of task.
Conclusion and Future Work