Archive-It Annual Partners Meeting, 8/2/2016 Atlanta, GA (Society of American Archivists)

1. MESSY WEB, MEET
CLEAN(ER) WEB:
AN EXPERIMENT
IN SOCIAL MEDIA ARCHIVING
Archive-It Annual Partners Meeting
8/2/2016
Atlanta, GA
Rachel Trent
Digital Services Manager
George Washington University Libraries, Special Collections

2. THE WEB IS MESSY

3. TWITTER
FACEBOOK
WIKIPEDIA
INSTAGRAM

5. WHAT A PERSON THINKS A TWEET IS

6. WHAT A CRAWLER THINKS A TWEET IS
There is no such thing as a “tweet.”
All I see is HTML & hyperlinks.

7. WHAT AN APP THINKS A TWEET IS

8. TIMESTAMP
TWEET
TEXT
TWITTER HANDLE
(CREATOR)
TWITTER HANDLE
(MENTION)
URL, EXPANDED

9. SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
JSON
Render JSON
ELK (w/ Kibana)

10. DEMO
SOCIAL FEED MANAGER (SFM)
ELK
PYWB
SFM VIEWER
(screenshots if web fails us)

13. SFM
JSON
Process and
visualize data
API Heritrix
ELK (w/ Kibana)
(screenshots if web fails us)

18. ELK IS GREAT!
(FOR USERS WHO DO DATA ANALYSIS)
WHAT ABOUT OUR OTHER USERS?
- linked web content
- media (images, videos)
- browsing

19. SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
Render JSON
ELK (w/ Kibana)

20. DEMO
SOCIAL FEED MANAGER (SFM)
ELK
PYWB
SFM VIEWER
(screenshots if web fails us)

27. SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
Render JSON
ELK (w/ Kibana)

28. WHAT’S THE TAKEAWAY?
It’s possible to collect messy-web & social-media-web
. . . without creating access silos
. . . using existing technologies
. . . to make more users happy