NISO Forum, Denver, September 24, 2012: ResourceSync: Web-Based Resource Synchronization

ResourceSync:
Web-Based
Resource
Synchronization
Herbert Van de Sompel
Los Alamos National Laboratory
@hvdsomp

ResourceSync is funded by
#resourcesync The Sloan Foundation & JISC

ResourceSync – Herbert Van de Sompel
NISO Forum, September 24 2012, Denver, CO

ResourceSync Core Team – NISO & OAI

Los Alamos National Laboratory & OAI: Martin Klein, Robert
Sanderson, Herbert Van de Sompel

Cornell University & OAI: Berhard Haslhofer, Simeon
Warner

Old Dominion University & OAI: Michael L. Nelson

University of Michigan & OAI: Carl Lagoze

NISO: Todd Carpenter, Nettie Lagace, Peter Murray


ResourceSync Technical Group

•  Manuel Bernhardt, Delving B.V.
•  Kevin Ford, Library of Congress
•  Richard Jones, JISC
•  Graham Klyne, JISC
•  Stuart Lewis, JISC
•  David Rosenthal, LOCKSS
•  Christian Sadilek, Red Hat
•  Shlomo Sanders, Ex Libris, Inc.
•  Sjoerd Siebinga, Delving B.V.
•  Ed Summers, Library of Congress
•  Jeff Young, OCLC Online Computer Library Center


ResourceSync

ResourceSync: What & Why?

Problem Perspective & Conceptual Approach

Possible Technical Choices

Q&A


Synchronize What?

•  Web resources – things with a URI that can be dereferenced and
are cache-able (no dependency on underlying OS, technologies
etc.)

•  Small websites/repositories (a few resources) to large
repositories/datasets/linked data collections (many millions of
resources)

•  That change slowly (weeks/months) or quickly (seconds), and
where latency needs may vary

•  Focus on needs of research communication and cultural heritage
organizations, but aim for generality


Why?

… because lots of projects and services are doing synchronization
but have to resort to ad-hoc, case by case, approaches!

•  Project team involved with projects that need this

•  Experience with OAI-PMH: widely used in repos but
o  XML metadata only

o  Attempts at synchronizing actual content via OAI-PMH

(complex object formats, dc:identifier) not successful.
o  Web technology has moved on since 1999

•  Devise a shared solution for data, metadata, linked data?


Use Cases – The Basics


Use Cases - More


Out Of Scope (For Now)

•  Bidirectional synchronization

•  Destination-defined selective synchronization (query)

•  Bulk URI migration


Use Case: arXiv Mirroring

•  1M article versions, ~800/day created or
updated at 8 PM US Eastern Time

•  Metadata and full-text for each article

•  Accuracy important

•  Want low barrier for others to use

•  Look for more general solution than current
homebrew mirroring (running with minor
modifications since 1994!) and occasional rsync
(filesystem layout specific, auth issues)


Use Case: DBpedia Live Duplication

•  Average of 2 updates per second
•  Want low latency => need a push technology


ResourceSync Problem

•  Consideration:
•  Source (server) A has resources that change over time: they
get created, modified, deleted
•  Destination (servers) X, Y, and Z leverage (some) resources
of Source A.
•  Problem:
•  Destinations want to keep in step with the resource changes
at source A: resource synchronization.
•  Goal:
•  Design an approach for resource synchronization aligned
with the Web Architecture that has a fair chance of adoption
by different communities.
•  The approach must scale better than recurrent HTTP
HEAD/GET on resources.


Destination: 3 Basic Synchronization Needs

1.  Baseline synchronization – A destination must be able to
perform an initial load or catch-up with a source
-  avoid out-of-band setup

2.  Incremental synchronization – A destination must have some
way to keep up-to-date with changes at a source
-  subject to some latency; minimal: create/update/delete
-  allow to catch-up after destination has been offline

3.  Audit – A destination should be able to determine whether it is
synchronized with a source
-  subject to some latency


Source Capability 1: Describing Content

In order to advertise the resources that a source wants destinations
to know about, it may describe them:

o  Publish an inventory of resource URIs and possibly
associated metadata
-  Destination GETs the Content Description
-  Destination GETs listed resources by their URI


Source Capability 2: Communicating Change Events

In order to achieve lower latency, a source may communicate about
changes to its resources:

o  2.1. Change Set: Publish a list of recent change events
(create, update, delete resource)
-  Destination acts upon change events, e.g. GETs created/
updated resources, removes deleted resources.


Source Capability 2: Communicating Change Events

In order to achieve lower latency, a source may communicate about
changes to its resources:

o  2.1. Change Set: Publish a list of recent change events
(create, update, delete resource)

o  2.2. Push Change Set: Push a list of recent change events
(create, update, delete resource) towards (a) destination(s)


Source Capability 3: Providing Access to Versions

In order to allow a destination to catch up with missed changes, a
source may support:

o  3.1. Historical Change Sets: Provide access to change events that
occurred prior to the ones listed in the current Change Set


Source Capability 3: Providing Access to Versions

In order to allow a destination to catch up with missed changes, a
source may support:

o  3.1. Historical Change Sets: Provide access to change events that
occurred prior to the ones listed in the current Change Set

o  3.2. Historical Content: Provide access to prior resource versions


Source Capability 4: Transferring Content

By default, content is transferred in response to a GET issued by a
destination against a URI of a source’s resource. But a source may
support additional mechanisms:

o  4.1. Dump: Publish a package of resource representations
and necessary metadata
-  Destination GETs the Dump
-  Destination unpacks the Dump


Source Capability 4: Transferring Content

By default, content is transferred in response to a GET issued by a
destination against a URI of a source’s resource. But a source may
support additional mechanisms:

o  4.1. Dump: Publish a package of resource representations
and necessary metadata
-  Destination GETs the Dump
-  Destination unpacks the Dump

o  4.2. Alternate Content Transfer: Support alternative
mechanisms to optimize getting content (see later)


Source: Advertise Capabilities

A source needs to advertise the capabilities it supports to allow a
destination to discover them

•  Some capabilities may be provided by a third party, not the
source itself
o  e.g. Historical Change Sets, Historical Content
o  But the source should still make those third party capabilities
discoverable - trust


ResourceSync: A Framework of Capabilities

•  Modular framework allowing selective deployment of
capabilities

•  A Source selects which capabilities to support in order
to meet local and community needs

•  A Source’s Capabilities can be discovered via
capability descriptions


BY REFERENCE!

BY VALUE!


Sitemap

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://example.com/res1</loc>
<lastmod>2012-08-08T08:15:00Z</lastmod>
</url>
<url>
<lastmod>2012-08-08T13:22:00Z</lastmod>
</url>
</urlset>


Baseline Matching - Sitemap

•  Periodic publication of up-to-date Sitemap, which is a “by
reference” inventory of a Source’s resources

•  Use ”as is” with resource location and last modification date as
core elements

•  Introduce extension elements aimed at supporting audit: e.g.
MD5 hash of content


robots.txt!

discovery


Baseline Matching – Dump

•  A Dump is a “by-value” inventory of a Source’s resources

•  Periodic publication of an up-to-date Dump

•  Possible technology: ZIP file consisting of:

•  Special-purpose Sitemap that acts as a manifest for
resources contained in the ZIP file
•  Introduce an element to express correspondence
between resource URI and filename in the ZIP file
•  Resource bitsteams

•  Possible technology: WARC file


Change Communication – Pull Change Sets

•  Periodic publication of a Change Set that describes recent
changes

•  A Change Set is a Sitemap-style document, enhanced to
express change events rather than inventory. Per change event,
convey:
•  About the event:
•  datetime
•  event type: create/update/delete (maybe move/copy)
•  About the changed resource:
•  URI
•  Information relevant for audit, e.g. fixity, size, mime type
•  Further information to aide accessing the resource (see
later)


Change Set, Based on Sitemap

<urlset rs:type="changeset”
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<url>
<lastmod rs:type="updated">2012-08-08T08:15:00Z</lastmod>
</url>
<url>
<lastmod rs:type="created">2012-08-08T10:22:00Z</lastmod>
</url>
</urlset>


Change Set, from Scratch

<changeset xmlns="http://www.openarchives.org/rs/changeset">
<change>
<link rel="created" length="1234" type="text/html”
href="http://example.com/res1.html"/>
<date>2012-09-25T09:00:00Z</date>
<fixity>ni:///sha-256;f4OxZX_x_FO5LcGBSKHWXfwtSx</fixity>
</change>
</changeset>


Change Communication – Push Change Sets

•  Use a push technology to convey changes

•  Express changes using same Sitemap-style document
•  A Change Set in this case might convey only one change
event

•  Possible technology: XMPP PubSub


<XMPP PubSub Intermezzo>

XMPP Publish-Subscribe: Client to Subscription Service,
Subscription Service to Client(s) communication

•  One of the XMPP (Extensible Messaging and Presence Protocol)
extensions http://xmpp.org/extensions/xep-0060.html

•  Apple Notifications based on XMPP PubSub

•  Both client and server tools widely available


</XMPP PubSub Intermezzo>

Source PubSub Server Destination


Change Communication Memory

•  Publication of one or more Change Sets that convey historical
(rather than recent) changes

•  All historical Change Sets use same Sitemap-style document

•  Same approach irrespective of whether pull or push is used for
Change Communication


Resource Transfer

•  Resources are obtained in bulk by obtaining a Dump

•  An individual resource is, by default, obtained by dereferencing a
resource’s URI listed in:
•  Sitemap
•  Change Set

•  Alternative access mechanisms are introduced to obtain an
individual resource:
•  From a mirror site
•  Access to diff with previous version instead of access to the
entire changed resource
•  Resource version


Resource Memory

•  Requires a (short or long term) archive of resource versions

•  Access to specific version can be expressed as an alternative
access mechanism in e.g. Change Set.
•  Via a link to a version resource that is the result of the
change expressed in the Change Set
•  Via a link to a Memento TimeGate that supports access to all
available prior versions


<Memento Intermezzo>

http://www.mementoweb.org/


Original Resources and Mementos


Bridge from Present to Past


Bridge from Past to Present


Memento Framework


Memento Framework

Original Resource: http://lanlsource.lanl.gov/pics/picoftheday.png


Time Travel across Versions of a Picture of the Day

Movie at: http://www.mementoweb.org/demo/picoftheday.mov

Memento Framework

Original Resource: http://dbpedia.org/resource/France


Time-Series Analysis across DBpedia Versions

Data collected through HTTP Navigation

Paper at http://arxiv.org/abs/1003.3661


</Memento Intermezzo>

http://www.mementoweb.org/


ResourceSync Timeline
•  August 2012
o  First draft spec shared for feedback with ResourceSync team

•  September 2012
o  Problem Statement paper in D-Lib Magazine

o  In-person meeting of ResourceSync Team

•  October 2012
o  Revise spec, conduct experiments

o  Solicit broad feedback

•  December 2012 – Finalize specification (?)


Pointers
•  First ResourceSync draft spec (do not implement!):
http://www.openarchives.org/rs/0.1/resourcesync!

•  ResourceSync Simulator code on github
http://github.org/resync/simulator!

•  NISO ResourceSync workspace
http://www.niso.org/workrooms/resourcesync/!

•  Memento
http://mementoweb.org!


ResourceSync:
Get the Sticker!

Herbert Van de Sompel
Los Alamos National Laboratory
@hvdsomp

ResourceSync is funded by
The Sloan Foundation & JISC


NISO Forum, Denver, September 24, 2012: ResourceSync: Web-Based Resource Synchronization

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (7)

Similar a NISO Forum, Denver, September 24, 2012: ResourceSync: Web-Based Resource Synchronization

Similar a NISO Forum, Denver, September 24, 2012: ResourceSync: Web-Based Resource Synchronization (20)

Más de National Information Standards Organization (NISO)

Más de National Information Standards Organization (NISO) (20)

Último

Último (20)

NISO Forum, Denver, September 24, 2012: ResourceSync: Web-Based Resource Synchronization