1. HISTORICAL STATE: DISCOVERING NC STATE HISTORY ADAM BERENBAK BRIAN DIETZ TODD KOSMERICK CATE PUTIRSKIS JASON RONALLO } NCSU LIBRARIES LAUNC-CH. 03.08.2010
36. CONTACT US ADAM BERENBAK, SCRC [email_address] BRIAN DIETZ, SCRC [email_address] TODD KOSMERICK, SCRC [email_address] CATE PUTIRSKIS, SCRC [email_address] JASON RONALLO, DLI [email_address]
Notas del editor
Before Historical State there were various resources dispersed throughout the NCSU Libraries website. There was an accumulation of disparate content created over several years. Everything static HTML pages.
4 timelines: university history athletics history African American history library history
History of women chancellors and presidents GI Bill
Internal and external links, including historical information on departmental websites back issues of university publications etc.
Published histories of the university, colleges, departments, and other units.
Finding aids (or collection guides) accessed from another part of website. Search function broke after a few years.
UAPC digitization project, commenced 2003 Luna [old name] server, metadata not crawled by search engines Highly requested materials
Beginning in 2006 course catalogs, some annual reports digitizied Future plans for yearbook and student newspaper Catalogs highly requested; yearbook and newspaper expected to be also Accessed through Dspace instance.
Challenge is how to have one interface to multiple content types from various sources without having it become some mutant giant grasshopper. I’ll start with an overview of what technologies I’ve used and then talk about a little of the new development I did.
Ruby on Rails
Lot’s ‘o plugins. As the sole developer, I needed as much help as I could get and the Rails framework and its many open source plugins helped me along.
Probably the biggest help came from the Rails plugin Blacklight. Blacklight is the OPAC replacement out of University of Virginia and used and developed by Stanford as well. Blacklight has become a good tool for getting a UI for the full text search power of Solr with little work. It allows for different formats to easily have different views, so it is very well suited to the task at hand.
One thing to understand is the use of two databases. Use each what it is good for. MySQL as a relational db, you get all of the common conveniences of working with a standard db such as good support in the framework. Solr provides the full-text searching. So with all of these off-the-shelf open source tools, what new development was done other than customization of the interface and hooking in all the plugins? One area was a Pageturner. We currently have a use case where we show page images of yearbooks, but do not provide a link to download a PDF of the whole thing. The pageturner uses Solr as well for good things like snippets and hit highlighting. Each page is stored in the index as a separate Solr document. Another area of a lot of development was in getting all of the data into the system. And this involves both MySQL and Solr. I’ll conclude my portion by talking a bit about how that developed and what I learned along the way.
This is how the search was first conceived as a harvesting records and dumping them in Solr. This is how an OPAC replacement would work. In this case though we’re taking records out of our home grown digital assets metadata management system. To simplify this I created an API over top of the system which returns JSON. From there, very little needs to be done to insert the record into the Solr index for search and display.
But this only worked for some of the content. We also have curated content to get in there as well. This was managed through spreadsheets and static HTML, so I created some simple web forms for data entry. Administrator functionality that looks like it wouldn’t work but it basically does. All of this gets stored in the MySQL db and allows us to assocated resources like colleges with departments.
Here’s what the process looked like at one point—some crazy contraption. The curated content goes to MySQL and as it gets saved there it also gets indexed in Solr. Digital Assets harvested records were going directly into the Solr index. This makes sense with millions of records to only have one database for the application saving time. But we don’t have millions of digital objects yet so there was no need to optimize prematurely. This became a problem when I wanted a simple way to associate comments with both the stuff in the relational db as well as stuff that never got to MySQL. One or the other would have been easy, but I didn’t want to have more than one place to manage comments. I didn’t want to figure out how to
The answer was simple. Don’t buck against the framework.
See how that looks cleaner now the path for everything to solr is the same—save it in the relational db first. Makes keeping the data in sync easier and reindexing content faster by having a cache to work from also gets all kinds of convenience methods I now have one comments model which works for all the different models of data in the db
Finally, we are just about to take the discovery of university history to a whole new level with the development of the repackaged Historical State. This resource makes all the finding aids, digitized objects and curated content (such as timelines, departmental histories) accessible in one place and facilitates searching across formats. The collecting, processing and digitizing work that fills our staffs’ days is fully leveraged for discovery via a new, developed in-house, open source, cross-search application. The new Historical State is an innovation in user driven discovery of university history.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
All of this is driven by content and metadata created by our archivists and innovative programming developed by our digital libraries initiatives team. Historical State Search is a Ruby on Rails application utilizing the Blacklight plugin for Solr-powered searching. A MySQL database is used for managing curated content like events and "Did you know?" content. Metadata is harvested from the web API of a homegrown digital assets management system. An integrated pageturner using a separate Solr index is used for displaying multi-page documents, like course catalogs and yearbooks. All of this technology takes advantage of description/cataloging and research to deliver just what the user is looking for and opening up university history in a way that has never been possible before.
