Slides from a 15-minute developer overview presented at the IAIC session at the 2014 ICAR meeting.

1. Arabidopsis Information Portal: A
new approach to data sharing and
araport.org
cooperative development
Matt Vaughn
Director, Life Sciences Computing
Texas Advanced Computing Center

2. araport.org
SAVE THE DATE
• First AIP Developer Workshop
• Nov 5-6, 2014 in Austin, TX @
Texas Advanced Computing Center
• Come learn and do with the AIP
team
– Interactive Science Apps development
– Data API development
– General hacking and good times
• Email araport@jcvi.org if interested

3. araport.org
Overview
• Rationale for the AIP
• Strategic objectives
• Science Apps Architecture
• Data API Architecture
• How you can participate

4. araport.org
The Rationale for AIP
• Loss of TAIR as a publicly funded shared
resource for data mining and basic
bioinformatics
• Centralization as a key contributing factor
– Loading of new data into database
– Development of new user experience
– Curation and annotation
– Community support mission
• AIP is designed to be de-centralized

5. araport.org
IAIC Workshop Design

6. araport.org
AIP Proposed Architecture

7. araport.org
The AIP Strategy (1)
• Objectives
– Develop a community web resource
• Sustainably fundable and community-extensible
• Hosts diverse analysis & visualization tools + user data spaces
– Support Federation to integrate diverse data sets from
distributed data sources
– Maintain the Col-0 gold standard annotation
• Methods
– Assimilate TAIR10 data
– Host an Arabidopsis InterMine
– Develop a strategy to allow federation
– Offer and consume well-designed RESTful web services
– Interoperate with iPlant (and other projects) wherever
possible

8. araport.org
The AIP Strategy (2)
• Key Design Decisions
– Centralized (but powerful) data warehousing capability PLUS
infrastructure enabling data federation
– Jbrowse as a genome browser platform
– WebApollo + Tripal for community annotation
– App store model for graphical data interfaces (complete with 3rd
party developer path)
– Data store model for data sources
– Accessible languages and frameworks
– Secure & modern single-sign on
– Web service access to Arabidopsis data for powerful
bioinformatics
– Geo-replication and high availability
– Code re-use from other projects wherever possible
– Full code release in real time via GitHub

9. araport.org
Araport Bill of Materials
• AIP is currently built using
– InterMine*
– Jbrowse 1.11.3*
– Drupal 7.25*
• Developer-oriented content management system
– Angular.js, Bootstrap.js and other web toolkits
– Agave Software as a Service platform
• Developed by the iPlant Collaborative
• Bulk data, metadata, authentication, HPC app & job
management, notifications & events, and more
• OAuth2 single-sign-on
– Internally-developed API manager
*With extensive customization

10. araport.org
Araport Architecture
CLI clients,
Scripts, 3rd party
applications
Agave Enterprise Service Bus
Agave Services
systems apps
jobs
Physical
resources
profile
meta
files
HPC | Files | DB
ADAMA
enroll
manage
a b c d e f
AIP & 3rd party data
providers
• Single-sign on
• Throttling
• Unified logging
• API versioning
• Automatic
HTTPS
API Mediators
• Simple proxy
• Mediator
• Aggregator
• Filter
REST-like
REST*
SOAP
POX
Cambrian CGI

11. araport.org
What is a Science App?
– Written in HTML/CSS/Javascript
• Uses standard frameworks
– Presented via web browser
• Query or Analyze, Present, Persist
– Developed by AIP and the community
• Deployed in AIP “app store”
• Choose which ones you want installed in your Araport
“dashboard”
– Uses AIP Data Architecture
• Data services: Local and remote query/retrieval
• Data integration and aggregation services
• Computation services

12. How are Science Apps Developed?
• Objective: 3rd party developers can create
a fully functional Araport science app on
their local dev machine, then make it
available for all Araport users
• We make use of Node.js, Yeoman, Grunt
to begin accomplishing this
araport.org

13. Setting an a Science App Skeleton
araport.org
$ mkdir icar_test_app
$ cd icar_test_app
$ npm install generator-aip-science-app
$ yo aip-science-app
$ grunt
1. Develop locally
2. Iterate, test, refine
3. Package up with “grunt dist”
and sent to Araport team for
hosting.
Media and script dependencies
are either provided by AIP
directly (common ones) or
bundled up in your package.

14. araport.org
IntAct Viewer
https://github.com/Arabidopsis-Information-Portal/InteractionScienceApp
• Clone from git
repo
• Launch
locally
• Dig around
internals and
learn how it
works
• Make
improvements
• Submit them
back to us

15. ADAMA: Araport DAta Mediator API
Objectives: Facile development by
end users; simple, secure
deployment to AIP systems;
reasonable performance
• Docker.io for packaging
• Ultra-portable dev
environment
• Wide language
araport.org
support
• Python,
Javascript, Lua,
Java
• Implicit security
model
• Scales horizontally
for performance
• Data API is package of
metadata + a Docker file
registered with a central
arbiter service
• Also used for services
written natively for AIP
AGAVE
API MANAGER
https://github.com/waltermoreira/apim

16. Data API Design Details (1)
• 100% RESTful services
• Queries are JSON objects (conforming to
a JSON schema)
• Providers REGISTER their services with
Araport
• Science Apps access the services using
Araport as a proxy
• Solves Cross-domain scripting issues that
complicate web app development
araport.org

17. Data API Design Details (2)
• To enroll a new 3rd party data service in
API Manager
– Specify the mapping between AIP’s query
fields and your service’s fields
– Map common query terms to minimal
controlled vocabulary
– Describe all service-specific parameters
– Describe the outputs of your data service
using some standard language
araport.org

18. araport.org
Data API Details (3)
When simple field:field mapping isn’t enough:
• Code-based transformations can be specified
via
– Python
– Java
– Ruby
– Javascript
• This code runs on Araport API servers – no
need for you to host additional services to
work with us!

19. araport.org
Data API Details (4)
• Results returned in a standard JSON
format*
– status, message, result
• Result is an array of JSON objects
• These objects conform to specific JSON
schemas
– drafts on AIP GitHub soon for comment *Unless there’s an operational reason not to

20. araport.org
Data API Details (5)
• Data APIs will implement extra utility
functions
– Count: How many records found?
– Pagination: Return only subsets
– Help: Return a usage page
– Convert: JSON (native), XML, CSV, etc.
• Result: 3rd party Arabidopsis data APIs are
centrally discoverable and usable

21. araport.org
Araport Data API Store

22. araport.org
Araport Data API Store
curl -X GET -k -v -L -b cookies
https://api.araport.org/store/site/blocks/api/listing/ajax/list.jag?action=getAllPublishe
dAPIs
{
"apis": [
{"name":"InteractionBrowser",
"provider":"vaughn",
"version":"pr2-0.1",
"context":"/data/BioAnalyticResource/interactionBrowser",
"status":"Deployed",
"thumbnailurl":"images/api-default.png",
"visibility":null,
"visibleRoles":null,
"description":"InteractionBrowser",
"apiOwner":"vaughn",
"isAdvertiseOnly":false},

23. araport.org
ADAMA Quickstart
Objective: 3rd party developers can create a fully functional
Araport web service on their local dev machine, then make it
available it as a production service for all Araport users.
• Docs:
http://rawgit.com/waltermoreira/apim/master/docs/index.html
• Runnable ADAMA VM (for local development): ETA end of
August 2014
• Develop service adapters right in an iPython notebook!
• We’re ready to start having developers kick the tires on Data
API development – shoot us an email

24. araport.org
Summary
• Next-generation MOD allowing community
participation in its development
• Powerful interactive query and analysis
functions available today
• Developing a data federation model
• New data sets and functions coming at a
quick pace
• Be on the lookout for participation
opportunities

25. araport.org
SAVE THE DATE
• First AIP Developer Workshop
• Nov 5-6, 2014 in Austin, TX @
Texas Advanced Computing Center
• Come learn and do with the AIP
team
– Interactive Science Apps development
– Data API development
– General hacking and good times
• Email araport@jcvi.org if interested

26. Chris Nelson
Project
Manager
araport.org
Chris Town, PI
Lisa McDonald
Education and
Outreach
Coordinator
Jason Miller, Co-PI
JCVI Technical Lead
Vivek Krishnakumar
Bioinf. Engineer
Erik Ferlanti
Software Engineer
Svetlana Karamycheva
Bioinf Engineer
Gos Micklem, co-PI Sergio Contrino
Eva Huala
Project lead, TAIR
Software Engineer
Bob Muller
Technical lead, TAIR
Matt Vaughn
co-PI
Steve Mock
Portal Engineer
Rion Dooley,
API Engineer
Matt Hanlon,
Portal Engineer
Maria Kim
Bioinf Engineer
Ben Rosen
Bioinf
Analyst
Joe Stubbs,
API
Engineer
Walter Moreira,
API Engineer

27. araport.org
Questions?
vaughn@tacc.utexas.edu
cdtown@jcvi.org

28. araport.org

29. araport.org
Araport architecture (2)
Consumer Applications
Secure, rationalized REST services
API Manager + Enterprise Service Bus
Simple
Proxy
ThaleMine, Data
integration, other
services
Cache
XML-to-
JSON
SOAP-to-
REST
Throttle
CGI-to-
REST
Legacy
API A
Legacy
API B
Simple
Proxy
REST
API C
• Single-sign on
• Throttling
• Unified
logging
• API versioning
• Mediation and
translation
• Dev-friendly
interfaces
• Rationalized
REST for
consumer
apps
Mediators

30. araport.org
Science Objectives
• Make more, varied data available to the
Arabidopsis (and other) communities
within a unified user experience
• Enhance the innate value of data by
offering enhanced search, retrieval, and
display capabilities
• Facilitate analysis of user data
• Enable community participation in
functional annotation

31. araport.org
Technical Objectives
• Deploy a responsive, flexible community-extensible
system
• Provide APIs everywhere!
• Promote and facilitate data integration
• Enable language- and region-specific
presentation of scientific content
• Meet mobile computing on its own terms

32. araport.org
Local vs. Data-driven Apps
Photoshop Express KAYAK Pro
Resources are local and
inherently offline. Operating
on local data using local
computing.
Resources are cloud-based and
inherently online. Multiple data
streams integrated, queried,
presented in context of broader
objective.

33. araport.org
Araport Bill of Materials
• Araport is currently built using
– Drupal 7.25
• Developer-oriented content management system
– Bootstrap.js and some other Javascript toolkits
– InterMine (with modifications)
– Bioinformatics infrastructure + misc. other bits
– Agave 2.0 Software as a Service platform
• Developed by iPlant Collaborative project
• Bulk data, metadata, authentication, HPC app and job
management, notifications & events, and more
• OAuth2 out of the box
• Enterprise service bus (ESB) architecture
• http://agaveapi.co/

34. Araport APIM Architecture (1)
SNP by Locus REST Indel by Position REST Enroll Manage
araport.org
Agave wso2 interface
Cache (Technology TBD)
Listen Respond
CSV
Form
POLYMORPH CGI
Listen Respond
Input Key
Map
Output
Key Map
Input
Transform
Output
Transform
Send Listen
Input Key
Map
Output
Key Map
Input
Transform
Output
Transform
Send Listen
Araport API
Manager
JSON Query JSON Response
ElasticSearch
Remote Services

35. Araport Architecture: Use Cases (1)
araport.org
• 1001 Genomes POLYMORPH tools
– Provides variation data via locus or positional
search
– Total of seven variant types available for search
– Search parameterization depends a lot on variant
type
– Example of a plain-text CGI service
– Returns results as CSV with named columns
• Objective: Transform into a RESTful API that
expects and returns rationalized JSON
http://polymorph.weigelworld.org

36. Araport Architecture: Use Cases (2)
araport.org
• ThaleMine
– Has native REST interface for general queries
– Has templates which can form basis of specific
services
• Objective: Offer both Intermine-native and
AIP-conformant interfaces as Data APIs
• Current path
– Enroll native services in our APIM
– Develop template-based AIP-conformant services
http://polymorph.weigelworld.org

37. araport.org
Data APIs: Getting Started
Service Queries Notes
BAR eFP Locus
BAR Expressologs Locus
BAR Interactions Locus
COGe Position Special case – output transform only
NASC $SERVICE Locus
SOAP based but may be offline
permanently
OrthologFinder Locus Based on a Thalemine template
POLYMORPH Locus, Position Actually seven CGI services
SUBA3 Locus
Compiling example queries, parameter mapping and description, and ideal
results for use in implementing the system

38. araport.org
Developing a Data API
• In order, we prefer that you have ready
• Well-documented REST
• Moderately well-documented REST
• SOAP services (plus WSDL or WADL)
• Plain Old XML
• Plaintext CGI
• HTML CGI
• No web services at all
• Work with us to enroll your services as a data
source. This will involve a minor amount of
coding.

39. Computational App Model (1)
Host OS
Host file
systems
araport.org
Docker.io
Centos
6.4
custom-repo
Container
/scratch
/database
REST API x JSON objects
Agave apps, data, jobs
sftp
Host FS (250 GB)
TACC Corral (PB+)

40. araport.org
Science Apps: Grid View
• Current Scheme
• 2-3 column view w
draggable apps
• Apps are normal, full-size,
or collapsed
• Single app screen
• Later in 2014
• N x X grid scheme
implementing resizable
app “tiles” like one sees
in Android or Win8.x
• App SDK libraries will
have “help” for enabling
resizable design
• Multiple app screens

41. araport.org
Data API Details (2)
• For service-specific parameters
– Provide human-readable names mapped to original
parameter names
– Offer minimal descriptive text
– Specify validation
• Cardinality
• Pattern validator (regex)
• Type (number, string, etc.)
– Indicate whether required
– Indicate whether they should be visible in a UI
– Specify reasonable default values
• Seems familiar?
– This approach is used to to abstract command line apps
– Allows automatic generation of minimally functional UI

42. Data APIs: Response types (1)
araport.org
• locus_relationship – pairwise
relationship between A and B
– Directionality
– Type
– Array of scores (weights, etc.)
• sequence_feature – positional attribute
– Extension of GFF model plus
– Build
– Attributes array

43. Data APIs: Response types (2)
araport.org
• locus_feature – key-value attributes per locus
– Optional controlled vocabulary* for keys
– Support for both slots and arrays
• raw – for returning images or other binary formats
– Source and other metadata carried in X-headers instead of
JSON result
– Outbound transformation still supported
– Not a preferred response mode
• text – returning either native service response or a
non-conformant JSON document
– Source and other metadata carried in X-headers instead of
JSON result
– Not a preferred response mode

44. araport.org
Data API Details (6)
• Transparent caching will compensate for
transient remote service failures
• Automatic indexing of certain response
types via ElasticSearch, allowing for
sophisticated global search
– ElasticSearch allows us to index everything
we “know about” and return it quickly
– iPlant uses it to live-index >700 TB user data

45. araport.org
Developing an app
• Understand and document the user stories you’re
addressing with your app
• Identify all requisite data sources AND
• Help us prepare them as Data APIs
– This may involve coding
• Understand the data integration or aggregation needs
of your app
– This may involve coding
• Develop the user interface(s) for your app using our
tool kits and suggested practices
– This will involve coding.
– But you will learn tools like jQuery, Bootstrap, & D3 and will
thus be eminently employable!