CSSDP Presentation from NEP Synergy Workshop

1. Canadian Space Science Data Portal
Technical Architecture
CANARIE Synergy Workshop
April 21 – 22, 2009

2. Technical Architecture
PROBLEM DOMAIN
CSSDP - NEP 21 2

3. Space Weather
CSSDP - NEP 21 3

4. CSSDP - NEP 21 4

5. Key Domain Characteristics
Governance-related
• Many independent Data Providers
• Data usage restrictions and auditing
Process-related
• Research groups / teams
• Resource search & discovery
• Routine steps to prepare data for research
Data-related
• Large size and volume of data sets
• Heterogeneity: instruments, data formats
Technology-related
• Heterogeneity: repository technologies, analytic / visualization
technologies
• Cost of technology vs funding models
CSSDP - NEP 21 5

6. Heterogeneity
CSSDP - NEP 21 6

7. Technical Architecture
SOLUTION DOMAIN
CSSDP - NEP 21 7

8. Virtual
Observatory
CSSDP - NEP 21 8

9. Portal
CSSDP - NEP 21 9

10. Primary Design Elements
CSSDP - NEP 21 10

11. Federating Resources
Local Site Overall System
Local Site
Local Site
Federated
Local Site
System
Local
Data
Repository Federated
Instrument System
Instrument
Data Site Data
Data Copied to Portal
Local Site
Enterprise Service Bus
Instrument
Researcher Data
Data Copied to Portal Instrument
Portal Federated
Data Site
System
Data Federated
Local Site
by Portal
Instrument Federated
Data Site System
Local Site
Local Analytics & Instrument
Visualizations Data
Federated
System
Overall System is a Federation of
Each Local Site is a Federation
Federations
CSSDP - NEP 21 11

12. Block Diagram
UI
Presentation Forms Reports Charts Visualizations
External Process Executon &
Layer Page Flow
Management
Java Business Logic
Analytics
POJOs
Business
Business Business
Components Rules
Layer
Events and
Resource
Notifications
Management
Scheduler Digital Rights
Management
Transport Resource
Factory Registry
Transaction
Data Data Set Management
Transport
Layer
Storage
Persistence
Data Set Management
Transformation
Common Services
Collaboration Configuration
Business Process & Instrumentation, Auditing &
Workflow Logging
Security & Identity Messaging & Web Services
CSSDP - NEP 21 12

13. Workflow
CSSDP - NEP 21 13

14. Pipes Filters
CSSDP - NEP 21 14

15. Scientific Workflows
D a ta L o a d e r
G e n e ra te
D a ta
A cq u ire D a ta
A va ila b ility
E ve n t
S ch e d u le r
M o n ito r /
In itia te
M o n ito r R e trie ve N o tify E ve n t
W o rkflo w s
W o rkflo w s E ve n ts S u b scrib e rs
fo r D a ta S e t
W o rkflo w
Run S to re R u n D a ta S to re Run S to re
A n a lytics R e su lts M a rke rs R e su lts V isu a liza tio n s R e su lts
D a ta M a rk e r
A n an a ly D a ta
A lyze S to re E ve n ts
R e s e a rc h e r
R e ce ive R e ce ive
R e vie w E ve n ts
D a ta S e t R e vie w P e rfo rm D a ta M a rke r
and
A va ila b ility D a ta S e ts A n a lysis E ve n t
V isu a liza tio n s
N o tifica tio n s N o tifica tio n s
CSSDP - NEP 21 15

16. Technology
CSSDP - NEP 21 16

17. Data
Layer
Layer
Layer
Business
Presentation
HTMLw/
Data Files HTTP
JavaScript
log4j
commons-logging
XML
Workflow Engine
(TBD)
Registry
Resource
Message Bus or
CSSDP - NEP 21
ESB (TBD)
JEE 5
XSLT
Spring
Jasper
Wicket
Reports
OGSA
JDBC
Spring
Sun AppServer
JFreeChart
Quartz
Scheduler
Technologies
REST / WS-* Web Services
Confluence
SQL
Hibernate
Postgres
Data IDL, Matlab, FORTRAN, etc.
Transport
Unix
Protocols
LDAP
Shell Scripts
(SFTP, etc.)
17

18. SPASE XML
CSSDP - NEP 21 18

19. Registration: JiBX and Java
CSSDP - NEP 21 19

20. Wicket
CSSDP - NEP 21 20

21. class Class Mo...
«<Presentati...
WebPage
Sample
Implementation
«Presentatio... «Presentatio... «Presentatio...
DataProv ider ResultsPage FindDataPage
«BusinessLayer» «BusinessLa...
DataChunkDataProv ider SearchCriteria
«DataLayer» «<DataLayer... «Spring» «ServiceLay...
HibernateSession RegistryDaoImpl BeanFactory RegistryServ ice
«DataLayer» «DataLayer» «ServiceLayer»
DataChunk RegistryDao RegistryServ iceImpl
«DataLayer»
PersistedObj ect
21

22. Some of the Lessons Learned (So Far)
MySQL vs Postgres
Human-readable vs automated URI formation
Wicket vs Struts
Hibernate query formation
Patterns
XML binding
CSSDP - NEP 21 22

23. More Lessons Learned (So Far)
Scrum and kanban (BVC)
UI prototyping
Unit vs integration testing
Administration and operations
External dependencies
Make it work, make it work better
CSSDP - NEP 21 23

24. Outstanding Questions
Web service file transfer
– MTOM? Redirection?
REST vs WS-* (or both)
– SPASE vs IVOA, OGSA
Workflow engine
– Scripting language? Taverna? ESB++?
CSSDP - NEP 21 24