Next-generation sequencing: Data mangement
•Download as ODP, PDF•
4 likes•3,588 views
Next-generation sequencing is producing vast amounts of data. Providing storage and compute is only half the battle. Researchers and IT staff need to be able to "manage" data, in order to stay productive. Talk given at BIO-IT World, Europe 2010.
![Next-Gen Sequencing: Data Management Guy Coates Wellcome Trust Sanger Institute [email_address]](https://image.slidesharecdn.com/bioit-2010-110301083353-phpapp02/85/Next-generation-sequencing-Data-mangement-1-320.jpg)
![About the Institute ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Next-generation sequencing: Data mangement
Similar to Next-generation sequencing: Data mangement (20)
Recently uploaded
Recently uploaded (20)
Next-generation sequencing: Data mangement
- 1. Next-Gen Sequencing: Data Management Guy Coates Wellcome Trust Sanger Institute [email_address]
- 6. The Scary Graph Instrument upgrades Peak Yearly capillary sequencing
- 9. Sequencing cost: T d =12 months
- 15. Historically sequencing and IT were budgeted separately.
- 16. Makes Pis aware of the IT costs, even if it does not cover 100%.
- 18. Assume from day 1 we will be adding more.
- 20. Currently using LSF to manage workflow. LSF Fast scratch disk Archival / Warehouse disk Network
- 25. Sequencing data flow. Automated processing and data management Sequencer Analysis/ alignment Internal repository EGA / SRA (EBI) compute-farm High-performance storage Manual data movement
- 27. Are we keeping control of our “private” datasets?
- 33. Groups need to exchange data.
- 35. Sequencing data flow. Automated processing and data management Manual Sequencer Analysis/ alignment Internal repository EGA / SRA (EBI) compute-farm High-performance storage Managed data movement
- 37. iRODS ICAT Catalogue database Rule Engine Implements policies Irods Server Data on disk User interface WebDAV, icommands,fuse Irods Server Data in database Irods Server Data in S3
- 39. Replicates data.
- 41. First implementation Automated processing and data management Manual Sequencer Analysis/ alignment Internal repository EGA / SRA (EBI) compute-farm High-performance storage
- 45. Example access: $ icd /seq/5307 $ ils /seq/5307: 5307_1.bam 5307_2.bam 5307_3.bam $ ils -l 5307_1.bam srpipe 0 res-g2 1987106409 2010-09-24.13:35 & 5307_1.bam srpipe 1 res-r2 1987106409 2010-09-24.13:36 & 5307_1.bam
- 46. Metadata imeta ls -d /seq/5307/5307_1.bam AVUs defined for dataObj /seq/5307/5307_1.bam: attribute: type value: bam units: ---- attribute: sample value: BG81 units: ---- attribute: id_run value: 5307 units: ---- attribute: lane value: 1 units: ---- attribute: study value: TRANSCRIPTION FACTORS IN HAEMATOPOIESIS - MOUSE units: ---- attribute: library value: BG81 449223 units:
- 47. Query imeta qu -d study = "TRANSCRIPTION FACTORS IN HAEMATOPOIESIS - MOUSE" collection: /seq/5307 dataObj: 5307_1.bam ---- collection: /seq/5307 dataObj: 5307_2.bam ---- collection: /seq/5307 dataObj: 5307_3.bam ----
- 48. So what...?
- 49. Next steps Sanger iRODs Datacentre 2 Datacentre 1 Replicate EGA/ERA Automated release/purge Collaborator iRODs Federate
- 50. Wishlist: HPC Integration Data is staged in/out to filesystem Archive / Metadata system Fast Storage / POSIX filesystem Compute farm Fast Storage / POSIX filesystem + Metadata sytem Compute farm System can do rule/metadata based ops and standard POSIX ops too.
- 53. Storage and servers spread across several locations. Fast link Storage Storage Storage Storage CPU CPU CPU CPU CPU medium link slow link
- 58. Hot datasets change over time.
- 61. Phil Butcher
- 63. Gen-Tao Chiang
- 64. Pete Clapham
- 66. Chris Duddington
- 67. Da Xu