Recommended
More Related Content
What's hot
What's hot (20)
Similar to Ruby on bioinformatics
Similar to Ruby on bioinformatics (20)
More from Tse-Ching Ho
More from Tse-Ching Ho (9)
Recently uploaded
Recently uploaded (20)
Ruby on bioinformatics
- 1. Ruby Conference Taiwan 2014 Ruby on Bioinformatics Tse-Ching Ho ! 何澤清! @tsechingho! 2014 / 4 / 26
- 2. Horse + Stripe = Zebra
- 4. Age of Big Data
- 5. Age of Data Science
- 6. High Through Put Data ❖ Big Data! ❖ file size is small but there are many files! ❖ file size is large but there are just few files! ❖ Data size of bioinformatics! ❖ 1,000,000,000 records for a subject (person) is normal
- 7. The Storage Demand is Increasing from Dr. Yu-Tai Wang
- 8. Data Size of Sequencing After 5 Years https://www.nanoporetech.com 70,000 New Born Baby X 500 GB = 35 TB 30,000 patients X 10,000 cells X 500 GB = 1.5 X 1011 GB = 150 EB from Dr. Yu-Tai Wang 1. count by current NGS data! 2. not include civil medical institutes
- 9. Computing Power is Required ❖ HPC! ❖ Infiniband cluster! ❖ Amazon EC2 cluster! ❖ Hadoop cluster! ❖ Many cores of CPU! ❖ Large Memory! ❖ High IO efficiency http://arstechnica.com/business/2012/05/amazons-hpc-cloud-supercomputing-for-the-99/
- 10. http://arstechnica.com/business/2012/04/4829-per-hour-supercomputer-built-on-amazon-cloud-to-fuel-cancer-research/ $4,828.85 per hour 51,132 cores, 58.78TB RAM 6,742 Amazon EC2 instances 2012! Protein simulation! Cycle Computing System! Ganglia HPC clusters! Deployed by Opscode Chef
- 11. http://www.hpcwire.com/2013/07/08/infiniband_snaps_up_strong_super_share/ Is 10 GB network enough for I/O? embarrassingly parallel: The calculations are independent of each other.
- 12. http://glennklockwood.blogspot.tw/2013/12/high-performance-virtualization-sr-iov_14.html Infiniband is good at I/O efficiency • Interconnect speed.! • I/O performance.! • Infiniband system is about 3.8GB/s of Bandwidth.! • 10 GB network is about 400MB/s of Bandwidth.
- 13. Data science is about DATA!
- 14. Data Scientist Concerns ❖ Data quality! ❖ Factors of filter! ❖ Statistics! ❖ Visualization! ❖ Interpretation
- 15. Programmer also Concerns ❖ High through put data (Big Data) handling! ❖ Data format / File format! ❖ Data parsing! ❖ Statistic tools! ❖ Visualization! ❖ Profit / Markets
- 16. Biology
- 17. http://businessintelligence.com/bi-insights/the-personalized-medicine-revolution-is-almost-here/
- 18. A Dream of Personalized Medicine from Dr. Yen-Hua Huang
- 20. Cure by Medicines http://scienceroll.com/2008/04/25/personalized-medicine-real-clinical-examples/
- 26. No Teach for Reading DNA http://intellimedix.com
- 27. Do The Right Things http://www.dnadirect.com
- 28. http://biodbnet.abcc.ncifcrf.gov/dbInfo/netGraph.php ID mapping of Databases Each node is a database.! Each database has it’s unique id.! These ids connected as a network.! I think handling these complexity should be easy for the people seating here.
- 29. Bioinformatics Sites for Rubists
- 36. Ruby Sites for Bioinformatists
- 43. What programming language is best for a bioinformatics beginner?
- 44. Mapping Sequence Data from Jui-Tse Hsu
- 45. Simple Mapping Sequence Data Convert to SAM Compress to BAM Index, Sort, Remove duplicate PCR (Rmdup) 1. .seq -> fastq 2. Illumina score -> Phred score 1. cleaned bam file 2. quality control, get statistics, mapped, unmapped, etc. 1. SNVs in VCFs 2. structural variants 3. copy number changes, etc. Aligner (soap2, bwa, bowtie, etc.) from Jui-Tse Hsu Illumina Exome sequence reads Aligned reads Aligned reads! (sam file) Aligned reads! (bam file) Useful reads data Call variants Visualization in browsers
- 46. C/C++ ❖ Key Algorithms! ❖ Written by C/C++! ❖ Foundation Tools! ❖ BWA! ❖ Bowtie / Bowtie2! ❖ samtools / bamtools! ❖ GMAP / GSNAP! ❖ BLAT! ❖ Tophat
- 47. http://genomebiology.com/2010/11/12/220 Analysis Pipeline Overview of the RNA-seq analysis pipeline for detecting differential expression
- 48. Perl ❖ First language! ❖ Bioperl! ❖ Ensembl http://millionchimpanzees.blogspot.tw/2011/09/book-review-learning-perl-sixth-edition.html
- 49. Java ❖ good part of java! ❖ GATK! ❖ Taverna! ❖ Hadoop http://shop.oreilly.com/product/9780596803742.do
- 50. R ❖ Statistic tools! ❖ Bioconductor! ❖ EdgeR! ❖ Data Mining and Analysis Books http://exploringdata.github.io/data-visualization-books/analysis/
- 51. Python ❖ young people! ❖ Galaxy http://news.oreilly.com/2008/08/python-for-unix-and-linux-syst.html
- 52. The Ruby Way in Bioinformatics
- 53. What kinds of libraries would you think it is important?
- 54. Foundation gems ❖ activerecord! ❖ nokogiri! ❖ ffi! ❖ parallel! ! ! ❖ bioruby! ❖ sciruby! ! ! !
- 55. C binding & wrapper ❖ bio-samtools! ❖ bio-bwa! ❖ bio-affy! ❖ bio-faster! ❖ mpi-ruby! ❖ bio-grid! ❖ gsl! ❖ rb-gsl! ❖ nmatrix! ❖ sambamba - D language! !
- 56. Data parser / analyser ❖ bio-genomic-interval! ❖ bio-blastxmlparser! ❖ bio-assembly! ❖ bio-gff3! ❖ bio-gff3-pltools! ❖ bio-alignment! ❖ bio-maf! ❖ bio-table! ❖ bio-rdf! ❖ bio-vcf! ❖ bio-velvet! ❖ bio-gngm! ❖ bio-gag! ❖ bio-dbsnp
- 57. Data parser / analyser ❖ bio-phyloxml! ❖ bio-jplace! ❖ bio-gex! ❖ bio-ipcress! ❖ bio-stockholm! ❖ bio-synreport! ❖ bio-cigar! ❖ bio-wolf_psort_wrapper! ❖ bio-hmmer3_report! ❖ bio-dbla-finder! ❖ bio-newbler_outputs! ❖ bio-sra_fastq_dumper! ❖ bigbio!
- 58. Data parser / analyser - protein ❖ protk! ❖ mascot-dat! ❖ bio-protparam! ❖ bio-plasmoap! ❖ bio-signalp! ❖ bio-exportpred! ❖ bio-hydropathy! ❖ bio-epitope! ❖ bio-bio-orthomcl! ❖ bio-isoelectric_point! ❖ bio-octopus! ❖ bio-tm_hmm! ❖ bio-aliphatic_index!
- 59. Database / Web API ❖ ruby-ensembl-api! ❖ bio-ucsc-api! ❖ bio-liftover! ❖ intermine! ❖ bio-eupathdb! ❖ bio-krona! ❖ bio-sra! ❖ bio-sradl http://www.ensembl.org
- 60. Statistics ❖ statsample! ❖ statsample-sem! ❖ statsample-optimization! ❖ statsample-timeseries! ❖ distribution! ❖ rinruby http://www.ncss.com/software/ncss/survival-analysis-in-ncss
- 61. SVG & Graph ❖ rubyvis! ❖ plotrb! ❖ bio-svgenes! ❖ bio-vis! ❖ gnuplot http://rubyvis.rubyforge.org
- 62. Tools ❖ minimization! ❖ integration! ❖ quorum - rails engine
- 63. I am Not Analyst, I am Programmer.
- 64. What can I get involved?
- 65. Pipeline / Workflow Galaxy - python! Taverna - java! ??? - Ruby
- 66. Web System ❖ Data warehouse! ❖ Pipeline management! ❖ Coordination center! ❖ Visualisation
- 67. Cloud / Distributed / Parallel http://www.mynamesnotmommy.com/yes-there-are-dumb-questions/question-mark/
- 68. What We Are Doing By Ruby?
- 69. Ensembl Virtual Machine ❖ Powered by VeeWee, Vagrant and Chef! ❖ Automatic build versioned Ensembl system (perl)! ❖ Include database, queuing services and analysis tools! ❖ Multi sites, multi species in one virtual machine! ❖ Help to build local & custom system from Tse-Ching Ho
- 70. Ensembl Virtual Machine Use existed vagrant box Prepare SOP for Chef recipes Provision VM with Chef recipes Write Chef recipes Export VM by Virtualbox Setup Vagrantfile Create Vagrant box by Veewee Write definition of Vagrant box by Veewee Ensembl VM Automation from Tse-Ching Ho
- 71. Ensembl Virtual Machine Web view of Ensembl from Tse-Ching Ho
- 72. DR. RAW ❖ Derived from DRAW and SneakPeek! ❖ Composed of C/C++, bash, perl, java, ruby! ❖ Have both DNA and RNA re-sequence analysis! ❖ Enhanced quality control for DNA and RNA! ❖ Distributed computing pipeline! ❖ Support PBS, LSF, SGE platforms (queuing system) from Hannah Lin
- 73. DR. RAW Analysis Tools Analysis Pipeline Quality Control Resource Manager System DNA QC Forward : Reverse RNA QC! Forward : Reverse BWA-0.7.7! Samtools-0.1.19! GATK-3.1 GSNAP-13-10-25! Cufflink-13-11! FusionGene … DNA Sequencing data RNA Sequencing data SGE (Sun Grid Engine) PBS (Portable Batch System)! LSF (Load Sharing Facility) Green: new components! Red: updated components from Hannah Lin
- 74. DR. RAW Web view by Rails from Hannah Lin
- 75. Neo4j - JRuby Data Parser ❖ Graph database for data integration of discrete clinical research documents! ❖ Origin data are excel/csv files collected in different time, by different people! ❖ Neo4j is good for cleanup such massive data set! ❖ Cooperation between biologist and programmer from Wei-Ming Wu, Chia-Hsuan Lee
- 76. Neo4j - JRuby Data Parser from Wei-Ming Wu, Chia-Hsuan Lee
- 77. Neo4j - JRuby Data Parser from Wei-Ming Wu, Chia-Hsuan Lee Collision Rate of Input Data: 1.3 %
- 78. API Server for Third Party Firm ❖ API server based on Rails, run by JRuby! ❖ ActiveRecord models for Oracle database! ❖ activerecord-oracle_enhanced-adapter gem! ❖ Import excel files to third party GUI client ! ❖ Third party server send XML request to API server from Wei-Ming Wu, Sean Wang
- 79. API Server for Third Party Firm TCHC server API server (rails, jruby) CSIS (java, oracle) Send data by XML Write into database Read data by client program Upload data Parse request Third Party Our Servers Windows GUI from Wei-Ming Wu, Sean Wang
- 80. Daily Checking Rule ❖ Based on Rails, run by JRuby! ❖ ActiveRecord models for Oracle database! ❖ activerecord-oracle_enhanced-adapter gem! ❖ User can define rules for checking data, usually values in filled forms! ❖ Run checking rules daily, not before filling forms from Wei-Ming Wu, Sean Wang
- 81. Daily Checking Rule from Wei-Ming Wu, Sean Wang
- 82. Daily Checking Rule from Wei-Ming Wu, Sean Wang
- 83. Daily Checking Rule from Wei-Ming Wu, Sean Wang
- 84. Daily Checking Rule from Wei-Ming Wu, Sean Wang
- 85. Patient Randomization ❖ Based on Rails, run by JRuby! ❖ ActiveRecord models for Oracle database! ❖ activerecord-oracle_enhanced-adapter gem! ❖ Assign patients into different groups by randomization method! ❖ Cooperation between statistician and programmer from Wei-Ming Wu, Sean Wang
- 86. Patient Randomization from Wei-Ming Wu, Sean Wang
- 87. Patient Randomization from Wei-Ming Wu, Sean Wang
- 88. Patient Randomization from Wei-Ming Wu, Sean Wang Assign patients to treatment groups
- 89. Database Statistics Dashboard ❖ Based on Rails, run by JRuby! ❖ ActiveRecord models for Oracle database! ❖ activerecord-oracle_enhanced-adapter gem! ❖ google_visualr gem for visualization! ❖ Count number of projects, forms, fields, records and patients from Wei-Ming Wu, Winnie Lui
- 90. Database Statistics Dashboard from Wei-Ming Wu, Winnie Lui
- 91. Education
- 92. Learning Bioinformatics ❖ http://www.nature.com/nbt/journal/v31/n11/full/ nbt.2740.html! ❖ http://www.liacs.nl/~hoogeboo/mcb/ nature_primer.html! ❖ http://www.mygoblet.org - python, R! ❖ http://www.biotnet.org
- 94. Python Book for Bioinformatics http://shop.oreilly.com/product/9780596154516.do
- 95. Python is very successful in Teach than Ruby
- 96. Do we lack a killer application by Ruby? http://www.witardroadbaptist.org/im-new/im-not-sure-im-ready-for-church-yet/
- 97. We Need Human !!
- 98. Are You Ready To Be A Data Scientist Or A Binformactis?
- 99. Markets http://www.genengnews.com/gen-articles/personalized-medicine-health-economic-aspects/4824/
- 100. http://www.bccresearch.com/market-research/biotechnology/bioinformatics-market-technology-bio051b.html Under developing Do Asia have enough market sharing?
- 101. Topics to take in action ❖ data generation and data management! ❖ data analysis and software! ❖ data processing and storage! ❖ application of bioinformatics in pharma research and development http://www.giichinese.com.tw/report/bc268909- bioinformatics-technologies-global-markets.html
- 102. Health Care in Cloud ❖ Health promotion cloud! ❖ Vaccination cloud! ❖ Exercise cloud! ❖ Workplace wellness! ❖ Physical checkup cloud! ❖ Welfare cloud from Dr. Chi-Hung Lin
- 104. Q & A