SlideShare una empresa de Scribd logo
1 de 46
Descargar para leer sin conexión
📦 Active Storage - Modern File Storage? 📦
by Michael Yagudaev
About Michael
• Co-Founder @ Nano 3 Labs & PictureThat

• Full-stack Developer Rails/React

• Been working with Rails since 2012

• Web dev since 2001

• Passionate about Product Design & AR

• Fun Fact: offline snowboarding 🏂
Outline
• Battle Stories: file upload is hard…
• What is Modern File Storage?
• What is Active Record?
• Uploading a File
• Transforming a File
• Downloading/Previewing a File
• Alternatives to Active Storage
• Conclusion
Battle Stories
Story 1: Bail on Rails
• Back in 2012 (Rails 2.3) — small startup, terrible uploader. Users upload
entire hi-res albums (100s of photos). Blocking all other users from
doing anything on the platform! :(
• Sales staff hides in hope of not running into angry users
• Can you build us a better file uploader?
• Me: yes! Let’s use Node.js + Direct Upload to S3
• Me 3 months later: FML… node… ;(
• We created a robust uploader
• auto-retry failed uploads, priority resize images users requested,
pre-create resize jobs for all images sizes, custom uploader ui
Story 2: Rails re-discovered
• 0idle - our own startup helping event organizers find venues
• 2013: implemented multi-file upload
• Carrierwave + Carrierwave-Direct S3
• Only took a few days
• ❤ Rails
• Life is good :)
• Custom upload UI
Story 3: DDoS Day
• Summer 2018 - implemented ActiveStorage for PictureThat. Loved
the variants feature
• Fall 2018 -
• DDoS attack on one of our projects
• Targeted slow pdf conversation end-points
• Made us look into slow running requests
• Second slowest was asset upload
• A DevOps Engineer I was working with outlined “Modern File
Storage” to me… it got me thinking, is Active Storage Modern?
What is Modern File Storage?
• Modular
• Supports cloud providers
• Fast
• Scalable
• Cheap
• Easy to setup
• API Based
What is ActiveStorage?
• Extracted from BaseCamp
• Supports
• Cloud Providers (S3, digital ocean, GCP, Azure)
• Variants
• Mirroring
• Previews
• Accessible MetaData* (sorta we will see soon)
• Upload: documents, images and videos (yes!)
File Life Cycle
File Life Cycle
1. select file(s)
2. upload file
3. analyze file
4. transform file
5. preview / download
$ bundle open activestorage
File Life Cycle - AS Mapping
1. select file(s) - rails form / bring your own ui
2. upload file - activestorage/app/controllers/
3. analyze file - activestorage/jobs/…/analyze_job.rb
4. transform file - activestorage/lib/…/
transformers.rb
5. preview / download - activestorage/models/…/
variant.rb (or) activestorage/models/…/preview
1. File Selection
1. Select File(s) - Traditional
• Easy with rails forms
• Add <%= form.file_field :files, multiple: true,%>
• For S3 Direct Upload (work with Digital Ocean, GCP,
etc)
• <%= form.file_field :files, multiple: true, direct_upload:
true,%>
• Use direct upload, no long requests
Rails Form File Field Be Like…
1. Select File(s) - BYOU
• Use modern Javascript
framework like React
• Try uppy.io
• Can work as Multi-part
XHR Upload
• But a two-step API based
upload is better
• API should support upload
from iOS and Android apps
2. Uploading
2. Uploading
• Considerations
• size of files (resumable?)
• # files
• types of files
• file selection method in step #1
• Offload away from Rails, it wasn’t meant for it
• Maintain short connection timeouts. Let S3 handle upload
Multipart form file upload
• Sends metadata & file content in one request
• Chunks file and sends using a boundary, looks like
• Content-Type: multipart/form-data; boundary=——
WebKitFormBoundarydmmGebTloNbZGViV
• Old and meant for forms, not APIs
• Can be used with XHR request
2-Step API Upload
• Instead use 2 requests
1. Upload Meta data (file name, size, type, etc)
2. Get Location back from step #1 for direct upload
• Active Storage Direct does this!
• POST http://localhost:3000/rails/active_storage/direct_uploads
• { blob: { filename: …, content_type: “image/jpeg”, byte_size:
123, checksum: “aoEPYm6I0kn9sFqekqy5yg==" }}
• RESPONSE: { direct_upload:{url:”<s3_signed_url>”, headers: {…}} }
AS Schema
• Run: $ rails active_storage:install
• active_storage_blobs => Meta data about file
• byte_size
• content_type
• checksum (md5 as base64)
• metadata (height, width, etc)
• active_storage_attachments => Relationship between your models and blobs
• blob_id
• record_id
• record_type (polymorphic association)
Custom API
• Simple all you need is
• Then just
• POST /api/v1/files { blob: { filename, content_type,
byte_size, checksum } }
class	Api::V1::FilesController	<	ActiveStorage::DirectUploadsController	
		skip_forgery_protection	
end
3. Analyzing
3. Analyze
• File has already been uploaded
• File is in temp folder or cloud storage
• If in temp
• Easy, run appropriate tool like`identify` on images
• Compare meta to client received ones
• Save meta data back
3. Analyze
• If on cloud storage (s3), either:
• Download to tmp file, and run analysis locally (AS)
• Run cloud analysis and connect to DB + update or update
through a webhook (no help from AS here)
• Ideally run background job / cloud function
• Might want to run things like a virus scan
• ActiveStorage has an AnalyzeJob, use it
• What happens if user requests resource right away?
4. Transformations
4. Transformation
• Resize images, extract display frame from video
• Either pre-process or lazy-evaluate (see step 5)
• Active Storage uses background jobs
• Alternatively, cloud function + cloud storage
• But…
Active Storage on S3…
ATM: No way to customize paths for human readable names :(
no file extensions?!
5. Downloading / Previewing
5. Download / Preview
• Viewing the original is simple
• <%= image_tag @user.avatar %>
• To get S3 url: @user.avatar.service_url
• Return it in your API
Displaying Transformed
Version
• upload.variable? (i.e. images)
• png, jpeg, etc (not SVG)
• upload.previewable? (i.e. can get an image from them)
• pdf, mpeg (video), etc
Variants
• only applies to images, can change size
• <%= image_tag @user.avatar.variant(resize:
“400x400”) %>
• 💡 dynamically generate an image size based on
device
• to link to full-size image simply do:
• <%= link_to image_tag(@user.avatar.variant(resize:
“400x400”)), @user.avatar %>
PictureThat AR
App
PictureThat Backend
PictureThat Backend
Variants
• ActiveStorage can transform(resize, etc) documents on
the fly
• in process
• bad performance
• works for small images only (due to connection
timeouts)
• in background process
• dispatches a resize job
File Previews
• As easy as this!
• <%= image_tag file.preview(resize: '400x400') %>
• Works for documents: pdf, videos!
• Can build your own custom previewers
Behind the Scenes
• hides your files behind a rails route
• /rails/active_storage/representations/a84…3b/file_name.jpg
• great for
• switching between environments or mirrors
• dynamically generating sizes
• no more size having to call resize on all images in the database
• lazily evaluated
• bad for
• performance, another call to rails as opposed to going to nginx or directly to S3
Advanced Transformation
• none of these are supported by ActiveStorage
• on-the-fly resize with s3
• AWS ServerLess-Thumbor -> https://github.com/
awslabs/serverless-image-handler
• http://thumbor-server/filters:watermark(http://
my.site.com/img.png,10p,-20p,50)/some/image.jpg
Alternative: Shrine
• Good
• Been around for longer
• Retains file extension on storage
• Customizable with Plugins
• Supports Cloud Processing using Plugins
• Works with uppy.io out of the box
• Bad
• More work to setup
Alternative: No Rails…
• Firebase Storage => easy to setup from an JS frontend
• Storage as a Service:
• https://transloadit.com/
• https://www.filestack.com/
• https://cloudinary.com
Conclusion
Conclusion
• ActiveStorage shows potentials and works for simple
cases (Admin interfaces)
• For all other, start with Shrine
• Don’t over-optimize. But keep your options open
References
• https://edgeguides.rubyonrails.org/active_storage_overview.html
• https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2
• https://gorails.com/episodes/how-to-create-an-active-storage-previewer
• https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/
• https://medium.com/typecode/a-strategy-for-handling-multiple-file-uploads-using-javascript-
eb00a77e15f
• https://www.netlify.com/blog/2016/11/17/serverless-file-uploads/
• https://bibwild.wordpress.com/2018/10/03/some-notes-on-whats-going-on-in-activestorage/
• https://github.com/rails/rails/issues/31419
• https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-
lambda-and-amazon-api-gateway/
• https://github.com/texpert/shrine-lambda
Questions?
• twitter: @yagudaev, @nano3labs
• come talk shop, tell me about your Rails project
• Our friends @spacelist are hiring remote F/T senior
ruby/rails dev

Más contenido relacionado

La actualidad más candente

The WordPress University
The WordPress UniversityThe WordPress University
The WordPress UniversityStephanie Leary
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryAlek Davis
 
Riding the Edge with Ember.js
Riding the Edge with Ember.jsRiding the Edge with Ember.js
Riding the Edge with Ember.jsaortbals
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your WebsiteAcquia
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryAlek Davis
 
The things we found in your website
The things we found in your websiteThe things we found in your website
The things we found in your websitehernanibf
 
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages ApplicationsIBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages Applicationsbeglee
 
Georgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal DevelopmentGeorgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal DevelopmentEric Sembrat
 
Play Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a ProposalPlay Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a ProposalMike Slinn
 
My site is slow
My site is slowMy site is slow
My site is slowhernanibf
 
Extreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NYExtreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NYMaximiliano Firtman
 
Introduction to Apache Roller
Introduction to Apache RollerIntroduction to Apache Roller
Introduction to Apache RollerMatt Raible
 
Flexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework WorldFlexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework WorldKevin Ball
 
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012Jagadish Prasath
 
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...Matt Raible
 
XPages: No Experience Needed
XPages: No Experience NeededXPages: No Experience Needed
XPages: No Experience NeededKathy Brown
 
User-percieved performance
User-percieved performanceUser-percieved performance
User-percieved performanceMike North
 
Tech Talk on Cloud Computing
Tech Talk on Cloud ComputingTech Talk on Cloud Computing
Tech Talk on Cloud ComputingITviec
 
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...Lucidworks
 

La actualidad más candente (20)

The WordPress University
The WordPress UniversityThe WordPress University
The WordPress University
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
Riding the Edge with Ember.js
Riding the Edge with Ember.jsRiding the Edge with Ember.js
Riding the Edge with Ember.js
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your Website
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
The things we found in your website
The things we found in your websiteThe things we found in your website
The things we found in your website
 
A Day of REST
A Day of RESTA Day of REST
A Day of REST
 
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages ApplicationsIBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
 
Georgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal DevelopmentGeorgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal Development
 
Play Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a ProposalPlay Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a Proposal
 
My site is slow
My site is slowMy site is slow
My site is slow
 
Extreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NYExtreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NY
 
Introduction to Apache Roller
Introduction to Apache RollerIntroduction to Apache Roller
Introduction to Apache Roller
 
Flexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework WorldFlexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework World
 
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
 
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
 
XPages: No Experience Needed
XPages: No Experience NeededXPages: No Experience Needed
XPages: No Experience Needed
 
User-percieved performance
User-percieved performanceUser-percieved performance
User-percieved performance
 
Tech Talk on Cloud Computing
Tech Talk on Cloud ComputingTech Talk on Cloud Computing
Tech Talk on Cloud Computing
 
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
 

Similar a Active Storage vs Modern File Storage

GraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platformGraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platformNeo4j
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQLKonstantin Gredeskoul
 
Training on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin SharmaTraining on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin SharmaMobileNepal
 
Mobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressMobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressDanilo Ercoli
 
GeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPressGeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPressGGDBologna
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?DATAVERSITY
 
Static Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource ConditionStatic Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource ConditionIWMW
 
Introduction to Android Development and Security
Introduction to Android Development and SecurityIntroduction to Android Development and Security
Introduction to Android Development and SecurityKelwin Yang
 
Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012Lee Klement
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesBrett Meyer
 
Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...boxuno
 
Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1Tuenti
 
Optimization of modern web applications
Optimization of modern web applicationsOptimization of modern web applications
Optimization of modern web applicationsEugene Lazutkin
 
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2Richard Esplin
 
Ship It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemShip It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemYi-Ting Cheng
 
Django Overview
Django OverviewDjango Overview
Django OverviewBrian Tol
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talksRuslan Meshenberg
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Derek Jacoby
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI
 

Similar a Active Storage vs Modern File Storage (20)

GraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platformGraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platform
 
Rails scaling
Rails scalingRails scaling
Rails scaling
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL
 
Training on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin SharmaTraining on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin Sharma
 
Mobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressMobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPress
 
GeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPressGeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPress
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
Static Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource ConditionStatic Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource Condition
 
Introduction to Android Development and Security
Introduction to Android Development and SecurityIntroduction to Android Development and Security
Introduction to Android Development and Security
 
Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
 
Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...
 
Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1
 
Optimization of modern web applications
Optimization of modern web applicationsOptimization of modern web applications
Optimization of modern web applications
 
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
 
Ship It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemShip It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails Ecosystem
 
Django Overview
Django OverviewDjango Overview
Django Overview
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Untangling - fall2017 - week 9
Untangling - fall2017 - week 9
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 

Último

Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 

Último (20)

Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 

Active Storage vs Modern File Storage

  • 1. 📦 Active Storage - Modern File Storage? 📦 by Michael Yagudaev
  • 2. About Michael • Co-Founder @ Nano 3 Labs & PictureThat • Full-stack Developer Rails/React • Been working with Rails since 2012 • Web dev since 2001 • Passionate about Product Design & AR • Fun Fact: offline snowboarding 🏂
  • 3. Outline • Battle Stories: file upload is hard… • What is Modern File Storage? • What is Active Record? • Uploading a File • Transforming a File • Downloading/Previewing a File • Alternatives to Active Storage • Conclusion
  • 5. Story 1: Bail on Rails • Back in 2012 (Rails 2.3) — small startup, terrible uploader. Users upload entire hi-res albums (100s of photos). Blocking all other users from doing anything on the platform! :( • Sales staff hides in hope of not running into angry users • Can you build us a better file uploader? • Me: yes! Let’s use Node.js + Direct Upload to S3 • Me 3 months later: FML… node… ;( • We created a robust uploader • auto-retry failed uploads, priority resize images users requested, pre-create resize jobs for all images sizes, custom uploader ui
  • 6. Story 2: Rails re-discovered • 0idle - our own startup helping event organizers find venues • 2013: implemented multi-file upload • Carrierwave + Carrierwave-Direct S3 • Only took a few days • ❤ Rails • Life is good :) • Custom upload UI
  • 7. Story 3: DDoS Day • Summer 2018 - implemented ActiveStorage for PictureThat. Loved the variants feature • Fall 2018 - • DDoS attack on one of our projects • Targeted slow pdf conversation end-points • Made us look into slow running requests • Second slowest was asset upload • A DevOps Engineer I was working with outlined “Modern File Storage” to me… it got me thinking, is Active Storage Modern?
  • 8. What is Modern File Storage? • Modular • Supports cloud providers • Fast • Scalable • Cheap • Easy to setup • API Based
  • 9. What is ActiveStorage? • Extracted from BaseCamp • Supports • Cloud Providers (S3, digital ocean, GCP, Azure) • Variants • Mirroring • Previews • Accessible MetaData* (sorta we will see soon) • Upload: documents, images and videos (yes!)
  • 11. File Life Cycle 1. select file(s) 2. upload file 3. analyze file 4. transform file 5. preview / download
  • 12. $ bundle open activestorage
  • 13. File Life Cycle - AS Mapping 1. select file(s) - rails form / bring your own ui 2. upload file - activestorage/app/controllers/ 3. analyze file - activestorage/jobs/…/analyze_job.rb 4. transform file - activestorage/lib/…/ transformers.rb 5. preview / download - activestorage/models/…/ variant.rb (or) activestorage/models/…/preview
  • 15. 1. Select File(s) - Traditional • Easy with rails forms • Add <%= form.file_field :files, multiple: true,%> • For S3 Direct Upload (work with Digital Ocean, GCP, etc) • <%= form.file_field :files, multiple: true, direct_upload: true,%> • Use direct upload, no long requests
  • 16. Rails Form File Field Be Like…
  • 17. 1. Select File(s) - BYOU • Use modern Javascript framework like React • Try uppy.io • Can work as Multi-part XHR Upload • But a two-step API based upload is better • API should support upload from iOS and Android apps
  • 19. 2. Uploading • Considerations • size of files (resumable?) • # files • types of files • file selection method in step #1 • Offload away from Rails, it wasn’t meant for it • Maintain short connection timeouts. Let S3 handle upload
  • 20. Multipart form file upload • Sends metadata & file content in one request • Chunks file and sends using a boundary, looks like • Content-Type: multipart/form-data; boundary=—— WebKitFormBoundarydmmGebTloNbZGViV • Old and meant for forms, not APIs • Can be used with XHR request
  • 21. 2-Step API Upload • Instead use 2 requests 1. Upload Meta data (file name, size, type, etc) 2. Get Location back from step #1 for direct upload • Active Storage Direct does this! • POST http://localhost:3000/rails/active_storage/direct_uploads • { blob: { filename: …, content_type: “image/jpeg”, byte_size: 123, checksum: “aoEPYm6I0kn9sFqekqy5yg==" }} • RESPONSE: { direct_upload:{url:”<s3_signed_url>”, headers: {…}} }
  • 22. AS Schema • Run: $ rails active_storage:install • active_storage_blobs => Meta data about file • byte_size • content_type • checksum (md5 as base64) • metadata (height, width, etc) • active_storage_attachments => Relationship between your models and blobs • blob_id • record_id • record_type (polymorphic association)
  • 23. Custom API • Simple all you need is • Then just • POST /api/v1/files { blob: { filename, content_type, byte_size, checksum } } class Api::V1::FilesController < ActiveStorage::DirectUploadsController skip_forgery_protection end
  • 25. 3. Analyze • File has already been uploaded • File is in temp folder or cloud storage • If in temp • Easy, run appropriate tool like`identify` on images • Compare meta to client received ones • Save meta data back
  • 26. 3. Analyze • If on cloud storage (s3), either: • Download to tmp file, and run analysis locally (AS) • Run cloud analysis and connect to DB + update or update through a webhook (no help from AS here) • Ideally run background job / cloud function • Might want to run things like a virus scan • ActiveStorage has an AnalyzeJob, use it • What happens if user requests resource right away?
  • 28. 4. Transformation • Resize images, extract display frame from video • Either pre-process or lazy-evaluate (see step 5) • Active Storage uses background jobs • Alternatively, cloud function + cloud storage • But…
  • 29. Active Storage on S3… ATM: No way to customize paths for human readable names :( no file extensions?!
  • 30. 5. Downloading / Previewing
  • 31. 5. Download / Preview • Viewing the original is simple • <%= image_tag @user.avatar %> • To get S3 url: @user.avatar.service_url • Return it in your API
  • 32. Displaying Transformed Version • upload.variable? (i.e. images) • png, jpeg, etc (not SVG) • upload.previewable? (i.e. can get an image from them) • pdf, mpeg (video), etc
  • 33. Variants • only applies to images, can change size • <%= image_tag @user.avatar.variant(resize: “400x400”) %> • 💡 dynamically generate an image size based on device • to link to full-size image simply do: • <%= link_to image_tag(@user.avatar.variant(resize: “400x400”)), @user.avatar %>
  • 37. Variants • ActiveStorage can transform(resize, etc) documents on the fly • in process • bad performance • works for small images only (due to connection timeouts) • in background process • dispatches a resize job
  • 38. File Previews • As easy as this! • <%= image_tag file.preview(resize: '400x400') %> • Works for documents: pdf, videos! • Can build your own custom previewers
  • 39. Behind the Scenes • hides your files behind a rails route • /rails/active_storage/representations/a84…3b/file_name.jpg • great for • switching between environments or mirrors • dynamically generating sizes • no more size having to call resize on all images in the database • lazily evaluated • bad for • performance, another call to rails as opposed to going to nginx or directly to S3
  • 40. Advanced Transformation • none of these are supported by ActiveStorage • on-the-fly resize with s3 • AWS ServerLess-Thumbor -> https://github.com/ awslabs/serverless-image-handler • http://thumbor-server/filters:watermark(http:// my.site.com/img.png,10p,-20p,50)/some/image.jpg
  • 41. Alternative: Shrine • Good • Been around for longer • Retains file extension on storage • Customizable with Plugins • Supports Cloud Processing using Plugins • Works with uppy.io out of the box • Bad • More work to setup
  • 42. Alternative: No Rails… • Firebase Storage => easy to setup from an JS frontend • Storage as a Service: • https://transloadit.com/ • https://www.filestack.com/ • https://cloudinary.com
  • 44. Conclusion • ActiveStorage shows potentials and works for simple cases (Admin interfaces) • For all other, start with Shrine • Don’t over-optimize. But keep your options open
  • 45. References • https://edgeguides.rubyonrails.org/active_storage_overview.html • https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2 • https://gorails.com/episodes/how-to-create-an-active-storage-previewer • https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/ • https://medium.com/typecode/a-strategy-for-handling-multiple-file-uploads-using-javascript- eb00a77e15f • https://www.netlify.com/blog/2016/11/17/serverless-file-uploads/ • https://bibwild.wordpress.com/2018/10/03/some-notes-on-whats-going-on-in-activestorage/ • https://github.com/rails/rails/issues/31419 • https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws- lambda-and-amazon-api-gateway/ • https://github.com/texpert/shrine-lambda
  • 46. Questions? • twitter: @yagudaev, @nano3labs • come talk shop, tell me about your Rails project • Our friends @spacelist are hiring remote F/T senior ruby/rails dev