(PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014

•

32 likes•49,524 views

This session drills deep into the Amazon S3 technical best practices that help you maximize storage performance for your use case. We provide real-world examples and discuss the impact of object naming conventions and parallelism on Amazon S3 performance, and describe the best practices for multipart uploads and byte-range downloads.

Technology

in data transfer from S3
not including Amazon Web Services use

Architecture
Choosing a region
Building a naming scheme
Considering LISTs
Optimizing PUTs
Multipart upload
Demo
Optimizing GETs
Using CloudFront
Range-based GETs
Demo
Customer Case
BigDataCorp

Request Rate and Performance Considerations
http://amzn.to/18oF5LC
TIP

1
2
5
8
100/8 = 12.5 events/sec
100,000 users @ 10 events an hour = 224 TPS

<my_bucket>/2013_11_13-164533125.jpg
<my_bucket>/2013_11_13-164533126.jpg
<my_bucket>/2013_11_13-164533127.jpg
<my_bucket>/2013_11_13-164533128.jpg
<my_bucket>/2013_11_12-164533129.jpg
<my_bucket>/2013_11_12-164533130.jpg
<my_bucket>/2013_11_12-164533131.jpg
<my_bucket>/2013_11_12-164533132.jpg
<my_bucket>/2013_11_11-164533133.jpg
<my_bucket>/2013_11_11-164533134.jpg
<my_bucket>/2013_11_11-164533135.jpg
<my_bucket>/2013_11_11-164533136.jpg

1
2
N
1
2
N
Partition
Partition
Partition
Partition

<my_bucket>/521335461-2013_11_13.jpg
<my_bucket>/465330151-2013_11_13.jpg
<my_bucket>/987331160-2013_11_13.jpg
<my_bucket>/465765461-2013_11_13.jpg
<my_bucket>/125631151-2013_11_13.jpg
<my_bucket>/934563160-2013_11_13.jpg
<my_bucket>/532132341-2013_11_13.jpg
<my_bucket>/565437681-2013_11_13.jpg
<my_bucket>/234567460-2013_11_13.jpg
<my_bucket>/456767561-2013_11_13.jpg
<my_bucket>/345565651-2013_11_13.jpg
<my_bucket>/431345660-2013_11_13.jpg

•Store objects as a hash of their name
–add the original name as metadata
•“deadmau5_mix.mp3” 0aa316fb000eae52921aab1b4697424958a53ad9
–prependkeyname withshort hash
•0aa3-deadmau5_mix.mp3
•Epoch time (reverse)
–5321354831-deadmau5_mix.mp3

<my_bucket>/images/521335461-2013_11_13.jpg
<my_bucket>/images/465330151-2013_11_13.jpg
<my_bucket>/movies/293924440-2013_11_13.jpg
<my_bucket>/movies/987331160-2013_11_13.jpg
<my_bucket>/thumbs-small/838434842-2013_11_13.jpg
<my_bucket>/thumbs-small/342532454-2013_11_13.jpg
<my_bucket>/thumbs-small/345233453-2013_11_13.jpg
<my_bucket>/thumbs-small/345453454-2013_11_13.jpg

fasterflexibleset of partspresents all parts as a single objectparallelpausingresumingbeginning uploads before you know the total object size

DEMOAmazon CloudFrontvs. Amazon S3 download performance

DynamoDB
Amazon RDS
Amazon CloudSearch
Amazon EC2

Maestro
(Reserved Instance)
List of crawl
URLs
Main workers
Execute crawling and process data
Spot Instances
Secondary workers(queue listeners)
Reprocess data, query additional services, store data on MongoDB
Spot Instances
Secondary
work queues –
processed data
MongoDBcluster
Command and
Control Queue

gfelipe@amazon.comthoran@bigdatacorp.com.br

Please give us your feedback on this
presentation

What's hot

Getting Started with the Hybrid Cloud: Enterprise Backup and Recovery

Amazon Web Services

Getting started with amazon aurora - Toronto

Amazon Web Services

In this session, storage experts will walk you through Amazon S3 and Amazon Glacier, bulk data repositories that can deliver 99.999999999% durability and scale past trillions of objects worldwide – with cost points competitive against tape archives. Learn about the different ways you can accelerate data transfer into S3 and get a close look at new tools to secure and manage your data more efficiently. Hear about Amazon Glacier and new capabilities to get access to your data faster with expedited retrievals. Learn how AWS customers have built solutions that turn their data from a cost into a strategic asset, and bring your toughest questions straight to our experts.

SRV403 Deep Dive on Object Storage: Amazon S3 and Amazon Glacier

Amazon Web Services

Explore Amazon DynamoDB capabilities and benefits in detail and learn how to get the most out of your DynamoDB database. We go over best practices for schema design with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others. We explore designing efficient indexes, scanning, and querying, and go into detail on a number of recently released features, including JSON document support, DynamoDB Streams, and more. We also provide lessons learned from operating DynamoDB at scale, including provisioning DynamoDB for IoT.

SRV404 Deep Dive on Amazon DynamoDB

Amazon Web Services

ABCs of AWS: S3

Mark Cohen

Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance families. We will also provide an overview of the newest instances announced at re:Invent, including the latest generation of Memory and Compute Optimized Instances R4 and C5 instances, new Storage Optimized High I/O I3 instances, and new larger T2 instances. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances. Learning Objectives: • Get an overview of the EC2 instance platform, key platform features, and the concept of instance generations • Learn about the latest generation of Amazon EC2 Instances • Learn best practices around instance selection to optimize performance

Deep Dive on Amazon EC2 Instances - January 2017 AWS Online Tech Talks

Amazon Web Services

AWS Storage Gateway is a service that connects an on-premises software appliance with AWS storage. It simplifies the adoption of cloud-based storage within on-premises environments, giving customers a secure, reliable, and cost-effective alternative to local storage. In this session, we take a detailed look at how to use Storage Gateway to backup and archive on-premises data. We discuss the three types of storage and how to select the right type for your environment. We walk through setup and configuration of the on-premises gateway appliance, data restoration, and daily management, such as monitoring performance and managing storage. The session is intended for customers who perform on-premises backup and archive today, and want to learn how to include cloud storage in their environment.

(STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Amazon Web Services

Masterclass Live: Amazon EMR

Amazon Web Services

After we launched Amazon Aurora, a cloud-native relational database with region-wide durability, high availability, fast failover, up to 15 read replicas, and up to five times the performance of MySQL, many of you asked us whether we could deliver the same features - but with PostgreSQL compatibility. We are now delivering a preview of Amazon Aurora with this functionality: we have built a PostgreSQL-compatible edition of Amazon Aurora, sharing the core Amazon Aurora innovations with the object-oriented capabilities, language interfaces, JSON compatibility, ANSI:SQL:2008 compliance, and broad functional richness of PostgreSQL. Amazon Aurora will provide full PostgreSQL compatibility while delivering more than twice the performance of the community PostgreSQL database on many workloads. At this session, we will be discussing the newest addition to Amazon Aurora in detail.

NEW LAUNCH! Introducing PostgreSQL compatibility for Amazon Aurora

Amazon Web Services

"Amazon EMR provides a managed framework which makes it easy, cost effective, and secure to run data processing frameworks such as Apache Hadoop, Apache Spark, and Presto on AWS. In this session, you learn the key design principles behind running these frameworks on the cloud and the feature set that Amazon EMR offers. We discuss the benefits of decoupling compute and storage and strategies to take advantage of the scale and the parallelism that the cloud offers, while lowering costs. Additionally, you hear from AOL’s Senior Software Engineer on how they used these strategies to migrate their Hadoop workloads to the AWS cloud and lessons learned along the way. In this session, you learn the benefits of decoupling storage and compute and allowing them to scale independently; how to run Hadoop, Spark, Presto and other supported Hadoop Applications on Amazon EMR; how to use Amazon S3 as a persistent data-store and process data directly from Amazon S3; dDeployment strategies and how to avoid common mistakes when deploying at scale; and how to use Spot instances to scale your transient infrastructure effectively."

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

Amazon Web Services

Amazon Athena is a new interactive query service that makes it easy to analyze data in Amazon S3, using standard SQL. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing your data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3. In this session, we will show you how easy is to start querying your data stored in Amazon S3, with Amazon Athena. First we will use Athena to create the schema for data already in S3. Then, we will demonstrate how you can run interactive queries through the built-in query editor. We will provide best practices and use cases for Athena. Then, we will talk about supported queries, data formats, and strategies to save costs when querying data with Athena.

NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.

Amazon Web Services

Amazon DynamoDB is a fully managed NoSQL database service for applications that need consistent, single-digit millisecond latency at any scale. This talk explores DynamoDB capabilities and benefits in detail and discusses how to get the most out of your DynamoDB database. We go over schema design best practices with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others. We also explore designing efficient indexes, scanning, and querying, and go into detail on a number of recently released features, including JSON document support, Streams, and more.

Deep Dive on Amazon DynamoDB

Amazon Web Services

In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We’ll cover how each service might help support your application, how much each service costs, and how to get started.

Intro to AWS: Database Services

Amazon Web Services

If you are interested to know more about AWS Chicago Summit, please use the following to register: http://amzn.to/1RooPPL Amazon S3 and Amazon Glacier provide developers and IT teams with secure, durable, highly-scalable object storage with no minimum fees or setup costs. In this webcast, we will provide an introduction to each service, dive deep into key features of Amazon S3 and Amazon Glacier, and explore different use cases that these services optimize. Learning Objectives: • Business value of Amazon S3 and Amazon Glacier • Leveraging S3 for web applications, media delivery, big data analytics and backup • Leveraging Amazon Glacier to build cost effective archives • Understand the life cycle management of AWS' storage services

AWS May Webinar Series - Getting Started: Storage with Amazon S3 and Amazon G...

Amazon Web Services

Amazon Elastic MapReduce is one of the largest Hadoop operators in the world. Since its launch five years ago, AWS customers have launched more than 5.5 million Hadoop clusters. In this talk, we introduce you to Amazon EMR design patterns such as using Amazon S3 instead of HDFS, taking advantage of both long and short-lived clusters and other Amazon EMR architectural patterns. We talk about how to scale your cluster up or down dynamically and introduce you to ways you can fine-tune your cluster. We also share best practices to keep your Amazon EMR cluster cost efficient. Speakers: Ian Meyers, AWS Solutions Architect Ian McDonald, IT Director, SwiftKey

Deep Dive - Amazon Elastic MapReduce (EMR)

Amazon Web Services

In the first Webinar of the 2014 Masterclass Series AWS Technical Evangelist Ian Massingham dives deep into the Amazon Simple Storage Service, S3. He starts by providing an overview of the high level architecture of S3 and the fundamental characteristics of the service before moving on to take a tour through the various features of S3 including storage classes, namespaces, encryption, access controls, transitions and lifecycle management. He also covers related AWS services such as Glacier and the AWS content distribution network, CloudFront, as well as explaining how you can use Amazon S3 to serve static web content.

Masterclass Webinar - Amazon Simple Storage Service S3

Amazon Web Services

In this session, storage experts will walk you through Amazon S3 and Amazon Glacier, bulk data repositories that can deliver 99.999999999% durability and scale past trillions of objects worldwide - with cost points competitive against tape archives. Learn about the different ways you can accelerate data transfer into S3 and get a close look at new tools to secure and manage your data more efficiently. See how Amazon Athena runs "query in place" analytics on your data and hear about the new expedited and bulk retrievals from Amazon Glacier. Learn how AWS customers have built solutions that turn their data from a cost into a strategic asset, and bring your toughest questions straight to our experts. Learn More: https://aws.amazon.com/government-education/

Deep Dive on Object Storage: Amazon S3 and Amazon Glacier | AWS Public Sector...

Amazon Web Services

In this session, learn about the managed relational database services Amazon RDS and Amazon Aurora. Amazon RDS enables you to launch an optimally configured, secure, and highly available relational database with just a few clicks, and it has seven popular database engines to choose from. Amazon Aurora is a relational database built for the cloud and provides high availability, high performance, and full compatibility with MySQL and PostgreSQL. We take a closer look at how Amazon RDS and Amazon Aurora work, and we cover some of the key innovations in the Aurora database engine and storage layers. We also describe recently announced features, such as Aurora Serverless, Aurora Multi-Master, and Aurora Parallel Query.

Amazon RDS & Amazon Aurora: Relational Databases on AWS - SRV206 - Atlanta AW...

Amazon Web Services

This session will begin with an introduction to non-relational (NoSQL) databases and compare them with relational (SQL) databases. We will also explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service. Learn the fundamentals of DynamoDB and see the new DynamoDB console first-hand as we discuss common use cases and benefits of this high-performance key-value and JSON document store.

Getting Started with Amazon DynamoDB

Amazon Web Services

Want to get ramped up on how to use Amazon's big data web services and launch your first big data application on AWS? Join us on our journey as we build a big data application in real-time using Amazon EMR, Amazon Redshift, Amazon Kinesis, Amazon DynamoDB, and Amazon S3. We review architecture design patterns for big data solutions on AWS, and give you access to a take-home lab so that you can rebuild and customize the application yourself.

(BDT205) Your First Big Data Application On AWS

Amazon Web Services

What's hot (20)

Getting Started with the Hybrid Cloud: Enterprise Backup and Recovery

Getting started with amazon aurora - Toronto

SRV403 Deep Dive on Object Storage: Amazon S3 and Amazon Glacier

SRV404 Deep Dive on Amazon DynamoDB

ABCs of AWS: S3

Deep Dive on Amazon EC2 Instances - January 2017 AWS Online Tech Talks

(STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Masterclass Live: Amazon EMR

NEW LAUNCH! Introducing PostgreSQL compatibility for Amazon Aurora

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.

Deep Dive on Amazon DynamoDB

Intro to AWS: Database Services

AWS May Webinar Series - Getting Started: Storage with Amazon S3 and Amazon G...

Deep Dive - Amazon Elastic MapReduce (EMR)

Masterclass Webinar - Amazon Simple Storage Service S3

Deep Dive on Object Storage: Amazon S3 and Amazon Glacier | AWS Public Sector...

Amazon RDS & Amazon Aurora: Relational Databases on AWS - SRV206 - Atlanta AW...

Getting Started with Amazon DynamoDB

(BDT205) Your First Big Data Application On AWS

Similar to (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014

Hand-on Resources II: Extending SCMSWeb

Sugree Phatanapherom

EVOLVE'15 | Enhance | Norberto Leite | Effectively Scale and Operate AEM with...

Evolve The Adobe Digital Marketing Community

The session will discuss a library of solutions implemented at clients for transferring between applications in separate pods. Each configuration has its own merits and use case. The four main categories that will be discussed are - 1. Trickle Feed - uses a combination of inter-pod REST API connection, data management load rule, groovy scripting and scheduled EPM Automate job on a jump server to pick-up the files from source and push to target. 2. Focused On-save Push - pushes an intersection from source to target using inter-pod REST API connection, data management load rule and groovy scripting. 3. Scheduled Push- uses a combination of windows or Linux job, inter-pod REST API connection, groovy scripting, data management load rule and EPM Automate commands to extract and push data en masse from source to target. 4. Json Extract and Load - uses a combination of groovy scripting and inter-pod REST API connection to extract and push an intersection on-save. The audience will walk-away with learnings and understanding of inter-pod configurations, mainly for EPM Cloud planning applications. Snippets of code will form the "gold dust" takeaway from the session.

nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud

Alithya

Java one2013 con4540-keenan

ddkeenan

Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

Amazon Web Services

Optimising Web Application Frontend

tkramar

Oracle SQL Tuning

Alex Zaballa

Improving 3rd Party Script Performance With IFrames

Philip Tellis

Mangled Or Managed

Niall Litchfield

Антон Серпутько “Testing and optimization of client-side performance”

Dakiry

Gonchik Tsymzhitov will share stories of improving the performance of DC (+Server) installations. His stories will start from simple examples to complex ones where it is necessary to trace HTTP requests to the DBMS server. At the end of the report, you will receive an approximate range of steps to improve your installation. Also, the speaker will be happy to discuss the Atlassian ecosystem and Observability tools (such as NewRelic, glowroot etc.,) as they allow you to practically proactively respond to possible incidents.

TsymzhitovGB - Jira Day

Gonchik Tsymzhitov

Jlook web ui framework

HongSeong Jeon

GE Predix 新手入门赵锴物联网_IoT

Kai Zhao

Oracle SQL Tuning

Alex Zaballa

Effectively Deploying MongoDB on AEM

Norberto Leite

This presentation was prepared for a Webcast where John Yerhot, Engine Yard US Support Lead, and Chris Kelly, Technical Evangelist at New Relic discussed how you can scale and improve the performance of your Ruby web apps. They shared detailed guidance on issues like: Caching strategies Slow database queries Background processing Profiling Ruby applications Picking the right Ruby web server Sharding data Attendees will learn how to: Gain visibility on site performance Improve scalability and uptime Find and fix key bottlenecks See the on-demand replay: http://pages.engineyard.com/6TipsforImprovingRubyApplicationPerformance.html

6 tips for improving ruby performance

Engine Yard

As a Tester you need to level up. You can do more than functional verification or reporting Response Time In my Performance Clinic Workshops I show you real life exampls on why Applications fail and what you can do to find these problems when you are testing these applications. I am using Free Tools for all of these excercises - especially Dynatrace which gives full End-to-End Visibility (Browser to Database). You can test and download Dynatrace for Free @ http://bit.ly/atd2014challenge

From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Andreas Grabner

Crack the complexity of oracle applications r12 workload v2

Ajith Narayanan

Large Data Management Strategies

Salesforce Developers

TeraGrid's GRAM Auditing & Accounting, & its Integration with the LEAD Scienc...

marcuschristie

Similar to (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014 (20)

Hand-on Resources II: Extending SCMSWeb

EVOLVE'15 | Enhance | Norberto Leite | Effectively Scale and Operate AEM with...

nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud

Java one2013 con4540-keenan

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

Optimising Web Application Frontend

Oracle SQL Tuning

Improving 3rd Party Script Performance With IFrames

Mangled Or Managed

Антон Серпутько “Testing and optimization of client-side performance”

TsymzhitovGB - Jira Day

Jlook web ui framework

GE Predix 新手入门赵锴物联网_IoT

Oracle SQL Tuning

Effectively Deploying MongoDB on AEM

6 tips for improving ruby performance

From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Crack the complexity of oracle applications r12 workload v2

Large Data Management Strategies

TeraGrid's GRAM Auditing & Accounting, & its Integration with the LEAD Scienc...

More from Amazon Web Services

Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS. In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Amazon Web Services

La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup. Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti. Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Amazon Web Services

Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi

Esegui pod serverless con Amazon EKS e AWS Fargate

Amazon Web Services

Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.

Costruire Applicazioni Moderne con AWS

Amazon Web Services

L’utilizzo dei container è in continua crescita. Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili. I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!

Come spendere fino al 90% in meno con i container e le istanze spot

Amazon Web Services

In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th. Event Agenda : Open banking so far (short recap) • PSD2, OB UK, OB Australia, OB LATAM, OB Israel Intro to Open Finance marketplace • Scope • Features • Tech overview and Demo The role of the Cloud The Future of APIs • Complying with regulation • Monetizing data / APIs • Business models • Time to market One platform for all: a Strategic approach Q&A

Open banking as a service

Amazon Web Services

Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc. AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta. Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

Amazon Web Services

Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity. AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet. Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Amazon Web Services

Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Amazon Web Services

Computer Vision con AWS

Amazon Web Services

Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti. Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.

Database Oracle e VMware Cloud on AWS i miti da sfatare

Amazon Web Services

Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti. Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire. Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito. In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

Amazon Web Services

Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline. Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.

API moderne real-time per applicazioni mobili e web

Amazon Web Services

Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali. In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Amazon Web Services

Tools for building your MVP on AWS

Amazon Web Services

How to Build a Winning Pitch Deck

Amazon Web Services

Building a web application without servers

Amazon Web Services

Fundraising Essentials

Amazon Web Services

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Amazon Web Services

Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.

Introduzione a Amazon Elastic Container Service

Amazon Web Services

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Esegui pod serverless con Amazon EKS e AWS Fargate

Costruire Applicazioni Moderne con AWS

Come spendere fino al 90% in meno con i container e le istanze spot

Open banking as a service

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Computer Vision con AWS

Database Oracle e VMware Cloud on AWS i miti da sfatare

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

API moderne real-time per applicazioni mobili e web

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Tools for building your MVP on AWS

How to Build a Winning Pitch Deck

Building a web application without servers

Fundraising Essentials

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Introduzione a Amazon Elastic Container Service

Recently uploaded

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Jago de Vreede

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Exploring Multimodal Embeddings with Milvus

Zilliz

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Recently uploaded (20)

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

How to Troubleshoot Apps for the Modern Connected Worker

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Exploring Multimodal Embeddings with Milvus

2024: Domino Containers - The Next Step. News from the Domino Container commu...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Corporate and higher education May webinar.pptx

Artificial Intelligence Chap.5 : Uncertainty

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Axa Assurance Maroc - Insurer Innovation Award 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

(PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014

2. in data transfer from S3 not including Amazon Web Services use

3. Architecture Choosing a region Building a naming scheme Considering LISTs Optimizing PUTs Multipart upload Demo Optimizing GETs Using CloudFront Range-based GETs Demo Customer Case BigDataCorp

6. Request Rate and Performance Considerations http://amzn.to/18oF5LC TIP

7. 1 2 5 8 100/8 = 12.5 events/sec 100,000 users @ 10 events an hour = 224 TPS

8. <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/2013_11_11-164533134.jpg <my_bucket>/2013_11_11-164533135.jpg <my_bucket>/2013_11_11-164533136.jpg

9. 1 2 N 1 2 N Partition Partition Partition Partition

10. <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg <my_bucket>/456767561-2013_11_13.jpg <my_bucket>/345565651-2013_11_13.jpg <my_bucket>/431345660-2013_11_13.jpg

11. 1 2 N 1 2 N Partition Partition Partition Partition

12. •Store objects as a hash of their name –add the original name as metadata •“deadmau5_mix.mp3” 0aa316fb000eae52921aab1b4697424958a53ad9 –prependkeyname withshort hash •0aa3-deadmau5_mix.mp3 •Epoch time (reverse) –5321354831-deadmau5_mix.mp3

13.

14. <my_bucket>/images/521335461-2013_11_13.jpg <my_bucket>/images/465330151-2013_11_13.jpg <my_bucket>/movies/293924440-2013_11_13.jpg <my_bucket>/movies/987331160-2013_11_13.jpg <my_bucket>/thumbs-small/838434842-2013_11_13.jpg <my_bucket>/thumbs-small/342532454-2013_11_13.jpg <my_bucket>/thumbs-small/345233453-2013_11_13.jpg <my_bucket>/thumbs-small/345453454-2013_11_13.jpg

15. Request Rate and Performance Considerations http://amzn.to/18oF5LC TIP

16.

17.

18.

19. fasterflexibleset of partspresents all parts as a single objectparallelpausingresumingbeginning uploads before you know the total object size

20.

21.

22. DEMOMultipart Uploads

23.

24.

25.

26. DEMOAmazon CloudFrontvs. Amazon S3 download performance

27. •Align your ranges with your parts!

28. DEMORange based GETs

29.

30.

31. DynamoDB Amazon RDS Amazon CloudSearch Amazon EC2

32.

33.

34.

35.

36. Maestro (Reserved Instance) List of crawl URLs Main workers Execute crawling and process data Spot Instances Secondary workers(queue listeners) Reprocess data, query additional services, store data on MongoDB Spot Instances Secondary work queues – processed data MongoDBcluster Command and Control Queue

37.

38. Architecture Choosing a region Building a naming scheme Considering LISTs Optimizing PUTs Multipart upload Demo Optimizing GETs Using CloudFront Range-based GETs Demo Customer Case BigDataCorp

39. gfelipe@amazon.comthoran@bigdatacorp.com.br

40. Please give us your feedback on this presentation

(PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014

Similar to (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

(PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014