SlideShare a Scribd company logo
1 of 28
ODTUG




Flexible Design and Modeling

    Planning for Constant Change

      Gwen Shapira and Robyn Sands
ODTUG




There is nothing
   permanent
 except change
ODTUG


                     Agile Values
Individuals and interactions OVER processes and tools.

Working software OVER comprehensive documentation.

Customer collaborations OVER contract negotiation.


Responding to change OVER following a plan.

           from ‘The Manifesto for Agile Development’ at agilemanifesto.org
ODTUG


                  Is this ‘Agile’?
• Database design is
  neglected
• Object models do not
  translate to good
  database design
• Scalability and
  performance are not last
  minute add-ons
• Systems can not be
  flexible unless you plan
  for change
ODTUG


       The Emperor has no clothes!
• Database performance
  and scalability left out
• Normalization and
  constraints are ‘old
  school
• ‘Just good enough’
  doesn’t scale
• If all ideas come from
  the user, where’s the
  innovation?
ODTUG


Why aren’t database people Agile?
 Data professionals are resistant to change.
 Missed the <blank>* revolution of the <time
 period>.
 Haven’t kept up with modern software
 development.
 Besides, DBAs hate all developers ....

          * insert favorite trend here (object, agile, extreme, etc)
                                   with the appropriate timeframe.
ODTUG


      It’s really development’s fault

Developers create performance problems and expect
DBAs to fix them later.
Missed the <blank>* foundational studies of the <time
period>.
Don’t take enough time to understand the data.
Developers just hate data people on principle.

                       * insert favorite university course title here
                                   with the appropriate timeframe.
ODTUG



Blame and finger-pointing doesn’t fix anything.

 We need the best of both worlds to succeed.
ODTUG



  Waterfall doesn’t work
    Chaos doesn’t work
We need controlled iterations
ODTUG




Normalization:
Stable Basis
For Flexible Application
ODTUG


Universal Modeling Language




                     Data in Motion
ODTUG


Data in Motion
ODTUG


Data at Rest
ODTUG


     What is important to a database?

•   Entities, relationships and data types
•   Access paths
•   Number of records
•   Distribution of the data
•   Lifespan of the information
ORM is not a Data Model
            • It hides the data model
            • Which is a good thing…
            • But if no one ever sees
              the data model…
            • How do you know it will
              scale?
ODTUG


    Building a refactorable database
Design for the inevitable change:
  • Structure schema for the data at rest.
  • Speed up data access with narrow tables,
    indexed organized tables or partitioning.
  • Delay normalization for unknowns, unused
    values.
  • Packages and procedures make possible to
    change the schema without changing the
    application code.
ODTUG


               Normalization
• Prevents redundant data
• Goal is a ‘single source of truth’
• Integrity and relationships protected by keys
  and constraints
• Denormalize for performance when you must
• Normalization is what makes a database
  ‘Agile’.
ODTUG


Framework
for Change
ODTUG


                 Transactional API
• Everything in PL/SQL calls
   – Application doesn’t touch tables
   – Suggestion of table design changes does not incite panic
• Clean interface between application and the
  database
   – Unit Testing
   – Simplify Builds
   – Minimize impact of DB changes
• Prepare for caching
ODTUG




The interface between the database and the application should
 be jointly designed by application and database developers.


    Meet in the ring and fight it out.
ODTUG


               Evolutionary design
Plan the interface:
  •     What does development need?
  •     How will you provide it?
Using old dogs for new tricks:
  •     Procedures, packages and views
  •     Table structures can change without impacting the
        application layer
Build the concept
      • Adapt as you gain knowledge
ODTUG


               Basic Design #Fail
1.   Inappropriate table and column names
2.   Mismatching data types
3.   Multi-purpose columns
4.   Lack of unique and non-volatile primary key
5.   Lack of common vocabulary
6.   Skipping the procedure - just this once.
7.   Hiding the data from the developer
8.   Duplicate processing
ODTUG




A duck is developer’s best friend
ODTUG


War Stories I – One Table to Rule Them All
ODTUG



War Stories II –
Hiding the Data
ODTUG


                  Keep the good
• Watch your end users to understand how they use data.
• Know your data and application goals, and anticipate the
  changes.
• Design applications for the process, design databases for
  data access.
• Create a normalized ERD to understand the entities and
  relationships.
• Use an evolutionary modeling with an iterative approach
• Normalize for performance AND agility.
• Use denormalization to ‘preassemble’ data when you must
  but avoid secondary 'sources of truth‘.
ODTUG


                  Lose the bias
              Agile is not ‘code and fix’.

             Normalization is not dead.

 Logical and physical database design will have a
 significant impact on your application’s success.
                                        (and performance)


Whether that impact is positive or negative is up to us.
ODTUG




Design ends when the
application is shelved.

As long as someone is
using the application –
expect changes…

And enjoy the ride.

More Related Content

What's hot

Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lakepunedevscom
 
Queues, Pools and Caches - Paper
Queues, Pools and Caches - PaperQueues, Pools and Caches - Paper
Queues, Pools and Caches - PaperGwen (Chen) Shapira
 
Hybrid Data Warehouse Hadoop Implementations
Hybrid Data Warehouse Hadoop ImplementationsHybrid Data Warehouse Hadoop Implementations
Hybrid Data Warehouse Hadoop ImplementationsDavid Portnoy
 
2012 10 bigdata_overview
2012 10 bigdata_overview2012 10 bigdata_overview
2012 10 bigdata_overviewjdijcks
 
Intro to Big Data and NoSQL
Intro to Big Data and NoSQLIntro to Big Data and NoSQL
Intro to Big Data and NoSQLDon Demcsak
 
Emergent Distributed Data Storage
Emergent Distributed Data StorageEmergent Distributed Data Storage
Emergent Distributed Data Storagehybrid cloud
 
SQL/NoSQL How to choose ?
SQL/NoSQL How to choose ?SQL/NoSQL How to choose ?
SQL/NoSQL How to choose ?Venu Anuganti
 
Hadoop and the Data Warehouse: When to Use Which
Hadoop and the Data Warehouse: When to Use Which Hadoop and the Data Warehouse: When to Use Which
Hadoop and the Data Warehouse: When to Use Which DataWorks Summit
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lakeJames Serra
 
Introduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLIntroduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLTushar Shende
 
Data Warehouse on Hadoop Based System In Action
Data Warehouse on Hadoop Based System In ActionData Warehouse on Hadoop Based System In Action
Data Warehouse on Hadoop Based System In ActionFrank Y
 
So You Want to Build a Data Lake?
So You Want to Build a Data Lake?So You Want to Build a Data Lake?
So You Want to Build a Data Lake?David P. Moore
 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's includedJames Serra
 
Introduction to Designing and Building Big Data Applications
Introduction to Designing and Building Big Data ApplicationsIntroduction to Designing and Building Big Data Applications
Introduction to Designing and Building Big Data ApplicationsCloudera, Inc.
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMark Kromer
 
Should I move my database to the cloud?
Should I move my database to the cloud?Should I move my database to the cloud?
Should I move my database to the cloud?James Serra
 
Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)James Serra
 
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Lviv Startup Club
 
Building a Data Lake - An App Dev's Perspective
Building a Data Lake - An App Dev's PerspectiveBuilding a Data Lake - An App Dev's Perspective
Building a Data Lake - An App Dev's PerspectiveGeekNightHyderabad
 

What's hot (20)

Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lake
 
Data lake
Data lakeData lake
Data lake
 
Queues, Pools and Caches - Paper
Queues, Pools and Caches - PaperQueues, Pools and Caches - Paper
Queues, Pools and Caches - Paper
 
Hybrid Data Warehouse Hadoop Implementations
Hybrid Data Warehouse Hadoop ImplementationsHybrid Data Warehouse Hadoop Implementations
Hybrid Data Warehouse Hadoop Implementations
 
2012 10 bigdata_overview
2012 10 bigdata_overview2012 10 bigdata_overview
2012 10 bigdata_overview
 
Intro to Big Data and NoSQL
Intro to Big Data and NoSQLIntro to Big Data and NoSQL
Intro to Big Data and NoSQL
 
Emergent Distributed Data Storage
Emergent Distributed Data StorageEmergent Distributed Data Storage
Emergent Distributed Data Storage
 
SQL/NoSQL How to choose ?
SQL/NoSQL How to choose ?SQL/NoSQL How to choose ?
SQL/NoSQL How to choose ?
 
Hadoop and the Data Warehouse: When to Use Which
Hadoop and the Data Warehouse: When to Use Which Hadoop and the Data Warehouse: When to Use Which
Hadoop and the Data Warehouse: When to Use Which
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lake
 
Introduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLIntroduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQL
 
Data Warehouse on Hadoop Based System In Action
Data Warehouse on Hadoop Based System In ActionData Warehouse on Hadoop Based System In Action
Data Warehouse on Hadoop Based System In Action
 
So You Want to Build a Data Lake?
So You Want to Build a Data Lake?So You Want to Build a Data Lake?
So You Want to Build a Data Lake?
 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's included
 
Introduction to Designing and Building Big Data Applications
Introduction to Designing and Building Big Data ApplicationsIntroduction to Designing and Building Big Data Applications
Introduction to Designing and Building Big Data Applications
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data Analytics
 
Should I move my database to the cloud?
Should I move my database to the cloud?Should I move my database to the cloud?
Should I move my database to the cloud?
 
Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)
 
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
 
Building a Data Lake - An App Dev's Perspective
Building a Data Lake - An App Dev's PerspectiveBuilding a Data Lake - An App Dev's Perspective
Building a Data Lake - An App Dev's Perspective
 

Similar to Flexible Design

Toad for Sybase Introduction
Toad for Sybase IntroductionToad for Sybase Introduction
Toad for Sybase IntroductionToad4Sybase
 
No Silver Bullet - Essence and Accidents of Software Engineering
No Silver Bullet - Essence and Accidents of Software EngineeringNo Silver Bullet - Essence and Accidents of Software Engineering
No Silver Bullet - Essence and Accidents of Software EngineeringAditi Abhang
 
Enabling Your Data Science Team with Modern Data Engineering
Enabling Your Data Science Team with Modern Data EngineeringEnabling Your Data Science Team with Modern Data Engineering
Enabling Your Data Science Team with Modern Data EngineeringJames Densmore
 
Bridging the gap in enterprise AI
Bridging the gap in enterprise AIBridging the gap in enterprise AI
Bridging the gap in enterprise AIMax Pumperla
 
UX, DX, DSX: Developers and Data Scientists as Users
UX, DX, DSX: Developers and Data Scientists as UsersUX, DX, DSX: Developers and Data Scientists as Users
UX, DX, DSX: Developers and Data Scientists as UsersUXDXConf
 
Three signs your architecture is too small for big data. Camp IT December 2014
Three signs your architecture is too small for big data.  Camp IT December 2014Three signs your architecture is too small for big data.  Camp IT December 2014
Three signs your architecture is too small for big data. Camp IT December 2014Craig Jordan
 
AMW43 - Unba.se, Distributed database for human interaction
AMW43 - Unba.se, Distributed database for human interactionAMW43 - Unba.se, Distributed database for human interaction
AMW43 - Unba.se, Distributed database for human interactionDaniel Norman
 
Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)ncoghlan_dev
 
Drupal and Devops , the Survey Results
Drupal and Devops , the Survey ResultsDrupal and Devops , the Survey Results
Drupal and Devops , the Survey ResultsKris Buytaert
 
Complexinterfaces
ComplexinterfacesComplexinterfaces
ComplexinterfacesBojhan
 
Path dependent-development (PyCon India)
Path dependent-development (PyCon India)Path dependent-development (PyCon India)
Path dependent-development (PyCon India)ncoghlan_dev
 
SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 CareerBuilder.com
 
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)Binary Studio
 
Enterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETEnterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETAnant Corporation
 
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Lucas Jellema
 
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Daniel Zivkovic
 
Which postgres is_right_for_me_20130517
Which postgres is_right_for_me_20130517Which postgres is_right_for_me_20130517
Which postgres is_right_for_me_20130517EDB
 

Similar to Flexible Design (20)

Toad for Sybase Introduction
Toad for Sybase IntroductionToad for Sybase Introduction
Toad for Sybase Introduction
 
No Silver Bullet - Essence and Accidents of Software Engineering
No Silver Bullet - Essence and Accidents of Software EngineeringNo Silver Bullet - Essence and Accidents of Software Engineering
No Silver Bullet - Essence and Accidents of Software Engineering
 
2020 | Metadata Day | LinkedIn
2020 | Metadata Day | LinkedIn2020 | Metadata Day | LinkedIn
2020 | Metadata Day | LinkedIn
 
Enabling Your Data Science Team with Modern Data Engineering
Enabling Your Data Science Team with Modern Data EngineeringEnabling Your Data Science Team with Modern Data Engineering
Enabling Your Data Science Team with Modern Data Engineering
 
Bridging the gap in enterprise AI
Bridging the gap in enterprise AIBridging the gap in enterprise AI
Bridging the gap in enterprise AI
 
UX, DX, DSX: Developers and Data Scientists as Users
UX, DX, DSX: Developers and Data Scientists as UsersUX, DX, DSX: Developers and Data Scientists as Users
UX, DX, DSX: Developers and Data Scientists as Users
 
Three signs your architecture is too small for big data. Camp IT December 2014
Three signs your architecture is too small for big data.  Camp IT December 2014Three signs your architecture is too small for big data.  Camp IT December 2014
Three signs your architecture is too small for big data. Camp IT December 2014
 
AMW43 - Unba.se, Distributed database for human interaction
AMW43 - Unba.se, Distributed database for human interactionAMW43 - Unba.se, Distributed database for human interaction
AMW43 - Unba.se, Distributed database for human interaction
 
Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)
 
Drupal and Devops , the Survey Results
Drupal and Devops , the Survey ResultsDrupal and Devops , the Survey Results
Drupal and Devops , the Survey Results
 
Complexinterfaces
ComplexinterfacesComplexinterfaces
Complexinterfaces
 
Path dependent-development (PyCon India)
Path dependent-development (PyCon India)Path dependent-development (PyCon India)
Path dependent-development (PyCon India)
 
Clean Architecture
Clean ArchitectureClean Architecture
Clean Architecture
 
SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018 SDSC18 and DSATL Meetup March 2018
SDSC18 and DSATL Meetup March 2018
 
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)
NoSQL vs SQL (by Dmitriy Beseda, JS developer and coach Binary Studio Academy)
 
Enterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETEnterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NET
 
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...
 
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
 
Which postgres is_right_for_me_20130517
Which postgres is_right_for_me_20130517Which postgres is_right_for_me_20130517
Which postgres is_right_for_me_20130517
 
Lean Analytics: How to get more out of your data science team
Lean Analytics: How to get more out of your data science teamLean Analytics: How to get more out of your data science team
Lean Analytics: How to get more out of your data science team
 

More from Gwen (Chen) Shapira

Velocity 2019 - Kafka Operations Deep Dive
Velocity 2019  - Kafka Operations Deep DiveVelocity 2019  - Kafka Operations Deep Dive
Velocity 2019 - Kafka Operations Deep DiveGwen (Chen) Shapira
 
Lies Enterprise Architects Tell - Data Day Texas 2018 Keynote
Lies Enterprise Architects Tell - Data Day Texas 2018  Keynote Lies Enterprise Architects Tell - Data Day Texas 2018  Keynote
Lies Enterprise Architects Tell - Data Day Texas 2018 Keynote Gwen (Chen) Shapira
 
Gluecon - Kafka and the service mesh
Gluecon - Kafka and the service meshGluecon - Kafka and the service mesh
Gluecon - Kafka and the service meshGwen (Chen) Shapira
 
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Gwen (Chen) Shapira
 
Papers we love realtime at facebook
Papers we love   realtime at facebookPapers we love   realtime at facebook
Papers we love realtime at facebookGwen (Chen) Shapira
 
Multi-Datacenter Kafka - Strata San Jose 2017
Multi-Datacenter Kafka - Strata San Jose 2017Multi-Datacenter Kafka - Strata San Jose 2017
Multi-Datacenter Kafka - Strata San Jose 2017Gwen (Chen) Shapira
 
Streaming Data Integration - For Women in Big Data Meetup
Streaming Data Integration - For Women in Big Data MeetupStreaming Data Integration - For Women in Big Data Meetup
Streaming Data Integration - For Women in Big Data MeetupGwen (Chen) Shapira
 
Kafka connect-london-meetup-2016
Kafka connect-london-meetup-2016Kafka connect-london-meetup-2016
Kafka connect-london-meetup-2016Gwen (Chen) Shapira
 
Fraud Detection for Israel BigThings Meetup
Fraud Detection  for Israel BigThings MeetupFraud Detection  for Israel BigThings Meetup
Fraud Detection for Israel BigThings MeetupGwen (Chen) Shapira
 
Kafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be thereKafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be thereGwen (Chen) Shapira
 
Nyc kafka meetup 2015 - when bad things happen to good kafka clusters
Nyc kafka meetup 2015 - when bad things happen to good kafka clustersNyc kafka meetup 2015 - when bad things happen to good kafka clusters
Nyc kafka meetup 2015 - when bad things happen to good kafka clustersGwen (Chen) Shapira
 
Data Architectures for Robust Decision Making
Data Architectures for Robust Decision MakingData Architectures for Robust Decision Making
Data Architectures for Robust Decision MakingGwen (Chen) Shapira
 
Kafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn MeetupKafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn MeetupGwen (Chen) Shapira
 
Kafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupKafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupGwen (Chen) Shapira
 

More from Gwen (Chen) Shapira (20)

Velocity 2019 - Kafka Operations Deep Dive
Velocity 2019  - Kafka Operations Deep DiveVelocity 2019  - Kafka Operations Deep Dive
Velocity 2019 - Kafka Operations Deep Dive
 
Lies Enterprise Architects Tell - Data Day Texas 2018 Keynote
Lies Enterprise Architects Tell - Data Day Texas 2018  Keynote Lies Enterprise Architects Tell - Data Day Texas 2018  Keynote
Lies Enterprise Architects Tell - Data Day Texas 2018 Keynote
 
Gluecon - Kafka and the service mesh
Gluecon - Kafka and the service meshGluecon - Kafka and the service mesh
Gluecon - Kafka and the service mesh
 
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17
Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17
 
Papers we love realtime at facebook
Papers we love   realtime at facebookPapers we love   realtime at facebook
Papers we love realtime at facebook
 
Kafka reliability velocity 17
Kafka reliability   velocity 17Kafka reliability   velocity 17
Kafka reliability velocity 17
 
Multi-Datacenter Kafka - Strata San Jose 2017
Multi-Datacenter Kafka - Strata San Jose 2017Multi-Datacenter Kafka - Strata San Jose 2017
Multi-Datacenter Kafka - Strata San Jose 2017
 
Streaming Data Integration - For Women in Big Data Meetup
Streaming Data Integration - For Women in Big Data MeetupStreaming Data Integration - For Women in Big Data Meetup
Streaming Data Integration - For Women in Big Data Meetup
 
Kafka at scale facebook israel
Kafka at scale   facebook israelKafka at scale   facebook israel
Kafka at scale facebook israel
 
Kafka connect-london-meetup-2016
Kafka connect-london-meetup-2016Kafka connect-london-meetup-2016
Kafka connect-london-meetup-2016
 
Fraud Detection for Israel BigThings Meetup
Fraud Detection  for Israel BigThings MeetupFraud Detection  for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
 
Kafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be thereKafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be there
 
Nyc kafka meetup 2015 - when bad things happen to good kafka clusters
Nyc kafka meetup 2015 - when bad things happen to good kafka clustersNyc kafka meetup 2015 - when bad things happen to good kafka clusters
Nyc kafka meetup 2015 - when bad things happen to good kafka clusters
 
Fraud Detection Architecture
Fraud Detection ArchitectureFraud Detection Architecture
Fraud Detection Architecture
 
Have your cake and eat it too
Have your cake and eat it tooHave your cake and eat it too
Have your cake and eat it too
 
Kafka for DBAs
Kafka for DBAsKafka for DBAs
Kafka for DBAs
 
Data Architectures for Robust Decision Making
Data Architectures for Robust Decision MakingData Architectures for Robust Decision Making
Data Architectures for Robust Decision Making
 
Kafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn MeetupKafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn Meetup
 
Kafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupKafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka Meetup
 
Twitter with hadoop for oow
Twitter with hadoop for oowTwitter with hadoop for oow
Twitter with hadoop for oow
 

Flexible Design

  • 1. ODTUG Flexible Design and Modeling Planning for Constant Change Gwen Shapira and Robyn Sands
  • 2. ODTUG There is nothing permanent except change
  • 3. ODTUG Agile Values Individuals and interactions OVER processes and tools. Working software OVER comprehensive documentation. Customer collaborations OVER contract negotiation. Responding to change OVER following a plan. from ‘The Manifesto for Agile Development’ at agilemanifesto.org
  • 4. ODTUG Is this ‘Agile’? • Database design is neglected • Object models do not translate to good database design • Scalability and performance are not last minute add-ons • Systems can not be flexible unless you plan for change
  • 5. ODTUG The Emperor has no clothes! • Database performance and scalability left out • Normalization and constraints are ‘old school • ‘Just good enough’ doesn’t scale • If all ideas come from the user, where’s the innovation?
  • 6. ODTUG Why aren’t database people Agile? Data professionals are resistant to change. Missed the <blank>* revolution of the <time period>. Haven’t kept up with modern software development. Besides, DBAs hate all developers .... * insert favorite trend here (object, agile, extreme, etc) with the appropriate timeframe.
  • 7. ODTUG It’s really development’s fault Developers create performance problems and expect DBAs to fix them later. Missed the <blank>* foundational studies of the <time period>. Don’t take enough time to understand the data. Developers just hate data people on principle. * insert favorite university course title here with the appropriate timeframe.
  • 8. ODTUG Blame and finger-pointing doesn’t fix anything. We need the best of both worlds to succeed.
  • 9. ODTUG Waterfall doesn’t work Chaos doesn’t work We need controlled iterations
  • 14. ODTUG What is important to a database? • Entities, relationships and data types • Access paths • Number of records • Distribution of the data • Lifespan of the information
  • 15. ORM is not a Data Model • It hides the data model • Which is a good thing… • But if no one ever sees the data model… • How do you know it will scale?
  • 16. ODTUG Building a refactorable database Design for the inevitable change: • Structure schema for the data at rest. • Speed up data access with narrow tables, indexed organized tables or partitioning. • Delay normalization for unknowns, unused values. • Packages and procedures make possible to change the schema without changing the application code.
  • 17. ODTUG Normalization • Prevents redundant data • Goal is a ‘single source of truth’ • Integrity and relationships protected by keys and constraints • Denormalize for performance when you must • Normalization is what makes a database ‘Agile’.
  • 19. ODTUG Transactional API • Everything in PL/SQL calls – Application doesn’t touch tables – Suggestion of table design changes does not incite panic • Clean interface between application and the database – Unit Testing – Simplify Builds – Minimize impact of DB changes • Prepare for caching
  • 20. ODTUG The interface between the database and the application should be jointly designed by application and database developers. Meet in the ring and fight it out.
  • 21. ODTUG Evolutionary design Plan the interface: • What does development need? • How will you provide it? Using old dogs for new tricks: • Procedures, packages and views • Table structures can change without impacting the application layer Build the concept • Adapt as you gain knowledge
  • 22. ODTUG Basic Design #Fail 1. Inappropriate table and column names 2. Mismatching data types 3. Multi-purpose columns 4. Lack of unique and non-volatile primary key 5. Lack of common vocabulary 6. Skipping the procedure - just this once. 7. Hiding the data from the developer 8. Duplicate processing
  • 23. ODTUG A duck is developer’s best friend
  • 24. ODTUG War Stories I – One Table to Rule Them All
  • 25. ODTUG War Stories II – Hiding the Data
  • 26. ODTUG Keep the good • Watch your end users to understand how they use data. • Know your data and application goals, and anticipate the changes. • Design applications for the process, design databases for data access. • Create a normalized ERD to understand the entities and relationships. • Use an evolutionary modeling with an iterative approach • Normalize for performance AND agility. • Use denormalization to ‘preassemble’ data when you must but avoid secondary 'sources of truth‘.
  • 27. ODTUG Lose the bias Agile is not ‘code and fix’. Normalization is not dead. Logical and physical database design will have a significant impact on your application’s success. (and performance) Whether that impact is positive or negative is up to us.
  • 28. ODTUG Design ends when the application is shelved. As long as someone is using the application – expect changes… And enjoy the ride.

Editor's Notes

  1. Data models assist in communications, planning, scalability and yes – flexibility.