Abstract: A apresentação centra-se na temática de ter forma de controlar, versionar e actualizar toda a parte de Base de Dados de um projecto. Estamos a falar, desde a produção de modelos ER, a versionamento de scripts, passando pelo deploy dos mesmos e terminado na documentação. A apresentação conta ainda com uma breve demonstração do uso da ferramenta Flyway para versionar e controlar a execução de scripts nos diversos ambientes de um projecto. Sobre o Nuno Alves: Chamo-me Nuno Alves nascido em Coimbra, Portugal e vivi maioritariamente em Leiria. Licenciado em Engenharia Informática na ESTG-IPLeiria (Escola Superior de Tecnologia e Gestão) onde o gosto por dados e bases de dados se começou a desenvolver. Daí, profissionalmente a minha área de actuação ser em torno de bases de dados e infra-estruturas. Tenho cerca de 10 anos de experiência repartidos pelas áreas Financeira, Seguros, Governo, Militar em tecnologias que vão desde Oracle, PostgreSQL, MSSQLServer a DB2.

1. KYSUC
Keep Your Schema Under Control
04.FEV.2016
Nuno Alexandre Alves

2. Speaker
• Nuno Alves
• Born at Coimbra / Lives in Leiria
• Degree in Informatics Engineering (ESTG-IPLeiria)
• Some experience with databases
• Nowadays working in Critical Software, SA
nuno.alexandre.alves@gmail.com
nuno.alexandre.alves

3. Disclaimer
This presentation is:
• From my personal experience
• My own suggestions
This presentation is NOT to:
• Pointing fingers
• Enumerate errors
• This is better than that
• Do not use this

4. Just another day in “paradise”
• How do you develop your scripts ?
• Do you have a ER model of your Database(s) ?
• Do you have your Database documented ?
• How do you control your scripts versions ?
• Do you have your test data under scripts ?
• How do you test your scripts ?
• How do you deploy your application and
database to a specific version ?

5. Agenda
1. Goals
2. Diagrams (ER models)
3. Scripts
4. Deploy
5. Documentation

6. Goals for Today
• Updated ER model
• Scripts versioned
• Easy Deploy
• Easy Documentation
• Schema Under Control !

7. Agenda
1. Goals
2. Diagrams (ER models)
3. Scripts
4. Deploy
5. Documentation

8. ER Model (1/4)
Start New Project (from requirements)
• Build your Conceptual Model
• First Physical Model
• First Scripts (version 1.0.0)
• Print (paper not PDF) your Physical Model and put it
in a Wall next to the team (old-school but it works)
Assumptions:
• High Level Architecture designed
• RBDMS almost decided (for the first scripts)

9. ER Model (2/4)
Start New Project (already in PROD) / Maintenance
• Ask for actual documentation (get lucky )
• DDL of actual production database
• Import DDL in your development servers
• Reverse Engineering
• Build Conceptual Model (from Physical) if
necessary
• Script Physical Model with actual application
version (ex: v3.5.2 – it’s your baseline)

10. ER Model (3/4)
Development / Maintenance
• Keep your database changes updated in
your ER Model
• Use Reverse Engineering to do
checkpoints (compare)

11. ER Model (4/4)
Tools
• Use a Case Tool
• PowerDesigner
• Enterprise Architect
• …

12. Agenda
1. Goals
2. Diagrams (ER models)
3. Scripts
4. Deploy
5. Documentation

13. Scripts (1/10)
Scripts Development (projects reality)
• Manually
• I don’t have scripts. My ORM tool does it
for me !?!
• Database is something to store some data
on it. I don’t care.

14. Scripts (2/10)
Scripts Development (projects reality)
• WTF ?!
• Your medical records relies on a database system
• Your bank account relies on a database system
• Your salary relies on a database system
• …
• Your life relies on too many database systems
• Take your own conclusions

15. Scripts (3/10)
Then someday
• How do I execute the database scripts ? Order By ?
• Which development database is updated ?
• How do you develop scripts ?
• One script vs multiple scripts ?
• Full Scripts vs Partial Scripts ?
• It’s your ORM your Database developer ?
• Naming Conventions for objects (where is the guide?)
• I have a app log with this error “constraint SYS_098374
violated”. What is that constraint ?
• The column CGTRTHY stores what ?

16. Scripts (4/10)
Database Scripts Under Control
• Why ?
• Is your source code under control ?
• Does your source code have versions ?
• Database scripts = source code ?
• I think Yes !
• Table definition
• Table initial data
• Function / Stored Procedure / Package

17. Scripts (5/10)
Decide your Database Storage Strategy
• Check production for rolling projects
• Decide according to hosting supplier or client needs
• Tablespaces
• Data files
• File groups
My Suggestion (minimum)
• One Tablespace for Data
• One Tablespace for Indexes
• One Tablespace for LOBs

18. Scripts (6/10)
Decide your Database / Schema(s) architecture
• One schema, multiple users with diff grants
• Multiple schemas, one/multiple users
My Suggestion
• Keep it simple
• One schema/database, multiple users (grants/views)
• FKs between multiple schemas not supported in all RDBMS
• Distributed Transactions (do you really want ?!)
• Multiple queries (joins in app server)
• Are you creating your own RDBMS ?!

19. Scripts (7/10)
Decide your Database Naming Conventions
• Easy identify a database object
• Name every object in database
• Don’t let default names under RDBMS

20. Scripts (8/10)
Script like source code
• Use SQL formatting (for human readable)
• Comment your SQL
• Define objects language: PT; EN; FR … (don’t mix)
Script for documentation
• Comment every object (mandatory: wait and see)
• Keep documentation in database dictionary
• Useful for Reverse Engineering
• Useful for Automatic Documentation
• COMMENT ON TABLE
• COMMENT ON COLUMN TABLE_NAME.COLUMN_NAME

21. Scripts (9/10)
Tools
• Manually (Notepad, VI, … is your friend!)
• Liquibase
• mybatis
• MIGRATEdb
• migrate4j
• Flyway
• …

22. Scripts (10/10)
Scripts Under Control ?

23. Flyway (1/6)
Overview
• Scripts: SQL, Java
• RDBMS Agnostic (JDBC driver)
• Inside your source code
• Placeholders (variables)
• Multiple Execution Modes
• Java API
• Maven / Ant
• Command line
• Ex: V2_0_0_1_0001__script_name.sql

24. Flyway (2/6)
Decide your Script Naming Convention
• V<JIRA_FIX_VERSION>_1_<4_digits>__<JIRA_KEY>.sql
• <4_digitos>
• Execution order

25. Flyway (3/6)
Put some code in your Script
• Add an header to your SQL file
• Add JIRA_KEY or JIRA URL as a comment
• Commit

26. Flyway (4/6)
Decide your Execution Mode
• Command line (java -jar …)
• Maven/ANT
• configure inside your IDE or in CLI
• Use Maven profiles is necessary (useful in multiple
databases/environments)

27. Flyway (5/6)
Flyway in action
• flyway:info
• Check schema version already installed
• You need flyway:baseline for the first time

28. Flyway (6/6)
Flyway in action
• flyway:migration
• Script execution
• With/without target release

29. Flyway
In Action

30. Agenda
1. Goals
2. Diagrams (ER models)
3. Scripts
4. Deploy
5. Documentation

31. Deploy (1/7)
Deploy Database Scripts
• Do you test your scripts ? (how many times)
• Do you re-create your schema from scratch ?
• Do you have automatic deployment for your app ?
• And your database ?
• Do you have night builds ?
• They include database too ? and automated tests ?
• Do you simulate your client releases in development
environments ?
• Do you have confidence in your own scripts ?

32. Deploy (2/7)
Deploy Database (full / upgrade)
• Use your Continuous Integration
• Jenkins
• Team City
• Bamboo
• Other …

33. Deploy (3/7)
Jenkins + Flyway = easy database deploy
• Why Jenkins ?
• Easy to configure
• Parametrized builds
• Many plugins (CVS, SVN, Maven, JIRA, Email, …)
• Scheduler
• Live Build Logs / Console output directly in browser
• Why not ?!

34. Deploy (4/7)
Jenkins + Flyway = easy database deploy
• Configure a Jenkins Job
• Checkout script from VCS
• Put some parameters
• if necessary for customs build
• for multiple versions
• for multiple databases
• Execute Flyway
• via Maven
• via custom Shell Script / Windows batch

35. Jenkins + Flyway
In Action

36. Deploy (5/7)
Continuous Monitor your database
• Oracle Enterprise Manager (Oracle)
• pgwatch (PostgreSQL)
• db2top (DB2)
• SQL Server Management Studio (SQLServer)
• MySQL Enterprise Monitor (MySQL)
• …
• Check administrator views (available in every RDBMS)

37. Deploy (6/7)
Continuous Monitor your database
• Check some random SQL Statements
• CPU
• I/O
• Get query plan for these statements
• Help development team write better SQL
• Create indexes according to business needs
• Monitor sessions
• Check Connection Pooling parameters
• Min / Max Pool sizes (warn: multiple Connections Pools)

38. Deploy (7/7)
Help Development Team to help yourself
• Write SQL Statements to log (debug mode)
• Ask for log4jdbc / P6Spy / … (with parameters)
• Write query times to log
• Analyze top time consuming queries
• Analyze top running queries
• Ask for application times vs database times (in log)
• Most of times the application consumes more time (yes it’s true)
• Try to analyze how many queries are made for a business operation
• Check the number of columns retrieved in every query (it’s really necessary 80
columns ?)

39. Agenda
1. Goals
2. Diagrams (ER models)
3. Scripts
4. Deploy
5. Documentation

40. Documentation (1/3)
How do you produce your database documentation ?
• Manually
• I don’t have database documentation
• “My client doesn’t need this documentation”
• And your team ?
• How do you maintain a project without documentation?
• Most (~All) of your production tickets are resolved by SQL
statements !
• Client business data is stored in “your” database (really?!)

41. Documentation (2/3)
Automatic Documentation
• Use RDBMS data dictionary
• information_schema.tables, …
• user_tables, …
• syscat.tables, …
• Remember COMMENT ON … from scripts ?
• “Export” to something useful
• Word Document/PDF ?
• Who reads a 800 pages document ?

42. Documentation (3/3)
Automatic Documentation
• Perl script (reads data dictionary, outputs One HTML file)
• Send to your client as a deliverable
• Publish in your wiki
• Open locally in your browser
• Produce custom CSS: 30m - 1h
• Update Documentation: 1 minute
• Remember COMMENT ON 

43. Summary (from initial goals) (1/2)
ER Model
• Use a Case Tool
• Keep your model updated
Scripts versioned
• Use Flyway / Liquibase / Other
• Script everything (data tests included)
• Review your scripts
• Naming Conventions
• Comments

44. Summary (from initial goals) (2/2)
Easy Deploy
• Use Jenkins / Team City / Bamboo / Other
• Keep your database under Continuous Integration
• Schedule your database builds
• Parametrize your database deploy
• Monitor your database close
• Release Matrix isn’t nice to have (is mandatory)
Easy Documentation
• Comment your objects in data dictionary

45. Main Goal
Updated ER Model
+ Scripts versioned
+ Easy Deploy
+ Easy Documentation
= KYSUC

46. Extra Credits
Define / ask for backup/restore plan
• Test your backups (restore is your friend)
• Check backup/restore times
• Check database growth
Compare data
• Compare your production data with development data:
reference data
• Try to equal PKs IDs across environments (for referencial data)

47. That’s It!
Thank you for your time.