Submit Search
Upload
Multi Master PostgreSQL Cluster on Kubernetes
•
13 likes
•
12,041 views
Ohyama Masanori
Follow
PGConf.Asia 2017 http://www.pgconf.asia/EN/2017/day-2/#B4
Read less
Read more
Software
Report
Share
Report
Share
1 of 92
Download now
Download to read offline
Recommended
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
Mastering PostgreSQL Administration
Mastering PostgreSQL Administration
EDB
Postgresql database administration volume 1
Postgresql database administration volume 1
Federico Campoli
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
PostgreSQL-Consulting
Operating PostgreSQL at Scale with Kubernetes
Operating PostgreSQL at Scale with Kubernetes
Jonathan Katz
PostgreSQL replication
PostgreSQL replication
NTT DATA OSS Professional Services
Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기
NeoClova
Getting started with postgresql
Getting started with postgresql
botsplash.com
Recommended
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
Mastering PostgreSQL Administration
Mastering PostgreSQL Administration
EDB
Postgresql database administration volume 1
Postgresql database administration volume 1
Federico Campoli
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
PostgreSQL-Consulting
Operating PostgreSQL at Scale with Kubernetes
Operating PostgreSQL at Scale with Kubernetes
Jonathan Katz
PostgreSQL replication
PostgreSQL replication
NTT DATA OSS Professional Services
Maria db 이중화구성_고민하기
Maria db 이중화구성_고민하기
NeoClova
Getting started with postgresql
Getting started with postgresql
botsplash.com
Postgres connections at scale
Postgres connections at scale
Mydbops
PostgreSQL9.3 Switchover/Switchback
PostgreSQL9.3 Switchover/Switchback
Vibhor Kumar
PostgreSQL Replication High Availability Methods
PostgreSQL Replication High Availability Methods
Mydbops
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suite
Kenny Gryp
Vacuum in PostgreSQL
Vacuum in PostgreSQL
Rafia Sabih
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
Command Prompt., Inc
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
Mydbops
MinIO January 2020 Briefing
MinIO January 2020 Briefing
Jonathan Symonds
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
Alexey Lesovsky
patroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deployment
hyeongchae lee
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
PGConf APAC
Ceph issue 해결 사례
Ceph issue 해결 사례
Open Source Consulting
PostgreSQL HA
PostgreSQL HA
haroonm
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
ScaleGrid.io
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
NoSQLmatters
Pro Postgres 9
Pro Postgres 9
Robert Treat
MySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software Test
I Goo Lee
PostgreSQL Replication Tutorial
PostgreSQL Replication Tutorial
Hans-Jürgen Schönig
M|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScale
MariaDB plc
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova
PostgreSQL Security. How Do We Think?
PostgreSQL Security. How Do We Think?
Ohyama Masanori
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
DoKC
More Related Content
What's hot
Postgres connections at scale
Postgres connections at scale
Mydbops
PostgreSQL9.3 Switchover/Switchback
PostgreSQL9.3 Switchover/Switchback
Vibhor Kumar
PostgreSQL Replication High Availability Methods
PostgreSQL Replication High Availability Methods
Mydbops
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suite
Kenny Gryp
Vacuum in PostgreSQL
Vacuum in PostgreSQL
Rafia Sabih
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
Command Prompt., Inc
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
Mydbops
MinIO January 2020 Briefing
MinIO January 2020 Briefing
Jonathan Symonds
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
Alexey Lesovsky
patroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deployment
hyeongchae lee
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
PGConf APAC
Ceph issue 해결 사례
Ceph issue 해결 사례
Open Source Consulting
PostgreSQL HA
PostgreSQL HA
haroonm
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
ScaleGrid.io
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
NoSQLmatters
Pro Postgres 9
Pro Postgres 9
Robert Treat
MySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software Test
I Goo Lee
PostgreSQL Replication Tutorial
PostgreSQL Replication Tutorial
Hans-Jürgen Schönig
M|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScale
MariaDB plc
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova
What's hot
(20)
Postgres connections at scale
Postgres connections at scale
PostgreSQL9.3 Switchover/Switchback
PostgreSQL9.3 Switchover/Switchback
PostgreSQL Replication High Availability Methods
PostgreSQL Replication High Availability Methods
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suite
Vacuum in PostgreSQL
Vacuum in PostgreSQL
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
MinIO January 2020 Briefing
MinIO January 2020 Briefing
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
patroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deployment
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
Ceph issue 해결 사례
Ceph issue 해결 사례
PostgreSQL HA
PostgreSQL HA
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...
Pro Postgres 9
Pro Postgres 9
MySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software Test
PostgreSQL Replication Tutorial
PostgreSQL Replication Tutorial
M|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScale
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
Similar to Multi Master PostgreSQL Cluster on Kubernetes
PostgreSQL Security. How Do We Think?
PostgreSQL Security. How Do We Think?
Ohyama Masanori
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
DoKC
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
DoKC
FDW-based Sharding Update and Future
FDW-based Sharding Update and Future
Masahiko Sawada
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
Kohei KaiGai
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
aspyker
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
SeungYong Oh
Criteo Labs Infrastructure Tech Talk Meetup Nov. 7
Criteo Labs Infrastructure Tech Talk Meetup Nov. 7
Shuo LI
HKNOG 6.0 Next Generation Networks - will automation put us out of jobs?
HKNOG 6.0 Next Generation Networks - will automation put us out of jobs?
Tom Paseka
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
Linaro
Apache AGE and the synergy effect in the combination of Postgres and NoSQL
Apache AGE and the synergy effect in the combination of Postgres and NoSQL
EDB
Webinar slides: How to automate and manage MongoDB & Percona Server for MongoDB
Webinar slides: How to automate and manage MongoDB & Percona Server for MongoDB
Severalnines
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Omid Vahdaty
Software Stacks to enable SDN and NFV
Software Stacks to enable SDN and NFV
Yoshihiro Nakajima
PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo
PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo
Equnix Business Solutions
GitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your Repos
Weaveworks
PostgreSQL Security. How Do We Think? at PGCon 2017
PostgreSQL Security. How Do We Think? at PGCon 2017
Ohyama Masanori
WekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound Again
inside-BigData.com
When it all GOes right
When it all GOes right
Pavlo Golub
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
Equnix Business Solutions
Similar to Multi Master PostgreSQL Cluster on Kubernetes
(20)
PostgreSQL Security. How Do We Think?
PostgreSQL Security. How Do We Think?
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
Why run Postgres in Kubernetes?
FDW-based Sharding Update and Future
FDW-based Sharding Update and Future
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Criteo Labs Infrastructure Tech Talk Meetup Nov. 7
Criteo Labs Infrastructure Tech Talk Meetup Nov. 7
HKNOG 6.0 Next Generation Networks - will automation put us out of jobs?
HKNOG 6.0 Next Generation Networks - will automation put us out of jobs?
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
Apache AGE and the synergy effect in the combination of Postgres and NoSQL
Apache AGE and the synergy effect in the combination of Postgres and NoSQL
Webinar slides: How to automate and manage MongoDB & Percona Server for MongoDB
Webinar slides: How to automate and manage MongoDB & Percona Server for MongoDB
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Software Stacks to enable SDN and NFV
Software Stacks to enable SDN and NFV
PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo
PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo
GitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your Repos
PostgreSQL Security. How Do We Think? at PGCon 2017
PostgreSQL Security. How Do We Think? at PGCon 2017
WekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound Again
When it all GOes right
When it all GOes right
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
Recently uploaded
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
Safe Software
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)
Hr365.us smith
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Ahmed Mohamed
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
Hironori Washizaki
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Natan Silnitsky
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
StefanoLambiase
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
Alina Yurenko
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Stefano Stabellini
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
team-WIBU
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
BrainSell Technologies
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
vyaparkranti
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
Sujith Sukumaran
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
andrehoraa
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
OnePlan Solutions
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
Dinusha Kumarasiri
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
VICTOR MAESTRE RAMIREZ
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Drew Moseley
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
Philip Schwarz
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
OnePlan Solutions
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
Velvetech LLC
Recently uploaded
(20)
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
Multi Master PostgreSQL Cluster on Kubernetes
1.
Copyright©2017 NTT corp.
All Rights Reserved. Multi-Master PostgreSQL Cluster on Kubernetes PGConf.Asia 2017 NTT OSSセンタ Masanori Oyama/⼤⼭真実
2.
2Copyright©2017 NTT corp.
All Rights Reserved. Who Am I Masanori Oyama / ⼤⼭ 真実 twitter @ooyamams1987 slideshare https://www.slideshare.net/ooyamams Work l PostgreSQL engineering support and consultation. l Extensive quality verification of PostgreSQL releases. l PostgreSQL Security Ø PostgreSQL Security. How Do We Think? at PGCon 2017 https://www.slideshare.net/ooyamams/postgresql-security-how-do-we-think-at-pgcon-2017 Prev work l Hadoop engineering support and consultation at NTT DATA Corp. Ø I had managed a big Hadoop cluster (1000 node, 200PB!).
3.
3Copyright©2017 NTT corp.
All Rights Reserved. I will talk about ... • Introduce Next Generation Multi Master PostgreSQL Cluster and share some details about a PoC of IoT log platform. • Share the knowledge of PostgreSQL cluster management using Kubernetes. Today’sTopic
4.
4Copyright©2017 NTT corp.
All Rights Reserved. I will talk about ... • Introduce Next Generation Multi Master PostgreSQL Cluster and share some details about a PoC of IoT log platform. • Share the knowledge of PostgreSQL cluster management using Kubernetes. Today’sTopic Please read the today's morning presentation material "Built-in Sharding“! I will focus on this today.
5.
5Copyright©2017 NTT corp.
All Rights Reserved. I will talk about ... • Introduce Next Generation Multi Master PostgreSQL Cluster and share some details about a PoC of IoT log platform. • Share the knowledge of PostgreSQL cluster management using Kubernetes. Today’sTopic Please read the today's morning presentation material "Built-in Sharding“! I will focus on this today.
6.
6Copyright©2017 NTT corp.
All Rights Reserved. 1. Introduction 2. Crunchy PostgreSQL Containers on OpenShift 3. Multi-Master PostgreSQL Cluster on OpenShift 4. Conclusion Table of Contents
7.
7Copyright©2017 NTT corp.
All Rights Reserved. 1. Introduction 2. Crunchy PostgreSQL Containers on OpenShift 3. Multi-Master PostgreSQL Cluster on OpenShift 4. Conclusion Table of Contents
8.
8Copyright©2017 NTT corp.
All Rights Reserved. Postgres-XC • Multi-Master PostgreSQL Cluster forked from PostgreSQL by NTT • Scale out for Read/Write workloads • The development ended in 2014 1. Introduction Coordinators DataNodes Global Transaction Manager Clients Now, We are contributing back this experiences to the PostgreSQL core. Please see “PostgreSQL Built-in Sharding” of today's morning presentation.
9.
9Copyright©2017 NTT corp.
All Rights Reserved. Why scale out? It is hard to improve write performance by scale up (by purchasing high- performance hardware). 1. Introduction Major bottleneck points of PostgreSQL • Get transaction snapshot • Write WAL There is a limit of performance improvement even if you use many core server and NVMe. To solve these bottlenecks, a drastic improvement is necessary. • New Transaction Mechanism for many core machine • Parallel WAL Mechanism or New WAL Writer Mechanism for NVMe Scale out of PostgreSQL is one of the workarounds to avoid these bottlenecks.
10.
10Copyright©2017 NTT corp.
All Rights Reserved. “Instagram” Way Tantan told about the homemade version similar to the “Instagram” way at last year PGconf.Asia. 1. Introduction http://www.pgconf.asia/JP/wp-content/uploads/2016/12/From-0-to-350bn-rows-in-2-years-PgConfAsia2016-v1.pdf Applications need to control which databases are accessed and satisfy the ACID properties with transaction. l 22TB / 350bn rows Biggest table. l 270k tuple writes / sec. l 1.3M TPS Aggregated over all databases
11.
11Copyright©2017 NTT corp.
All Rights Reserved. PostgreSQL Built-in Sharding 1. Introduction Worker node 3 PostgreSQL community is working very hard to realize PostgreSQL Built-in Sharding. Worker node 2Worker node 1
12.
12Copyright©2017 NTT corp.
All Rights Reserved. Even if we could get a Perfect PostgreSQL Cluster, we have to spend a lot of time to manage the cluster. For example, • Orchestration • HA and Recovery • Backup • Monitoring • Security • Resource Management • Load Balancing • Scale out (Add/Delete Coordinator/Shard) 1. Introduction
13.
13Copyright©2017 NTT corp.
All Rights Reserved. Distributed systems are hard!
14.
14Copyright©2017 NTT corp.
All Rights Reserved. The Difficulty of Scale Out For example, about High Availability. With a ordinary PostgreSQL, we have to use a middleware to achieve HA. 1. Introduction PG-REX PostgreSQL HA solution with Pacemaker How about the PostgreSQL cluster? https://www.slideshare.net/kazuhcurry/pgrexpacemaker (Japanese)
15.
15Copyright©2017 NTT corp.
All Rights Reserved. HA of PostgreSQL-XC 1. Introduction https://wiki.postgresql.o rg/images/4/44/Pgxc_H A_20121024.pdf
16.
16Copyright©2017 NTT corp.
All Rights Reserved. HA of PostgreSQL-XC 1. Introduction GTM Server Coordinator and DataNode Servers
17.
17Copyright©2017 NTT corp.
All Rights Reserved. HA of PostgreSQL-XC 10 shard How many HA settings are needed!? 1. Introduction GTM Server Coordinator and DataNode Servers
18.
18Copyright©2017 NTT corp.
All Rights Reserved. HA of PostgreSQL-XC 10 shard How many HA settings are needed!? 1. Introduction GTM Server Coordinator and DataNode Servers I don’t want to manage this by myself !
19.
19Copyright©2017 NTT corp.
All Rights Reserved. How do people solve this? PostgreSQL Related Slides and Presentations https://wiki.postgresql.org/wiki/PostgreSQL_Related_Slides_and_Presentations PGCONF.EU 2017 • USING KUBERNETES, DOCKER, AND HELM TO DEPLOY ON-DEMAND POSTGRESQL STREAMING REPLICAS https://www.postgresql.eu/events/sessions/pgconfeu2017/session/1559/slides/35/Using%20Kubernetes,% 20Docker,%20and%20Helm%20to%20Deploy%20On-Demand%20PostgreSQL%20Streaming%20Replicas.pdf Postgres Open 2017 • A Kubernetes Operator for PostgreSQL - Architecture and Design https://www.sarahconway.com/slides/postgres-operator.pdf • Containerized Clustered PostgreSQL http://jberkus.github.io/container_cluster_pg/#23 PGConf 2017 • Patroni - HA PostgreSQL made easy https://www.slideshare.net/AlexanderKukushkin1/patroni-ha-postgresql-made-easy • PostgreSQL High Availability in a Containerized World http://jkshah.blogspot.jp/2017/03/pgconf-2017-postgresql-high.html 1. Introduction
20.
20Copyright©2017 NTT corp.
All Rights Reserved. How do people solve this? PostgreSQL Related Slides and Presentations https://wiki.postgresql.org/wiki/PostgreSQL_Related_Slides_and_Presentations PGCONF.EU 2017 • USING KUBERNETES, DOCKER, AND HELM TO DEPLOY ON-DEMAND POSTGRESQL STREAMING REPLICAS https://www.postgresql.eu/events/sessions/pgconfeu2017/session/1559/slides/35/Using%20Kubernetes,% 20Docker,%20and%20Helm%20to%20Deploy%20On-Demand%20PostgreSQL%20Streaming%20Replicas.pdf Postgres Open 2017 • A Kubernetes Operator for PostgreSQL - Architecture and Design https://www.sarahconway.com/slides/postgres-operator.pdf • Containerized Clustered PostgreSQL http://jberkus.github.io/container_cluster_pg/#23 PGConf 2017 • Patroni - HA PostgreSQL made easy https://www.slideshare.net/AlexanderKukushkin1/patroni-ha-postgresql-made-easy • PostgreSQL High Availability in a Containerized World http://jkshah.blogspot.jp/2017/03/pgconf-2017-postgresql-high.html 1. Introduction Container and Kubernetes
21.
21Copyright©2017 NTT corp.
All Rights Reserved. 3.PostgreSQL on Kubernetes “standing on the shoulders of giants” for getting the Open Source Power! https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants
22.
22Copyright©2017 NTT corp.
All Rights Reserved. PostgreSQL on Kubernetes Projects • AtomicDB https://hackernoon.com/postgresql-cluster-into-kubernetes-cluster-f353cde212de • Crunchy PostgreSQL Containers (CPC) https://github.com/CrunchyData/crunchy-containers • Patroni https://github.com/zalando/patroni • Stolon https://github.com/sorintlab/stolon We are trying CPC on OpenShift origin! 1. Introduction
23.
23Copyright©2017 NTT corp.
All Rights Reserved. 1. Introduction 2. Crunchy PostgreSQL Containers on OpenShift 3. Multi-Master PostgreSQL Cluster on OpenShift 4. Conclusion Table of Contents
24.
24Copyright©2017 NTT corp.
All Rights Reserved. What is Kubernetes? • Kubernetes is an open-source system for management of containerized applications. • Pod is the basic scheduling unit which consists of one or more containers. • Kubernetes follows the master- slave architecture. 1. Introduction https://en.wikipedia.org/wiki/Kubernetes
25.
25Copyright©2017 NTT corp.
All Rights Reserved. 2.Crunchy PostgreSQL Containers on OpenShift https://blog.openshift.com/enterprise-ready-kubernetes/
26.
26Copyright©2017 NTT corp.
All Rights Reserved. 2.Crunchy PostgreSQL Containers on OpenShift Crunchy PostgreSQL Containers • Docker 1.12 and above • Openshift 3.4 and above • Kubernetes 1.5 and above https://github.com/CrunchyData/crunchy-containers
27.
27Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Pod manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: primary - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password volumeMounts: - mountPath: /pgdata name: pgdata readonly: false volumes: - name: pgdata emptyDir: {} Define the Pod state like SQL, not the procedure. • simple_postgres_pod.yml
28.
28Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Pod manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: primary - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password volumeMounts: - mountPath: /pgdata name: pgdata readonly: false volumes: - name: pgdata emptyDir: {} Define the Pod state like SQL, not the procedure. • simple_postgres_pod.yml PostgreSQL Container settings
29.
29Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Pod manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: primary - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password volumeMounts: - mountPath: /pgdata name: pgdata readonly: false volumes: - name: pgdata emptyDir: {} Define the Pod state like SQL, not the procedure. • simple_postgres_pod.yml pull the container image from DockerHub PostgreSQL Container settings
30.
30Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Pod manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: primary - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password volumeMounts: - mountPath: /pgdata name: pgdata readonly: false volumes: - name: pgdata emptyDir: {} Define the Pod state like SQL, not the procedure. PostgreSQL Container settings pull the container image from DockerHub PostgreSQL settings • simple_postgres_pod.yml
31.
31Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Pod manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: primary - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password volumeMounts: - mountPath: /pgdata name: pgdata readonly: false volumes: - name: pgdata emptyDir: {} Define the Pod state like SQL, not the procedure. PostgreSQL Container settings pull the container image from DockerHub PostgreSQL settings Pod storage settings • simple_postgres_pod.yml
32.
32Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Service manifest file. • simple_postgres_svc.yml 2.Crunchy PostgreSQL Containers on OpenShift kind: Service apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: ports: - protocol: TCP port: 5432 targetPort: 5432 nodePort: 0 selector: name: simple-pg type: ClusterIP sessionAffinity: None status: loadBalancer: {}
33.
33Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Write the Service manifest file. • simple_postgres_svc.yml 2.Crunchy PostgreSQL Containers on OpenShift kind: Service apiVersion: v1 metadata: name: simple-pg labels: name: simple-pg spec: ports: - protocol: TCP port: 5432 targetPort: 5432 nodePort: 0 selector: name: simple-pg type: ClusterIP sessionAffinity: None status: loadBalancer: {} This Service sends packets to TCP port 5432 on any Pod with the "name: simple-pg" label.
34.
34Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Execute “oc create” to create the Simple PostgreSQL pod and service. 2.Crunchy PostgreSQL Containers on OpenShift $ oc create -f simple_postgres_svc.yml service "simple-pg" created $ oc create -f simple_postgres_pod.yml pod "simple-pg" created service: simple-pg The simple-pg service sends SQLs to the simple-pg pod. pod: simple-pg labels: name: simple-pg simple-pg.mypj01.svc.cluster.local Clients
35.
35Copyright©2017 NTT corp.
All Rights Reserved. How to run a simple PostgreSQL Login to the hostname “simple-pg.mypj01.svc.cluster.local”. 2.Crunchy PostgreSQL Containers on OpenShift $ psql -h simple-pg.mypj01.svc.cluster.local -U testuser -d userdb Password for user testuser: userdb=> ¥d List of relations Schema | Name | Type | Owner ----------+--------------------+-------+---------- public | pg_stat_statements | view | postgres testuser | testtable | table | testuser service: simple-pg pod: simple-pg labels: name: simple-pg simple-pg.mypj01.svc.cluster.local Clients
36.
36Copyright©2017 NTT corp.
All Rights Reserved. Let’s try to add a replication! 2.Crunchy PostgreSQL Containers on OpenShift simple-pg.mypj01.svc.cluster.local service: simple-pg-rep pod: simple-pg-rep simple-pg-rep.mypj01.svc.cluster.local streaming replication Add replication Pod and Service Clients service: simple-pg pod: simple-pg
37.
37Copyright©2017 NTT corp.
All Rights Reserved. Let’s try to add a replication! Write the Pod and Service manifest file. 2.Crunchy PostgreSQL Containers on OpenShift kind: Pod apiVersion: v1 metadata: name: simple-pg-rep labels: name: simple-pg-rep spec: containers: - name: postgres image: crunchydata/crunchy-postgres:centos7- 10.1-1.7.0 ports: - containerPort: 5432 protocol: TCP env: - name: PGHOST value: /tmp - name: PG_PRIMARY_USER value: primaryuser - name: PG_PRIMARY_PORT value: '5432' - name: PG_MODE value: replica - name: PG_PRIMARY_PASSWORD value: password - name: PG_USER value: testuser - name: PG_PASSWORD value: password - name: PG_DATABASE value: userdb - name: PG_ROOT_PASSWORD value: password - name: PG_PRIMARY_HOST value: simple-pg - name: PG_PRIMARY_PORT value: '5432' volumeMounts: - mountPath: /pgdata name: pgdata readOnly: false volumes: - name: pgdata emptyDir: {} • simple_postgres_pod-rep.yml Crunchy-postgres runs a shell script for initialization when the container starts. The PG_MODE variable is used to determine the role(Primary Postgres or Replica Postgres).
38.
38Copyright©2017 NTT corp.
All Rights Reserved. Let’s try to add a replication! Execute “oc create” to create the Replication pod and service. 2.Crunchy PostgreSQL Containers on OpenShift $ oc create -f simple_postgres_rep_svc.yml service "simple-pg-rep" created $ oc create -f simple_postgres_rep_pod.yml pod "simple-pg-rep" created
39.
39Copyright©2017 NTT corp.
All Rights Reserved. Let’s try to add a replication! Check replication status 2.Crunchy PostgreSQL Containers on OpenShift $ psql -h simple-pg.mypj01.svc.cluster.local -U postgres -d userdb userdb=# select * from pg_stat_replication; -[ RECORD 1 ]----+------------------------------ pid | 329 usesysid | 16391 usename | primaryuser application_name | simple-pg-rep client_addr | 10.131.0.1 ... sync_state | async
40.
40Copyright©2017 NTT corp.
All Rights Reserved. Where is PostgreSQL running on OpenShif Cluster? 2.Crunchy PostgreSQL Containers on OpenShift master01 infra01 worker01 worker02 worker03 worker04 worker05 worker06 OpenShift Cluster (8 virtual machines) simple-pg simple-pg-rep $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE simple-pg 1/1 Running 0 7m 10.129.2.20 worker04 simple-pg-rep 1/1 Running 0 5m 10.131.0.16 worker05 We don’t have to manage these IP addresses and where these containers are located!
41.
41Copyright©2017 NTT corp.
All Rights Reserved. At first, what should we think about when running PostgreSQL on OpenShift(Kubernetes)? • What should we use for a Persistent storage? • Local disk • (Shared) Network storage • What should we do in case of server/container failures? • Failover • Restart PostgreSQL 2.Crunchy PostgreSQL Containers on OpenShift
42.
42Copyright©2017 NTT corp.
All Rights Reserved. At first, what should we think about when running PostgreSQL on OpenShift(Kubernetes)? • What should we use for a Persistent storage? • Local disk • (Shared) Network storage • What should we do in case of server/container failures? • Failover • Restart PostgreSQL 2.Crunchy PostgreSQL Containers on OpenShift
43.
43Copyright©2017 NTT corp.
All Rights Reserved. worker01 worker02 What should we use for storage? 2.Crunchy PostgreSQL Containers on OpenShift Local disk Each worker node has several HDD/SSD.
44.
44Copyright©2017 NTT corp.
All Rights Reserved. What should we use for storage? 2.Crunchy PostgreSQL Containers on OpenShift (Shared) Network storage Each worker node connects to one or more network storages. worker01 worker02
45.
45Copyright©2017 NTT corp.
All Rights Reserved. PersistentVolume plug-in 2.Crunchy PostgreSQL Containers on OpenShift OpenShift Origin supports the following PersistentVolume plug-ins: • NFS • HostPath • GlusterFS • Ceph RBD • OpenStack Cinder • AWS Elastic Block Store (EBS) • GCE Persistent Disk • iSCSI • Fibre Channel • Azure Disk • Azure File • VMWare vSphere • Local https://docs.openshift.org/latest/architecture/additional_concepts/storage.html apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle nfs: path: /tmp server: 172.17.0.2 NFS example
46.
46Copyright©2017 NTT corp.
All Rights Reserved. At first, what should we think about when running PostgreSQL on OpenShift(Kubernetes)? • What should we use for a Persistent storage? • Local disk • (Shared) Network storage • What should we do in case of server/container failures? • Failover • Restart PostgreSQL 2.Crunchy PostgreSQL Containers on OpenShift
47.
47Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 replication Primary Secondary worker03 Watch Container
48.
48Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 replication Primary Secondary worker03 Watch Container Execute pg_isready repeatedly Check a trigger file repeatedly
49.
49Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 replication Primary Secondary worker03 Watch Container Execute pg_isready repeatedly Check a trigger file repeatedly
50.
50Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 Primary Secondary worker03 Watch Container Check a trigger file repeatedly Set a trigger file
51.
51Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 Primary Secondary worker03 Watch Container Set a trigger file Promote!
52.
52Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift • Failover CPC provides a Watch Container to manage failover. worker01 worker02 Primary worker03 Watch Container Set a trigger file Promote! Secondary -> Primary Change the label
53.
53Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Mount the network storage
54.
54Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Execute pg_isready repeatedly Liveness Probe Mount the network storage
55.
55Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Execute pg_isready repeatedly Liveness Probe Mount the network storage Check Liveness Probe results.
56.
56Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Liveness Probe Check Liveness Probe results.
57.
57Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Restart the PostgreSQL pod on other worker nodes
58.
58Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Restart the PostgreSQL pod on other worker nodes
59.
59Copyright©2017 NTT corp.
All Rights Reserved. What should we do in case of server/container failures? 2.Crunchy PostgreSQL Containers on OpenShift Primary • Restart PostgreSQL OpenShift(Kubernetes) provides Liveness Probe to confirm whether the container is alive. worker01 worker02 master01 OpenShift Master Restart the PostgreSQL pod on other worker nodes Mount the same network storage
60.
60Copyright©2017 NTT corp.
All Rights Reserved. Choose a suitable method for your environments. Persistent storage • Local disk • (Shared) Network storage in case of server/container failures • Failover • Restart PostgreSQL 2.Crunchy PostgreSQL Containers on OpenShift
61.
61Copyright©2017 NTT corp.
All Rights Reserved. CPC other examples 2.Crunchy PostgreSQL Containers on OpenShift crunchy-collect The crunchy-collect container gathers different metrics from the crunchy-postgres container and pushes these to the Prometheus Promgateway. https://github.com/CrunchyData/crunchy-containers/blob/master/docs/metrics.adoc
62.
62Copyright©2017 NTT corp.
All Rights Reserved. Advantages of PostgreSQL on Kubernetes Easy to run PostgreSQL (including HA, monitoring etc.). ØDefine the state, not the procedure. The procedure is packed into the container with portability. ØThe network is abstracted. This makes easier to define the state. Disadvantages of PostgreSQL on Kubernetes Hard to manage Kubernetes cluster. (maybe...) ØIt seems to me that the complexity of PostgreSQL layer is shifted to Kubernetes layer. ØBut if you are already running a Kubernetes cluster (and that will come soon?) I believe this will not be a problem. 2.Crunchy PostgreSQL Containers on OpenShift
63.
63Copyright©2017 NTT corp.
All Rights Reserved. 1. Introduction 2. Crunchy PostgreSQL Containers on OpenShift 3. Multi-Master PostgreSQL Cluster on OpenShift 4. Conclusion Table of Contents
64.
64Copyright©2017 NTT corp.
All Rights Reserved. Try to install Built-in Sharding base Multi-Master PostgreSQL Cluster on OpenShift. 3.Multi-Master PostgreSQL Cluster on OpenShift Clients pg-coordinators FDW FDW pg-shards parent table partition table • The PostgreSQL Container contains the same source code which was demonstrated in the today's morning session "Built-in Sharding Special Version” and the modified scripts of Crunchy-postgres container. $ docker pull ooyamams/postgres-dev • Using NFS as a shared network storage for test purposes. • pg-coordinator Pod and pg-shard Pod are created by Templates and controlled by DeploymentConfig, not StatefulSet.
65.
65Copyright©2017 NTT corp.
All Rights Reserved. kind: Template apiVersion: v1 metadata: name: pg-coordinator creationTimestamp: null annotations: description: PostgreSQL Multi-Master Build in Sharding Example iconClass: icon-database tags: database,postgresql parameters: - name: PG_PRIMARY_USER description: The username used for primary / replica replication value: primaryuser ...(omit)... objects: - kind: Service apiVersion: v1 ...(omit)... - kind: DeploymentConfig apiVersion: v1 metadata: name: ${PG_PRIMARY_SERVICE_NAME} creationTimestamp: null spec: strategy: type: Recreate resources: {} triggers: - type: ConfigChange replicas: 2 3.Multi-Master PostgreSQL Cluster on OpenShift selector: name: ${PG_PRIMARY_SERVICE_NAME} template: metadata: creationTimestamp: null labels: name: ${PG_PRIMARY_SERVICE_NAME} spec: serviceAccount: pg-cluster-sa containers: - name: server image: 172.30.81.49:5000/mypj01/postgres- dev:build-in-sharding-4 livenessProbe: exec: command: - /opt/cpm/bin/liveness.sh initialDelaySeconds: 90 timeoutSeconds: 1 ports: - containerPort: 5432 protocol: TCP env: - name: PG_PRIMARY_HOST value: ${PG_PRIMARY_SERVICE_NAME} ...(omit)... resources: {} terminationMessagePath: /dev/termination-log securityContext: privileged: false volumeMounts: - mountPath: /pgdata name: pgdata readOnly: false volumes: - name: pgdata emptyDir: {} restartPolicy: Always dnsPolicy: ClusterFirst status: {} pg-coordinator.yml
66.
66Copyright©2017 NTT corp.
All Rights Reserved. Try to install Built-in Sharding base Multi-Master PostgreSQL Cluster on OpenShift. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc create -f pg-coordinator.yml • Create Template for pg-coordinator. $ oc process pg-coordinator | oc create -f - • Create Service and DeploymentConfig from the Template. pg-coordinator pg-coordinator pg-coordinator.mypj01 .svc.cluster.local pg-coordinator clients
67.
67Copyright©2017 NTT corp.
All Rights Reserved. Try to install Built-in Sharding base Multi-Master PostgreSQL Cluster on OpenShift. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc create -f pg-coordinator.yml • Create Template for pg-coordinator. $ oc process pg-coordinator | oc create -f - • Create Service and DeploymentConfig from the Template. Round robin load balancing pg-coordinator pg-coordinator pg-coordinator.mypj01 .svc.cluster.local pg-coordinator clients
68.
68Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift kind: Template apiVersion: v1 metadata: name: pg-shared creationTimestamp: null annotations: description: PostgreSQL Multi-Master Build in Sharding Example iconClass: icon-database tags: database,postgresql parameters: - name: PG_PRIMARY_USER description: The username used for primary / replica replication value: primaryuser ...(omit)... - name: PGDATA_PATH_OVERRIDE value: shared-01 objects: - kind: Service apiVersion: v1 metadata: name: ${PG_PRIMARY_SERVICE_NAME} labels: name: ${PG_PRIMARY_SERVICE_NAME} spec: ports: - protocol: TCP port: 5432 targetPort: 5432 nodePort: 0 selector: name: ${PG_PRIMARY_SERVICE_NAME} clusterIP: None - kind: DeploymentConfig apiVersion: v1 metadata: name: ${PG_PRIMARY_SERVICE_NAME} creationTimestamp: null spec: serviceName: ${PG_PRIMARY_SERVICE_NAME} strategy: ...(omit)... labels: name: ${PG_PRIMARY_SERVICE_NAME} spec: serviceAccount: pg-sa containers: - name: ${PG_PRIMARY_SERVICE_NAME} image: 172.30.81.49:5000/mypj01/postgres- dev:build-in-sharding-1 livenessProbe: exec: command: - /opt/cpm/bin/liveness.sh initialDelaySeconds: 90 timeoutSeconds: 1 ports: - containerPort: 5432 protocol: TCP env: - name: PG_PRIMARY_HOST value: ${PG_PRIMARY_SERVICE_NAME} ...(omit)... resources: {} terminationMessagePath: /dev/termination-log securityContext: privileged: false volumeMounts: - name: pgdata mountPath: /pgdata readOnly: false volumes: - name: pgdata persistentVolumeClaim: claimName: crunchy-pvc pg-shard.yml
69.
69Copyright©2017 NTT corp.
All Rights Reserved. Try to install Built-in Sharding base Multi-Master PostgreSQL Cluster on OpenShift. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc create -f pg-shard.yml • Create Template for pg-shard. $ oc process pg-shard -p PG_PRIMARY_SERVICE_NAME=pg-shard-00 -p PGDATA_PATH_OVERRIDE=pg-shard-00 | oc create -f - $ oc process pg-shard -p PG_PRIMARY_SERVICE_NAME=pg-shard-01 -p PGDATA_PATH_OVERRIDE=pg-shard-01 | oc create -f - • Create Service and DeploymentConfig from the Template. pg-shard-00 pg-shard-00.mypj01 .svc.cluster.local pg-shard-01 pg-shard-01.mypj01 .svc.cluster.local pg-shard-00 pg-shard-01 pg-shard-01 pg-shard-00 pg-coordinator pg-coordinator pg-coordinator.mypj01 .svc.cluster.local pg-coordinator clients Round robin load balancing
70.
70Copyright©2017 NTT corp.
All Rights Reserved. Check Physical cluster state. 3.Multi-Master PostgreSQL Cluster on OpenShift infra01 worker02 $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE pg-coordinator-1-drtsv 1/1 Running 0 4m 10.131.0.37 worker05 pg-coordinator-1-rmkb8 1/1 Running 0 4m 10.130.0.50 worker03 pg-shard-00-1-qqwtr 1/1 Running 0 4m 10.129.0.68 worker02 pg-shard-01-1-ht4jn 1/1 Running 0 3m 10.130.0.51 worker03 master01 worker01 worker04 worker03 worker06 worker05 NFS server OpenShift Cluster (8 virtual machines) pg-coordinator pg-shard legends pg-shard-01 pg-shard-00 OpenShift Master
71.
71Copyright©2017 NTT corp.
All Rights Reserved. Execute SQL to the cluster. The same schema and data as "Sharding in Build" in the morning session. 3.Multi-Master PostgreSQL Cluster on OpenShift $ psql -h pg-coordinator.mypj01.svc.cluster.local -U postgres -d postgres ¥ -c "explain analyze select * from flight_bookings"; QUERY PLAN ----------------------------------------------------------------------------------------- Append (cost=100.00..254.09 rows=974 width=144) (actual time=1.945..40.289 rows=7499 loops=1) -> Foreign Scan on flight_bookings1 (cost=...) (actual time=...) -> Foreign Scan on flight_bookings0 (cost=...) (actual time=...) Planning time: 1.930 ms Execution time: 60.319 ms (5 rows) pg-shard-00 pg-shard-00.mypj01 .svc.cluster.local pg-shard-01 pg-shard-01.mypj01 .svc.cluster.local pg-shard-00 pg-shard-01 pg-shard-01 pg-shard-00 pg-coordinator pg-coordinator pg-coordinator.mypj01 .svc.cluster.local pg-coordinator clients Round robin load balancing
72.
72Copyright©2017 NTT corp.
All Rights Reserved. Try worker03 to crash. 3.Multi-Master PostgreSQL Cluster on OpenShift Execute this command to generate kernel panic on worker 03. $ oc get node NAME STATUS AGE VERSION infra01 Ready 26d v1.7.6+a08f5eeb62 master01 Ready,SchedulingDisabled 26d v1.7.6+a08f5eeb62 worker01 Ready 26d v1.7.6+a08f5eeb62 worker02 Ready 26d v1.7.6+a08f5eeb62 worker03 NotReady 26d v1.7.6+a08f5eeb62 worker04 Ready 26d v1.7.6+a08f5eeb62 worker05 Ready 26d v1.7.6+a08f5eeb62 worker06 Ready 26d v1.7.6+a08f5eeb62 $ sudo sh -c "echo c > /proc/sysrq-trigger" Worker 03 status changes “NotReady”.
73.
73Copyright©2017 NTT corp.
All Rights Reserved. Containers are re-creating. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE pg-coordinator-1-drtsv 1/1 Running 0 8m 10.131.0.37 worker05 pg-coordinator-1-rmkb8 1/1 Terminating 0 8m 10.130.0.50 worker03 pg-coordinator-1-wk4wx 0/1 ContainerCreating 0 2s <none> worker01 pg-shard-00-1-qqwtr 1/1 Running 0 8m 10.129.0.68 worker02 pg-shard-01-1-ht4jn 1/1 Terminating 0 8m 10.130.0.51 worker03 pg-shard-01-1-ztdn4 0/1 ContainerCreating 0 2s <none> worker01 infra01 worker02 master01 worker01 worker04 worker03 worker06 worker05 NFS server OpenShift Cluster (8 virtual machines) pg-coordinator pg-shard legends pg-shard-01 pg-shard-00 OpenShift Master New container
74.
74Copyright©2017 NTT corp.
All Rights Reserved. Re-execute SQL. 3.Multi-Master PostgreSQL Cluster on OpenShift $ psql -h pg-coordinator.mypj01.svc.cluster.local -U postgres -d postgres ¥ -c "explain analyze select * from flight_bookings"; QUERY PLAN --------------------------------------------------------------------------------------- Append (cost=100.00..254.09 rows=974 width=144) (actual time=15.531..63.278 rows=7499 loops=1) -> Foreign Scan on flight_bookings1 (cost=...) (actual time=...) -> Foreign Scan on flight_bookings0 (cost=...) (actual time=...) Planning time: 1.716 ms Execution time: 116.989 ms (5 rows) pg-shard-00 pg-shard-00.mypj01 .svc.cluster.local pg-shard-01 pg-shard-01.mypj01 .svc.cluster.local pg-shard-00 pg-shard-01 pg-shard-01 pg-shard-00 pg-coordinator pg-coordinator pg-coordinator.mypj01 .svc.cluster.local pg-coordinator clients Round robin load balancing
75.
75Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift pg-coordinators don't have persistent storage. So they lost the cluster configurations if restarted. worker01 worker03 pg-coordinators worker02 pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 mount parent table FDW parent table FDW
76.
76Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift pg-coordinators don't have persistent storage. So they lost the cluster configurations if restarted. worker01 worker03 pg-coordinators worker02 pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 mount parent table FDW parent table FDW
77.
77Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift pg-coordinators don't have persistent storage. So they lost the cluster configurations if restarted. worker01 worker03 pg-coordinators worker02 pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 re-mount parent table FDW parent table FDW
78.
78Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift pg-coordinators don't have persistent storage. So they lost the cluster configurations if restarted. worker01 worker03 pg-coordinators worker02 pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 re-mount parent table FDW parent table FDW
79.
79Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift pg-coordinators don't have persistent storage. So they lost the cluster configurations if restarted. worker01 worker03 pg-coordinators worker02 pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 re-mount FDW parent table FDW empty Lost configurations! ? parent table
80.
80Copyright©2017 NTT corp.
All Rights Reserved. Custom Resource Definitions (CRD) We try to use CDR(CustomResourceDefinitions) for sharing the cluster configurations. 3.Multi-Master PostgreSQL Cluster on OpenShift • A image of the HTTP API space of kubernetes /api/v1/mypj01 /api/v1/mypj01/pods/... /api/v1/mypj01/services/... /api/v1/mypj01/....
81.
81Copyright©2017 NTT corp.
All Rights Reserved. Custom Resource Definitions (CRD) We try to use CDR(CustomResourceDefinitions) for sharing the cluster configurations. 3.Multi-Master PostgreSQL Cluster on OpenShift -> CDR can extend the kubernetes core API. • A image of the HTTP API space of kubernetes /api/v1/mypj01 /api/v1/mypj01/pods/... /api/v1/mypj01/services/... /api/v1/mypj01/.... /api/v1/mypj01/pg-cluster/...
82.
82Copyright©2017 NTT corp.
All Rights Reserved. Write the CDR manifest file. 3.Multi-Master PostgreSQL Cluster on OpenShift $ cat pg-cluster-shard-information.yml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: shard-informations.pg-cluster.example.com spec: group: pg-cluster.example.com version: v1 scope: Namespaced names: plural: shard-information singular: shard-information kind: Shard-Info shortNames: - si This CRD can be access by “oc get shard-information“ or “oc get si“ $ oc get shard-information No resources found.
83.
83Copyright©2017 NTT corp.
All Rights Reserved. Write the shard-information manifest file. 3.Multi-Master PostgreSQL Cluster on OpenShift $ cat defines/sharding_info_coordinator.yml apiVersion: "pg-cluster.example.com/v1" kind: Shard-Info metadata: name: multi-master-demo-pgconfasia spec: coordinator: "create extension if not exists postgres_fdw; create server shard0 foreign data wrapper postgres_fdw options (host 'pg-shard-00.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create server shard1 foreign data wrapper postgres_fdw options (host 'pg-shard-01.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create user mapping for postgres server shard0 OPTIONS (user 'postgres', password 'password'); create user mapping for postgres server shard1 OPTIONS (user 'postgres', password 'password'); create table hotel_bookings (id serial, user_id int, booked_at timestamp, city_name text, continent text, flight_id int) partition by list (continent); create table flight_bookings (id serial, user_id int, booked_at timestamp, from_city text, from_continent text, to_city text, to_continent text) partition by list (to_continent); create table users (id serial, name text, age int); create foreign table flight_bookings0 partition of flight_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table hotel_bookings0 partition of hotel_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table flight_bookings1 partition of flight_bookings for values in ('Europe', 'Africa') server shard1; create foreign table hotel_bookings1 partition of hotel_bookings for values in ('Europe', 'Africa') server shard1;
84.
84Copyright©2017 NTT corp.
All Rights Reserved. Write the shard-information manifest file. 3.Multi-Master PostgreSQL Cluster on OpenShift $ cat defines/sharding_info_coordinator.yml apiVersion: "pg-cluster.example.com/v1" kind: Shard-Info metadata: name: multi-master-demo-pgconfasia spec: coordinator: "create extension if not exists postgres_fdw; create server shard0 foreign data wrapper postgres_fdw options (host 'pg-shard-00.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create server shard1 foreign data wrapper postgres_fdw options (host 'pg-shard-01.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create user mapping for postgres server shard0 OPTIONS (user 'postgres', password 'password'); create user mapping for postgres server shard1 OPTIONS (user 'postgres', password 'password'); create table hotel_bookings (id serial, user_id int, booked_at timestamp, city_name text, continent text, flight_id int) partition by list (continent); create table flight_bookings (id serial, user_id int, booked_at timestamp, from_city text, from_continent text, to_city text, to_continent text) partition by list (to_continent); create table users (id serial, name text, age int); create foreign table flight_bookings0 partition of flight_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table hotel_bookings0 partition of hotel_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table flight_bookings1 partition of flight_bookings for values in ('Europe', 'Africa') server shard1; create foreign table hotel_bookings1 partition of hotel_bookings for values in ('Europe', 'Africa') server shard1; There are SQLs for setting up to pg-coordinator.
85.
85Copyright©2017 NTT corp.
All Rights Reserved. We can get these SQLs through “oc” command. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc get si multi-master-demo-pgconfasia -o json | jq -r '.spec.coordinator' "create extension if not exists postgres_fdw; create server shard0 foreign data wrapper postgres_fdw options (host 'pg-shard-00.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create server shard1 foreign data wrapper postgres_fdw options (host 'pg-shard-01.mypj01.svc.cluster.local', dbname 'postgres', port '5432'); create user mapping for postgres server shard0 OPTIONS (user 'postgres', password 'password'); create user mapping for postgres server shard1 OPTIONS (user 'postgres', password 'password'); create table hotel_bookings (id serial, user_id int, booked_at timestamp, city_name text, continent text, flight_id int) partition by list (continent); create table flight_bookings (id serial, user_id int, booked_at timestamp, from_city text, from_continent text, to_city text, to_continent text) partition by list (to_continent); create table users (id serial, name text, age int); create foreign table flight_bookings0 partition of flight_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table hotel_bookings0 partition of hotel_bookings for values in ('Asia', 'Oceania') server shard0; create foreign table flight_bookings1 partition of flight_bookings for values in ('Europe', 'Africa') server shard1; create foreign table hotel_bookings1 partition of hotel_bookings for values in ('Europe', 'Africa') server shard1; pg-coordinator can also take out these SQLs from the Shard-Info CDR.
86.
86Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift worker03 pg-coordinators pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 mount mount parent table FDW empty Lost configurations! ? master01 OpenShift Master pg-shards pg-cluster.example.com/Shard-Info multi-master-demo-pgconfasia: spec: coordinator: create... pg-coordinator takes out the setting SQLs from the Shard- Info CDR when it is started.
87.
87Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift worker03 pg-coordinators pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 mount mount parent table FDW empty Lost configurations! ? master01 OpenShift Master pg-shards oc get si ... > /tmp/setup.sql psql -U postgres < tmp/setup.sql pg-cluster.example.com/Shard-Info multi-master-demo-pgconfasia: spec: coordinator: create... pg-coordinator takes out the setting SQLs from the Shard- Info CDR when it is started. Take out from the Shard-Info.
88.
88Copyright©2017 NTT corp.
All Rights Reserved. 3.Multi-Master PostgreSQL Cluster on OpenShift worker03 pg-coordinators pg-shards worker04 NFS server pg-shard-01 pg-shard-00 worker05 worker06 mount mount parent table FDW master01 OpenShift Master pg-shards parent table FDW configured ! pg-cluster.example.com/Shard-Info multi-master-demo-pgconfasia: spec: coordinator: create... pg-coordinator takes out the setting SQLs from the Shard- Info CDR when it is started.
89.
89Copyright©2017 NTT corp.
All Rights Reserved. Let’s scale out pg-coordinator. 3.Multi-Master PostgreSQL Cluster on OpenShift $ oc scale dc pg-coordinator --replicas=3 deploymentconfig "pg-coordinator" scaled • Set “replicas” 2 -> 3 $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE pg-coordinator-1-drtsv 1/1 Running 0 20m 10.131.0.37 worker05 pg-coordinator-1-wk4wx 0/1 Running 0 11m 10.128.0.76 worker01 pg-coordinator-1-87ddq 0/1 Running 0 1m 10.129.2.45 worker04 pg-shard-00-1-qqwtr 1/1 Running 0 20m 10.129.0.68 worker02 pg-shard-01-1-ztdn4 0/1 Running 0 11m 10.128.0.77 worker01 • Create Service and DeploymentConfig from the Template. infra01 worker02 master01 worker01 worker04 worker03 worker06 worker05 NFS server OpenShift Cluster (8 virtual machines) pg-coordinator pg-shard legends pg-shard-01 pg-shard-00 OpenShift Master New container
90.
90Copyright©2017 NTT corp.
All Rights Reserved. 4.Conclusion l PostgreSQL on Kubernetes is easy to run PostgreSQL even if Multi-Master PostgreSQL Cluster. • including HA, monitoring etc. • easy to scale out. l Custom Resource Definitions is useful for sharing the cluster configurations. but you have to consider ... Ø Persistent storage. Ø In case of server/container failure. Ø Kubernetes cluster management. Please try PostgreSQL on kubernetes! You too! And share the knowledge. Future Plan We try to use Red Hat Ceph Storage as a distributed block storage instead of NFS. This can make the storage layer scalable! And evaluate the Performance.
91.
91Copyright©2017 NTT corp.
All Rights Reserved. 2 examples of Multi-Master PostgreSQL Cluster Appendix • For High availability • For Performance BDR (Bi-Directional Replication) Postgres-XC way Logical Replication Coordinators DataNodes
92.
92Copyright©2017 NTT corp.
All Rights Reserved. OpenShift provides Web Console. Appendix
Download now