At my current company, we embed PostgreSQL based technologies in various applications shipped as shrink-wrapped software. In this session we talk about the experience of embedding PostgreSQL where it is not directly exposed to end-user and the issues encountered on how they were resolved. We will talk about business reasons,technical architecture of deployments, upgrades, security processes on how to work with embedded PostgreSQL databases.

1. My
experience
with
embedding
PostgreSQL
Postgres
Conf
EU,
Dublin,
Ireland
October
2013
Jignesh
Shah,
Sr
Manager,
VMware
© 2013 VMware, Inc. All rights reserved

2. About
me
• Sr
Manager,
VMware
Data
– Leads
Postgres
team
at
VMware
– Blog
at
hKp://jkshah.blogspot.com
• Previously
Staff
Engineer,
Sun
Microsystems
– Team
Member
of
the
first
published
SpecJAppServer
2004
benchmark
with
PostgreSQL
– Performance
of
PostgreSQL
on
Solaris/Sun
systems
• Working
with
PostgreSQL
circa
2005
© 2013 VMware, Inc. All rights reserved

3. Agenda
• Business
reasons
for
embedding
database
• Technical
architecture
journey
© 2013 VMware, Inc. All rights reserved

4. The
IniXal
Problem
• >30
#
products
and
product
components
• >20
requires
DB
• Customers
have
to
get
their
own
DB
license
• Increases
the
Total
Cost
of
Deployment
– As
database
load
increases,
CPU
cores
required
increases,
so
does
the
total
license
cost
of
DB
© 2013 VMware, Inc. All rights reserved

5. SQL
Server
Express
• Currently
shipped
for
some
Windows
based
products
– However
limits
not
enough
– Max
10GB
database
usage
• Standard
and
Enterprise
EdiXons
get
expensive
preKy
quickly
• Not
Available
on
Linux
© 2013 VMware, Inc. All rights reserved

6. Proposal:
PostgreSQL?
• PostgreSQL
is
free
and
open
sourced
• Long
history
of
stability
• Friendly
BSD
License
• Plus
I
am
at
a
Postgres
Conference
J
© 2013 VMware, Inc. All rights reserved

7. IniXal
External
Feedback
• Actually
got
pushback
from
customers,
partners
– PostgreSQL
not
part
of
their
database
standards
– No
in-­‐house
PostgreSQL
DBA
skills
– Lack
of
integrated
tools
,
monitor,
ease-­‐of-­‐use
– Large
enterprises
have
to
convince
their
managed
hosXng
partners
to
support
the
database
also
• AKendees
here
are
going
to
be
educated
about
it
– but
we
are
worried
about
the
rest
of
the
world
© 2013 VMware, Inc. All rights reserved

8. IniXal
Internal
Feedback
• DB
Support
Matrix
complexity
increases
– Every
new
DB
support
increases
N
plaborm
test
– Large
scale
tests
are
serialized
due
to
resources
© 2013 VMware, Inc. All rights reserved

9. Plan:
Embedded
Default
DB
• Embed
PostgreSQL
as
part
of
the
product
• Make
it
the
default
choice
opXon
for
most
deployment
• Do
not
expose
database
directly
to
endusers
• ApplicaXon
will
manage
the
DB
• Support
it
for
producXon
deployment
• Customer
can
sXll
select
to
use
an
external
database
of
their
choice
or
save
DB
License
cost
with
the
embedded
one
© 2013 VMware, Inc. All rights reserved

10. Ground
Work:
MigraXon
• Products
used
various
databases
– SQL
Server
– Oracle
– MySQL
• Few
had
about
1500
stored
procedures
• Some
used
exoXc
features
• For
example
MySQL
used
InMemory
Engine,
InnodB
engine,
MyISAM
engine
in
the
same
workload
© 2013 VMware, Inc. All rights reserved

11. MigraXon
• Leveraged
contractors
to
migrate
stored
procedures
• Performance
bake-­‐off
tests
to
convince
product
teams
• Integrated
into
the
build
infrastructure
to
ease
pulling
components
© 2013 VMware, Inc. All rights reserved

12. Example:
vCenter
Server
Appliance
© 2013 VMware, Inc. All rights reserved

13. Plan:
Backup
&
Recovery
• DB
not
the
only
thing
to
be
backed
up
• ConfiguraXon
files
• ApplicaXon
Status
• All
data
to
be
backed
up
went
into
a
virtual
disk
backed
up
with
vADP
• pg_dump
/
pg_restore
used
in
few
cases
© 2013 VMware, Inc. All rights reserved

14. IniXal
Embedded
DB
Deployments
• Each
Product
team
is
different
–
has
different
levels
of
DB
Development
skills
• Each
architect
comes
up
with
their
own
design
– Postgres
Service
designed
(Start/Stop)
– own
DB
Layout
design
– own
DB
Security
design
– Some
Tuned
their
DB
– Many
did
not
© 2013 VMware, Inc. All rights reserved

15. Early
Results
• YES
!
Customer
started
seeing
benefit
• However
– Experience
of
PostgreSQL
different
on
each
product
– Random
security
holes
idenXfied
due
to
lack
of
knowledge
of
PostgreSQL
nuances
– Performance
configuraXon
was
an
issue
– Upgrade
to
new
versions
of
PostgreSQL
was
an
akerthought
(What
no
in-­‐place
upgrade?)
– Legal
work
required
for
OSS
increased
– Resulted
in
PostgreSQL
version
sXckiness
© 2013 VMware, Inc. All rights reserved

16. Some
encounters
• Portable
binaries
–
Shrink
wrapped
sokware
(not
compiled
on
same
systems)
– OSS
Libraries
used
for
compiling
PostgreSQL
have
to
be
shipped
– Manage
security
patches
for
OSS
Libraries
• Need
Standard
drivers
like
ODBC,
JDBC
© 2013 VMware, Inc. All rights reserved

17. MulX-­‐Plaborm
Requirement
• As
company
grew,
acquisiXons
grew
• Some
products
only
worked
on
Windows
• Few
used
exclusively
.NET
framework
© 2013 VMware, Inc. All rights reserved

18. SoluXon:
Embedded
DBaaS
• Provide
End
to
End
Experience
with
PostgreSQL
© 2013 VMware, Inc. All rights reserved

19. InstallaXon
• Extend
installaXon
to
setup
a
service
• Configurable
to
set
up
as
internal
service
or
external
service
(mostly
embedded)
• Security
handled
as
part
of
service
setup
to
follow
best
pracXcesfor
embedded
• Security
based
on
access
and
minimum
roles
defined
© 2013 VMware, Inc. All rights reserved

20. InstallaXon
• RPM
installaXon
– Dependent
OSS
Libraries
to
make
RPM
portable
– Server
and
client
RPMs
– Service
Setup
configurator
to
setup
a
default
service
•
•
•
•
User
setup
initdb
setup
pg_hba
setup
Password
handling
– Provide
tools
to
quickly
setup
DB
Owner,
DB
for
various
users
of
the
service
© 2013 VMware, Inc. All rights reserved

21. Installer
on
Windows
•
•
•
•
PostgreSQL
Installer
on
Windows
Service
setup
on
Windows
ODBC
Support
Npgsql
Support
for
.NET
Plaborms
© 2013 VMware, Inc. All rights reserved

22. Backup
&
Recovery
• While
other
backups
strategy
are
available
– Nothing
beats
standard
database
backups
– Design
cron
jobs
for
criXcal
data
– Design
Archive
On
Demand
for
semng
up
replicaXon
quickly
– Most
customer
do
not
want
to
hack
with
backup/
restore
scripts.
– Should
work
out
of
the
box
© 2013 VMware, Inc. All rights reserved

23. Upgrades
• Seamless
upgrade
to
new
version
– RPM
Upgrades
needs
to
be
transparent
for
minor
versions
– RPM
upgrades
for
major
version
needs
to
migrate
data
from
the
older
version
in
the
default
service
© 2013 VMware, Inc. All rights reserved

24. Upgrades
• Problems
sXll
not
completely
resolved
– Reduce
downXme
for
major
release
upgrades
– Generally
migraXon
since
in-­‐place
upgrade
does
not
provide
ways
to
roll
back
the
upgrade
© 2013 VMware, Inc. All rights reserved

25. Performance
• Default
PostgreSQL
semng
are
conservaXve
• Most
deployments
use
default
values
• Started
doing
auto-­‐tuning
– Modified
PostgreSQL
core
engine
with
intelligence
– For
checkpoint
segments,
shared_memory
• Added
tool
to
auto
tune
postgresql.conf
– Based
on
memory
share
for
DB
Service
– Product
workload
dependent
© 2013 VMware, Inc. All rights reserved

26. Performance
related
tools
• Provided
tools
to
understand
what’s
happening
with
Postgres
– pg_top
with
changes
• Changes
already
contributed
back
to
community
© 2013 VMware, Inc. All rights reserved

27. Example:
pg_top
© 2013 VMware, Inc. All rights reserved

28. Performance
related
future
work
• ExisXng
tools
sXll
meant
for
expert
users
• Lot
of
demand
for
simple
performance
reports
• Looking
into
pg_statspack
© 2013 VMware, Inc. All rights reserved

29. Availability
• Created
a
watchdog
for
PostgreSQL
to
restart
PostgreSQL
if
database
cannot
be
accessed
for
long
• If
restart
fails
then
it
can
throw
alerts
• Also
created
plugins
for
monitoring
the
service
© 2013 VMware, Inc. All rights reserved

30. Related
Problems:
Crashes
• PostgreSQL
started
throwing
errors
with
“Out
of
Memory”
errors
in
stress
tests
• MulXple
applicaXons
can
fight
for
memory
• Made
Performance
semngs
less
aggressively
© 2013 VMware, Inc. All rights reserved

31. Supportability
• When
customers
hit
a
problem
– Hard
to
get
access
to
the
system
• Created
support
package
tool
to
collect
relevant
informaXon
– Collect
config,
status
in
a
tar
file
for
analysis
© 2013 VMware, Inc. All rights reserved

32. Usability
• Experts
love
command
line
uXliXes
• Novices
love
GUI,
WebUI
• It
needs
to
match
the
other
GUIs
VMware
ship
© 2013 VMware, Inc. All rights reserved

33. Pluggable
UI
• Matching
VMware’s
GUI
Plaborm
© 2013 VMware, Inc. All rights reserved

34. Embedded
DBaaS
Architecture
VMware
Product
Service
Configurator
WebUI
Pg_top
JDBC
PostgreSQL
ODBC
Linux
Based
VM
Watchdog
Perf-­‐Tuner
Monitoring
Support
package
tool
vSphere
Plaborm
© 2013 VMware, Inc. All rights reserved

35. Virtual
Appliances
© 2013 VMware, Inc. All rights reserved

36. Virtual
Appliances
© 2013 VMware, Inc. All rights reserved

37. ReplicaXon
between
vApps
• Provide
simple
uXliXes
to
quickly
– archive_command
– create_replicaXon_user
– run_as_replica
– show_replica_status
– promote_replica_to_primary
© 2013 VMware, Inc. All rights reserved

38. In
Closing
• Free
Sokware
is
nice
• Experience
–
What
business
users
care
– User
Experience
– DBA
Experience
– Support
Experience
– etc
• Improve
the
Experience
with
the
technology
© 2013 VMware, Inc. All rights reserved

39. Thanks.
QuesXons?
Download
VMware
vFabric
Postgres
hKp://vmware.com/go/download-­‐vfabric-­‐postgres
Follow us on twitter: @vPostgres
Data Platform Blog: http://blogs.vmware.com/dataplatforms
© 2013 VMware, Inc. All rights reserved