Developer and Fusion Middleware 1 | Mark Nelson | Continuous Integration for ...
Database & Technology 1 _ Martin Power _ Delivering Oracles hight availability database services in minutes.pdf
1. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Delivering
Oracle’s
High
Availability
Database
Services
in
minutes,
a
case
study
on
OEM
11G
• Single
Server
imaging
• Gold
Image
development
and
deployments
• Data
Guard
services
with
Observers
• Automated
Patch
Management
MarCn
Power,
Oracle
ACE
www.logicaltech.com.au
1300
858
117
2. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
• Large
Public
Sector
Organiza=on
• Employs
60,000
plus
staff
• Tradi=onally
used
Sun
E
series
servers
with
Solaris
• Goal
–
To
move
to
a
horizontally
scalable
High
Availability
database
environment
• Database
Systems
are
required
to
move
to
99.999%
availability
across
24
x
7
x
365
operaCng
window
with
2
scheduled
outage
per
year,
not
exceeding
4
hours
in
total
www.logicaltech.com.au
1300
858
117
3. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
What
was
used
to
achieve
these
requirements?
• Restricted
list
of
SOE
Hardware,
Intel
x86
–
64
bit
servers.
• OEL
5.5
with
hardening
• 11.2.0.2
Grid
Infrastructure
• 11.2.0.2
RAC
Database
• OEM
11g
Grid
Control
with
the
Provisioning
Pack
• Bare
Metal
and
Single
Server
Image
provisioning
• Gold
Image
Deployments
• OEM
Patch
Plans
and
Patch
Management
• Data
Guard
Automated
Failover
and
Fail
back
www.logicaltech.com.au
1300
858
117
4. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Bare
Metal
and
Single
Server
Imaging
Steps
1. Build
a
single
server
with
OS,
Hardening
and
RPMs
and
Kernel
Configura=on
2. Build
a
Bare
Metal
/
Single
Server
image
in
OEM.
3. Iden=fy
the
servers
to
Deploy
to,
PXE
boot
them
from
the
network
to
install
the
Bare
Metal
Image
4. Deploy
the
Single
server
image
to
the
servers
www.logicaltech.com.au
1300
858
117
5. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Bare
Metal
and
Single
Server
Imaging
!
www.logicaltech.com.au
1300
858
117
6. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
In
OEM,
Select,
1. Deployments
2. Images
3. Create
Image
!
www.logicaltech.com.au
1300
858
117
7. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
Select
1. OVS
Default
Image
Enter
details;
Eg.
Name:
OEL5_5_x86_64_v1
Descrip=on:
Prototype
Linux
Image
Hardened.
Product
name/Patch
number:
OEL
5.5
Product
version:
5.5
Vendor:
Oracle
CorporaCon
www.logicaltech.com.au
1300
858
117
8. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
Select
the
Target
Host
–
e.g
grid11g-‐n1.localdomain
!
!
www.logicaltech.com.au
1300
858
117
9. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
Configure
the
remaining
Image
op=ons
RPM
Repository:
select
local-‐repo
Root
Password/Confirm
Root
Password
Select
=mezone:
Australia/Brisbane
Deselect
Use
UTC
Clock
AddiConally
you
will
set
• Username
• Group
• Password
• And
enable
SUDO
Privileges
!
www.logicaltech.com.au
1300
858
117
10. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
On
the
Customize
screen,
accept
the
defaults
and
click
Next
!
www.logicaltech.com.au
1300
858
117
11. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
www.logicaltech.com.au
1300
858
117
Finally
OEM
submits
a
job
to
clone
Linux
OS
DONE!
!
www.logicaltech.com.au
1300
858
117
12. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
Deployment
1. PXE
boot
the
new
server(s)
2. Deploy
the
Bare
Metal
Image
3. Deploy
the
Single
Server
Image
www.logicaltech.com.au
1300
858
117
13. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
PXE
Boot
the
servers
Update
the
/etc/
dhcp.conf
file
on
the
PXE
boot
server
with
the
server
MAC
address
host
llama0
{
hardware
ethernet
04:4B:80:80:80:03;
fixed-‐address
192.168.0.254;
op=on
host-‐name
"llama0";
! filename
"pxelinux.
0";
}
www.logicaltech.com.au
1300
858
117
14. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
A
Single
Server
Image
contains
an
operaCng
system
based
on
the
Default
Image,
together
with
tailored
disk
layout,
profile
and
network
detail
components,
Set
Host
Preferred
CredenCals
!
!
www.logicaltech.com.au
1300
858
117
15. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
In
OEM
Grid
Control,
click
through
to
the
Deployments
>
Provisioning
>
Assignments
Create
Image
Assignment
!
Single
Server
Image
!
!
www.logicaltech.com.au
1300
858
117
16. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
1.
Select
buion
beside
the
New
Network
2.
Choose
Network
screen,
select
the
required
Network
Component
!
!
3.
Edit
Advanced
ProperCes
www.logicaltech.com.au
1300
858
117
17. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
On
the
Edit
Advanced
ProperCes
screen,
enter
the
required
custom
details:
Agent
Base
Installa=on
Directory:
/opt/oracle
Oracle
Central
Inventory:
/opt/oracle/oraInventory
Enable
SELinux:
Permissive
Kernel
Parameter
Sekngs:
change
the
following
line
from:
net.ipv4.conf.all.log_marCans
=
1
…to:
net.ipv4.conf.all.log_marCans
=
0
(This
is
a
bug
that
will
leads
to
broadcasCng
of
errors
by
the
network
cards)
Post
Sec=on
Scripts:
add
the
following
commands:
mkdir
/nfs
mount
lgest:/orastage
/nfs
-‐o
nolock
cat
/nfs/scripts/post-‐install.sh
>
/etc/init.d/S90post_install
cp
/nfs/scripts/conf-‐files.tar
/opt/oracle/
chmod
777
/etc/init.d/S90post_install
ln
-‐s
/etc/init.d/S90post_install
/etc/rc.d/rc3.d/
umount
/nfs
!
www.logicaltech.com.au
1300
858
117
18. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
Click
the
Add
buion
1300
858
117
select
the
required
Hostname
!
!
www.logicaltech.com.au
1300
858
117
19. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
ConfirmaCon
screen
is
shown,
indicaCng
the
image
deployment
is
ready
!
Now
we
return
to
the
server
and
reboot
to
pick
up
the
Single
Server
Image.
www.logicaltech.com.au
1300
858
117
20. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
Select
OEM
Grid
Control
under
Deployments
>
Provisioning
>
Assignments;
click
the
Provision
!
On
the
Provision
Assignments
page:
Set
the
Schedule
Type:
One
Time
(Immediately)
Enter
(and
confirm)
the
Agent
RegistraCon
Password
Select
the
(previously)
Assigned
Image
www.logicaltech.com.au
1300
858
117
21. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
On
the
Assignment
Status
page,
click
the
Schedule’s
Job
ID
!
www.logicaltech.com.au
1300
858
117
22. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
Press
the
refresh
bugon
to
watch
the
Single
Server
Image
provisioning
process
execute.
!
Successful
comple=on
!
!
www.logicaltech.com.au
1300
858
117
23. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
Issues
1300
858
117
Symptoms:
NB:
The
following
issue
must
be
handled
immediately
once
the
Provisioning
process
has
been
started
and
must
be
completed
before
the
server
reboots.
The
single-‐server
image
tar
and
directories
created
by
the
assignment
staging
process
are
located
on
OMS
in
the
/oracle/stage/Images
directory;
and
comprise
an
oms.proper=es
file,
together
with
a
tar
file
and
corresponding
sub-‐directories
that
match
the
provisioned
single-‐server
images.
These
single-‐server
image
tar/directories
are
named
according
to
the
NIC
MAC
address
associated
with
the
assignment.
An
example
is:
oms.proper=es
File
:
9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308.tar
File
9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308
Directory
9DB50A0232DA68E3E04400144FD42308
Directory
:
An
Oracle
Service
Request
(SR)
has
been
raised
to
resolve
an
issue
with
the
above
oms.properCes
file,
which
is
created
with
missing
entries.
www.logicaltech.com.au
1300
858
117
24. Expert@Oracle Bare
Metal
and
Single
Server
Imaging
Deployment
www.logicaltech.com.au
1300
858
117
Cause:
Fault/bug
in
the
provisioning
process.
Solu=on:
The
oms.proper=es
file
has
to
be
manually
updated
with
the
missing
details;
namely
the
OMS
Host
and
Port
details.
As
an
example,
the
original
file
contains:
s_encrSecurePwd="05a61eae5282b1219ee52898b7bd189cd077b40996815fde61"
…and
should
be
updated
(as
the
root
user)
to:
s_encrSecurePwd="05a61eae5282b1219ee52898b7bd189cd077b40996815fde61"
s_OMSHost="qhsz007p.LT-‐test.logicaltech.com.au"
s_OMSPort="1159”
Once
updated,
the
oms.proper=es
file
needs
to
be
copied
into
the
single-‐server
image’s
subdirectory;
for
example:
9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308
…overwri=ng
the
default
one
created
therein
by
the
assignment
process.
ised
to
resolve
an
issue
with
the
above
oms.properCes
file,
which
is
created
with
missing
entries.
www.logicaltech.com.au
1300
858
117
25. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Gold
Image
Deployment
Sequencing
1. Install
11gR2
Grid
Infrastructure
and
Database
as
a
Single
Node
Cluster.
2. Create
a
Component
in
OEM
(Image
of
the
server)
3. IdenCfy
the
servers
to
Deploy
to.
4. Deploy
the
Gold
Image
to
the
servers
www.logicaltech.com.au
1300
858
117
26. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Create
the
Gold
Image
!
www.logicaltech.com.au
1300
858
117
27. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Create
the
Gold
Image
IdenCfy
the
• Host
• Grid
Home
• Grid
User
&
Group
• Etc.
!
www.logicaltech.com.au
1300
858
117
28. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Create
the
Gold
Image
Job
is
Created
and
run
!
Confirms
compleCon
!
www.logicaltech.com.au
1300
858
117
29. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Deploying
the
Gold
Image
!
www.logicaltech.com.au
1300
858
117
30. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Deploying
the
Gold
Image
Select
Image
and
schedule
deployment
!
www.logicaltech.com.au
1300
858
117
31. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Deploying
the
Gold
Image
Set
the
• Cluster
Name,
• Hosts
• Scan
DNS
Name
• Scan
Port
• Network
Bonding
Details
!
www.logicaltech.com.au
1300
858
117
32. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Deploying
the
Gold
Image
Set
the
• Discovery
Path
• ASM
Disk
Details
• OCR
Disk
Group
!
www.logicaltech.com.au
1300
858
117
33. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Deploying
the
Gold
Image
A
job
is
scheduled
• View
the
job
process
!
www.logicaltech.com.au
1300
858
117
34. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Comparison
of
2
build
methods
• Build
a
3
Node
Linux
Cluster,
2
to
3
days
• Deploy
3
Node
Linux
Cluster
with
Single
server
Images,
6
–
10
minutes
• Build
a
3
Node
Grid
Infrastructure
&
Database
Cluster,
1
day
• Deploy
a
3
Node
RAC
Cluster,
10
minutes
All
nodes
are
idenCcal….
Use
DBCA
to
create
RAC
databases
www.logicaltech.com.au
1300
858
117
35. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Data
Guard
Deployment
with
Observer
www.logicaltech.com.au
1300
858
117
36. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Data
Guard
Deployment
with
Observer
From
OEM
select,
• RAC
Database
• Select
Availability
tab
• Select
the
Add
Standby
Database
OEM
Provisioning
runs
through
all
the
step
just
like
the
Gold
Image
!
Deployment
!
!
In
5
minute
you
will
have
a
single
node
standby
database
www.logicaltech.com.au
1300
858
117
37. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Data
Guard
Deployment
with
Observer
Convert
to
Cluster
Database:
Cluster
CredenCals
page,
enter/select:
In
the
Cluster
CredenCals
sec=on:
Oracle
Home:
for
example
/u02/app/
oracle/product/11.2.0.2/db_1
Username
(of
the
o/s
Oracle
so@ware
owner):
oracle
Password:
(as
required)
In
the
ASM
CredenCals
sec=on:
!
Username
(of
the
SYSDBA
user
running
the
ASM
instance):
sys
Password:
(as
required)
Connect
As:
SYSDBA
Tick
the
Save
as
Preferred
CredenCals
checkbox
! www.logicaltech.com.au
1300
858
117
38. Expert@Oracle Data
Guard
Deployment
with
Observer
www.logicaltech.com.au
1300
858
117
Issues
dgmgrl
sys/[password]
show
database
verbose
tadbsb;
Database
-‐
tadbsb
Role:
PHYSICAL
STANDBY
Intended
State:
APPLY-‐ON
Transport
Lag:
0
seconds
Apply
Lag:
0
seconds
Real
Time
Query:
OFF
Instance(s):
tadbsb_1
(apply
instance)
tadbsb_2
Proper=es:
DGConnectIdenCfier
=
'(DESCRIPTION
=
(LOAD_BALANCE
=
ON)(ADDRESS
=
(PROTOCOL
=
TCP)(HOST
=
qhrac2-‐002t-‐v)(PORT
=
1521))(CONNECT_DATA
=
(SERVICE_NAME
=
tadbsb)))'
ObserverConnectIden=fier
=
''
LogXptMode
=
'ASYNC'
Host
address
is
wrong
DelayMins
=
'0'
It
should
be
the
SCAN
address
www.logicaltech.com.au
1300
858
117
39. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Patching
(Out
of
Place
Patching)
Clone
the
Oracle
Home
tar
-‐cf
-‐
*
-‐-‐exclude=*.aud
|
(cd
../db_2;tar
-‐xvf
-‐
)
Update
the
OraInventory
./runInstaller
-‐silent
-‐clone
ORACLE_HOME="/u02/app/oracle/product/11.2.0.2/db_2"
ORACLE_HOME_NAME="OraDb11g_home2"
ORACLE_BASE="/u02/app/oracle"
Amend
the
database
home
details
held
in
cluster
services
srvctl
stop
database
-‐d
tadbnew
srvctl
modify
database
-‐d
tadbnew
-‐o
/u02/app/oracle/product/11.2.0.2/db_2
srvctl
start
database
-‐d
tadbnew
www.logicaltech.com.au
1300
858
117
40. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Patching
(Out
of
Place
Patching)
Create
Patch
Plans
Patch
Cache
Maximum
Size
(MB):
700
Use
Dump
Directory
Radio
check
buion
set
to
Yes
/var/tmp/
www.logicaltech.com.au
1300
858
117
41. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Patching
(Out
of
Place
Patching)
Awer
selec=ng
the
patches
press
the
upload
buion
The
patches
are
then
staged
and
a
success
message
returned
Now
click
the
apply
to
schedule
the
job
that
runs
the
patching
process
www.logicaltech.com.au
1300
858
117
42. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Patching
(Out
of
Place
Patching)
www.logicaltech.com.au
1300
858
117
43. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Schedule
the
applicaCon
of
the
patch
set.
www.logicaltech.com.au
1300
858
117
44. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Schedule
the
applicaCon
of
the
patch
set.
• Set
the
details
for
the
patch
type
• Select
the
patch
sets
www.logicaltech.com.au
1300
858
117
45. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Schedule
the
applicaCon
of
the
patch
set.
• IdenCfy
the
databases
to
patch
• Watch
the
job
execute
www.logicaltech.com.au
1300
858
117
46. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
Patching
• Based
on
Agreed
Patch
Plan
Standards
• Rolling
Hot
Patch
ApplicaCon
• 3
Data
Centers
www.logicaltech.com.au
1300
858
117
47. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Case
Study
1
Patching
Sequence
Normal
Opera=on
Primary
Database
Standby
Database
Patch
the
databases
cold
4
2
Switch
to
DR
site
Create
a
third
at
the
primary
1
data
center
(Alternate
Servers)
3
Primary
Database
Standby
synchronizes
Standby
Database
www.logicaltech.com.au
1300
858
117
48. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Switch
back
to
original
Primary
5
8
Patch
the
databases
cold
Create
a
third
Standby
Primary
Database
6
at
the
primary
7
data
center
(Alternate
Servers)
9
Standby
synchronizes
Drop
the
temporary
Standby
database
Standby
Database
Normal
Opera=on
10
Resynchronize
the
standby
Primary
Database
Standby
Database
11
Drop
the
temporary
Standby
database
www.logicaltech.com.au
1300
858
117
49. Expert@Oracle High Availability in Minutes
www.logicaltech.com.au
1300
858
117
Summary
Case
Study
Benefits
• Build
3
Linux
Cluster
Nodes,
2
to
3
days
($2500
-‐
$4000)
• Deploy
3
Linux
Nodes
with
Single
server
Images,
6
–
10
minutes
($25.00)
• Build
a
3
Grid
Infrastructure
&
Database
Cluster,
1
day
($1400)
• Deploy
a
3
Node
RAC
Cluster,
10
minutes
($25.00)
• The
hardware
and
licensing
costs
delivered
a
30%
to
50%
saving
on
tradiConal
costs
with
3
5
x
9’s
of
availability
www.logicaltech.com.au
1300
858
117
50. Expert@Oracle
www.logicaltech.com.au
1300
858
Business
Solu=ons
Guaranteed
Service
Availability
Fixed
Monthly
Cost
Scales
with
you
business 117
www.logicaltech.com.au
1300
858
117