Deep review of LMS process

©OraInternals Riyaj Shamsudeen
RAC Hack:
Deep review of LMS/LGWR process
By
Riyaj Shamsudeen

LMS Processing (over simplified)
Rx Msg
CR / CUR
block build
Msg to LGWR
(if needed)
Wakeup
Log buffer
processing
Log file write
Signal
LMS
Wake up
Send Block
OS,Network
stack
©OraInternals Riyaj Shamsudeen 2
Send GC
Message
OS,Network
stack
User session
processing
Copy to SGA
OS,Network
stack
User LMSx LGWR
Node 1 Node 2

Processing in the
remote nodes
GC CR latency
 GC CR latency ~=
Time spent in sending message to LMS +
LMS processing (building blocks etc) +
LGWR latency ( if any) +
LMS send time +
Wire latency
Averages can be misleading. Always review both total time
and average to understand the issue.

LMS process – A deep dive
 LMS process uses pollsys system call and listens for incoming
packets, with a 10ms timeout.
 Sockets are file descriptors in UNIX.
truss -d -E -v all -p 1485 |more
1.8531 0.0000 pollsys(0xFFFFFD7FFFDFBA70, 7, 0xFFFFFD7FFFDFBA20, 0x00000000) = 0
fd=36 ev=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND rev=0
timeout: 0.010000000 sec
Timeout 10ms
1.8635 0.0000 pollsys(0xFFFFFD7FFFDFBA70, 7, 0xFFFFFD7FFFDFBA20, 0x00000000) = 0

LMS sockets
 Pfiles shows that these file descriptors are sockets, essentially,
LMS process is sending and receiving messages in the ports.
Pfiles 1845
36: S_IFSOCK mode:0666 dev:298,0 ino:10517 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
SOCK_DGRAM
SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.224.0.0)
sockname: AF_INET 127.0.0.1 port: 33320
SOCK_DGRAM
SOCK_DGRAM
Demo: demo_lms_truss.ksh demo_lms_pfiles.ksh

Pollsys call voluntarily releases CPU until a new packet is
arrived to a port or a timeout.
Uses very little of CPU if there is no work to be done.
Without any work, just 92 Micro seconds of CPU used in
a 10,242 micro seconds window.
LMS CPU usage
 Just because LMS process runs in RT mode, does not mean that
LMS process is consuming all that CPU.
#./trace_syscall_preempt_size.sh 1485
0 => pollsys timestamp : 45075622139230
0 | swtch:pswitch oracle sysinfo: timestamp : 45075622155047
0 | swtch:pswitch Vol context switch : 45075622155697 pswitch genunix`cv_timedwait_sig_hires+0x2ab
0 | resume:off-cpu On cpu 0 for: 92460
0 | resume:on-cpu Off cpu for: 10242512
0 <= pollsys timestamp : 45075632406018 elapsed : 10266788
Demo: as root trace_syscall_preempt_size.sh

LMS process was woken up in 29 Micro seconds.
LMS – early wakeup
 Kernel will schedule LMS process if there is a network packet
arriving to that port.
#./trace_syscall_preempt_size.sh 1485
0 => pollsys timestamp : 45075592390763
0 | swtch:pswitch oracle sysinfo: timestamp : 45075592402281
0 | swtch:pswitch Vol context switch : 45075592402933 pswitch genunix`cv_timedwait_sig_hires+0x2ab
0 | resume:off-cpu On cpu 0 for: 55099
0 | resume:on-cpu Off cpu for: 29660449
0 <= pollsys timestamp : 45075622072662 elapsed : 29681899
Demo: as root trace_syscall_preempt_size.sh

LMS count
 Even in busy environments, I have seen LMS to be busy only
50% of the time.
 To schedule a process, CPU scheduler loads the CPU registers,
instruction pipeline etc, a costly process.
 If you have many LMS processes, then the workload will be
distributed among them. Due to RT priority, they will be moving
in and out of CPU.
 In a multi-processor environment, this becomes more
complicated.
 Version 11.2 uses much more meaningful values for LMS count.

 It breaks down the process in to micro accounting percentages.
In this case,
Breakdown of LMS CPU usage is:
LMS – prstat
 In Solaris, another way to check the efficiency of LMS process is
through prstat micro accounting.
# prstat -mL -p 18243
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
18243 prod 6.4 5.9 0.0 0.0 0.0 0.0 88 0.0 2K 0 30K 187 oracle/1
6.4% USR mode
5.9% SYS mode
0% CPU latency
88% Sleep
Demo: prstat command
2K voluntary context switches
0 involuntary context swiches

LMS process in Node 4 is busy serving CR blocks. Why?
LMS – session
 LMS session level statistics can be used to measure the workload
distribution. Of course, this is from the start of instances.
@lms_workload_perc
INST_ID PGM NAME VAL PROC_TO_INST PROC_TO_TOT INST_TO_TOT
---------- ------- ------------------------------ ---------- ------------ ----------- -----------
1 (LMS0) gc cr blocks served 62960382 15 3 25
...
...
...
...
Demo: lms_workload_distr.sql – to measure workload from the instance start

LMS processes in Node 4 is busy here.
LMS - workload
 In few cases, it is prudent to measure the current rate, instead of
relying upon prior rate.
@gc_lms_workload_distr_diff.sql
Enter value for search_string: gc cr blocks served
Enter value for sleep: 60
---------|--------------|----------------|----------|---------------|---------------|-------------|
Inst | Pgm | value | totvalue | instvalue | proc2inst |inst2total |
---------|--------------|----------------|----------|---------------|---------------|-------------|
1 | (LMS0)| 348| 16931| 2993| 11| 2|
1 | (LMS1)| 335| 16931| 2993| 11| 1|
...
2 | (LMS0)| 359| 16931| 2660| 13| 2|
2 | (LMS1)| 375| 16931| 2660| 14| 2|
...
3 | (LMS0)| 132| 16931| 1231| 10| 0|
3 | (LMS1)| 194| 16931| 1231| 15| 1|
...
4 | (LMS0)| 1164| 16931| 10047| 11| 6|
4 | (LMS1)| 1784| 16931| 10047| 17| 10|
---------|--------------|----------------|----------|---------------|---------------|-------------|
Demo: gc_lms_workload_distr_diff.sql

 Following session level stats can be reviewed to determine the
performance counter for undo blocks applied.
@get_sesstat_sid
Enter the wildcard character (Null=All):undo
Enter value threshold :1
Enter sid :11000
NAME VALUE
---------------------------------------------------------------- ----------
transaction tables consistent reads - undo records applied 13180
data blocks consistent reads - undo records applied 474036
474K undo records were applied to create CR blocks.
LMS – applying undo
 LMS process applies undo blocks to construct the CR buffer to
send.
Demo: get_sesstat_sid.sql

-- Session Snapper v2.01 by Tanel Poder ( http://www.tanelpoder.com )
----------------------------------------------------------------------------------------------------------------------
SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, %TIME, GRAPH
----------------------------------------------------------------------------------------------------------------------
11000, (LMS0) , STAT, cleanouts and rollbacks - consistent rea, 633, 42.2,
11000, (LMS0) , STAT, immediate (CR) block cleanout applicatio, 689, 45.93,
11000, (LMS0) , STAT, commit txn count during cleanout , 56, 3.73,
11000, (LMS0) , STAT, active txn count during cleanout , 633, 42.2,
11000, (LMS0) , STAT, cleanout - number of ktugct calls , 690, 46,
11000, (LMS0) , TIME, background cpu time , 696907, 46.46ms, 4.6%, |@ |
11000, (LMS0) , TIME, background elapsed time , 696907, 46.46ms, 4.6%, |@ |
11000, (LMS0) , WAIT, gcs remote message , 13987778, 932.52ms, 93.3%, |@@@@@@@@@@|
11000, (LMS0) , WAIT, events in waitclass Other , 150638, 10.04ms, 1.0%, |@ |
-- End of snap 2, end=2011-06-28 22:28:01, seconds=15
LMS – snapper.sql
 Tanel’s ultra-cool snapper is useful to find the rate of few
statistics on LMS session.
@session_snapper out,gather=stw 15 4 11000
Demo: snapper.sql

@gc_traffic_print.sql
---------|--------------|---------|----------------|---------|---------------|---------|-------------|---------|
Inst | CR blocks Rx | CR Rx% | CUR blocks Rx | CUR RX %| CR blocks Tx | CR TX % | CUR blks TX | CUR TX% |
---------|--------------|---------|----------------|---------|---------------|---------|-------------|---------|
1 | 283| 3.6| 950| 27.12| 214| 3.33| 665| 16.8|
2 | 7185| 91.47| 1327| 37.89| 256| 3.98| 1117| 28.22|
3 | 119| 1.51| 886| 25.29| 5798| 90.22| 1617| 40.86|
4 | 268| 3.41| 339| 9.68| 158| 2.45| 558| 14.1|
In that sampling interval, node 3 was transmitting 90% of the CR blocks and
node 2 was receiving those blocks. This insight is useful to measure the
workload distribution, with a larger sampling interval.
GC TX/RX %
 You can find the percentage for TX and RX packets using the
gc_trafic_print.sql script too. These percentages are at instance
level.
Demo: gc_traffic_print.sql

gcs log flush sync
 Before sending a reconstructed CR block or CUR block, LMS
will verify that corresponding redo vectors are flushed to disk.
 If the redo vector are not flushed, LMS need to wait for ‘gcs log
flush sync’ event after requesting LGWR for a log flush,
analogous to ‘log file sync’ event.
 This is not an idle event, even though some old documentation
suggest that.

Gcs log flush sync - ASH
 ASH shows that LMS waits for ‘gcs log flush sync’ event.
select event, count(*) from v$active_session_history
where session_id in (select sid from v$session where program like '%LMS0%')
and sample_time > sysdate -(4/24)
group by event
order by 2 desc;
EVENT COUNT(*)
---------------------------------------- ----------
767
gcs log flush sync 265
latch: KCL gc element parent latch 4
 In this database, there is no issue and so, waits for ‘gcs log flush
sync’ is not high.

Gcs log flush sync – v$session_event
 But, v$session_event for the LMS process shows that there are
no waits for gcs log flush sync!
select event, trunc(time_waited_micro/1000) wait_milli, total_waits
from v$session_event where sid in (select sid from v$session where program like '%LMS0%')
order by 2 desc;
EVENT WAIT_MILLI TOTAL_WAITS
---------------------------------------- ---------- -----------
gcs remote message 218407919 83934373
events in waitclass Other 3970180 5237156
buffer busy waits 356 2897
latch: cache buffers chains 316 5197
latch: row cache objects 0 2
latch: shared pool 0 3
Event ‘gcs log flush sync’ is combined with other events in the wait_class
“other”.

Gcs log file sync - histogram
 To review the impact of ‘gcs log file sync’ waits, you should
review v$event_histogram.
 73% of the waits complete under 1ms. This is probably not an
issue.
@event_histogram.sql
Enter value for event_name: gcs log flush sync
INST_ID EVENT WAIT_TIME_MILLI WAIT_COUNT PER
---------- ---------------------------------------------------------------- --------------- ---------- ----------
1 gcs log flush sync 1 24490064 73.42
1 gcs log flush sync 16 142603 .42
1 gcs log flush sync 64 66 0

Gcs log file sync – Not so good
 Following histogram shows an example when there is a
performance issue with LFS.
 If you have LFS waits and GC waits, then you should consider
tuning log file sync before tuning GC events.
@event_histogram.sql
Enter value for event_name: gcs log flush sync
INST_ID EVENT WAIT_TIME_MILLI WAIT_COUNT PER
---------- ---------------------------------------------------------------- --------------- ---------- ----------
1 gcs log flush sync 128 2 0

Gcs log file sync – LGWR interaction
 If LGWR is suffering from performance issues, then LMS
process can be seen waiting on ‘gcs log flush’ wait event in a tight
10ms loop.
LMS trace file:
...
WAIT #0: nam='gcs log flush sync' ela= 10281 waittime=3 poll=0 event=136 obj#=-1 tim=1381909996
...
 If you have LFS waits and GC waits, then you should consider
tuning log file sync before tuning GC events.

GCS log flush sync - Example
Excessive waits for log file sync for the
foreground processes.
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
log file sync 2,054 23,720 11548 45.8 Commit
gc buffer busy acquire 19,505 10,382 532 20.0 Cluster
gc cr block busy 5,407 4,655 861 9.0 Cluster
enq: SQ - contention 140 3,432 24514 6.6 Configurat
db file sequential read 38,062 1,305 34 2.5 User I/O
Host CPU (CPUs: 24 Cores: 24 Sockets: 24)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
--------- --------- --------- --------- --------- ---------
1.18 1.16 2.7 2.6 0.0 94.7

GCS log flush sync – GC waits
Average waits for CR RX was at 222ms
Global Cache and Enqueue Services - Workload Characteristics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg global enqueue get time (ms): 7.4
Avg global cache cr block receive time (ms): 222.0
Avg global cache current block receive time (ms): 27.5
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.1
Global cache log flushes for cr blocks served %: 2.7
Avg global cache cr block flush time (ms): 15879.9
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.1
Global cache log flushes for current blocks served %: 0.3
Avg global cache current block flush time (ms): 1701.3
High flush time indicating
waits for LGWR process

GCS log flush sync – GC waits
Background processes waiting for excessive gcs log flush
sync events.
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
-------------------------- ------------ ----- ---------- ------- -------- ------
gcs log flush sync 80,695 51 1,862 23 34.7 32.9
log file parallel write 44,129 0 880 20 19.0 15.6
Log archive I/O 1,607 0 876 545 0.7 15.5
gc cr block busy 729 71 752 1031 0.3 13.3
db file parallel write 25,752 0 434 17 11.1 7.7
enq: CF - contention 166 64 307 1850 0.1 5.4
High waits for log file parallel writes

LGWR is important
 So, if you think, LGWR performance is important in single
instance, then it is ultra-important in RAC.
 If you have LGWR related performance issues, you can almost
discard other waits as symptoms.
 It’s a pity that LGWR does not run in RT mode ( or even FX
class).
 LMS processes runs in elevated priority, but LGWR does not run
in elevated priority, classic priority-inversion!

Thank you for attending!
Contact info:
Email: rshamsud@gmail.com
Blog : orainternals.wordpress.com
URL : www.orainternals.com

Deep review of LMS process

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Deep review of LMS process

Similar a Deep review of LMS process (20)

Más de Riyaj Shamsudeen

Más de Riyaj Shamsudeen (11)

Último

Último (20)

Deep review of LMS process