SlideShare una empresa de Scribd logo
1 de 232
1 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Download this slide
http://ouo.io/MsNQB
SPARC M6/M5-32 Server –
Technical Overview
Insert Presenter’s Name Here
Insert Presenter’s Title Here
3 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
The following is intended to outline our general product direction. It is
intended for information purposes only,
and may not be incorporated into any contract. It is
not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality
described for Oracle’s products remains at the sole discretion of Oracle.
4 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC M6/M5-32 Server
5 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
New l SPARC M6 Servers
32-Socket High-End Datacenter Server
 Compute
– Up to 32x SPARC M6 12 core 3.6GHz CPUs
– Allows mixing with SPARC M5 6 core processors
– Up to 1024x DDR3 DIMMs for max memory of up to 32TB
 I/O and storage
– 32x 2.5” SAS-2 internal drives
– 64x PCIe Gen3 low profile internal slots
 Scalability and investment protection
– Upgradable with M6 processor
 Availability and management
– Advanced RAS with redundant and hot swap of key components
– Extensive virtualization with Oracle VM Server for SPARC + hard partitions
– Integrated Oracle ILOM system management
6 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
High-end Product Data Sheet
7 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32
Processor M6 3.6 GHz, 48 MB L3$
Max 32 (384 cores)
Memory Max 32TB
(32GB x 1024 DIMMs)
Internal Disks Max 32 (2.5” HDD or SSD)
RemovableMedia rKVMS
CMU Max 16
IOU Max 4
PCIe Slots Max 64 PCIe
Domains Max 4 physical
Max 512 logical
Power Options 3-phase/Dual Grid
Power Consumption 25 KW
System Management ILOM (main service processor)
High-end Product Comparison
8 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
2 TB
DDR3 DIMMs)
CIe Gen 3
Dual Grid
Max= 25KW
vice processor)
.1”Dx78.7”H
(1677 kg)
M9000-32 M5-32/M6-32
Processor SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88GHz, 6 MB L2$
Max 64 (256 cores)
SPARC64 VII+ M5 3.6 GHz, 48 MB L3$
3.0 GHz,12 MB L2$ Max 32 (192 cores)
Max 64 (256 cores) M6 3.6 GHz, 48 MB L3$
Max 32 (384 cores)
Memory Max 4 TB Max
3
(8GB x 512 DDR2 DIMMs) (32GB x 1024
Internal Disks Max 64 (2.5” SAS) Max 32 (2.5” HDD or SSD)
RemovableMedia DVD, DAT rKVMS
CMU and IOU Max 16 each Max 16 CMU and 4 IOU
PCI Slots Max 64 PCIe Gen 1 Max 64 P
Domains Max 24 physical Max 4 physical, Max 512 logical
Power Options 1-phase/3-phase/DualGrid 3-phase/
Power Consumption Min= 12KW, Max= 14.6KW Min=23KW,
System Management XSCF (service processor) ILOM (main ser
MemoryBandwidth 737 GB/s 1,980 GB/s
I/O Bandwidth 244 GB/s 1,024 GB/s
Dimensions 33.5”Wx49.6”Dx70.9”H 35.6”Wx57
Weight (Max) Base cab: 2068 lbs.(940kg),Power cab: 770 lbs.(350 kg) 3697 lbs.
Key Differences from Earlier M-series
9 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M8000/M9000 M6-32
SP XSCF, running XCP SP, running ILOM
Each DCU has it’s own SP Proxy (SPP)
Domains Hard Domains: up to 24 Hard Domains: up to 4, Logical Domains: up to 512
Memory Clock DDR2 at 480MHz DDR3 at 1066MHz
Lowest Latency 387ns 158 ns (Bounded Domain), 160 ns (Regular Domain)
IO PCIe Gen 1 PCIe Gen 3
Smallest Hard Domain 1 CPU + 8 DIMMs (Quad-XSB) 4 CPUs + 128 DIMMs (2 CMU boards)
Internal network PPP 10/100 Ethernet
Clock and Interconnect Cold service for failed clock board and crossbar board Hot-plug replacement of failed clock and SBB boards
Power Feed Single phase, or 3-phase. DPF optional
Customer provides own cable for 3-phase
3-phase, DPF standard
Cable is orderable
Airflow Bottom to top Front to back
Base IO Card Must use existing PCIe slot Dedicated slots just for Base IO card
DVD Built-in. Can be assigned to any Base IO card N/A. Must use remote media of rKVMS.
Cable Routing Bottom only Top or bottom
Term Chart
10 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Term Definition
CPU memoryboard Unit (CMU) Board equipped with two M6 CPU modules and up to 64 DIMMs of memory
Chip Multi-Processor(CMP) Another term for a multi-core processor
Buffer on Board (BoB) An ASIC that interfaces between the DIMMs and the MCU on the M6
Memory Controller Unit (MCU) Each M6 has two MCUs in order to communicatewith each BoB.
IO board unit (IOU) Unit equipped with 16 PCIe Gen 3, 8 disk drives (HDD or SSD), and 8 10GbE ports
DomainConfigurationUnit (DCU) A grouping of up to 4 CMUs, 1 IOU controlled by its own SP Proxy
Physical Domain (PDom) An electrically fault-isolatedhard domain comprised of one or more DCUs
Hypervisor The SPARC hypervisor is a small firmware layer that provides a stable virtualized machine
architectureto which an operating system can be written.
Front Cabinet Components
1. System status indicators, key switch, and ESD
grounding connector
2. 12 Power Supplies (PS)
3. 18 Fan Modules (FM)
4. Leveling Feet
5. 2 Power Supply Distribution Boards (PSDB)
6. 2 Service Processors (SP)
7. 2 Clock Boards
8. 12 Scalability Switch Boards (SSB)
9. Mounting Brackets
11 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Rear Cabinet Components
1. Left cable management bracket
2. 4 IO Units (IOU)
3. 4 Service Processor Proxies (SPP)
4. 16 CPU Memory Units (CMU)
5. Leveling feet
6. 6 AC input filters (power cord connectors)(3+3)
7. Right cable management bracket
8. 16 PCIe hot-plug carrier for low-profile PCIe cards
9. 4 Express Module SAS for disks (EMS)
10. 32 Hard disk drives
11. System status indicators and ESD grounding connector
12. Mounting brackets
12 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 Front and Rear Isometric Views
13 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
CPU Memory Unit (CMU)
 Two M6 CPU sockets with
12 cores each
 Each CPU runs at 3.6GHz
 Memory is type DDR3L RDIMM
at 1066MHz
 32 DIMM slots per M6 socket
 16 and 32 GB DIMMs supported
 Min of 8 DIMMs per CPU
M6 #1MEM #1
MEM #0 M6 #0
14 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 also supports CMUs with M5 processors !
CMU Numbering CMU0
15 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
CMU15
As seen from rear of chassis
MEM1
MEM0
CMP1
CMP0
Zakim (ZK) – Systems Memory Interface
16 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Also referred to as a BoB (Buffer-on-Board)
 Features and Technology:
– 4 ZKs per M5/M6 memory controller
– 2 ZKs per T5 memory controller
– Supports DDR3 DIMMs
– Memory Link to DDR Interface
– 2 Memory Link Ports, 2 DDR ports
– 8 Write FIFOs
– Pass through commands to DIMMs
– Core frequency: 1066 MHz
Elements of a CMU
 The BoB is really the Zakim ASIC
 4 DDR3 DIMMs per BoB
 4 BoBs per M6 memory controller
 32 DIMMs per M6 processor
 The DIMMs for each processor are
mounted on a memory module
 2 memory modules per CMU
 Each M6 has 7 ports to allow direct
communication with other M6s in the
domain configuration unit (DCU)
CMU
Mem Mod 0
BoB BoB BoB BoB
BoB BoB BoB BoB
M6-0
Mem Ctlr Mem Ctlr
Mem Mod 0
BoB BoB BoB BoB
BoB BoB BoB BoB
Mem Ctlr Mem Ctlr
17 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-1
Internode Coherency Overview
 Glueless 1-hop scaling to eight sockets
– Glueless means no hub required to connect all
8 CPUs
 Aprecise directory tracks all L3s in the
system
– striped across all processors
– stored in on-chip SRAMs
– flexible for different socket counts
 Higher BW efficiency than snoop-based
protocols enables better scaling
– 50% more effective bandwidth than
comparable snoopy implementation
M5/
M6
M5/
M6
18 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M5/
M6
M5/
M6
M5/
M6
M5/
M6
M5/
M6
M5/
M6
8 Socket Local Coherency & Data Interconnect
DIMMS
M5/M6/T5
M5/M6/T5
M5/M6/T5
M5/M6/T5
M5/M6/T5
M5/M6/T5
M5/M6/T5 M5/M6/T5 DIMMSDIMMS
DIMMS
POINT-TO-POINT
LOCAL
INTERCONNECT
DIMMS
PCIe Gen3 Bandwidth
8 diff pairs per ports
At 8Gb/sec ~8GB/sec/direction
19 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
DIMMS
DIMMS
DIMMS
All-to-All Interconnect
Where the Node-to-Node Fabric is 12 diff
pairs per link in each direction.
T5 interconnect bandwidth= 157.5 GB/sec
T5-8 interconnect bandwidth= 1260 GB/sec
DDR3-1066
MemoryBandwidth
T5 is 133 GB/sec
T5-8 is 1064 GB/s
~8Gb/sec/lane X 8 lanes = 64 Gb/s
X 2 directions
X 2 Ports/chip
= 256Gb/s/chip
= 32 GB/s per chip
M5/M6 Global Coherency & Data Interconnect
DIMM
32
M5/M6
M5/M6
M5/M6
M5/M6
M5/M6
M5/M6
M5/M6 M5/M6
DIMM
32
DIMM
32
DIMM
32
DIMM
32
DIMM
32
DIMM
32
DIMM
32
POINT-TO-POINT
LOCAL
INTERCONNECT
Memory Bandwidth
Per M5/M6 = 68GB/sec
Or 544GB/sec for 8 M5/M6
All-to-All Local Interconnect
Where the Node-to-Node
Fabric is 7x12 per link in
each direction
At 16Gb/sec/direction
Bisection Bandwidth 16
Links x 24 GB/sec/Link
X 2 directions = 768GB/sec
Each CPU connects to 6 BXs
Wherethe M5/M6-to-BX
Fabric is 6 x 4
Links in each direction
At 16Gb/sec
~8GB/sec/direction/link
Global interconnect
To and from the BXs
For 8 CPUs is 48 x 4 Links
~768GB/sec/DCU
32 DIMMs per M5/M6
128 DIMMs / DCU
~4TB with 32GB
DIMMs
20 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
CMU: Root Complexes
21 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Each /SYS/CMU has 2 CMPs (CPUs), and each
/SYS/CMU/CMP has 2 IOS (2 root complexes). NAC
Format for root complex: /SYS/CMU[0-15]/CMP[0-
1]/IOS[0-1]
 PCIe DevPath for a root complex:
dev:////pci@<x>/pci@1, where <x> is the Base
address shown in this table
CMP0
IOS0
CMP0
IOS1
CMP1
IOS0
CMP1
IOS1
CMU0 300 340 380 3c0
CMU1 500 540 580 5c0
CMU2 400 440 480 4c0
CMU3 600 640 680 6c0
CMU4 700 740 780 7c0
CMU5 900 940 980 9c0
CMU6 800 840 880 8c0
CMU7 a00 a40 a80 ac0
CMU8 b00 b40 b80 bc0
CMU9 d00 d40 d80 dc0
CMU10 c00 c40 c80 cc0
CMU11 e00 e40 e80 ec0
CMU12 f00 f40 f80 fc0
CMU13 1100 1140 1180 11c0
CMU14 1000 1040 1080 10c0
CMU15 1200 1240 1280 12c0
Memory Population Rules
22 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 There are two channels from each BoB
 Each channel talks with 1 or 2 DIMMs
 CMUs are either ¼ or ½ populated with DIMMs
 Populate each CMU with the same size/rank DIMMs
 It is allowable to mix different DIMM sizes on different CMU boards
 DIMMs in the same position (0,1,2, or 3) within a MCU, must be homogeneous
– must be of the same type (x4 or x8)
– same dram device capacity (2Gb or 4Gb)
– same number of ranks (SR, DR, or QR)
 DIMMs on the same channel must have the same rank capacity
M5/M6 Memory Configuration ¼ Populated
Best performance for
eight DIMMs
0
1
2
3
0
1
2
3
0
1
2
3
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
0
1
2
3
BoB0BoB1 BoB2BoB3
BoB5 BoB4BoB6BoB7
M6
fsr2
fsr0fsr1
fsr3
23 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M5/M6 Memory Configuration ½ Populated
Best performance for
16 DIMMs
0
1
2
3
0
1
2
3
0
1
2
3
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
0
1
2
3
BoB0BoB1 BoB2BoB3
BoB5 BoB4BoB6BoB7
M6
fsr2
fsr0fsr1
fsr3
24 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
0
1
2
3
3
2
1
0
3
2
1
0
0
1
2
3
M5/M6 Memory Configuration Fully Populated
Best performance for
32 DIMMs
0
1
2
3
3
2
1
0
3
2
1
0
0
1
2
3
BoB0BoB1 BoB2BoB3
BoB5 BoB4BoB6BoB7
M6
fsr2
fsr0fsr1
fsr3
25 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Memory Module
26 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
27 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
IO Switch Board
PCIe
Switch
28 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
I/O Unit (IOU)
29 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 1x IOU per physical domain
 16 PCIe Gen3 Low Profile slots
 8 2.5” HDD or SSD disk drives
 Four Base I/O Cards (used to access the disk drives)
 8 10GbE per IOU
 PCIe carrier used to allow hot-plug of PCIe cards
M5/M6 IOU LP PCIe Carriers (8)
(Hot Plug)
Slots 9-16
Base IO Cards (2)
EMS3, EMS4
Base IO Cards (2)
EMS1, EMS2
LP PCIe Carriers (8)
(Hot Plug)
Slots 1-8
Two independent IO bays.
30 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Switch Board 1
(Hot Plug)
10 GbE (8)
Hot Plug HDD Drives (4)
0-3
Hot Plug HDD Drives (4)
4-7
Switch Board 0
(Hot Plug)
Elements of a IOU
 2 I/O bays per IOU (0 & 1)
 8 PCIe Gen 3 slots per bay
 8 SAS 2 disk drives
 Disk drives are in 2 groups
– Group 0: Disks 0-3
– Group 1: Disks 4-7
 I/O Bay 0 is controlled by I/O Switch Board 0
– Disk group 0 is controlled via HD Backplane 0
 I/O Bay 1 is controlled by I/O Switch Board 1
– Disk group 1 is controlled via HD Backplane 1
IOU
Bay
0
Bay
1
Disks
0-3
Disks
4-7
9
10
11
12
Base 3
13
14
15
16
Base 4
Base 1
1
2
3
4
31 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Base 2
5
6
7
8
Base IO Card
32 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 This board is required in order to access the internal disk drives
 Each board provides 2 10GbE interfaces
M6-32 2 Socket DCU PCIe Block Diagram (Bounded PDom)
IOB1
CMU2 CMU3
CMP0 CMP1 CMP0 CMP1
pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15
33 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
EMS3
EMS4
9 10 11 12 13 14 15 16
M6-32 4 Socket DCU PCIe Block Diagram
34 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 4 Socket DCU PCIe Block Diagram (Bounded PDom)
IOB1
CMU2 CMU3
CMP0 CMP1 CMP0 CMP1
pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15
35 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
EMS3
EMS4
9 10 11 12 13 14 15 16
M6-32 6 Socket DCU PCIe Block Diagram (Bounded PDom)
36 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 8 Socket DCU PCIe Block Diagram
37 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 Base IO
Block Diagram
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T 24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
DiscDrive3
DiscDrive2
DiscDrive1
DiscDriveo
DiscDrive7
DiscDrive6
DiscDrive5
DiscDrive4
SAS-BP
A B A B A B A B
SAS-BP
A B A B A B A B
To Switch
Board 1
To Switch
Board 1To Switch Board 0
To Switch Board 0
Base IO
Board 1
Base IO
Board 2
Base IO
Board 3
Base IO
Board 4
x8 x8 x8 x8
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
RJ RJ
45 45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
Disk Drives Are Dual Ported!
38 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 Base IO
Block Diagram
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T 24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
DiscDrive3
DiscDrive2
DiscDrive1
DiscDriveo
DiscDrive7
DiscDrive6
DiscDrive5
DiscDrive4
SAS-BP
A B A B A B A B
SAS-BP
A B A B A B A B
To Switch
Board 1
To Switch
Board 1To Switch Board 0
To Switch Board 0
Base IO
Board 1
Base IO
Board 2
Base IO
Board 3
Base IO
Board 4
x8 x8 x8 x8
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
RJ RJ
45 45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
Disk Drives Are Dual Ported!
39 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 Base IO
Block Diagram
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T 24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
DiscDrive3
DiscDrive2
DiscDrive1
DiscDriveo
DiscDrive7
DiscDrive6
DiscDrive5
DiscDrive4
SAS-BP
A B A B A B A B
SAS-BP
A B A B A B A B
To Switch
Board 1
To Switch
Board 1To Switch Board 0
To Switch Board 0
Base IO
Board 1
Base IO
Board 2
Base IO
Board 3
Base IO
Board 4
x8 x8 x8 x8
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
RJ RJ
45 45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
Disk Drives Are Dual Ported!
40 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T 24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
DiscDrive3
DiscDrive2
DiscDrive1
DiscDriveo
DiscDrive7
DiscDrive6
DiscDrive5
DiscDrive4
SAS-BP
A B A B A B A B
SAS-BP
A B A B A B A B
To Switch
Board 1
To Switch
Board 1To Switch Board 0
To Switch Board 0
Base IO
Board 1
Base IO
Board 2
Base IO
Board 3
Base IO
Board 4
x8 x8 x8 x8
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
RJ RJ
45 45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
M6-32 Base IO
Block Diagram
Disk Drives Are Dual Ported!
41 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T 24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
24p Gen2
Switch
SAS2
x8
LSI
2008
Intel
X540
Dual
10GBA
SE-T
DiscDrive3
DiscDrive2
DiscDrive1
DiscDriveo
SAS-BP
A B A B A B A B
DiscDrive7
DiscDrive6
DiscDrive5
DiscDrive4
SAS-BP
A B A B A B A B
To Switch
Board 1
To Switch
Board 1To Switch Board 0
To Switch Board 0
Base IO
Board 1
Base IO
Board 2
Base IO
Board 3
Base IO
Board 4
x8 x8 x8 x8
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
X4
SAS
X8
PCIe
RJ RJ
45 45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
RJ
45
M6-32 Base IO
Block Diagram
Disk Drives Are Dual Ported!
42 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
EMS to Disk Mapping
43 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Redundant Disk Paths In a Domain
 EMS1 and EMS3 provide
SAS paths to the four
drives
 Access all of the drives if
one EMS fails
 Configure them for
redundancy using the
Oracle Solaris I/O
multipathing feature
44 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
PCIe Links
45 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 An IOU contains:
 2 IOBs (I/O Switch Boards)
 16 slots for LP-carriers (low-profile PCIe card
carriers for user-selected add-in I/O)
 4 slots for Base IO cards (on-board fixed I/O in
a removable form-factor)
 The LP-carrier slots are labeled PCIE1-PCIE16,
and the Base IO slots are labeled EMS1-EMS4.
 The PCIe data path originates from a root complex
(in a CMU), connects to a PCIE_SWITCH (in an
IOB) which fans out the signal to PCIe or EMS
slots (in the IOU) or to KVMS targets on an SPP
(Service Processor Proxy)
CMUs IOB Slots
CMU[0-1] IOU0/IOB0 IOU0/EMS[1-2], IOU0/PCIE[1-8], SPP0
CMU[2-3] IOU0/IOB1 IOU0/EMS[3-4], IOU0/PCIE[9-16]
CMU[4-5] IOU1/IOB0 IOU1/EMS[1-2], IOU1/PCIE[1-8], SPP1
CMU[6-7] IOU1/IOB1 IOU1/EMS[3-4], IOU1/PCIE[9-16]
CMU[8-9] IOU2/IOB0 IOU2/EMS[1-2], IOU2/PCIE[1-8], SPP2
CMU[10-11] IOU2/IOB1 IOU2/EMS[3-4], IOU2/PCIE[9-16]
CMU[12-13] IOU3/IOB0 IOU3/EMS[1-2], IOU3/PCIE[1-8], SPP3
CMU[14-15] IOU3/IOB1 IOU3/EMS[3-4], IOU3/PCIE[9-16]
Root Complex
4 CMUs in DCU
46 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Root Complex
2 CMUs in DCU
47 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Disk Device Paths
4 CMUs in DCU
48 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Disk Device Paths
2 CMUs in DCU
49 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
T5 and M5/M6 PCIe Carrier Card
 Supports standard low-profile PCIe cards
Air Flow
50 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
PCIe Retimer
x16 Connector
(x8 electrical)
About the F40 Flash Card (Aura2)
51 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Supported on T5-2, T5-4, T5-8, and M5-32/M6-32
 Hot-plug supported
 Can not be placed in Slot 8 of IOU on M5-32/M6-32 due to
thermal issues.
Base IO Card
SAS
Controller
PCIe
Switch
Intel
10GbE
RJ45
Ports
52 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Bixby (BX)– M5/M6 Scalability ASIC
53 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Features and Technology:
– 16-port Coherency switch router for M5/M6 CPUs > 8 Sockets
– 8Address Serialization Pipelines
– Supports 4 directory lookups per cycle
– Each SSB contains one BX
 12 switches in a M6-32
– For unbounded physical domains, each BX has a portion of the L3$
system directory:
 6 BXs hold 50% of the system directory for a M6-32
– For bounded physical domains, the BX is not used
Scalability Switch Board (SSB)
54 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
BX
Coherency Switch Connectivity
55 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 12 Switch cards (SSB), each with a BX ASIC
 M6-32
- Even numbered CMUs connect to odd numbered BXs
- Odd numbered CMUs connect to even numbered BXs
 Remember, each M6 only has 6 scalability links. All 6 go to either even or odd
SSBs.
 The need to separate BX boards in to even or odd, is due to the limited number
of pin outs on the BX chip
 Each BX uses a 26-bit tag + 7-bit ECC to track one L3$. To track the 32P
system, each BX has 8*32*12*424*2 tags (7.41MB)
Additional BX Details
56 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 The 12 SSBs are grouped into two sets of 6. One set for even numbered CMU
boards, and the other set for odd numbered CMU boards.
 The directory is oversized so that if 1 BX fails in a set, the system can be re-
configured to operate with any 5 of the 6 BX's in a set. This allows for
redundancy in the system directory.
 A DCU can only communicate data with other DCUs via the BX ASICs on the
SSB.
 Even CMUs can only communicate with even CMUs in other DCUs. Same for
odd.
 For an even CMU to communicate with an odd CMU in another DCU, it must
hop at the end.
Scalability Switch Assembly
SP0 Connector SP1 Connector
Clock0 Connector Clock1 Connector
SSB Connectors
57 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Central
Coherency
Directory
And Data
Switch
12x BX
ASICs
I/O
Connectivity
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
I/O
Connectivity
I/O
Connectivity
I/O
Connectivity
58 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6-32 Block Diagram
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
Central Coherency Directory
And Data Switch
IO Switch
Board 1
IO Switch
Board 0
IO Switch
Board 1
IO Switch
Board 1
IO Switch
Board 1
BX 0
BX 1
BX 2
BX 3
BX 4
BX 5
BX 6
BX 7
BX 8
BX 9
BX 10
BX 11
Even numbered CMU
boards connect to odd
numbered SSB boards.
IO Switch
Board 0
IO Switch
Board 0
IO Switch
59 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Board 0
M6-32 Block Diagram
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
Central Coherency Directory
And Data Switch
BX 0
BX 1
BX 2
BX 3
BX 4
BX 5
BX 6
BX 7
BX 8
BX 9
BX 10
BX 11
IO Switch
Board 1
IO Switch
Board 0
IO Switch
Board 1
IO Switch
Board 1
IO Switch
Board 1
IO Switch
Board 0
IO Switch
Board 0
IO Switch
60 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Board 0
Even numbered CMU
boards connect to odd
numbered SSB boards.
M6-32 Block Diagram
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
Central Coherency Directory
And Data Switch
BX 0
BX 1
BX 2
BX 3
BX 4
BX 5
BX 6
BX 7
BX 8
BX 9
BX 10
BX 11
IO Switch
Board 1
IO Switch
Board 0
IO Switch
Board 1
IO Switch
Board 1
IO Switch
Board 1
IO Switch
Board 0
IO Switch
Board 0
IO Switch
61 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Board 0
M6-32 Block Diagram
61 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Central Coherency Directory And Data Switches
12 13 14 15
3
IO Switch
Board 1
IO Switch
Board 0
0
1
0
1
0
1
0
1
DCU 4
Disks Disks
0-3 4-7
0 1 2 3
0
IO Switch
Board 1
IO Switch
Board 0
0
1
0
1
0
1
0
1
DCU 1
Disks Disks
0-3 4-7
4 5 6 7
1
IO Switch
Board 1
IO Switch
Board 0
0
1
0
1
0
1
0
1
DCU 2
Disks Disks
0-3 4-7
8 9 10 11
2
IO Switch
Board 1
IO Switch
Board 0
0
1
0
1
0
1
0
1
DCU 3
Disks Disks
0-3 4-7
M6-32 Block Diagram
62 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Central Coherency Directory And Data Switches
DCU 1 DCU 2 DCU 3 DCU 4
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch
Board 0 Board 1 Board 0 Board 1 Board 0 Board 1 Board 0 Board 1
Disks Disks Disks Disks Disks Disks Disks Disks
0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7
0 1 2 3
Data Path Example
CMU0-CPU0 to PCIe slot 61
Clock Board
64 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 2 clock boards, in a redundant configuration
 Each clock board has dual clock sources with a dynamic multiplexer on each pair.
 There are 3 clock types distributed through out the system. System core clock 133MHz,
Memory clock 133MHz and PCIe clock 100MHz.
 If one of the clock source fails (in a pair), the dynamic multiplexer automatically switches
to secondary synthesizer without bring the system down. The domain keeps running.
 If more than one clock source in a pair fails, the system will reboot and the alternate
clock board will become the active source.
 A failed clock board that has been marked inactive, it can be replaced while the system
is running as there are 2 clock boards, in a redundant configuration.
Service Processor (SP)
65 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Also known as the Integrated Lights Out Management, or ILOM
 Two redundant SP modules per M6-32
 Provides primary platform configuration and management
 Works with the SPP in each DCU to configure and monitor the
components in each DCU
SP Board
Serial
Network
Pilot3
Processor
66 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SSP Board
Pilot3 Module
sits here
67 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SP and SPP Connectivity
Corporate LAN
SP 0 SP 1
SPP 0 SPP 1 SPP 2 SPP 3
Ethernet
16 Port Switch
SP
Mezzanine
100 Base-T 100 Base-T
1 GbE1 GbE
100 Base-T
SerialSerial
Ethernet 16
Port Switch
SP
Mezzanine
68 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
rKVMS
69 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 rKVMS is remote keyboard/mouse/storage connectivity
 The domain creates a USB keyboard, mouse and storage, as well as a
graphics console window that is sent to the SPP
 The SP opens a port to initiate a graphical console interface so a user
can manage the domain through this remote console
 Only the Golden SPP will have the path to the rKVMS
Front/Rear Status Panels
70 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
No
.
LED Description
1 System Locator LED
(white)
The Locator LED can be turned on to identify the server. When on, it blinks rapidly. There are two methods for turning on the
LocatorLED:
• Issuing the Oracle ILOM commandset /SYS/LOCATE value=Fast_Blink
• Pressing the Locator button on the front of the server.
2 System Service Required
LED (amber)
Indicates that service is required.
• The Oracle ILOM show faulty commandprovides details about any faults that cause this indicator to light.
• Under some fault conditions, individual component fault LEDs light in addition to the Service Required LED.
3 System Power OK LED
(green)
Indicates the followingconditions:
• Off – System is not running in its normal state. System power might be off. The SPs might still be running.
• Steady on – System is powered on and is running in its normal operating state. No service actions are required.
• Fast blink – System is running in standby mode and can be quickly returned to full function.
• Slow blink – A normal but transitoryactivity is taking place. Slow blinking might indicate that system diagnostics are running or
that the system is booting.
4 Power On/Standby
button
The recessed Power On/Standby button toggles the system on or off.
• Press once to power the server on and return the server to its previously configured operating configuration.
• If the key switch is set to service mode, press once to shut the server down in a normal manner.
• If the key switch is set to service mode, press and hold for at least 4 seconds to perform an emergency shut down.
Caution - After using the Power On/Standby button, you must switch off the circuit breakers on your facilitypower grid in order to
power down the server completely. The server will remain on standby power until you switch the circuit breakers off.
Front/Rear Status Panels
71 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
No
.
LED Description
5 SP LED Indicates the following conditions:
• Off – Indicates the AC power mighthave been connected to the power supplies.
• Steady on, green – SPs are running in their normal operating state.
• Blink, green – SPs are initializingthe Oracle ILOM firmware.
• Steady on, amber – A SP error has occurred and service is required.
6 Front Component LEDs
(amber)
The front panel componentrequires service.
7 Rear Component LEDs
(amber)
The rear panel component requires service.
8 System key switch Using a supplied key, you can switch between normal and service modes.
Normal operation mode:
• You can power on the server by pressing the Power On/Standby button.
• You cannot shut the server down using the Power On/Standbybutton.
Service mode:
• Server is placed in service mode
• You can shut down the server using the Power On/Standby button.
• The key cannot be removed from the key switch while in the Service position.
9 Antistatic wrist strap
connector
The server has two 10-mm connectors where you can attach an antistaticwrist strap prior to installing or servicing the server.
M5 / M6 Differences,
Configuration rules and
PCIe Slot Guidelines
72 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M5 / M6 Configuration and Replacement Rules
73 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Mixing of M5 and M6 CMUs within a DCU is not allowed.
 All CMUs in a DCU must be either M5 or M6. However DCUs with
M5 CMUs can co-exist with M6 CMUs in a domain in different DCUs
 Mixed (M5 and M6) clock boards in M5 chassis are allowed
 Mixed (M5 and M6) clock boards in a M6 chassis will not inhibit boot
but will generate an alert to the user.
 Anytime the system detects mixed (M5 and M6) clock boards, and
both boards are usable, it will use the M6 clock board as the active
Note : The M6 clock board in mixed configurations being favored and detection of the mix in an M6
system as well as generating an alert, these changes will be post RR. FW will not stop the user
from mixing clock boards.
M5 / M6 Configuration Guidelines
74 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Fill Dark Green Slots First
Fill Light Green Slots Second
Fill Yellow Slots Last
Factory configuration guidelines
PCIe IO Slot Priority
HDD Slot HDD/SSD EMS Slot BASE IO CARD CMU/IOB IO Slot
HALFDCU
0 X 1 X CMU0 – IOB0 1* 2
1 X 3 4*
2 2 5 6*
3 7* 8
4 X 3 X CMU3 – IOB1 9 10*
5 X 11* 12
6 4 13* 14
7 15 16*
FULLDCU
0 X 1 X CMU0/1 – IOB0 1 2
1 X 3 4
2 X 2 X 5 6
3 X 7 8
4 X 3 X CMU2/3 – IOB1 9 10
5 X 11 12
6 X 4 X 13 14
7 X 15 16
 A PCIE card marked with an asterisk ( * ) is removed or added, the OBP
device path may change, dependent on the state of the ILOM property
/HOSTx/ioreconfigure and CMUs present.
 We recommend prioritizing the installation of PCIE cards ( grouped
functionally ) based on the color scheme in the table :
1)Infiniband and Storage PCIE cards starting with dark green first, followed by
light green slots without the asterisks.
2)10GigE Network PCIE cards on available dark and light green marked slots
with the asterisks.
3) 10/100/1GigE Network cards on the light green and yellow marked with the
asterisks.
 Due to thermal requirements F40/Aura2 cards must not use PCIe slot 8.
75 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M5 / M6 Configuration Guidelines
The SPARC M6 Processor
76 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
25+ Years of SPARC Processors
1987 1992 1996 2000 20051988 1995 2002
Sunrise:
1st SPARC
Processor
SUNRAY
UltraSPARC I
SuperSPARC I
UltraSPARC II
UltraSPARC III
UltraSPARC IIIi
UltraSPARC IV+
UltraSPARC T1
SPARC T4
SPARC M5
2007 2010
UltraSPARC T3
UltraSPARC T2
SUNRAY
2011 2013
SPARC T5
77 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
1987 1988 1992 1995 1996 2000 2002 2005 2007 2010 2011 2013
Sunrise:
1st SPARC
Processor UltraSPARC IV+
UltraSPARC II SPARC T5
SuperSPARC I
UltraSPARC IIIi
UltraSPARC T2 SPARC T4
SUNRAY
SUNRAY UltraSPARC I UltraSPARC III UltraSPARC T1 UltraSPARC T3 SPARC M5
SPARC S3 Core Processor Family
78 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6 Processor
 12 S3 cores @ 3.6Ghz
– Single or multi-threaded operation per core
– 6-48 total strands
 SPARC S3 Core
– 1-8 Strand Dynamically Threaded Pipeline
– Vertically Threaded 2-decode, Out-of-Order (OoO)
2-Issue
– 16KB 4-way L1 I$
– 16KB 4-way L1 D$
– 128KB 8-way L2$
– ISA-based Crypto-acceleration
 Shared 48MB L3$
 Memory Controllers
– 2x Dual Channel DDR3L MCU's
– Cascadable Buffer-on-Board (BoB)
 Integrated I/O
– 2 x8 Lane PCIe Gen3 @ 8GT/s
– 32GB/s IO Bandwidth per processor
 System Scalability
– Up to 8-socket glueless
– 32-socket Systems via scalability links
M6
79 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC M6: Processor Overview
 12 SPARC S3 cores, 96 threads
 48MB shared L3 cache
 4 DDR3 schedulers, maximum of
1TB of memory per socket
 2 PCIe 3.0 x8 lanes
 Up to 8 sockets glue-less scaling
 Up to 32 sockets glued scaling
 4.1 Tbps total link bandwidth
 4.27 billion transistors
Coherence&Scalability
MIO
Crossbar
L3 L3
L3 L3
SPARC SPARC L3
Core Core Ctl
SPARC SPARC L3
Core Core Ctl
L3 SPARC SPARC
Ctl Core Core
SerDes
SerDes
SerDes
SerDesMCU
SPARC
Core
SPARC
Core
L3 SPARC SPARC
Ctl Core Core
SerDes SerDesMCU
SPARC
Core
SPARC
Core
PCIe
PCIe
80 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC M6 CPU Block Diagram
2x8PCIe3.0@8GBps
16GBpseachdirection
Crypto
8 threads per
Core
IO
Subsystem
Memory
Control
Coherency2x4Switch
Memory
Control
Coherency Links
12.8 Gbps per lane
- 12 lanes per link
(1075 Gbps)
12 x 5 Crossbar (~620GBps bandwidth)
C0 C1
L3$ B0L3$ B0
1MB,16-way
128 KB L2$
SLC
16 KB L1I$
16 KB L1D$
FGU
SPARC
S3 Core
BoB BoB BoBBoB
Coherence Unit Coherence Unit
SLC
SLC
SLC
SLC
SLC
Link 0
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
C2 C3 C4 C5 C6 C7 C8
DDR3 – 1066 MHz DDR3 – 1066 MHz
BoB BoB BoBBoB
DDR3 – 1066 MHz DDR3 – 1066 MHz
Scalability Links
12 Gbps per lane
– 4 lanes per link
(144 Gbps)
C9
81 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
C10 C11
L3$ B0
12MB,16-way
L3$ B2
12MB,16-way
L3$ B1
12MB,16-way
L3$ B3
12MB,16-way
SPARC M6: Processor Key Features
82 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 SPARC M6 based on SPARC S3 core: Same core as used in T4, T5 and M5
– 12 S3 cores, dual PCI Express 3.0 root complexes
– Up to 32 M6 processors per system, 12 cores x 8 threads → 96 per M6, or up
to 3072 threads
– Clock frequency is 3.6 GHz
– Each SPARC S3 core contains:
 2 Integer pipelines
 1 FGU pipeline (consisting of 3 physical sub-pipelines):
– FPX pipeline
– FGX pipeline
– FPD pipeline
 1 Load-Store (Memory) pipeline
S3 Core Recap
83 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Out-of-order, dual-issue
 High frequency achieved with 3.6GHz
 16 stage integer pipeline
 Dynamically threaded, one to eight strands
 Accelerates 16 encryption algorithms and random number generation
SPARC S3 Core
 Dual-issue, out-of-order
 Integrated encryption
acceleration instructions
 Enhanced instruction set to
accelerate Oracle SW stack
 Dynamically threaded pipeline
– 1 to 8 threads
84 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC Core Roadmap
T3 Servers
S2 Core
M-Series
SPARC64 VII/VII+ Core
M5 Servers
T5 Servers
T4 Servers
S3 Core
M6 Servers
S4 Core
85 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
S3 Core Overview
86 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 8-way threaded, dual-issue, OoO execution, in order commit
 Dynamically threaded with hardware-optimized resource sharing
 Support for Critical Threads
 Deep pipeline for high frequency operation (3 GHz in 40 nm)
 Balanced single-thread and multi-thread performance
 Enhanced instruction set to accelerate Oracle SW stack
- PAUSE, fused compare-branch
 Integrated user-level cryptographic acceleration
- DES/3DES,AES, Kasumi, Camellia, MD5, SHA-1, SHA-224/256/384/512,
RSA, DSA, CRC32c
 Foundation core for future technology / product nodes
Oracle SPARC S3 Core
87 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Enables T4, T5, M5, M6
• Design Goals
- Develop a common replacement core for T-series processors
- Significantly improve the single strand performance of the T3 processor
- Significantly improve the throughput performance of the M3 processor
- Improve the RAS and power management capabilities
- Maintain backward ISA, Solaris and OVM for SPARC compatibility
Core S1: M2, M3 Core S2: T2, T2+, T3 Core S3: T4, T5, M5, M6
Frequency 2.4 – 3.0 GHz 1.4 – 1.65 GHz T4: 2.85 – 3.0 GHz
T5, M5, M6: 3.6 GHz
L1 InstructionCache 64KB 16KB 16KB
L1 Data Cache 64KB 8KB 16KB
L2 Cache - - 128KB
# of Pipelines 1 2 1
# of Threads per Pipeline 2 4 1-8 Dynamic
Instructionsper Thread 4 per cycle 1 per cycle 2 per cycle
Out of Order Issue Yes No Yes (36 instr window)
CryptographyAcceleration None SPU ISA Based
OVM for SPARC Compatible No Yes Yes
SPARC V9 ISA Compatible Yes Yes Yes
SPARC Security
Securing All the Stack Layers
 Secure database queries 43% faster; Low overhead for encryption
 WebLogic SSL & WS-Security 2x – 3x faster
 ZFS Filesystem Crypto 3x faster than x86
 Java crypto 2x – 3x faster than x86
 OpenSSL4.3x faster for single-thread security versus Power7
Results:
88 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC T4/T5/M5/M6 Leads in On-Chip Encryption Acceleration
89 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
On-Chip
Accelerators
Common Use
Examples
Mainly Used SPARC T4/T5/M5/M6 IBM Power7+
Only 3 cryptos for 8
cores
Intel
Westmere
/SandyBridge
Asymmetric /Public
Key Encryption
Web Browsers, phone
calls, VPN, Secure FTP
To protect data or
files in transit
RSA, DH, DSA, ECC RSA, ECC RSA, ECC
Symmetric Key /
Bulk Encryption
Databases, credit card
and social security
numbers, private info
To protect data or
files at rest or stored:
disks, backup tapes,
etc.
AES, DES, 3DES,
Camellia, Kasumi
AES (Modes: ECB,
CBC, CTR, CCM, CCA,
GCM, GCA, GMAC,
CM, F8, XBC-MAC-96;
Key lengths: 128b,192b,
256b) (Lacks AES-CFB
used by Oracle
database!!)
AES
Message Digest /
Hash Functions
Data lookup &
authentication, digital
signatures, message
authentication codes
(MAC)
To compress/create a
short summary from a
data chunk and not
expose it; To detect
duplicate or corrupt
data
CRC32c, MD5, SHA-1,
SHA-2 (SHA-224, SHA-
256, SHA-384, SHA-512)
MD5, SHA-1, Partial
SHA-2 (SHA-256, SHA-
512), HMAC
supported for SHA (?)
none
Random Number
Generation
Ubiquitous To generate keys Supported Supported none till Ivy
Bridge
Next Generation l SPARC Processors
90 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
T5 M5 M6
Processor 16 S3 cores, 128 threads @
3.6GHz
6 S3 cores, 48 threads @
3.6GHz
12 S3 cores, 96 threads @
3.6GHz
S3 Core
Features
1-8 Thread, Dynamically threaded pipeline
ISA-based Crypto-acceleration
Shared L3$ 8MB 48MB
Integrated I/O 2x8 Lane PCIe 3.0 @ 8GT/s
System
Scalability
7 Coherence Ports for scalability to 8S
Additional 6 Scalability Ports for scaling to 32S
Power
Management
Dynamic Voltage Frequency Scaling
Enterprise
Features
OVM Server for SPARC (LDoms), Solaris Zones
Scalability Port Redundancy, Dynamic Domains, Hot-Plug SP
S3 Core: Dynamic Threading
Page <#>91 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 While software can activate up to 8 strands on each core at a time, hardware
dynamically and seamlessly allocates core resources such as instruction, data,
and L2 caches and TLBs, and out-of-order execution resources such as the
128-entry re-order buffer in the core among the active strands.
 Software activates strands by sending an interrupt to a HALTed strand.
Software deactivates strands by executing a HALT instruction on each strand it
wants to deactivate. No strand has special hardware characteristics; all strands
have identical hardware capabilities.
S3 Core: Dynamic Threading
Page <#>92 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Since the core dynamically allocates resources among the active strands,
there is no explicit "single-thread mode" or "multi-thread mode" for software to
activate or deactivate.
 If software effectively halts all strands except one on a core via Critical Thread
Optimization, the core devotes all of its resources to the sole running strand.
Thus, that strand will run as quickly as possible
S3 Core: Critical Thread Optimization
93 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 The S3 core, starting with Oracle Solaris 10 8/11, can optimize performance by
assigning one software thread exclusive access to all of a core's hardware
resources. That software thread is considered to be a "critical thread."
 Solaris automatically detects opportunities to perform this assignment: when
one software thread has high CPU utilization and there are more cores than
runnable threads. We recommend that users allow Solaris to automatically
perform Critical Thread assignment.
 A privileged user can tell Solaris that a particular software thread should be a
critical thread, via the nice(1) command. Solaris will then assign that thread to
a core, even if there are more runnable threads than cores.
S3 Core: Critical Thread Optimization
94 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Solaris Critical Threads optimization for S3 core, tries to provide exclusive
access of certain hardware resources to certain application threads
 Since the core dynamically allocates resources among the active strands,
there is no explicit "single-thread mode" or "multi-thread mode" for software to
activate or deactivate
 There is no new API for declaring threads as ‘critical’; that would require
significant changes to source code
 Rather, to invoke Critical Thread Optimization, use the following CLI or system
calls to flag a thread as critical by raising its priority to 60:
– priocntl(1)
– priocntl(2)
– priocntlset(2)
S3 Core: Critical Thread Optimization
95 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Starting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised
to priority 60; the thread can be in any scheduling class
 In Oracle Solaris 11, to be considered critical by the scheduler, a thread must
be:
– in the FX (Fixed Priority) or RT (Real-Time) scheduling classes
– be raised to priority 60 by one of the previously mentioned mechanisms
 In either of the above instances, this one thread will run as quickly as possible
as it has exclusive access to all core resources
Critical Threads for key applications
96 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Applicability Opportunity Current Status
Database Logwriter, LMS Up to 30% improvement in
efficiency
LMS is already CT ready.
LGWR planned for 12c
JAVA (JVM) Compiler threads, GC
and priority mapping
support
Up to 2x improvement for app
startup, Smooth GC
Support for JVM and JAVA
apps to be CT aware is
integrated in JDK7U4
Coherence Packet writer, service
thread
Up to 20% improvement in
throughput
Integrated in Coherence
version 3.7.1 Patch 1
Solaris S11U1 / S10U11 Improve CT perf to be within
10% of best case (hand
optimized)
Optimizations for decayed
PG util and stealing being
integrated in S11U1
Reliability,
Availability,
Serviceability (RAS)
97 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Definition of Terms
98 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Hot-plug:
• refers to the fact that a component can be plugged and unplugged without powering
down the platform. It applies to both hot swap and hot service.
• Hot service:
• refers to the ability to perform hot-plug operations, with the additional necessity of
some operator actions (invocation of a CLI or actuating a hot service button on the
component to be removed).
• The system will notify the user when it is safe to remove the component.
• Typical examples would be PCIe Express modules.
• Hot swap:
• refers to an operation where a component is unplugged and plugged in with no
interaction with the ILOM or domain required.
• Typical examples here are a single RAID disk or a power supplies.
SPARC M5/M6/T5 System RAS Overview
99 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Designed to minimize part count and operating temperature to enhance
reliability
• End-to-end data protection detecting and correcting errors throughout server –
ECC everywhere
• Processor and Memory protection
• CPU core and thread off-lining
• Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failover
• Major components redundant & hot-pluggable
• Fan, Power Supply, and internal disks
• RAID capability for internal disks
• Fault Management Architecture (FMA) support on ILOM
End-to-End RAS
 Built RAS from the inside out
 Start with the processor, then
memory, system and IO,
virtualization layer, and the OS
 Add Oracle Solaris Cluster software
for additional service availability
 Fault Management Architecture
(FMA) binds all the layers together
100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
T5/M5/M6 Systems RAS
• Diagnosis engine on SP
• Auto reconfigure on failure
• Soft Error Rate Discrimination(SERD)
• Bad page retirement
• OS and SP watchdogs
• FMA Component hot-upgradeable
T5/M5 Processor
• L1$ Tag, Status $ Data
• Parity protection
• Retry on error
• L2$/L3$ Data
• SEC/DED protection
• Cache-line Sparing
• L2$/L3$ Tags
• SEC/DED protection
• Inline Correction
• Cache-line Sparing
• L2$/L3$ Status & Directory
• SEC/DED protection
• Inline Correction
• Architectural RegistersL2 Cache
• SEC/DED protection
• Precise Trap and
Hypervisor Correction and Retry
Power and Cooling
• Advanced Power Management
• Redundant hot-swap fans
• Redundant hot-swap AC/DC
• Dual grid power
System I/O
S11 FMA Hypervisor
System
• Redundant SPs with automatic failover
• Redundant clock boards
• Dual synthesizers per clock board
• Diagnosis to the FRU level on first fault
•PCI-Express end-to-end CRC
• PCI Express link retry
• Hot-plug low profile PCI Express cards
• Redundant, hot-plug boot disks
•Alternate connections between M5 and IO
controllers
Memory
• SDRAM Soft Errors
• ECC Protection and Correction
• Extended ECC Protection
• 4-bit Correction
• Pin Steering
• Channel Interconnect
• CRC protection/Message Retry
• Lane Sparing
•Enables software partitioning (LDoms)
virtualization and failure containment
•Processor support for error clearing, correction
and collection
Central Directory and Switch
•SEC/DED protection with in line correction
• Physical domain isolation
•CRC protected System Interconnect with message retry and
lane sparing
• Deconfigurable directory chips, no loss of functionality,
minimized bandwidth loss
• Redundant ScalabilitySwitch Boards
100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Unique to M5/M6
SPARC M6 RAS: End-to-End Protection
 Internal Logic: parity and ECC
– Architectural Registers
– Cache structures
– Internal networks
 Links: CRC retry
CRC
ECC with line retire
Data-ECC,Address-parity
Other (Parity, Retry etc)
DFT, Debug etc.
102 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC M6 RAS: General Error Handling Flow
Hardware
Detection and Clean-up
Gather Signature
De-configure Resources
Hypervisor
Assist with clean-up
Collect Hardware Data
Generate Report for SP
Manage De-configuration
Service Processor
Analyze Hypervisor Report
Update Error History
Initiate Service Call
Initiate De-configuration
Solaris
De-configure
User-Visible Resource
Offline
SerDes Lanes
Retire
Cache Lines
Activate DIMM
Spare Column
Retire Threads
Retire Cores
Retire Pages
103 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPARC M6 RAS: Memory Errors
 ECC optimized for device failures
 Inline correction and auto write-back
 “Scrubber” prevents accumulation of upsets
 “E-retry” characterizes soft vs persistent
 Cell or word-line fail: Solaris retires page(s)
 Bit-line or pin fail: firmware deploys DIMM
spare column
 Device fail: inline correction
M6
Scheduler
Normal
RD/WR
Scrubber
RD
Eretry
RD/WR
DIMM
ECC
Gen
ECC
Chk/Corr
Err
104 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
HW Aspects of RAS – New and Interesting
Features (1 of 3)
105 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Isolation of memory UE’s to a single DIMM
- Cachelines (the unit of memory access) are fully contained within a single DIMM
- Most (all?) past Oracle/Sun systems had cachelines that straddled two or four
DIMMs
- This means UE’s can be diagnosed to a single DIMM rather than generating a list of
possibilities
 Memory pin sparing
- ADIMM has 72 data pins (DQ’s) and we use a burst length of 8 when accessing
DIMMs => 572 total bits of data
- Only 564 bits used for data + ECC + control
- Remaining 8 unused bits sourced from a single DQ
- MCU can dynamically remap the layout to change which DQ is unused
HW Aspects of RAS – New and Interesting
Features (2 of 3)
106 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Deconfigurability of Scalability Switch Boards
- The 12 SSBs logically function as two groups of 6
- Agroup is capable of functioning with just 5
- Requires a system restart because it completely changes the mapping of SSBs
to physical addresses that are tracked
- In the case of a fatal error internal to an SSB, it will automatically be deconfigured
and the system restarted
- Replacement can happen while system is running but a manual restart will be
required to re-integrate
- In the case of non-fatal errors leading to service required, can be serviced live but
requires 2 manual restarts
- Note: single DCU physical domains can be configured to not use the SSBs
HW Aspects of RAS – New and Interesting
Features (3 of 3)
107 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Dual clock synthesizers per clock board
 Reconfigurability of PCI fabric
- All PCI slots are accessible by 2 different CMUs
- Primary or alternate path programmed at the time the physical domain
is started
- Allows for full IO accessibility in the case of a faulty or missing CMU
- Necessary because the PCI root complexes are on the M5/M6 chip
- Note: complications arise when multiple logical domains are configured, so
reconfiguration is suppressed once the system is “virtualized”
SW Aspects of RAS
108 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Fault diagnosis done on SP => covered under FMA
- Failures typically isolated to a single FRU, with the exception of
interconnect failures that span FRU boundaries
- Automatic reconfiguration of resources
 Configuring for Availability => covered separately
- Primarily addresses IO configuration using Logical Domains, and how to
account for multipathing
- LDoms also provide a level of isolation for some classes of uncorrectable
errors, since the impact will only be to that LDom
Fault Management
109 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Knowledge Articles in MOS
 ILOM fdd Diagnosis
 Faults and Alerts
 NoALOM Compatibility
 ILOM FMACaptive Shell
 Sideband Service Processor Network Connection
 New ILOM Fault Notification (SNMP Trap)
 ASR Support
FMA – restricted shell in SPSH
110 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 To get FMA details in SPSH
– start -script /SP/faultmgmt/shell
– Returns a faultmgmtsp> prompt
 Available build-in commands:
– echo - Display information to user.
 Typical use: echo $?
– help
– Exit – exits restircted shell
 External commands:
– fmadm - Administers the fault management service
– fmdump - Displays contents of the fault and ereport/error logs
– fmstat - Displays statistics on fault management operations
– etcd - ereport injector
ASR Support
111 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 SPARC M5/M6 servers will be supported byASR (Automatic Service
Request)
 Continues use of sunHwTrapFaultDiagnosed SNMP notification
 Telemetry for ILOM fdd diagnosis
 Supports platform and FRU identity
 Supports multi-suspect list
M6-32 FRU Serviceability
*
Reboot required to activate new board.
M6-32
112 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
ILOM
113 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Service Processor Software (ILOM) on M6
Systems
114 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 ILOM looks/behaves just like ILOM on other platforms
 Simple (user-visible) set of extensions to support Physical Domains
 Extensions to support Service Processor Proxies and redundant
Service Processors
- Minimal impact on user experience
Oracle ILOM Key Functions
115 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Management Interfaces
• CLI, BUI, IPMI, SNMP
• Firmware Updates
• Remote Host Management
• Inventory and Component Management
• System Monitoring and Alert/Fault Management
• UserAccount Management
• Power Consumption Management
ILOM on M6 – Enterprise Features
116 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 ILOM extensions to support Enterprise Features
– Some of these are conceptually leveraged from XSCF SW
 SP Tracing Facility
– Very useful for tracing inter-process activity
– Reliable performance (elapsed time) measurements
– Allows for tracing interactions between SPs and SPPs etc
– Enterprise systems have lower volume, more complex configurations and high RAS expectations
 We cannot expect customers to reproduce bugs
 Need to collect as much debug info as possible on live system as it occurs on customer site
– Coredump compression and snapshot collection
– Unified snapshot from all SPs and SPPs
ILOM on M6 – Enterprise Features
117 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Confstore distributed config database
– Exploitable by CMM/Blades
 System Identity is maintained across FRU / SP replacement
– aka TLI (Top-Level-Identifier)
 Flash Images are signed to avoid compromised images
 “hardened” edits of config files
– Transaction oriented – before or after, no intermediate results
 Tunables framework for MAX_USERS etc
 Sensor Broadcast (SSBCAST) enhanced to support
ILOM – New Enterprise Features
118 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Creation of Dynamic Domains (i.e. PDoms)
 Native FMA support
 Redundant SPs
 SPP support
ILOM User Roles
119 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 ILOM supports a maximum of 61 user accounts (configured under /SP/users) on the M6-32 SP
 M6-32 platforms introduce the concept of separately managed Physical Domains (aka PDom). Each
PDom can be separately controlled.
 Each user account can be optionally configured with specific roles for an individual PDom.
 The user roles are additive. For example, a user account is permitted to do reset operations on a
PDom if:
1. the user has 'r' role for the platform (in /SP/users/username/role) _OR_
2. the user has 'r' role for the specific PDom (in /SP/users/username/host_roles/hostX_role)
Roles and Capabilities
120 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Role Capabilities
none user has no user roles, and prevents the user’s roles from being looked up in LDAP
user (u) • Create, delete, disable, and enable user accounts.
• Change a user’s password and password properties.
• Change a user’s roles.
• View all platform states.
administrator(a) • Perform all Service Processor configurationother than the useradm and auditadm tasks.
• Assign and unassign hardware to or from domains.
• Perform Service Processor power operations.
• Perform Service Processor failover operations on systems with more than one Service Processor.
• Perform all operations on domainhardware.
• View all platform and physical domain states.
console (c) • Access the Oracle ILOM Remote Console and the SP console.
• View and change the state of the Oracle ILOM console configurationvariables.
reset (r) • View all states of the hardware assigned to the domain(s)on which this role is held.
• View all states of the domain(s)on which this role is held.
• Operate the system, which includes performingpower operations, resetting the system, hot-pluggingdevices, enabling and
disablingcomponents,and fault managementfor the specified domain.
read-only operator (o) • View all platform states.
• Change the password and the Session Time-Out setting for their own user account.
Whenthis role is defined at the domainlevel, users can:
• View all states of the hardware assigned to the domain(s)on which this role is held.
• View all states of the domain(s)on which this role is held.
field engineering (s) perform all operations reserved for field engineers.
Required User Roles
121 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Aspecific role is required for certain tasks:
Task Required User Role
Assign and unassign DCUs to a PDomain admin (a)
Manage ILOM passwords user (u)
Connect to a PDomain console (c)
Perform power operations (start, stop, and reset) reset (r)
Configure user accounts user (u)
Configure host groups user (u)
New to ILOM 3.1
122 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Simplified Data Model (SDM)
 Three-Level Model
– Level One: System Summary Info
– Level Two: Subsystem Summary
– Level Three: Logical Topology
 Subsystems: Cooling, Power, CPUs, Memory,
 Storage, and Networking
 Also Blades, DCUs, CMUs, CPU Modules, I/O Modules on some
platforms
 Open Problems unifies fault management with SDM
ILOM 3.2: New Linux distro and compilers
123 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 M6 will release with ILOM 3.2
 Linux version 2.6.27.43, “SQUEEZE”
– was 2.6.16.4, “SARGE”
 gcc version 4.4.5
– was 3.3.6
 Why:
– Old distro no longer supported
– Security and bug fixes
– Posix threads instead of Linux threads
WEB CLI
LUMAIN
SDM BACKEND
Platform xml
LIBHDL
Hw service
CAPI SSM API
SDM Architecture
124 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SDM CLI
125 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
CLI is reorganized.
 /System target(tree) is introduced
 Different components of the system are grouped and organized into
sub targets of /System
 At every level of the tree, the critical properties are shown along with
any sub targets
 The applicable cli commands are supported at different levels of the
/System tree.
 All targets and properties under /System are case insensitive
SDM CLI (cont.)
126 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Health and health details are two of the common properties shown at
every level to indicate the over all health of that sub tree.
 Open_Problems target shows the detailed descriptions of the faults in
the system
 /SYS and /Storage targets are made legacy
 Continue to exist but hidden by default
 The legacy targets can be made visible by enabling
/SP/cli/legacy_targets property.
SDM CLI (Summary level targets)
127 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
-> show /System -d targets
/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
SDM CLI (Summary level properties)
128 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
-> show /System -d properties
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = Exadata X2-3
part_number = 8124854
serial_number = 2229CNL124
component_model = SUN FIRE X4170 M3
component_part_number = 7013743
component_serial_number = 1118CNL013
system_identifier = sysidentifier
system_fw_version = 3.1.0.10
primary_operating_system = Not Available
host_primary_mac_address = 00:21:28:d5:c0:b2
ilom_address = 10.153.55.201
ilom_mac_address = 00:21:28:D5:C0:B6
locator_indicator = Off
power_state = Off
actual_power_consumption = 5 watts
action = (none)
SDM CLI (Processors Subsystem)
129 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
-> show /System/Processors/
Targets:
CPUs
Properties:
health = OK
health_details = -
architecture = x86 64-bit
summary_description = Two Intel Xeon Processor E5
Series
installed_cpus = 2
max_cpus = 2
-> show /System/Processors/CPUs/
Targets:
CPU_0
CPU_1
ILOM on M6 – SP-Proxy Design
130 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 M6-32 is a large system with 32 CPU chips, 1024 DIMMs, 2160+ sensors
– Too large for one SP to manage – I2C, JTAG traces etc
– For comparison, T5-8 has 8 CPU chips, 128 DIMMs, 400+ sensors
 Divide-and-conquer: each SPP is responsible for one DCU (8 CPUs/256 DIMMs)
– Re-use ILOM functionality on a per-DCU basis
 POD, Poller, rKVMS, USB-over-Ethernet
– Offloads work from Main-SP to SPPs.
 Main-SP (SP0/SP1) “aggregates” all SPs/SPPs and presents a unified system view.
 SP administration can be done on the Main-SP or drilldown to the SPP for PDom-specific
management
ILOM on M5 – SP-Proxy Design
131 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 One SPP-per-PDom is chosen as “Golden-SPP”
– Provides RKVMS service
– Serves as the “Management-SP” for the Pdom
 Externally accessible target for IPMI, SNMP, spsh operations
– Some functions from Main-SP are forwarded to Golden-SPP for service
 e.g. /HOSTx/SP/services/kvms, /HOSTx/SP/network/interconnect
 Configuration data from Golden-SPPs is backed to Main-SP
– Confstore is a distributed extension of ILOM's memstore database
– Maintains unique config data subset for every HOSTx
– As SPPs are moved between HOSTx, each SPP will obtain the correct config data
subset for its current HOSTx
 Audit logs, faults, etc forwarded from SPPs to Main-SP (and replicated)
ILOM on M6 – Redundant SP Support
132 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Redundant pair of Main-SPs
 Active-SP will be the management SP for the platform.
– Communicate all configuration data to the Standby
– Standby ready to take over if the Active fails
 Heartbeat is maintained between SPs and SPPs
– Active-SP → Standby-SP (SP0 or SP1)
– Active-SP → SP-Proxies (SPP0, .. SPP3)
 Implemented checkpointing support
– Use of checkpoint to recover from inconsistent state due to poweron/DR
operations interrupted by SP Failover/SPP failures
SP
133 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Service Processors (SP)
SP0 SP1
134 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6 M6
Service Processor Proxy (SPP)
 The SPP is a local SP dedicated to the DCU
 Each DCU has its own SPP
 It purpose is to
– Initialize, manage the CPU, Memory controller,
DIMMs with the DCU
– Environmental monitoring such as temp sensors
and adjust 8 Fans
 In a multi-DCU domain scenario, the SPP with
the lowest number available in the domain will
automatically become the Golden SPP that will
manage the domain for all multiple DCUs
 In case the Golden SPP fails, the next higher
numbers SPP takes over.
DCU
SP
To other SPP Units
Disks
0-3
Disks
4-7
M6 M6
M6 M6
M6 M6
135 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPP
SPP Function
136 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 SPPs are commanded by the main System SPs
 The communication is ethernet between the SP and SPPs
 Ethernet communication in SP used to manage the SPPs as well as
rKVMS
SPP Features
137 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Xilinx FPGA to provide the SSI boot interface to the CMU
 RKVMS (Remote Keyboard, Mouse and storage):
- This feature allow a domain console to be shown at a remote display. It is able to send keyboard,
mouse, and storage commands through Ethernet Packets.
- The graphical pixel scraper as contained in a Matrox PCIE VGA inside the Pilot3 Controller.
- Avirtual USB hub (not a USB controller) that can transmit and receive USB connections from an
external USB UPD727200 controller. The Hub resides in the Pilot3 controller.
- The Pilot3 converts these devices in to Ethernet packets and provides remote console
functionality for the domain.
- Logical Domain Control communication will use Ethernet over USB to share the global SRAM
needed for CMU’s to communicate between each other.
 100 Mhz PCIE reference clock distribution to CMU’s and IO boards.
 Ethernet interfaces to the SP board via the capacitive coupled scheme.
Physical Domains
138 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M6 Systems Terminology
139 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 M6 Systems consist of CPU, Memory and IO
- CPU and Memory are on CPU/Memory boards (CMUs)
 Each CMU contains 64 DIMM slots
 Domain Configuration Units (DCUs) are groups of 4 CMUs, and one
IOU, connected by the local coherency interconnect
 ADynamic Domain consists of one or more DCUs
 ADynamic Domain is also referred to as a Physical Domain (or
PDom)
Domain Configuration Unit (DCU)
 Since the M6-32 forms groups of 8-way bounded domains,
4 CMU boards and a single IOU form a single
configuration unit, or DCU.
 All communication to other components in the DCU do not
need to pass through a BX ASIC.
 ABX is only needed when data from one CMU needs to
go to a CMU in a different DCU.
 Aphysical domain (PDom) can be made from one or more
DCUs.
 Each DCU is controlled and managed by its own SP Proxy
 The admin manages the DCU via the main SP
M6 1 M6 2
DCU
SP
To other SPP Units
Disks
0-3
Disks
4-7
M6 7 M6 8
M6 5 M6 6
M6 3 M6 4
140 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
SPP
Similar DCUs On Other Systems
141 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
E25K M9000-32 M5-32/M6-32
1 Uniboard
+
1 I/O Board
1) 1 CPU + 8 DIMMs +
2 PCIe slots
2) 1 CMU + 1 IOU
4 CMU + 1 IOU
4 DCUs
 ADCU has:
– 1 IOU
– 1 SPP
– Up to 4 CMUs
 DCUs communicate using
the SSB
DCU0
142 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
DCU1 DCU2
DCU3
DCU Population Rules
143 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Must install minimum of 4 CMU boards
– minimum 2 CMU per DCU
 Additional CMU boards must be installed in pairs
– DCU slots 0 & 3 first, then slots 1 & 2
– Need both even & odd in DCU for proper SSB communication
 HDD or SSD requirement per PDom
– Minimum of 2 HDD
– Minimum of 2 Base IO cards for the initial pair of HDD
– Additional Base IO cards depend on placement of subsequent pairs of HDD
DCU Memory Population Rules
144 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Memory on CMUs are either ¼, ½, or fully populated
– All the memory on the CMU must be the same density and rank
– All the CMU boards in a DCU must be populated exactly the same way, and with the
same exact DIMMs
– CMU boards in different DCUs can be populated differently
M5/M6 Population Rules
145 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Both processors on a CMU must be the same (both M5 or
both M6)
 All CMUs in a DCU must be the same (all M5 or all M6
based CMUs)
 APDom can be made from a combination of M5 and M6
based DCUs
– If DCU0 has four M5 cpus and DCU1 has eight M6 cpus, they
can both be in the same PDom
 If a DCU has empty CMU slots, they can only be filled with
CMUs exactly the same as the existing CMUs in that DCU
Mixing M5 and M6 CPUs Rules
146 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 All the CMUs in a DCU must be identical
 No mixing of M5 and M6 CMUs in a DCU
 APDom can be made from multiple DCUs,
– some DCUs are M5 populated
– other DCUs are M6 populated
Dynamic Domains
147 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 There can be up to four physical domains (PDoms or PDomain) in the M6-32
 Each PDom operates like an independent server that has full hardware
isolation from other PDoms on the server
 Domain configurable units (DCUs) are the building blocks of PDoms
 Each PDom is represented as /Servers/PDomains/PDomain_x/HOST in
Oracle ILOM where x ranges from 0 to one less than the maximum number of
possible PDoms in the system (PDomain_0, PDomain_1, PDomain_2,
PDomain_3).
Expandable Dynamic Domains
148 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 A PDom has an attribute of expandability (expandable flag)
 By default, a Physical Domain consisting of a single DCU will run as a
regular Physical Domain (i.e. expandable=true)
 A non-expandable PDom is called a Bounded PDom (or Bounded
Physical Domain) (i.e. expandable=false)
 A PDom can be set to either a Bounded Physical Domain, or a Regular
Physical Domain
 However, a Bounded Physical Domain is not affected by the loss of an
SSB
Bounded Dynamic Domains
149 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 ABounded Physical Domain (or Bounded PDom)
– does not use the SSB for L3$ directory storage
– for CMU to CMU access within the DCU
 Lower latency
 No impacted by loss of SSB
 Can never grow beyond a DCU
 Changing a domain to Bounded, or from Bounded to regular, requires
the domain be halted before changing the “expandable” variable.
Tradeoffs
150 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Pros Cons
Dynamic
Domain (PDom)
• Expandable beyond 8 M6 CPUs
• SSB stores L3$ directory for faster
lookup (benefits remote DCU)
• higher latency
• loss of any SSB causes PDom reset
Bounded
Dynamic
Domain (PDom)
• Lower latency
• loss of any SSB has no effect
• Can not expand beyond 8 M6 CPUs
PDoms Examples
= Bounded DD
= DD
PDom 0 PDom 0 PDom 1 PDom 0 PDom 1
PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2
PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
151 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
DCU and Dynamic Reconfiguration
152 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Individual CMU/IOU boards can not be logically moved between PDoms
 Only virtual CPUs within a PDom can be moved between logical domains in
the same PDom
 A Dynamic Domain (i.e. PDom) can have its resources changed as a cold-
service event.
Latency for M6 and M8000/M9000 Systems
153 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Latency (ns) M8000 M9000-32 M6-32
PDom 8-socket
M6-32
“Bounded PDom” 8-socket
Within CMU 342 387 160 158
Within XB Group/DCU 402 447 222 221
Within Cabinet 402 464 329 -
Domain Boot Sequence
154 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 M6 starts executing Hostconfig code from flash prom
 Hostconfig does the following
– Initialization and configuration of CPUs, Memory, etc.
– Generates MDs and PRI for Hypervisor and Guest
– Invokes diagnostics and applies platform policies to configure system around failed
components
– Jumps to Hypervisor on master CPU; others parked
 Hypervisor proceeds to
– Copy itself from ROM to RAM
– Initializes itself based on HV MD
– Starts the guest (OpenBoot is the first guest)
 OpenBoot probes I/O devices based on MD and sets up the device tree for Solaris.
Starts Solaris boot.
Internal VLAN
155 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Internal VLAN network is not exposed to the customer network.
 End users should not have the need to login directly to the SPPs.
Target IP: 169.254.10.xx
SP 1
SP0 2
SP1 3
SSP0 17
SSP1 18
SSP2 19
SSP3 20
Update Firmware
156 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 The SP firmware can be updated for all SPs at once
 The firmware image includes the Oracle ILOM firmware, OpenBoot PROM
firmware, POST firmware, and miscellaneous files
 The firmware image is installed in the SP flash memory
 A user must have the administrator (a) or field engineering (s) role to update the
firmware
Relative Domain Boot Times
Boot Times In Minutes
157 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
M9000-64w/ 1TB M6-32 w/ 4TB
No Post No Post Max Post
AC Power On -> ILOM Shell 12:22 12:15 12:15
Start Host -> OBP OK 25:50 27:04 76:40
OBP OK -> Solaris Login 02:11 04:02 04:02
Total Power On to Solaris 40:23 43:21 92:57
Solaris Halt -> OBP 00:56 04:15 04:15
OBP -> Power Off 03:58 03:23 03:23
Total Solaris Halt to Power Off 04:54 07:38 07:38
SPARC Virtualization
Technologies for The M5
158 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Oracle Solaris and SPARC Virtualization
Better Resource Utilization for a More Efficient Datacenter
Dynamic Domains Oracle VM Server
for SPARC
M-Series T-Series, M5/M6
Oracle Solaris
Zones
Oracle Solaris
DW DB
Domain A
OLTP DB
OLTP DB
Domain B
Domain A
App
Domain B
App
Domain C
Web
Web
Web
159 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
OracleSolaris8Zone
OracleSolaris9Zone
OracleSolarisZone
OracleSolarisZone
Web DB App Web
Virtualization on M6 Systems
160 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 High degree of virtualization
1. Physical domains
- Granularity of physical domains is 8 CPU sockets
2. Logical domains
- Includes support of LDoms within physical domains
3. Oracle Solaris Zones for OS virtualization
 Oracle Enterprise Manager Ops Center provides an administrator-
friendly integration of these different virtualization levels
SPARC
Physical Domain
Oracle Solaris 11 Oracle Solaris 10
Oracle VM Server for SPARC
Solaris 10
Zone
Physical Domain
Oracle Solaris 11 Oracle Solaris 10
Oracle VM Server for SPARC
Solaris 11 Zone
Most Extensive Virtualization Infrastructure
Virtualization Can Be Layered
Solaris
11
Zone
Solaris
Legacy
Zone
Solaris
10
Zone
Solaris Legacy Zone
161 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Oracle Solaris Zones
Built-in Virtualization on Any Oracle Solaris System
162 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Same virtualization technology for all SPARC, x86 systems
 Simple; lowest overhead; highest performance
- Ideally suited to leverage multithreading hardware
 Mission-critical deployments
- Largest Sun financial and Telco customers
all run Oracle Solaris Zones
- In production on 25+% of installed Oracle Solaris systems
 Ideal for a variety of scenarios
- Lightweight test environments
- Dynamic environments with resource sharing
- Rapid prototyping test beds on same hardware and OS
- Zones cloning/migration/instant restart
Built-in Virtualization
Oracle Solaris 11 Zones
• Secure, light-weight virtualization
• Scales to 100s of zones/ node
• Delegated administration
• ZFS datasets, boot environments
• Observability via zonestat
• Solaris 10 Zones
• NFS Server
• Network stack isolation and
resource management
Co-engineered with installation, security, ZFS, networking, IPS, SPARC and x86 hypervisors
163 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Cloud-Scale Networking
 Virtualize, consolidate network infrastructure
– Increase performance and reduce costs
– Secure Isolation
 Integrated functionality
– Routing, Firewalling, Load Balancing, Bridging, High
Availability
• Parallel networking stack. Built to scale.
• Hardware assisted Network Resource Management
• Optimized for performance at every level
•Ease of Use
• Automatic Networking mode
• Fine grained observability
• VLAN isolation, dynamic VLAN provisioning
164 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Oracle VM Server for SPARC
The Virtualization Platform combining the best of Oracle Solaris and
SPARC for Your Enterprise Server Workloads
Isolated OS and
applications in each
logical (or virtual)
domain
Firmware-based hypervisor
Each logical domain runs in
dedicated CPU thread(s)
M6-32 Server
Oracle Solaris 10
Oracle Solaris 11
Database Domain
Oracle Solaris 10
Oracle Solaris 11
Database Domain
Oracle Solaris 10
Oracle Solaris 11
Database Domain
Oracle Solaris 10
Oracle Solaris 11
Database Domain
GP Domain GP DomainGP Domain GP Domain
SPARC Hypervisor
165 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Alignment with SPARC– designed for Threads
166 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Traditional VM based on assumption “CPUs are scarce, so we must
over-commit and time-slice them”
• Overhead for time-slicing different contexts
• Intercept for “privileged operations”
• Latency servicing every interrupt
• T5/M5/M6 systems are “thread-rich” - so we can dedicate CPU
threads to each domain for native CPU performance
• Eliminates CPU latency and overhead
• Context switches in a single clock on cache miss or interval
• Some VM systems also over-commit RAM,
• Causes overhead and requires complex memory management
• Ok for lightweight, occasional workloads, very bad for enterprise apps
Roles of Domains
167 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Control domain
• Creates and manages other logical domains and services
• Control domain usually also a service and I/O domain
• I/O domains
• own physical I/O bus or devices. May run apps using physical I/O for native
performance
• Service domains
• provide virtual network and disk devices. Typically an I/O domain
• Guest domain:
• run applications on virtual I/O devices provided by service domain
Domain Components
168 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Hypervisor and Logical Domains
Oracle VM for SPARC
169 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 A logical domain (LDom) is a virtual machine comprised of a discrete logical
grouping of resources
 Each LDom runs its own instance of Solaris
 Each LDom can be created, destroyed, reconfigured, and rebooted
independently
 The hypervisor enforces the partitioning of the server's resources, and the OS
and applications running in those partitions (i.e. LDoms)
 The hypervisor allocates a subset of the overall CPU, memory, and I/O
resources of a server to a given logical domain
 Up to 128 Logical Domains per hypervisor
 Minimum version supported is 3.1
Oracle VM Server for SPARC: M6 Servers
 A physical domain is made of one or more Domain
Configuration Units (DCU)
 Each physical domain must have a hypervisor
 Maximum of 128 logical domains per physical domain
Isolated OS and applications in each
logical (or virtual) domain
Firmware-based hypervisor
Each logical domain runs in dedicated
CPU thread(s), and memory
Physical Domain
Hypervisor
DCU
IOU
CMU
CMU
CMU
CMU
DCU
IOU
CMU
CMU
CMU
CMU
Up to 4
Domain DomainDomain Domain
SOA DB
170 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
LDom Example
171 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 This shows the following layers that make up the Logical Domains
functionality:
- User/services, or applications
- Kernel, or operating systems
- Firmware, or hypervisor
- Hardware, including CPU, memory, and I/O
Combinations of Physical Domains
172 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Max LDoms: 128 Max LDoms: 256
SOA DB SOA DB SOA DB
Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain
Hypervisor Hypervisor Hypervisor
DCU DCU DCU DCU DCU DCU DCU DCU
CMU CMU CMU CMU CMU CMU CMU CMU
Domain
Max LDoms: 256
SOA DB SOA DB
Domain Domain Domain Domain Domain Domain
Hypervisor Hypervisor
DCU DCU DCU DCU
IOU IOU IOU IOU
CMU CMU CMU CMU
CMU CMU CMU CMU
CMU CMU CMU CMU
CMU CMU CMU CMU
Domain
Max LDoms: 384
SOA DB SOA DB
Domain Domain Domain Domain Domain Domain Domain Domain
Hypervisor Hypervisor
DCU DCU DCU
IOU IOU IOU
CMU CMU CMU
CMU CMU CMU
CMU CMU CMU
CMU CMU CMU
Domain SOA DB
Domain Domain
Hypervisor
DCU
IOU
CMU
CMU
CMU
CMU
Domain
SOA DB
Domain Domain Domain Domain Domain
Hypervisor
DCU
IOU
CMU
CMU
CMU
CMU
Max LDoms: 512
SOA DB SOA DB
Domain Domain Domain Domain Domain Domain Domain
Hypervisor Hypervisor
DCU DCU
IOU IOU
CMU CMU
CMU CMU
CMU CMU
CMU CMU
SOA DB
Domain Domain Domain
Hypervisor
DCU
IOU
CMU
CMU
CMU
CMU
Domain
DCU DCU
Assign PDUs to the PDom
 PDom 0 can be either a Bounded PDom, or a regular PDom
 PDom 1 is a regular PDom
 If PDom 0 is regular, then a DCU from PDom 1 can be logically moved
from PDom 1 to PDom 0
Hardware
DCU
PDom 0 PDom 1 Hardware
173 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
DCU DCU
Create Guests
Hardware
DCU
Hardware
Logical
Domain A
Logical
Domain B
Logical
Domain A
Logical
Domain B
Logical
Domain C
Hypervisor Hypervisor
174 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
PDom 0 PDom 1
DCU DCU
Create Zones In Each Guest
Hardware
DCU
Hardware
Kernel
Hypervisor Hypervisor
App
App
App App
App
App
Solaris 11 Solaris 11 Solaris 10U9 + Patches Solaris 10U11Solaris 11
Logical
Domain A
Zone A
App
Logical
Domain B
Logical
Domain A
Logical
Domain B
App
App
Zone A
App
Logical
Domain C
Zone B
User/Services
App
App
175 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
PDom 0 PDom 1
Configuring The Hypervisors
176 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Each Physical Domain can have its hypervisor configured differently
than the others
- Config 1 - All devices are virtualized and run through the Control Domain,
or Service Domain.
- Config 2 - All CPU and memory are virtualized and run through the Control
Domain.All I/O is provided through the I/O Domain, which owns a PCIe
root complex, or offers direct I/O to a specific PCIe device.
- Config 3 – No devices are virtualized. All devices are directly accessed
from the Control Domain, in which the Logical Domain Manager is optional.
- Config 1 and Config 2 can be combined if needed.
VM
External Shared Storage
Oracle VM Server Pool
Secure Live Migration
Eliminates Application Downtime
 Live migration available on SPARC
systems (w/o encryption)
– SPARC M5/M6
– SPARC T5
– SPARC T4
– SPARC T3
– UltraSPARC T2 Plus
– UltraSPARC T2
 On-chip crypto accelerators deliver
secure, wire speed encryption for live
migration
– No additional hardware required
– Eliminates requirement for dedicated network
 More secure, more flexible
Secure Live Migration (SSL)VM VM
SPARC servers
177 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Cross CPU Migration - Architecture
178 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Allows migration of domains across sun4v architecture platforms
• Supports migration among platforms
• Will be extended to support new platforms as they are introduced
– new platforms might not be migration-compatible with all previous platforms
• Allows migration among same CPU architecture with different system clocks
frequencies
• Dependent on guest domain having Solaris 11
– Solaris introduces a generic sun4v CPU module, simulated 1GHz system
clock if HW not available, other changes.
• LDoms Manager introduces domain cpu-arch property
Cross CPU Migration - Solaris
179 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Supported in guests running
– Solaris 11.1
– Solaris 10 1/13
• Introduces new generic CPU module: sun4v-cpu
– Domain service extension to identify CPU module capabilities
– CPU Module has a major/minor version number, used by domain manager to
determine capabilities of the guest
• Simulates 1GHz system clock if needed
– Kernel routines for read tick/stick modified to emulate clock rate: emulate 1GHz in
generic mode; emulate boot frequency after migration in native mode to system with
different clock frequency
Cross CPU Migration – Generic Domains
180 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• LDom Manager/Firmware/Solaris must be of sufficient revision to
support Cross CPU Migration
– Firmware must support LDom Live Migration on both source and target
domains
– Guest domain must be Solaris 11 FCS or newer
– Migration is for the most part unchanged
• At the start of the migration, domain capabilities and generic CPU
module version are retrieved and sent to the target
• Check on target ensures that the target processor supports the generic
CPU module version
Live Migration Requirements
181 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
• Live Migration with encryption
– Only support when source and target systems are the same type (i.e., both
T3, both T4, etc…) using cpu-arch=generic
– Under OVM for SPARC 3.1, live migration is supported across S3 core
based systems (i.e. T4 to M5, or T5 to T4, etc….)
Solaris Support
182 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Software Support on M6-32
183 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
 Solaris 11.1 is required to run in the control domain. This means
M5 only supports Solaris 11.1 on bare metal.
 Solaris 10 1/13 is supported in guest, I/O, root, and service
domains.
 S10U9, S10U10, and S11 + appropriate Patch Bundle is
supported in guest, I/O, root, and service domains.
 M6 platform supports Dynamic Domains
 OVM for SPARC 3.1
Power Management
184 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Power Management Features
185 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
Existing M Series (M3000-M9000) have no active power management features
Feature T4 T5 M5/M6 Comments
Dynamic Voltage & Frequency
Scaling
(DVFS)
No   New to SPARC, already exists on x86
Cycle Skipping    T4 whole socket granularity
T5/M5 sub socket granularity
Coherency Link Scaling No * No T5-2 only
Power Supplies Gold+ A261A A254 T5 PS (A261): Goal Platinum
M5/M6 (A254): 3 Phase goal similar to
platinum
IFS (Intelligent Fan Control)    (Technically not CPU feature)
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overview

Más contenido relacionado

La actualidad más candente

Oracle RAC features on Exadata
Oracle RAC features on ExadataOracle RAC features on Exadata
Oracle RAC features on ExadataAnil Nair
 
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)オラクルエンジニア通信
 
Simple Solutions for Complex Problems
Simple Solutions for Complex Problems Simple Solutions for Complex Problems
Simple Solutions for Complex Problems Apcera
 
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)オラクルエンジニア通信
 
The DDS Tutorial - Part I
The DDS Tutorial - Part IThe DDS Tutorial - Part I
The DDS Tutorial - Part IAngelo Corsaro
 
OpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design SolutionsOpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design SolutionsShane Gibson
 
Architecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPUArchitecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPUGlobalLogic Ukraine
 
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing WorldOMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing WorldAllan Cantle
 
Oracle Security Presentation
Oracle Security PresentationOracle Security Presentation
Oracle Security PresentationFrancisco Alvarez
 
Expanding Asterisk with Kamailio
Expanding Asterisk with KamailioExpanding Asterisk with Kamailio
Expanding Asterisk with KamailioFred Posner
 
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive Glen Hawkins
 
Best Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open SourceBest Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open SourceNGINX, Inc.
 
Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001jucaab
 
Decoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMIDecoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMIAllan Cantle
 
Oracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12cOracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12cTrivadis
 
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020 Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020 Oracle
 
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...Edge AI and Vision Alliance
 
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3オラクルエンジニア通信
 

La actualidad más candente (20)

Oracle RAC features on Exadata
Oracle RAC features on ExadataOracle RAC features on Exadata
Oracle RAC features on Exadata
 
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
 
Simple Solutions for Complex Problems
Simple Solutions for Complex Problems Simple Solutions for Complex Problems
Simple Solutions for Complex Problems
 
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
 
The DDS Tutorial - Part I
The DDS Tutorial - Part IThe DDS Tutorial - Part I
The DDS Tutorial - Part I
 
OpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design SolutionsOpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design Solutions
 
Architecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPUArchitecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPU
 
Securefile LOBs
Securefile LOBsSecurefile LOBs
Securefile LOBs
 
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing WorldOMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
 
Oracle Security Presentation
Oracle Security PresentationOracle Security Presentation
Oracle Security Presentation
 
Expanding Asterisk with Kamailio
Expanding Asterisk with KamailioExpanding Asterisk with Kamailio
Expanding Asterisk with Kamailio
 
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
 
Best Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open SourceBest Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open Source
 
Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001
 
OCI GoldenGate Overview 2021年4月版
OCI GoldenGate Overview 2021年4月版OCI GoldenGate Overview 2021年4月版
OCI GoldenGate Overview 2021年4月版
 
Decoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMIDecoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMI
 
Oracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12cOracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12c
 
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020 Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
 
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
 
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
 

Similar a Presentation sparc m6 m5-32 server technical overview

Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterpriseSparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprisesolarisyougood
 
Sparc t3 2 technical presentation
Sparc t3 2 technical presentationSparc t3 2 technical presentation
Sparc t3 2 technical presentationsolarisyougood
 
Fujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilitiesFujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilitiessolarisyougood
 
Computer Systems And Networks Configuration
Computer Systems And Networks ConfigurationComputer Systems And Networks Configuration
Computer Systems And Networks ConfigurationTara Daly
 
Power 7 Overview
Power 7 OverviewPower 7 Overview
Power 7 Overviewlambertt
 
Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached Fuad Arshad
 
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSnehaLatha68
 
April 2014 IBM announcement webcast
April 2014 IBM announcement webcastApril 2014 IBM announcement webcast
April 2014 IBM announcement webcastHELP400
 
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I   Core of Embedded SystemsSYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I   Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded SystemsArti Parab Academics
 
SUN主机产品介绍.ppt
SUN主机产品介绍.pptSUN主机产品介绍.ppt
SUN主机产品介绍.pptPencilData
 
Sparc t4 4 system technical overview
Sparc t4 4 system technical overviewSparc t4 4 system technical overview
Sparc t4 4 system technical overviewsolarisyougood
 
Overview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit MicrocontrollersOverview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit MicrocontrollersPremier Farnell
 
AMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor ArchitectureAMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor ArchitectureAMD
 
system unit and Motherboard
system unit and Motherboardsystem unit and Motherboard
system unit and Motherboardromeodait
 
Sparc t4 1 system technical overview
Sparc t4 1 system technical overviewSparc t4 1 system technical overview
Sparc t4 1 system technical overviewsolarisyougood
 
Sparc t3 4 system technical overview
Sparc t3 4 system technical overviewSparc t3 4 system technical overview
Sparc t3 4 system technical overviewsolarisyougood
 
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_OverviewRichard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_OverviewRichard Baker
 

Similar a Presentation sparc m6 m5-32 server technical overview (20)

Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterpriseSparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprise
 
The Cell Processor
The Cell ProcessorThe Cell Processor
The Cell Processor
 
Sparc t3 2 technical presentation
Sparc t3 2 technical presentationSparc t3 2 technical presentation
Sparc t3 2 technical presentation
 
Fujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilitiesFujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilities
 
Computer Systems And Networks Configuration
Computer Systems And Networks ConfigurationComputer Systems And Networks Configuration
Computer Systems And Networks Configuration
 
Power 7 Overview
Power 7 OverviewPower 7 Overview
Power 7 Overview
 
Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached
 
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
 
April 2014 IBM announcement webcast
April 2014 IBM announcement webcastApril 2014 IBM announcement webcast
April 2014 IBM announcement webcast
 
Phytium 64 core cpu preview
Phytium 64 core cpu previewPhytium 64 core cpu preview
Phytium 64 core cpu preview
 
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I   Core of Embedded SystemsSYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I   Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
 
SUN主机产品介绍.ppt
SUN主机产品介绍.pptSUN主机产品介绍.ppt
SUN主机产品介绍.ppt
 
Sparc t4 4 system technical overview
Sparc t4 4 system technical overviewSparc t4 4 system technical overview
Sparc t4 4 system technical overview
 
Overview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit MicrocontrollersOverview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit Microcontrollers
 
AMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor ArchitectureAMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor Architecture
 
system unit and Motherboard
system unit and Motherboardsystem unit and Motherboard
system unit and Motherboard
 
IBM HPC Transformation with AI
IBM HPC Transformation with AI IBM HPC Transformation with AI
IBM HPC Transformation with AI
 
Sparc t4 1 system technical overview
Sparc t4 1 system technical overviewSparc t4 1 system technical overview
Sparc t4 1 system technical overview
 
Sparc t3 4 system technical overview
Sparc t3 4 system technical overviewSparc t3 4 system technical overview
Sparc t3 4 system technical overview
 
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_OverviewRichard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
 

Más de solarisyougood

Emc recoverpoint technical
Emc recoverpoint technicalEmc recoverpoint technical
Emc recoverpoint technicalsolarisyougood
 
Emc vmax3 technical deep workshop
Emc vmax3 technical deep workshopEmc vmax3 technical deep workshop
Emc vmax3 technical deep workshopsolarisyougood
 
EMC Atmos for service providers
EMC Atmos for service providersEMC Atmos for service providers
EMC Atmos for service providerssolarisyougood
 
Cisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overviewCisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overviewsolarisyougood
 
Designing your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guideDesigning your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guidesolarisyougood
 
Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...solarisyougood
 
Ibm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshopIbm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshopsolarisyougood
 
Power8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshopPower8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshopsolarisyougood
 
Power systems virtualization with power kvm
Power systems virtualization with power kvmPower systems virtualization with power kvm
Power systems virtualization with power kvmsolarisyougood
 
Power vc for powervm deep dive tips &amp; tricks
Power vc for powervm deep dive tips &amp; tricksPower vc for powervm deep dive tips &amp; tricks
Power vc for powervm deep dive tips &amp; trickssolarisyougood
 
Emc data domain technical deep dive workshop
Emc data domain  technical deep dive workshopEmc data domain  technical deep dive workshop
Emc data domain technical deep dive workshopsolarisyougood
 
Ibm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshopIbm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshopsolarisyougood
 
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopEmc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopsolarisyougood
 
Emc isilon technical deep dive workshop
Emc isilon technical deep dive workshopEmc isilon technical deep dive workshop
Emc isilon technical deep dive workshopsolarisyougood
 
Emc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshopEmc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshopsolarisyougood
 
Cisco mds 9148 s training workshop
Cisco mds 9148 s training workshopCisco mds 9148 s training workshop
Cisco mds 9148 s training workshopsolarisyougood
 
Cisco cloud computing deploying openstack
Cisco cloud computing deploying openstackCisco cloud computing deploying openstack
Cisco cloud computing deploying openstacksolarisyougood
 
Se training storage grid webscale technical overview
Se training   storage grid webscale technical overviewSe training   storage grid webscale technical overview
Se training storage grid webscale technical overviewsolarisyougood
 

Más de solarisyougood (20)

Emc vipr srm workshop
Emc vipr srm workshopEmc vipr srm workshop
Emc vipr srm workshop
 
Emc recoverpoint technical
Emc recoverpoint technicalEmc recoverpoint technical
Emc recoverpoint technical
 
Emc vmax3 technical deep workshop
Emc vmax3 technical deep workshopEmc vmax3 technical deep workshop
Emc vmax3 technical deep workshop
 
EMC Atmos for service providers
EMC Atmos for service providersEMC Atmos for service providers
EMC Atmos for service providers
 
Cisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overviewCisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overview
 
Designing your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guideDesigning your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guide
 
Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...
 
Ibm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshopIbm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshop
 
Power8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshopPower8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshop
 
Power systems virtualization with power kvm
Power systems virtualization with power kvmPower systems virtualization with power kvm
Power systems virtualization with power kvm
 
Power vc for powervm deep dive tips &amp; tricks
Power vc for powervm deep dive tips &amp; tricksPower vc for powervm deep dive tips &amp; tricks
Power vc for powervm deep dive tips &amp; tricks
 
Emc data domain technical deep dive workshop
Emc data domain  technical deep dive workshopEmc data domain  technical deep dive workshop
Emc data domain technical deep dive workshop
 
Ibm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshopIbm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshop
 
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopEmc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshop
 
Emc isilon technical deep dive workshop
Emc isilon technical deep dive workshopEmc isilon technical deep dive workshop
Emc isilon technical deep dive workshop
 
Emc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshopEmc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshop
 
Emc vplex deep dive
Emc vplex deep diveEmc vplex deep dive
Emc vplex deep dive
 
Cisco mds 9148 s training workshop
Cisco mds 9148 s training workshopCisco mds 9148 s training workshop
Cisco mds 9148 s training workshop
 
Cisco cloud computing deploying openstack
Cisco cloud computing deploying openstackCisco cloud computing deploying openstack
Cisco cloud computing deploying openstack
 
Se training storage grid webscale technical overview
Se training   storage grid webscale technical overviewSe training   storage grid webscale technical overview
Se training storage grid webscale technical overview
 

Último

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Último (20)

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 

Presentation sparc m6 m5-32 server technical overview

  • 1. 1 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 3. SPARC M6/M5-32 Server – Technical Overview Insert Presenter’s Name Here Insert Presenter’s Title Here 3 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 4. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 4 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 5. SPARC M6/M5-32 Server 5 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 6. New l SPARC M6 Servers 32-Socket High-End Datacenter Server  Compute – Up to 32x SPARC M6 12 core 3.6GHz CPUs – Allows mixing with SPARC M5 6 core processors – Up to 1024x DDR3 DIMMs for max memory of up to 32TB  I/O and storage – 32x 2.5” SAS-2 internal drives – 64x PCIe Gen3 low profile internal slots  Scalability and investment protection – Upgradable with M6 processor  Availability and management – Advanced RAS with redundant and hot swap of key components – Extensive virtualization with Oracle VM Server for SPARC + hard partitions – Integrated Oracle ILOM system management 6 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 7. High-end Product Data Sheet 7 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 Processor M6 3.6 GHz, 48 MB L3$ Max 32 (384 cores) Memory Max 32TB (32GB x 1024 DIMMs) Internal Disks Max 32 (2.5” HDD or SSD) RemovableMedia rKVMS CMU Max 16 IOU Max 4 PCIe Slots Max 64 PCIe Domains Max 4 physical Max 512 logical Power Options 3-phase/Dual Grid Power Consumption 25 KW System Management ILOM (main service processor)
  • 8. High-end Product Comparison 8 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 2 TB DDR3 DIMMs) CIe Gen 3 Dual Grid Max= 25KW vice processor) .1”Dx78.7”H (1677 kg) M9000-32 M5-32/M6-32 Processor SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88GHz, 6 MB L2$ Max 64 (256 cores) SPARC64 VII+ M5 3.6 GHz, 48 MB L3$ 3.0 GHz,12 MB L2$ Max 32 (192 cores) Max 64 (256 cores) M6 3.6 GHz, 48 MB L3$ Max 32 (384 cores) Memory Max 4 TB Max 3 (8GB x 512 DDR2 DIMMs) (32GB x 1024 Internal Disks Max 64 (2.5” SAS) Max 32 (2.5” HDD or SSD) RemovableMedia DVD, DAT rKVMS CMU and IOU Max 16 each Max 16 CMU and 4 IOU PCI Slots Max 64 PCIe Gen 1 Max 64 P Domains Max 24 physical Max 4 physical, Max 512 logical Power Options 1-phase/3-phase/DualGrid 3-phase/ Power Consumption Min= 12KW, Max= 14.6KW Min=23KW, System Management XSCF (service processor) ILOM (main ser MemoryBandwidth 737 GB/s 1,980 GB/s I/O Bandwidth 244 GB/s 1,024 GB/s Dimensions 33.5”Wx49.6”Dx70.9”H 35.6”Wx57 Weight (Max) Base cab: 2068 lbs.(940kg),Power cab: 770 lbs.(350 kg) 3697 lbs.
  • 9. Key Differences from Earlier M-series 9 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M8000/M9000 M6-32 SP XSCF, running XCP SP, running ILOM Each DCU has it’s own SP Proxy (SPP) Domains Hard Domains: up to 24 Hard Domains: up to 4, Logical Domains: up to 512 Memory Clock DDR2 at 480MHz DDR3 at 1066MHz Lowest Latency 387ns 158 ns (Bounded Domain), 160 ns (Regular Domain) IO PCIe Gen 1 PCIe Gen 3 Smallest Hard Domain 1 CPU + 8 DIMMs (Quad-XSB) 4 CPUs + 128 DIMMs (2 CMU boards) Internal network PPP 10/100 Ethernet Clock and Interconnect Cold service for failed clock board and crossbar board Hot-plug replacement of failed clock and SBB boards Power Feed Single phase, or 3-phase. DPF optional Customer provides own cable for 3-phase 3-phase, DPF standard Cable is orderable Airflow Bottom to top Front to back Base IO Card Must use existing PCIe slot Dedicated slots just for Base IO card DVD Built-in. Can be assigned to any Base IO card N/A. Must use remote media of rKVMS. Cable Routing Bottom only Top or bottom
  • 10. Term Chart 10 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Term Definition CPU memoryboard Unit (CMU) Board equipped with two M6 CPU modules and up to 64 DIMMs of memory Chip Multi-Processor(CMP) Another term for a multi-core processor Buffer on Board (BoB) An ASIC that interfaces between the DIMMs and the MCU on the M6 Memory Controller Unit (MCU) Each M6 has two MCUs in order to communicatewith each BoB. IO board unit (IOU) Unit equipped with 16 PCIe Gen 3, 8 disk drives (HDD or SSD), and 8 10GbE ports DomainConfigurationUnit (DCU) A grouping of up to 4 CMUs, 1 IOU controlled by its own SP Proxy Physical Domain (PDom) An electrically fault-isolatedhard domain comprised of one or more DCUs Hypervisor The SPARC hypervisor is a small firmware layer that provides a stable virtualized machine architectureto which an operating system can be written.
  • 11. Front Cabinet Components 1. System status indicators, key switch, and ESD grounding connector 2. 12 Power Supplies (PS) 3. 18 Fan Modules (FM) 4. Leveling Feet 5. 2 Power Supply Distribution Boards (PSDB) 6. 2 Service Processors (SP) 7. 2 Clock Boards 8. 12 Scalability Switch Boards (SSB) 9. Mounting Brackets 11 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 12. Rear Cabinet Components 1. Left cable management bracket 2. 4 IO Units (IOU) 3. 4 Service Processor Proxies (SPP) 4. 16 CPU Memory Units (CMU) 5. Leveling feet 6. 6 AC input filters (power cord connectors)(3+3) 7. Right cable management bracket 8. 16 PCIe hot-plug carrier for low-profile PCIe cards 9. 4 Express Module SAS for disks (EMS) 10. 32 Hard disk drives 11. System status indicators and ESD grounding connector 12. Mounting brackets 12 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 13. M6-32 Front and Rear Isometric Views 13 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 14. CPU Memory Unit (CMU)  Two M6 CPU sockets with 12 cores each  Each CPU runs at 3.6GHz  Memory is type DDR3L RDIMM at 1066MHz  32 DIMM slots per M6 socket  16 and 32 GB DIMMs supported  Min of 8 DIMMs per CPU M6 #1MEM #1 MEM #0 M6 #0 14 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 also supports CMUs with M5 processors !
  • 15. CMU Numbering CMU0 15 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal CMU15 As seen from rear of chassis MEM1 MEM0 CMP1 CMP0
  • 16. Zakim (ZK) – Systems Memory Interface 16 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Also referred to as a BoB (Buffer-on-Board)  Features and Technology: – 4 ZKs per M5/M6 memory controller – 2 ZKs per T5 memory controller – Supports DDR3 DIMMs – Memory Link to DDR Interface – 2 Memory Link Ports, 2 DDR ports – 8 Write FIFOs – Pass through commands to DIMMs – Core frequency: 1066 MHz
  • 17. Elements of a CMU  The BoB is really the Zakim ASIC  4 DDR3 DIMMs per BoB  4 BoBs per M6 memory controller  32 DIMMs per M6 processor  The DIMMs for each processor are mounted on a memory module  2 memory modules per CMU  Each M6 has 7 ports to allow direct communication with other M6s in the domain configuration unit (DCU) CMU Mem Mod 0 BoB BoB BoB BoB BoB BoB BoB BoB M6-0 Mem Ctlr Mem Ctlr Mem Mod 0 BoB BoB BoB BoB BoB BoB BoB BoB Mem Ctlr Mem Ctlr 17 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-1
  • 18. Internode Coherency Overview  Glueless 1-hop scaling to eight sockets – Glueless means no hub required to connect all 8 CPUs  Aprecise directory tracks all L3s in the system – striped across all processors – stored in on-chip SRAMs – flexible for different socket counts  Higher BW efficiency than snoop-based protocols enables better scaling – 50% more effective bandwidth than comparable snoopy implementation M5/ M6 M5/ M6 18 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M5/ M6 M5/ M6 M5/ M6 M5/ M6 M5/ M6 M5/ M6
  • 19. 8 Socket Local Coherency & Data Interconnect DIMMS M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 DIMMSDIMMS DIMMS POINT-TO-POINT LOCAL INTERCONNECT DIMMS PCIe Gen3 Bandwidth 8 diff pairs per ports At 8Gb/sec ~8GB/sec/direction 19 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal DIMMS DIMMS DIMMS All-to-All Interconnect Where the Node-to-Node Fabric is 12 diff pairs per link in each direction. T5 interconnect bandwidth= 157.5 GB/sec T5-8 interconnect bandwidth= 1260 GB/sec DDR3-1066 MemoryBandwidth T5 is 133 GB/sec T5-8 is 1064 GB/s ~8Gb/sec/lane X 8 lanes = 64 Gb/s X 2 directions X 2 Ports/chip = 256Gb/s/chip = 32 GB/s per chip
  • 20. M5/M6 Global Coherency & Data Interconnect DIMM 32 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 POINT-TO-POINT LOCAL INTERCONNECT Memory Bandwidth Per M5/M6 = 68GB/sec Or 544GB/sec for 8 M5/M6 All-to-All Local Interconnect Where the Node-to-Node Fabric is 7x12 per link in each direction At 16Gb/sec/direction Bisection Bandwidth 16 Links x 24 GB/sec/Link X 2 directions = 768GB/sec Each CPU connects to 6 BXs Wherethe M5/M6-to-BX Fabric is 6 x 4 Links in each direction At 16Gb/sec ~8GB/sec/direction/link Global interconnect To and from the BXs For 8 CPUs is 48 x 4 Links ~768GB/sec/DCU 32 DIMMs per M5/M6 128 DIMMs / DCU ~4TB with 32GB DIMMs 20 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 21. CMU: Root Complexes 21 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Each /SYS/CMU has 2 CMPs (CPUs), and each /SYS/CMU/CMP has 2 IOS (2 root complexes). NAC Format for root complex: /SYS/CMU[0-15]/CMP[0- 1]/IOS[0-1]  PCIe DevPath for a root complex: dev:////pci@<x>/pci@1, where <x> is the Base address shown in this table CMP0 IOS0 CMP0 IOS1 CMP1 IOS0 CMP1 IOS1 CMU0 300 340 380 3c0 CMU1 500 540 580 5c0 CMU2 400 440 480 4c0 CMU3 600 640 680 6c0 CMU4 700 740 780 7c0 CMU5 900 940 980 9c0 CMU6 800 840 880 8c0 CMU7 a00 a40 a80 ac0 CMU8 b00 b40 b80 bc0 CMU9 d00 d40 d80 dc0 CMU10 c00 c40 c80 cc0 CMU11 e00 e40 e80 ec0 CMU12 f00 f40 f80 fc0 CMU13 1100 1140 1180 11c0 CMU14 1000 1040 1080 10c0 CMU15 1200 1240 1280 12c0
  • 22. Memory Population Rules 22 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  There are two channels from each BoB  Each channel talks with 1 or 2 DIMMs  CMUs are either ¼ or ½ populated with DIMMs  Populate each CMU with the same size/rank DIMMs  It is allowable to mix different DIMM sizes on different CMU boards  DIMMs in the same position (0,1,2, or 3) within a MCU, must be homogeneous – must be of the same type (x4 or x8) – same dram device capacity (2Gb or 4Gb) – same number of ranks (SR, DR, or QR)  DIMMs on the same channel must have the same rank capacity
  • 23. M5/M6 Memory Configuration ¼ Populated Best performance for eight DIMMs 0 1 2 3 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 23 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 24. M5/M6 Memory Configuration ½ Populated Best performance for 16 DIMMs 0 1 2 3 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 24 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 25. 0 1 2 3 3 2 1 0 3 2 1 0 0 1 2 3 M5/M6 Memory Configuration Fully Populated Best performance for 32 DIMMs 0 1 2 3 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 25 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 26. Memory Module 26 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 27. 27 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 28. IO Switch Board PCIe Switch 28 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 29. I/O Unit (IOU) 29 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  1x IOU per physical domain  16 PCIe Gen3 Low Profile slots  8 2.5” HDD or SSD disk drives  Four Base I/O Cards (used to access the disk drives)  8 10GbE per IOU  PCIe carrier used to allow hot-plug of PCIe cards
  • 30. M5/M6 IOU LP PCIe Carriers (8) (Hot Plug) Slots 9-16 Base IO Cards (2) EMS3, EMS4 Base IO Cards (2) EMS1, EMS2 LP PCIe Carriers (8) (Hot Plug) Slots 1-8 Two independent IO bays. 30 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Switch Board 1 (Hot Plug) 10 GbE (8) Hot Plug HDD Drives (4) 0-3 Hot Plug HDD Drives (4) 4-7 Switch Board 0 (Hot Plug)
  • 31. Elements of a IOU  2 I/O bays per IOU (0 & 1)  8 PCIe Gen 3 slots per bay  8 SAS 2 disk drives  Disk drives are in 2 groups – Group 0: Disks 0-3 – Group 1: Disks 4-7  I/O Bay 0 is controlled by I/O Switch Board 0 – Disk group 0 is controlled via HD Backplane 0  I/O Bay 1 is controlled by I/O Switch Board 1 – Disk group 1 is controlled via HD Backplane 1 IOU Bay 0 Bay 1 Disks 0-3 Disks 4-7 9 10 11 12 Base 3 13 14 15 16 Base 4 Base 1 1 2 3 4 31 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Base 2 5 6 7 8
  • 32. Base IO Card 32 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  This board is required in order to access the internal disk drives  Each board provides 2 10GbE interfaces
  • 33. M6-32 2 Socket DCU PCIe Block Diagram (Bounded PDom) IOB1 CMU2 CMU3 CMP0 CMP1 CMP0 CMP1 pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15 33 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal EMS3 EMS4 9 10 11 12 13 14 15 16
  • 34. M6-32 4 Socket DCU PCIe Block Diagram 34 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 35. M6-32 4 Socket DCU PCIe Block Diagram (Bounded PDom) IOB1 CMU2 CMU3 CMP0 CMP1 CMP0 CMP1 pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15 35 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal EMS3 EMS4 9 10 11 12 13 14 15 16
  • 36. M6-32 6 Socket DCU PCIe Block Diagram (Bounded PDom) 36 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 37. M6-32 8 Socket DCU PCIe Block Diagram 37 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 38. M6-32 Base IO Block Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 38 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 39. M6-32 Base IO Block Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 39 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 40. M6-32 Base IO Block Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 40 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 41. 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 M6-32 Base IO Block Diagram Disk Drives Are Dual Ported! 41 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 42. 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo SAS-BP A B A B A B A B DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 M6-32 Base IO Block Diagram Disk Drives Are Dual Ported! 42 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 43. EMS to Disk Mapping 43 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 44. Redundant Disk Paths In a Domain  EMS1 and EMS3 provide SAS paths to the four drives  Access all of the drives if one EMS fails  Configure them for redundancy using the Oracle Solaris I/O multipathing feature 44 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 45. PCIe Links 45 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  An IOU contains:  2 IOBs (I/O Switch Boards)  16 slots for LP-carriers (low-profile PCIe card carriers for user-selected add-in I/O)  4 slots for Base IO cards (on-board fixed I/O in a removable form-factor)  The LP-carrier slots are labeled PCIE1-PCIE16, and the Base IO slots are labeled EMS1-EMS4.  The PCIe data path originates from a root complex (in a CMU), connects to a PCIE_SWITCH (in an IOB) which fans out the signal to PCIe or EMS slots (in the IOU) or to KVMS targets on an SPP (Service Processor Proxy) CMUs IOB Slots CMU[0-1] IOU0/IOB0 IOU0/EMS[1-2], IOU0/PCIE[1-8], SPP0 CMU[2-3] IOU0/IOB1 IOU0/EMS[3-4], IOU0/PCIE[9-16] CMU[4-5] IOU1/IOB0 IOU1/EMS[1-2], IOU1/PCIE[1-8], SPP1 CMU[6-7] IOU1/IOB1 IOU1/EMS[3-4], IOU1/PCIE[9-16] CMU[8-9] IOU2/IOB0 IOU2/EMS[1-2], IOU2/PCIE[1-8], SPP2 CMU[10-11] IOU2/IOB1 IOU2/EMS[3-4], IOU2/PCIE[9-16] CMU[12-13] IOU3/IOB0 IOU3/EMS[1-2], IOU3/PCIE[1-8], SPP3 CMU[14-15] IOU3/IOB1 IOU3/EMS[3-4], IOU3/PCIE[9-16]
  • 46. Root Complex 4 CMUs in DCU 46 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 47. Root Complex 2 CMUs in DCU 47 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 48. Disk Device Paths 4 CMUs in DCU 48 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 49. Disk Device Paths 2 CMUs in DCU 49 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 50. T5 and M5/M6 PCIe Carrier Card  Supports standard low-profile PCIe cards Air Flow 50 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PCIe Retimer x16 Connector (x8 electrical)
  • 51. About the F40 Flash Card (Aura2) 51 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Supported on T5-2, T5-4, T5-8, and M5-32/M6-32  Hot-plug supported  Can not be placed in Slot 8 of IOU on M5-32/M6-32 due to thermal issues.
  • 52. Base IO Card SAS Controller PCIe Switch Intel 10GbE RJ45 Ports 52 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 53. Bixby (BX)– M5/M6 Scalability ASIC 53 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Features and Technology: – 16-port Coherency switch router for M5/M6 CPUs > 8 Sockets – 8Address Serialization Pipelines – Supports 4 directory lookups per cycle – Each SSB contains one BX  12 switches in a M6-32 – For unbounded physical domains, each BX has a portion of the L3$ system directory:  6 BXs hold 50% of the system directory for a M6-32 – For bounded physical domains, the BX is not used
  • 54. Scalability Switch Board (SSB) 54 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal BX
  • 55. Coherency Switch Connectivity 55 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  12 Switch cards (SSB), each with a BX ASIC  M6-32 - Even numbered CMUs connect to odd numbered BXs - Odd numbered CMUs connect to even numbered BXs  Remember, each M6 only has 6 scalability links. All 6 go to either even or odd SSBs.  The need to separate BX boards in to even or odd, is due to the limited number of pin outs on the BX chip  Each BX uses a 26-bit tag + 7-bit ECC to track one L3$. To track the 32P system, each BX has 8*32*12*424*2 tags (7.41MB)
  • 56. Additional BX Details 56 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  The 12 SSBs are grouped into two sets of 6. One set for even numbered CMU boards, and the other set for odd numbered CMU boards.  The directory is oversized so that if 1 BX fails in a set, the system can be re- configured to operate with any 5 of the 6 BX's in a set. This allows for redundancy in the system directory.  A DCU can only communicate data with other DCUs via the BX ASICs on the SSB.  Even CMUs can only communicate with even CMUs in other DCUs. Same for odd.  For an even CMU to communicate with an odd CMU in another DCU, it must hop at the end.
  • 57. Scalability Switch Assembly SP0 Connector SP1 Connector Clock0 Connector Clock1 Connector SSB Connectors 57 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 59. DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And Data Switch IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 Even numbered CMU boards connect to odd numbered SSB boards. IO Switch Board 0 IO Switch Board 0 IO Switch 59 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 M6-32 Block Diagram
  • 60. DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And Data Switch BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 IO Switch Board 0 IO Switch Board 0 IO Switch 60 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 Even numbered CMU boards connect to odd numbered SSB boards. M6-32 Block Diagram
  • 61. DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And Data Switch BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 IO Switch Board 0 IO Switch Board 0 IO Switch 61 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 M6-32 Block Diagram
  • 62. 61 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Central Coherency Directory And Data Switches 12 13 14 15 3 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 4 Disks Disks 0-3 4-7 0 1 2 3 0 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 1 Disks Disks 0-3 4-7 4 5 6 7 1 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 2 Disks Disks 0-3 4-7 8 9 10 11 2 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 3 Disks Disks 0-3 4-7 M6-32 Block Diagram
  • 63. 62 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Central Coherency Directory And Data Switches DCU 1 DCU 2 DCU 3 DCU 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch Board 0 Board 1 Board 0 Board 1 Board 0 Board 1 Board 0 Board 1 Disks Disks Disks Disks Disks Disks Disks Disks 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0 1 2 3 Data Path Example CMU0-CPU0 to PCIe slot 61
  • 64. Clock Board 64 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  2 clock boards, in a redundant configuration  Each clock board has dual clock sources with a dynamic multiplexer on each pair.  There are 3 clock types distributed through out the system. System core clock 133MHz, Memory clock 133MHz and PCIe clock 100MHz.  If one of the clock source fails (in a pair), the dynamic multiplexer automatically switches to secondary synthesizer without bring the system down. The domain keeps running.  If more than one clock source in a pair fails, the system will reboot and the alternate clock board will become the active source.  A failed clock board that has been marked inactive, it can be replaced while the system is running as there are 2 clock boards, in a redundant configuration.
  • 65. Service Processor (SP) 65 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Also known as the Integrated Lights Out Management, or ILOM  Two redundant SP modules per M6-32  Provides primary platform configuration and management  Works with the SPP in each DCU to configure and monitor the components in each DCU
  • 66. SP Board Serial Network Pilot3 Processor 66 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 67. SSP Board Pilot3 Module sits here 67 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 68. SP and SPP Connectivity Corporate LAN SP 0 SP 1 SPP 0 SPP 1 SPP 2 SPP 3 Ethernet 16 Port Switch SP Mezzanine 100 Base-T 100 Base-T 1 GbE1 GbE 100 Base-T SerialSerial Ethernet 16 Port Switch SP Mezzanine 68 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 69. rKVMS 69 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  rKVMS is remote keyboard/mouse/storage connectivity  The domain creates a USB keyboard, mouse and storage, as well as a graphics console window that is sent to the SPP  The SP opens a port to initiate a graphical console interface so a user can manage the domain through this remote console  Only the Golden SPP will have the path to the rKVMS
  • 70. Front/Rear Status Panels 70 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal No . LED Description 1 System Locator LED (white) The Locator LED can be turned on to identify the server. When on, it blinks rapidly. There are two methods for turning on the LocatorLED: • Issuing the Oracle ILOM commandset /SYS/LOCATE value=Fast_Blink • Pressing the Locator button on the front of the server. 2 System Service Required LED (amber) Indicates that service is required. • The Oracle ILOM show faulty commandprovides details about any faults that cause this indicator to light. • Under some fault conditions, individual component fault LEDs light in addition to the Service Required LED. 3 System Power OK LED (green) Indicates the followingconditions: • Off – System is not running in its normal state. System power might be off. The SPs might still be running. • Steady on – System is powered on and is running in its normal operating state. No service actions are required. • Fast blink – System is running in standby mode and can be quickly returned to full function. • Slow blink – A normal but transitoryactivity is taking place. Slow blinking might indicate that system diagnostics are running or that the system is booting. 4 Power On/Standby button The recessed Power On/Standby button toggles the system on or off. • Press once to power the server on and return the server to its previously configured operating configuration. • If the key switch is set to service mode, press once to shut the server down in a normal manner. • If the key switch is set to service mode, press and hold for at least 4 seconds to perform an emergency shut down. Caution - After using the Power On/Standby button, you must switch off the circuit breakers on your facilitypower grid in order to power down the server completely. The server will remain on standby power until you switch the circuit breakers off.
  • 71. Front/Rear Status Panels 71 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal No . LED Description 5 SP LED Indicates the following conditions: • Off – Indicates the AC power mighthave been connected to the power supplies. • Steady on, green – SPs are running in their normal operating state. • Blink, green – SPs are initializingthe Oracle ILOM firmware. • Steady on, amber – A SP error has occurred and service is required. 6 Front Component LEDs (amber) The front panel componentrequires service. 7 Rear Component LEDs (amber) The rear panel component requires service. 8 System key switch Using a supplied key, you can switch between normal and service modes. Normal operation mode: • You can power on the server by pressing the Power On/Standby button. • You cannot shut the server down using the Power On/Standbybutton. Service mode: • Server is placed in service mode • You can shut down the server using the Power On/Standby button. • The key cannot be removed from the key switch while in the Service position. 9 Antistatic wrist strap connector The server has two 10-mm connectors where you can attach an antistaticwrist strap prior to installing or servicing the server.
  • 72. M5 / M6 Differences, Configuration rules and PCIe Slot Guidelines 72 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 73. M5 / M6 Configuration and Replacement Rules 73 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Mixing of M5 and M6 CMUs within a DCU is not allowed.  All CMUs in a DCU must be either M5 or M6. However DCUs with M5 CMUs can co-exist with M6 CMUs in a domain in different DCUs  Mixed (M5 and M6) clock boards in M5 chassis are allowed  Mixed (M5 and M6) clock boards in a M6 chassis will not inhibit boot but will generate an alert to the user.  Anytime the system detects mixed (M5 and M6) clock boards, and both boards are usable, it will use the M6 clock board as the active Note : The M6 clock board in mixed configurations being favored and detection of the mix in an M6 system as well as generating an alert, these changes will be post RR. FW will not stop the user from mixing clock boards.
  • 74. M5 / M6 Configuration Guidelines 74 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Fill Dark Green Slots First Fill Light Green Slots Second Fill Yellow Slots Last Factory configuration guidelines PCIe IO Slot Priority HDD Slot HDD/SSD EMS Slot BASE IO CARD CMU/IOB IO Slot HALFDCU 0 X 1 X CMU0 – IOB0 1* 2 1 X 3 4* 2 2 5 6* 3 7* 8 4 X 3 X CMU3 – IOB1 9 10* 5 X 11* 12 6 4 13* 14 7 15 16* FULLDCU 0 X 1 X CMU0/1 – IOB0 1 2 1 X 3 4 2 X 2 X 5 6 3 X 7 8 4 X 3 X CMU2/3 – IOB1 9 10 5 X 11 12 6 X 4 X 13 14 7 X 15 16
  • 75.  A PCIE card marked with an asterisk ( * ) is removed or added, the OBP device path may change, dependent on the state of the ILOM property /HOSTx/ioreconfigure and CMUs present.  We recommend prioritizing the installation of PCIE cards ( grouped functionally ) based on the color scheme in the table : 1)Infiniband and Storage PCIE cards starting with dark green first, followed by light green slots without the asterisks. 2)10GigE Network PCIE cards on available dark and light green marked slots with the asterisks. 3) 10/100/1GigE Network cards on the light green and yellow marked with the asterisks.  Due to thermal requirements F40/Aura2 cards must not use PCIe slot 8. 75 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M5 / M6 Configuration Guidelines
  • 76. The SPARC M6 Processor 76 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 77. 25+ Years of SPARC Processors 1987 1992 1996 2000 20051988 1995 2002 Sunrise: 1st SPARC Processor SUNRAY UltraSPARC I SuperSPARC I UltraSPARC II UltraSPARC III UltraSPARC IIIi UltraSPARC IV+ UltraSPARC T1 SPARC T4 SPARC M5 2007 2010 UltraSPARC T3 UltraSPARC T2 SUNRAY 2011 2013 SPARC T5 77 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 78. 1987 1988 1992 1995 1996 2000 2002 2005 2007 2010 2011 2013 Sunrise: 1st SPARC Processor UltraSPARC IV+ UltraSPARC II SPARC T5 SuperSPARC I UltraSPARC IIIi UltraSPARC T2 SPARC T4 SUNRAY SUNRAY UltraSPARC I UltraSPARC III UltraSPARC T1 UltraSPARC T3 SPARC M5 SPARC S3 Core Processor Family 78 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 79. M6 Processor  12 S3 cores @ 3.6Ghz – Single or multi-threaded operation per core – 6-48 total strands  SPARC S3 Core – 1-8 Strand Dynamically Threaded Pipeline – Vertically Threaded 2-decode, Out-of-Order (OoO) 2-Issue – 16KB 4-way L1 I$ – 16KB 4-way L1 D$ – 128KB 8-way L2$ – ISA-based Crypto-acceleration  Shared 48MB L3$  Memory Controllers – 2x Dual Channel DDR3L MCU's – Cascadable Buffer-on-Board (BoB)  Integrated I/O – 2 x8 Lane PCIe Gen3 @ 8GT/s – 32GB/s IO Bandwidth per processor  System Scalability – Up to 8-socket glueless – 32-socket Systems via scalability links M6 79 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 80. SPARC M6: Processor Overview  12 SPARC S3 cores, 96 threads  48MB shared L3 cache  4 DDR3 schedulers, maximum of 1TB of memory per socket  2 PCIe 3.0 x8 lanes  Up to 8 sockets glue-less scaling  Up to 32 sockets glued scaling  4.1 Tbps total link bandwidth  4.27 billion transistors Coherence&Scalability MIO Crossbar L3 L3 L3 L3 SPARC SPARC L3 Core Core Ctl SPARC SPARC L3 Core Core Ctl L3 SPARC SPARC Ctl Core Core SerDes SerDes SerDes SerDesMCU SPARC Core SPARC Core L3 SPARC SPARC Ctl Core Core SerDes SerDesMCU SPARC Core SPARC Core PCIe PCIe 80 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 81. SPARC M6 CPU Block Diagram 2x8PCIe3.0@8GBps 16GBpseachdirection Crypto 8 threads per Core IO Subsystem Memory Control Coherency2x4Switch Memory Control Coherency Links 12.8 Gbps per lane - 12 lanes per link (1075 Gbps) 12 x 5 Crossbar (~620GBps bandwidth) C0 C1 L3$ B0L3$ B0 1MB,16-way 128 KB L2$ SLC 16 KB L1I$ 16 KB L1D$ FGU SPARC S3 Core BoB BoB BoBBoB Coherence Unit Coherence Unit SLC SLC SLC SLC SLC Link 0 Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 C2 C3 C4 C5 C6 C7 C8 DDR3 – 1066 MHz DDR3 – 1066 MHz BoB BoB BoBBoB DDR3 – 1066 MHz DDR3 – 1066 MHz Scalability Links 12 Gbps per lane – 4 lanes per link (144 Gbps) C9 81 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal C10 C11 L3$ B0 12MB,16-way L3$ B2 12MB,16-way L3$ B1 12MB,16-way L3$ B3 12MB,16-way
  • 82. SPARC M6: Processor Key Features 82 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  SPARC M6 based on SPARC S3 core: Same core as used in T4, T5 and M5 – 12 S3 cores, dual PCI Express 3.0 root complexes – Up to 32 M6 processors per system, 12 cores x 8 threads → 96 per M6, or up to 3072 threads – Clock frequency is 3.6 GHz – Each SPARC S3 core contains:  2 Integer pipelines  1 FGU pipeline (consisting of 3 physical sub-pipelines): – FPX pipeline – FGX pipeline – FPD pipeline  1 Load-Store (Memory) pipeline
  • 83. S3 Core Recap 83 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Out-of-order, dual-issue  High frequency achieved with 3.6GHz  16 stage integer pipeline  Dynamically threaded, one to eight strands  Accelerates 16 encryption algorithms and random number generation
  • 84. SPARC S3 Core  Dual-issue, out-of-order  Integrated encryption acceleration instructions  Enhanced instruction set to accelerate Oracle SW stack  Dynamically threaded pipeline – 1 to 8 threads 84 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 85. SPARC Core Roadmap T3 Servers S2 Core M-Series SPARC64 VII/VII+ Core M5 Servers T5 Servers T4 Servers S3 Core M6 Servers S4 Core 85 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 86. S3 Core Overview 86 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  8-way threaded, dual-issue, OoO execution, in order commit  Dynamically threaded with hardware-optimized resource sharing  Support for Critical Threads  Deep pipeline for high frequency operation (3 GHz in 40 nm)  Balanced single-thread and multi-thread performance  Enhanced instruction set to accelerate Oracle SW stack - PAUSE, fused compare-branch  Integrated user-level cryptographic acceleration - DES/3DES,AES, Kasumi, Camellia, MD5, SHA-1, SHA-224/256/384/512, RSA, DSA, CRC32c  Foundation core for future technology / product nodes
  • 87. Oracle SPARC S3 Core 87 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Enables T4, T5, M5, M6 • Design Goals - Develop a common replacement core for T-series processors - Significantly improve the single strand performance of the T3 processor - Significantly improve the throughput performance of the M3 processor - Improve the RAS and power management capabilities - Maintain backward ISA, Solaris and OVM for SPARC compatibility Core S1: M2, M3 Core S2: T2, T2+, T3 Core S3: T4, T5, M5, M6 Frequency 2.4 – 3.0 GHz 1.4 – 1.65 GHz T4: 2.85 – 3.0 GHz T5, M5, M6: 3.6 GHz L1 InstructionCache 64KB 16KB 16KB L1 Data Cache 64KB 8KB 16KB L2 Cache - - 128KB # of Pipelines 1 2 1 # of Threads per Pipeline 2 4 1-8 Dynamic Instructionsper Thread 4 per cycle 1 per cycle 2 per cycle Out of Order Issue Yes No Yes (36 instr window) CryptographyAcceleration None SPU ISA Based OVM for SPARC Compatible No Yes Yes SPARC V9 ISA Compatible Yes Yes Yes
  • 88. SPARC Security Securing All the Stack Layers  Secure database queries 43% faster; Low overhead for encryption  WebLogic SSL & WS-Security 2x – 3x faster  ZFS Filesystem Crypto 3x faster than x86  Java crypto 2x – 3x faster than x86  OpenSSL4.3x faster for single-thread security versus Power7 Results: 88 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 89. SPARC T4/T5/M5/M6 Leads in On-Chip Encryption Acceleration 89 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal On-Chip Accelerators Common Use Examples Mainly Used SPARC T4/T5/M5/M6 IBM Power7+ Only 3 cryptos for 8 cores Intel Westmere /SandyBridge Asymmetric /Public Key Encryption Web Browsers, phone calls, VPN, Secure FTP To protect data or files in transit RSA, DH, DSA, ECC RSA, ECC RSA, ECC Symmetric Key / Bulk Encryption Databases, credit card and social security numbers, private info To protect data or files at rest or stored: disks, backup tapes, etc. AES, DES, 3DES, Camellia, Kasumi AES (Modes: ECB, CBC, CTR, CCM, CCA, GCM, GCA, GMAC, CM, F8, XBC-MAC-96; Key lengths: 128b,192b, 256b) (Lacks AES-CFB used by Oracle database!!) AES Message Digest / Hash Functions Data lookup & authentication, digital signatures, message authentication codes (MAC) To compress/create a short summary from a data chunk and not expose it; To detect duplicate or corrupt data CRC32c, MD5, SHA-1, SHA-2 (SHA-224, SHA- 256, SHA-384, SHA-512) MD5, SHA-1, Partial SHA-2 (SHA-256, SHA- 512), HMAC supported for SHA (?) none Random Number Generation Ubiquitous To generate keys Supported Supported none till Ivy Bridge
  • 90. Next Generation l SPARC Processors 90 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal T5 M5 M6 Processor 16 S3 cores, 128 threads @ 3.6GHz 6 S3 cores, 48 threads @ 3.6GHz 12 S3 cores, 96 threads @ 3.6GHz S3 Core Features 1-8 Thread, Dynamically threaded pipeline ISA-based Crypto-acceleration Shared L3$ 8MB 48MB Integrated I/O 2x8 Lane PCIe 3.0 @ 8GT/s System Scalability 7 Coherence Ports for scalability to 8S Additional 6 Scalability Ports for scaling to 32S Power Management Dynamic Voltage Frequency Scaling Enterprise Features OVM Server for SPARC (LDoms), Solaris Zones Scalability Port Redundancy, Dynamic Domains, Hot-Plug SP
  • 91. S3 Core: Dynamic Threading Page <#>91 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  While software can activate up to 8 strands on each core at a time, hardware dynamically and seamlessly allocates core resources such as instruction, data, and L2 caches and TLBs, and out-of-order execution resources such as the 128-entry re-order buffer in the core among the active strands.  Software activates strands by sending an interrupt to a HALTed strand. Software deactivates strands by executing a HALT instruction on each strand it wants to deactivate. No strand has special hardware characteristics; all strands have identical hardware capabilities.
  • 92. S3 Core: Dynamic Threading Page <#>92 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate.  If software effectively halts all strands except one on a core via Critical Thread Optimization, the core devotes all of its resources to the sole running strand. Thus, that strand will run as quickly as possible
  • 93. S3 Core: Critical Thread Optimization 93 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  The S3 core, starting with Oracle Solaris 10 8/11, can optimize performance by assigning one software thread exclusive access to all of a core's hardware resources. That software thread is considered to be a "critical thread."  Solaris automatically detects opportunities to perform this assignment: when one software thread has high CPU utilization and there are more cores than runnable threads. We recommend that users allow Solaris to automatically perform Critical Thread assignment.  A privileged user can tell Solaris that a particular software thread should be a critical thread, via the nice(1) command. Solaris will then assign that thread to a core, even if there are more runnable threads than cores.
  • 94. S3 Core: Critical Thread Optimization 94 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Solaris Critical Threads optimization for S3 core, tries to provide exclusive access of certain hardware resources to certain application threads  Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate  There is no new API for declaring threads as ‘critical’; that would require significant changes to source code  Rather, to invoke Critical Thread Optimization, use the following CLI or system calls to flag a thread as critical by raising its priority to 60: – priocntl(1) – priocntl(2) – priocntlset(2)
  • 95. S3 Core: Critical Thread Optimization 95 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Starting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised to priority 60; the thread can be in any scheduling class  In Oracle Solaris 11, to be considered critical by the scheduler, a thread must be: – in the FX (Fixed Priority) or RT (Real-Time) scheduling classes – be raised to priority 60 by one of the previously mentioned mechanisms  In either of the above instances, this one thread will run as quickly as possible as it has exclusive access to all core resources
  • 96. Critical Threads for key applications 96 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Applicability Opportunity Current Status Database Logwriter, LMS Up to 30% improvement in efficiency LMS is already CT ready. LGWR planned for 12c JAVA (JVM) Compiler threads, GC and priority mapping support Up to 2x improvement for app startup, Smooth GC Support for JVM and JAVA apps to be CT aware is integrated in JDK7U4 Coherence Packet writer, service thread Up to 20% improvement in throughput Integrated in Coherence version 3.7.1 Patch 1 Solaris S11U1 / S10U11 Improve CT perf to be within 10% of best case (hand optimized) Optimizations for decayed PG util and stealing being integrated in S11U1
  • 97. Reliability, Availability, Serviceability (RAS) 97 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 98. Definition of Terms 98 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Hot-plug: • refers to the fact that a component can be plugged and unplugged without powering down the platform. It applies to both hot swap and hot service. • Hot service: • refers to the ability to perform hot-plug operations, with the additional necessity of some operator actions (invocation of a CLI or actuating a hot service button on the component to be removed). • The system will notify the user when it is safe to remove the component. • Typical examples would be PCIe Express modules. • Hot swap: • refers to an operation where a component is unplugged and plugged in with no interaction with the ILOM or domain required. • Typical examples here are a single RAID disk or a power supplies.
  • 99. SPARC M5/M6/T5 System RAS Overview 99 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Designed to minimize part count and operating temperature to enhance reliability • End-to-end data protection detecting and correcting errors throughout server – ECC everywhere • Processor and Memory protection • CPU core and thread off-lining • Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failover • Major components redundant & hot-pluggable • Fan, Power Supply, and internal disks • RAID capability for internal disks • Fault Management Architecture (FMA) support on ILOM
  • 100. End-to-End RAS  Built RAS from the inside out  Start with the processor, then memory, system and IO, virtualization layer, and the OS  Add Oracle Solaris Cluster software for additional service availability  Fault Management Architecture (FMA) binds all the layers together 100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 101. T5/M5/M6 Systems RAS • Diagnosis engine on SP • Auto reconfigure on failure • Soft Error Rate Discrimination(SERD) • Bad page retirement • OS and SP watchdogs • FMA Component hot-upgradeable T5/M5 Processor • L1$ Tag, Status $ Data • Parity protection • Retry on error • L2$/L3$ Data • SEC/DED protection • Cache-line Sparing • L2$/L3$ Tags • SEC/DED protection • Inline Correction • Cache-line Sparing • L2$/L3$ Status & Directory • SEC/DED protection • Inline Correction • Architectural RegistersL2 Cache • SEC/DED protection • Precise Trap and Hypervisor Correction and Retry Power and Cooling • Advanced Power Management • Redundant hot-swap fans • Redundant hot-swap AC/DC • Dual grid power System I/O S11 FMA Hypervisor System • Redundant SPs with automatic failover • Redundant clock boards • Dual synthesizers per clock board • Diagnosis to the FRU level on first fault •PCI-Express end-to-end CRC • PCI Express link retry • Hot-plug low profile PCI Express cards • Redundant, hot-plug boot disks •Alternate connections between M5 and IO controllers Memory • SDRAM Soft Errors • ECC Protection and Correction • Extended ECC Protection • 4-bit Correction • Pin Steering • Channel Interconnect • CRC protection/Message Retry • Lane Sparing •Enables software partitioning (LDoms) virtualization and failure containment •Processor support for error clearing, correction and collection Central Directory and Switch •SEC/DED protection with in line correction • Physical domain isolation •CRC protected System Interconnect with message retry and lane sparing • Deconfigurable directory chips, no loss of functionality, minimized bandwidth loss • Redundant ScalabilitySwitch Boards 100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Unique to M5/M6
  • 102. SPARC M6 RAS: End-to-End Protection  Internal Logic: parity and ECC – Architectural Registers – Cache structures – Internal networks  Links: CRC retry CRC ECC with line retire Data-ECC,Address-parity Other (Parity, Retry etc) DFT, Debug etc. 102 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 103. SPARC M6 RAS: General Error Handling Flow Hardware Detection and Clean-up Gather Signature De-configure Resources Hypervisor Assist with clean-up Collect Hardware Data Generate Report for SP Manage De-configuration Service Processor Analyze Hypervisor Report Update Error History Initiate Service Call Initiate De-configuration Solaris De-configure User-Visible Resource Offline SerDes Lanes Retire Cache Lines Activate DIMM Spare Column Retire Threads Retire Cores Retire Pages 103 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 104. SPARC M6 RAS: Memory Errors  ECC optimized for device failures  Inline correction and auto write-back  “Scrubber” prevents accumulation of upsets  “E-retry” characterizes soft vs persistent  Cell or word-line fail: Solaris retires page(s)  Bit-line or pin fail: firmware deploys DIMM spare column  Device fail: inline correction M6 Scheduler Normal RD/WR Scrubber RD Eretry RD/WR DIMM ECC Gen ECC Chk/Corr Err 104 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 105. HW Aspects of RAS – New and Interesting Features (1 of 3) 105 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Isolation of memory UE’s to a single DIMM - Cachelines (the unit of memory access) are fully contained within a single DIMM - Most (all?) past Oracle/Sun systems had cachelines that straddled two or four DIMMs - This means UE’s can be diagnosed to a single DIMM rather than generating a list of possibilities  Memory pin sparing - ADIMM has 72 data pins (DQ’s) and we use a burst length of 8 when accessing DIMMs => 572 total bits of data - Only 564 bits used for data + ECC + control - Remaining 8 unused bits sourced from a single DQ - MCU can dynamically remap the layout to change which DQ is unused
  • 106. HW Aspects of RAS – New and Interesting Features (2 of 3) 106 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Deconfigurability of Scalability Switch Boards - The 12 SSBs logically function as two groups of 6 - Agroup is capable of functioning with just 5 - Requires a system restart because it completely changes the mapping of SSBs to physical addresses that are tracked - In the case of a fatal error internal to an SSB, it will automatically be deconfigured and the system restarted - Replacement can happen while system is running but a manual restart will be required to re-integrate - In the case of non-fatal errors leading to service required, can be serviced live but requires 2 manual restarts - Note: single DCU physical domains can be configured to not use the SSBs
  • 107. HW Aspects of RAS – New and Interesting Features (3 of 3) 107 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Dual clock synthesizers per clock board  Reconfigurability of PCI fabric - All PCI slots are accessible by 2 different CMUs - Primary or alternate path programmed at the time the physical domain is started - Allows for full IO accessibility in the case of a faulty or missing CMU - Necessary because the PCI root complexes are on the M5/M6 chip - Note: complications arise when multiple logical domains are configured, so reconfiguration is suppressed once the system is “virtualized”
  • 108. SW Aspects of RAS 108 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Fault diagnosis done on SP => covered under FMA - Failures typically isolated to a single FRU, with the exception of interconnect failures that span FRU boundaries - Automatic reconfiguration of resources  Configuring for Availability => covered separately - Primarily addresses IO configuration using Logical Domains, and how to account for multipathing - LDoms also provide a level of isolation for some classes of uncorrectable errors, since the impact will only be to that LDom
  • 109. Fault Management 109 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Knowledge Articles in MOS  ILOM fdd Diagnosis  Faults and Alerts  NoALOM Compatibility  ILOM FMACaptive Shell  Sideband Service Processor Network Connection  New ILOM Fault Notification (SNMP Trap)  ASR Support
  • 110. FMA – restricted shell in SPSH 110 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  To get FMA details in SPSH – start -script /SP/faultmgmt/shell – Returns a faultmgmtsp> prompt  Available build-in commands: – echo - Display information to user.  Typical use: echo $? – help – Exit – exits restircted shell  External commands: – fmadm - Administers the fault management service – fmdump - Displays contents of the fault and ereport/error logs – fmstat - Displays statistics on fault management operations – etcd - ereport injector
  • 111. ASR Support 111 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  SPARC M5/M6 servers will be supported byASR (Automatic Service Request)  Continues use of sunHwTrapFaultDiagnosed SNMP notification  Telemetry for ILOM fdd diagnosis  Supports platform and FRU identity  Supports multi-suspect list
  • 112. M6-32 FRU Serviceability * Reboot required to activate new board. M6-32 112 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 113. ILOM 113 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 114. Service Processor Software (ILOM) on M6 Systems 114 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  ILOM looks/behaves just like ILOM on other platforms  Simple (user-visible) set of extensions to support Physical Domains  Extensions to support Service Processor Proxies and redundant Service Processors - Minimal impact on user experience
  • 115. Oracle ILOM Key Functions 115 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Management Interfaces • CLI, BUI, IPMI, SNMP • Firmware Updates • Remote Host Management • Inventory and Component Management • System Monitoring and Alert/Fault Management • UserAccount Management • Power Consumption Management
  • 116. ILOM on M6 – Enterprise Features 116 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  ILOM extensions to support Enterprise Features – Some of these are conceptually leveraged from XSCF SW  SP Tracing Facility – Very useful for tracing inter-process activity – Reliable performance (elapsed time) measurements – Allows for tracing interactions between SPs and SPPs etc – Enterprise systems have lower volume, more complex configurations and high RAS expectations  We cannot expect customers to reproduce bugs  Need to collect as much debug info as possible on live system as it occurs on customer site – Coredump compression and snapshot collection – Unified snapshot from all SPs and SPPs
  • 117. ILOM on M6 – Enterprise Features 117 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Confstore distributed config database – Exploitable by CMM/Blades  System Identity is maintained across FRU / SP replacement – aka TLI (Top-Level-Identifier)  Flash Images are signed to avoid compromised images  “hardened” edits of config files – Transaction oriented – before or after, no intermediate results  Tunables framework for MAX_USERS etc  Sensor Broadcast (SSBCAST) enhanced to support
  • 118. ILOM – New Enterprise Features 118 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Creation of Dynamic Domains (i.e. PDoms)  Native FMA support  Redundant SPs  SPP support
  • 119. ILOM User Roles 119 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  ILOM supports a maximum of 61 user accounts (configured under /SP/users) on the M6-32 SP  M6-32 platforms introduce the concept of separately managed Physical Domains (aka PDom). Each PDom can be separately controlled.  Each user account can be optionally configured with specific roles for an individual PDom.  The user roles are additive. For example, a user account is permitted to do reset operations on a PDom if: 1. the user has 'r' role for the platform (in /SP/users/username/role) _OR_ 2. the user has 'r' role for the specific PDom (in /SP/users/username/host_roles/hostX_role)
  • 120. Roles and Capabilities 120 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Role Capabilities none user has no user roles, and prevents the user’s roles from being looked up in LDAP user (u) • Create, delete, disable, and enable user accounts. • Change a user’s password and password properties. • Change a user’s roles. • View all platform states. administrator(a) • Perform all Service Processor configurationother than the useradm and auditadm tasks. • Assign and unassign hardware to or from domains. • Perform Service Processor power operations. • Perform Service Processor failover operations on systems with more than one Service Processor. • Perform all operations on domainhardware. • View all platform and physical domain states. console (c) • Access the Oracle ILOM Remote Console and the SP console. • View and change the state of the Oracle ILOM console configurationvariables. reset (r) • View all states of the hardware assigned to the domain(s)on which this role is held. • View all states of the domain(s)on which this role is held. • Operate the system, which includes performingpower operations, resetting the system, hot-pluggingdevices, enabling and disablingcomponents,and fault managementfor the specified domain. read-only operator (o) • View all platform states. • Change the password and the Session Time-Out setting for their own user account. Whenthis role is defined at the domainlevel, users can: • View all states of the hardware assigned to the domain(s)on which this role is held. • View all states of the domain(s)on which this role is held. field engineering (s) perform all operations reserved for field engineers.
  • 121. Required User Roles 121 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Aspecific role is required for certain tasks: Task Required User Role Assign and unassign DCUs to a PDomain admin (a) Manage ILOM passwords user (u) Connect to a PDomain console (c) Perform power operations (start, stop, and reset) reset (r) Configure user accounts user (u) Configure host groups user (u)
  • 122. New to ILOM 3.1 122 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Simplified Data Model (SDM)  Three-Level Model – Level One: System Summary Info – Level Two: Subsystem Summary – Level Three: Logical Topology  Subsystems: Cooling, Power, CPUs, Memory,  Storage, and Networking  Also Blades, DCUs, CMUs, CPU Modules, I/O Modules on some platforms  Open Problems unifies fault management with SDM
  • 123. ILOM 3.2: New Linux distro and compilers 123 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  M6 will release with ILOM 3.2  Linux version 2.6.27.43, “SQUEEZE” – was 2.6.16.4, “SARGE”  gcc version 4.4.5 – was 3.3.6  Why: – Old distro no longer supported – Security and bug fixes – Posix threads instead of Linux threads
  • 124. WEB CLI LUMAIN SDM BACKEND Platform xml LIBHDL Hw service CAPI SSM API SDM Architecture 124 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 125. SDM CLI 125 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal CLI is reorganized.  /System target(tree) is introduced  Different components of the system are grouped and organized into sub targets of /System  At every level of the tree, the critical properties are shown along with any sub targets  The applicable cli commands are supported at different levels of the /System tree.  All targets and properties under /System are case insensitive
  • 126. SDM CLI (cont.) 126 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Health and health details are two of the common properties shown at every level to indicate the over all health of that sub tree.  Open_Problems target shows the detailed descriptions of the faults in the system  /SYS and /Storage targets are made legacy  Continue to exist but hidden by default  The legacy targets can be made visible by enabling /SP/cli/legacy_targets property.
  • 127. SDM CLI (Summary level targets) 127 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System -d targets /System Targets: Open_Problems (0) Processors Memory Power Cooling Storage Networking PCI_Devices Firmware BIOS
  • 128. SDM CLI (Summary level properties) 128 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System -d properties health = OK health_details = - open_problems_count = 0 type = Rack Mount model = Exadata X2-3 part_number = 8124854 serial_number = 2229CNL124 component_model = SUN FIRE X4170 M3 component_part_number = 7013743 component_serial_number = 1118CNL013 system_identifier = sysidentifier system_fw_version = 3.1.0.10 primary_operating_system = Not Available host_primary_mac_address = 00:21:28:d5:c0:b2 ilom_address = 10.153.55.201 ilom_mac_address = 00:21:28:D5:C0:B6 locator_indicator = Off power_state = Off actual_power_consumption = 5 watts action = (none)
  • 129. SDM CLI (Processors Subsystem) 129 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System/Processors/ Targets: CPUs Properties: health = OK health_details = - architecture = x86 64-bit summary_description = Two Intel Xeon Processor E5 Series installed_cpus = 2 max_cpus = 2 -> show /System/Processors/CPUs/ Targets: CPU_0 CPU_1
  • 130. ILOM on M6 – SP-Proxy Design 130 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  M6-32 is a large system with 32 CPU chips, 1024 DIMMs, 2160+ sensors – Too large for one SP to manage – I2C, JTAG traces etc – For comparison, T5-8 has 8 CPU chips, 128 DIMMs, 400+ sensors  Divide-and-conquer: each SPP is responsible for one DCU (8 CPUs/256 DIMMs) – Re-use ILOM functionality on a per-DCU basis  POD, Poller, rKVMS, USB-over-Ethernet – Offloads work from Main-SP to SPPs.  Main-SP (SP0/SP1) “aggregates” all SPs/SPPs and presents a unified system view.  SP administration can be done on the Main-SP or drilldown to the SPP for PDom-specific management
  • 131. ILOM on M5 – SP-Proxy Design 131 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  One SPP-per-PDom is chosen as “Golden-SPP” – Provides RKVMS service – Serves as the “Management-SP” for the Pdom  Externally accessible target for IPMI, SNMP, spsh operations – Some functions from Main-SP are forwarded to Golden-SPP for service  e.g. /HOSTx/SP/services/kvms, /HOSTx/SP/network/interconnect  Configuration data from Golden-SPPs is backed to Main-SP – Confstore is a distributed extension of ILOM's memstore database – Maintains unique config data subset for every HOSTx – As SPPs are moved between HOSTx, each SPP will obtain the correct config data subset for its current HOSTx  Audit logs, faults, etc forwarded from SPPs to Main-SP (and replicated)
  • 132. ILOM on M6 – Redundant SP Support 132 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Redundant pair of Main-SPs  Active-SP will be the management SP for the platform. – Communicate all configuration data to the Standby – Standby ready to take over if the Active fails  Heartbeat is maintained between SPs and SPPs – Active-SP → Standby-SP (SP0 or SP1) – Active-SP → SP-Proxies (SPP0, .. SPP3)  Implemented checkpointing support – Use of checkpoint to recover from inconsistent state due to poweron/DR operations interrupted by SP Failover/SPP failures
  • 133. SP 133 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 134. Service Processors (SP) SP0 SP1 134 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 135. M6 M6 Service Processor Proxy (SPP)  The SPP is a local SP dedicated to the DCU  Each DCU has its own SPP  It purpose is to – Initialize, manage the CPU, Memory controller, DIMMs with the DCU – Environmental monitoring such as temp sensors and adjust 8 Fans  In a multi-DCU domain scenario, the SPP with the lowest number available in the domain will automatically become the Golden SPP that will manage the domain for all multiple DCUs  In case the Golden SPP fails, the next higher numbers SPP takes over. DCU SP To other SPP Units Disks 0-3 Disks 4-7 M6 M6 M6 M6 M6 M6 135 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPP
  • 136. SPP Function 136 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  SPPs are commanded by the main System SPs  The communication is ethernet between the SP and SPPs  Ethernet communication in SP used to manage the SPPs as well as rKVMS
  • 137. SPP Features 137 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Xilinx FPGA to provide the SSI boot interface to the CMU  RKVMS (Remote Keyboard, Mouse and storage): - This feature allow a domain console to be shown at a remote display. It is able to send keyboard, mouse, and storage commands through Ethernet Packets. - The graphical pixel scraper as contained in a Matrox PCIE VGA inside the Pilot3 Controller. - Avirtual USB hub (not a USB controller) that can transmit and receive USB connections from an external USB UPD727200 controller. The Hub resides in the Pilot3 controller. - The Pilot3 converts these devices in to Ethernet packets and provides remote console functionality for the domain. - Logical Domain Control communication will use Ethernet over USB to share the global SRAM needed for CMU’s to communicate between each other.  100 Mhz PCIE reference clock distribution to CMU’s and IO boards.  Ethernet interfaces to the SP board via the capacitive coupled scheme.
  • 138. Physical Domains 138 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 139. M6 Systems Terminology 139 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  M6 Systems consist of CPU, Memory and IO - CPU and Memory are on CPU/Memory boards (CMUs)  Each CMU contains 64 DIMM slots  Domain Configuration Units (DCUs) are groups of 4 CMUs, and one IOU, connected by the local coherency interconnect  ADynamic Domain consists of one or more DCUs  ADynamic Domain is also referred to as a Physical Domain (or PDom)
  • 140. Domain Configuration Unit (DCU)  Since the M6-32 forms groups of 8-way bounded domains, 4 CMU boards and a single IOU form a single configuration unit, or DCU.  All communication to other components in the DCU do not need to pass through a BX ASIC.  ABX is only needed when data from one CMU needs to go to a CMU in a different DCU.  Aphysical domain (PDom) can be made from one or more DCUs.  Each DCU is controlled and managed by its own SP Proxy  The admin manages the DCU via the main SP M6 1 M6 2 DCU SP To other SPP Units Disks 0-3 Disks 4-7 M6 7 M6 8 M6 5 M6 6 M6 3 M6 4 140 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPP
  • 141. Similar DCUs On Other Systems 141 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal E25K M9000-32 M5-32/M6-32 1 Uniboard + 1 I/O Board 1) 1 CPU + 8 DIMMs + 2 PCIe slots 2) 1 CMU + 1 IOU 4 CMU + 1 IOU
  • 142. 4 DCUs  ADCU has: – 1 IOU – 1 SPP – Up to 4 CMUs  DCUs communicate using the SSB DCU0 142 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal DCU1 DCU2 DCU3
  • 143. DCU Population Rules 143 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Must install minimum of 4 CMU boards – minimum 2 CMU per DCU  Additional CMU boards must be installed in pairs – DCU slots 0 & 3 first, then slots 1 & 2 – Need both even & odd in DCU for proper SSB communication  HDD or SSD requirement per PDom – Minimum of 2 HDD – Minimum of 2 Base IO cards for the initial pair of HDD – Additional Base IO cards depend on placement of subsequent pairs of HDD
  • 144. DCU Memory Population Rules 144 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Memory on CMUs are either ¼, ½, or fully populated – All the memory on the CMU must be the same density and rank – All the CMU boards in a DCU must be populated exactly the same way, and with the same exact DIMMs – CMU boards in different DCUs can be populated differently
  • 145. M5/M6 Population Rules 145 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Both processors on a CMU must be the same (both M5 or both M6)  All CMUs in a DCU must be the same (all M5 or all M6 based CMUs)  APDom can be made from a combination of M5 and M6 based DCUs – If DCU0 has four M5 cpus and DCU1 has eight M6 cpus, they can both be in the same PDom  If a DCU has empty CMU slots, they can only be filled with CMUs exactly the same as the existing CMUs in that DCU
  • 146. Mixing M5 and M6 CPUs Rules 146 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  All the CMUs in a DCU must be identical  No mixing of M5 and M6 CMUs in a DCU  APDom can be made from multiple DCUs, – some DCUs are M5 populated – other DCUs are M6 populated
  • 147. Dynamic Domains 147 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  There can be up to four physical domains (PDoms or PDomain) in the M6-32  Each PDom operates like an independent server that has full hardware isolation from other PDoms on the server  Domain configurable units (DCUs) are the building blocks of PDoms  Each PDom is represented as /Servers/PDomains/PDomain_x/HOST in Oracle ILOM where x ranges from 0 to one less than the maximum number of possible PDoms in the system (PDomain_0, PDomain_1, PDomain_2, PDomain_3).
  • 148. Expandable Dynamic Domains 148 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  A PDom has an attribute of expandability (expandable flag)  By default, a Physical Domain consisting of a single DCU will run as a regular Physical Domain (i.e. expandable=true)  A non-expandable PDom is called a Bounded PDom (or Bounded Physical Domain) (i.e. expandable=false)  A PDom can be set to either a Bounded Physical Domain, or a Regular Physical Domain  However, a Bounded Physical Domain is not affected by the loss of an SSB
  • 149. Bounded Dynamic Domains 149 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  ABounded Physical Domain (or Bounded PDom) – does not use the SSB for L3$ directory storage – for CMU to CMU access within the DCU  Lower latency  No impacted by loss of SSB  Can never grow beyond a DCU  Changing a domain to Bounded, or from Bounded to regular, requires the domain be halted before changing the “expandable” variable.
  • 150. Tradeoffs 150 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Pros Cons Dynamic Domain (PDom) • Expandable beyond 8 M6 CPUs • SSB stores L3$ directory for faster lookup (benefits remote DCU) • higher latency • loss of any SSB causes PDom reset Bounded Dynamic Domain (PDom) • Lower latency • loss of any SSB has no effect • Can not expand beyond 8 M6 CPUs
  • 151. PDoms Examples = Bounded DD = DD PDom 0 PDom 0 PDom 1 PDom 0 PDom 1 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3 DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU 151 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 152. DCU and Dynamic Reconfiguration 152 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Individual CMU/IOU boards can not be logically moved between PDoms  Only virtual CPUs within a PDom can be moved between logical domains in the same PDom  A Dynamic Domain (i.e. PDom) can have its resources changed as a cold- service event.
  • 153. Latency for M6 and M8000/M9000 Systems 153 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Latency (ns) M8000 M9000-32 M6-32 PDom 8-socket M6-32 “Bounded PDom” 8-socket Within CMU 342 387 160 158 Within XB Group/DCU 402 447 222 221 Within Cabinet 402 464 329 -
  • 154. Domain Boot Sequence 154 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  M6 starts executing Hostconfig code from flash prom  Hostconfig does the following – Initialization and configuration of CPUs, Memory, etc. – Generates MDs and PRI for Hypervisor and Guest – Invokes diagnostics and applies platform policies to configure system around failed components – Jumps to Hypervisor on master CPU; others parked  Hypervisor proceeds to – Copy itself from ROM to RAM – Initializes itself based on HV MD – Starts the guest (OpenBoot is the first guest)  OpenBoot probes I/O devices based on MD and sets up the device tree for Solaris. Starts Solaris boot.
  • 155. Internal VLAN 155 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Internal VLAN network is not exposed to the customer network.  End users should not have the need to login directly to the SPPs. Target IP: 169.254.10.xx SP 1 SP0 2 SP1 3 SSP0 17 SSP1 18 SSP2 19 SSP3 20
  • 156. Update Firmware 156 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  The SP firmware can be updated for all SPs at once  The firmware image includes the Oracle ILOM firmware, OpenBoot PROM firmware, POST firmware, and miscellaneous files  The firmware image is installed in the SP flash memory  A user must have the administrator (a) or field engineering (s) role to update the firmware
  • 157. Relative Domain Boot Times Boot Times In Minutes 157 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M9000-64w/ 1TB M6-32 w/ 4TB No Post No Post Max Post AC Power On -> ILOM Shell 12:22 12:15 12:15 Start Host -> OBP OK 25:50 27:04 76:40 OBP OK -> Solaris Login 02:11 04:02 04:02 Total Power On to Solaris 40:23 43:21 92:57 Solaris Halt -> OBP 00:56 04:15 04:15 OBP -> Power Off 03:58 03:23 03:23 Total Solaris Halt to Power Off 04:54 07:38 07:38
  • 158. SPARC Virtualization Technologies for The M5 158 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 159. Oracle Solaris and SPARC Virtualization Better Resource Utilization for a More Efficient Datacenter Dynamic Domains Oracle VM Server for SPARC M-Series T-Series, M5/M6 Oracle Solaris Zones Oracle Solaris DW DB Domain A OLTP DB OLTP DB Domain B Domain A App Domain B App Domain C Web Web Web 159 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal OracleSolaris8Zone OracleSolaris9Zone OracleSolarisZone OracleSolarisZone Web DB App Web
  • 160. Virtualization on M6 Systems 160 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  High degree of virtualization 1. Physical domains - Granularity of physical domains is 8 CPU sockets 2. Logical domains - Includes support of LDoms within physical domains 3. Oracle Solaris Zones for OS virtualization  Oracle Enterprise Manager Ops Center provides an administrator- friendly integration of these different virtualization levels
  • 161. SPARC Physical Domain Oracle Solaris 11 Oracle Solaris 10 Oracle VM Server for SPARC Solaris 10 Zone Physical Domain Oracle Solaris 11 Oracle Solaris 10 Oracle VM Server for SPARC Solaris 11 Zone Most Extensive Virtualization Infrastructure Virtualization Can Be Layered Solaris 11 Zone Solaris Legacy Zone Solaris 10 Zone Solaris Legacy Zone 161 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 162. Oracle Solaris Zones Built-in Virtualization on Any Oracle Solaris System 162 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Same virtualization technology for all SPARC, x86 systems  Simple; lowest overhead; highest performance - Ideally suited to leverage multithreading hardware  Mission-critical deployments - Largest Sun financial and Telco customers all run Oracle Solaris Zones - In production on 25+% of installed Oracle Solaris systems  Ideal for a variety of scenarios - Lightweight test environments - Dynamic environments with resource sharing - Rapid prototyping test beds on same hardware and OS - Zones cloning/migration/instant restart
  • 163. Built-in Virtualization Oracle Solaris 11 Zones • Secure, light-weight virtualization • Scales to 100s of zones/ node • Delegated administration • ZFS datasets, boot environments • Observability via zonestat • Solaris 10 Zones • NFS Server • Network stack isolation and resource management Co-engineered with installation, security, ZFS, networking, IPS, SPARC and x86 hypervisors 163 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 164. Cloud-Scale Networking  Virtualize, consolidate network infrastructure – Increase performance and reduce costs – Secure Isolation  Integrated functionality – Routing, Firewalling, Load Balancing, Bridging, High Availability • Parallel networking stack. Built to scale. • Hardware assisted Network Resource Management • Optimized for performance at every level •Ease of Use • Automatic Networking mode • Fine grained observability • VLAN isolation, dynamic VLAN provisioning 164 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 165. Oracle VM Server for SPARC The Virtualization Platform combining the best of Oracle Solaris and SPARC for Your Enterprise Server Workloads Isolated OS and applications in each logical (or virtual) domain Firmware-based hypervisor Each logical domain runs in dedicated CPU thread(s) M6-32 Server Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain GP Domain GP DomainGP Domain GP Domain SPARC Hypervisor 165 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 166. Alignment with SPARC– designed for Threads 166 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Traditional VM based on assumption “CPUs are scarce, so we must over-commit and time-slice them” • Overhead for time-slicing different contexts • Intercept for “privileged operations” • Latency servicing every interrupt • T5/M5/M6 systems are “thread-rich” - so we can dedicate CPU threads to each domain for native CPU performance • Eliminates CPU latency and overhead • Context switches in a single clock on cache miss or interval • Some VM systems also over-commit RAM, • Causes overhead and requires complex memory management • Ok for lightweight, occasional workloads, very bad for enterprise apps
  • 167. Roles of Domains 167 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Control domain • Creates and manages other logical domains and services • Control domain usually also a service and I/O domain • I/O domains • own physical I/O bus or devices. May run apps using physical I/O for native performance • Service domains • provide virtual network and disk devices. Typically an I/O domain • Guest domain: • run applications on virtual I/O devices provided by service domain
  • 168. Domain Components 168 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 169. Hypervisor and Logical Domains Oracle VM for SPARC 169 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  A logical domain (LDom) is a virtual machine comprised of a discrete logical grouping of resources  Each LDom runs its own instance of Solaris  Each LDom can be created, destroyed, reconfigured, and rebooted independently  The hypervisor enforces the partitioning of the server's resources, and the OS and applications running in those partitions (i.e. LDoms)  The hypervisor allocates a subset of the overall CPU, memory, and I/O resources of a server to a given logical domain  Up to 128 Logical Domains per hypervisor  Minimum version supported is 3.1
  • 170. Oracle VM Server for SPARC: M6 Servers  A physical domain is made of one or more Domain Configuration Units (DCU)  Each physical domain must have a hypervisor  Maximum of 128 logical domains per physical domain Isolated OS and applications in each logical (or virtual) domain Firmware-based hypervisor Each logical domain runs in dedicated CPU thread(s), and memory Physical Domain Hypervisor DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Up to 4 Domain DomainDomain Domain SOA DB 170 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 171. LDom Example 171 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  This shows the following layers that make up the Logical Domains functionality: - User/services, or applications - Kernel, or operating systems - Firmware, or hypervisor - Hardware, including CPU, memory, and I/O
  • 172. Combinations of Physical Domains 172 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Max LDoms: 128 Max LDoms: 256 SOA DB SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor Hypervisor DCU DCU DCU DCU DCU DCU DCU DCU CMU CMU CMU CMU CMU CMU CMU CMU Domain Max LDoms: 256 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU DCU DCU IOU IOU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU Domain Max LDoms: 384 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU DCU IOU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU Domain SOA DB Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Domain SOA DB Domain Domain Domain Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Max LDoms: 512 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU SOA DB Domain Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Domain
  • 173. DCU DCU Assign PDUs to the PDom  PDom 0 can be either a Bounded PDom, or a regular PDom  PDom 1 is a regular PDom  If PDom 0 is regular, then a DCU from PDom 1 can be logically moved from PDom 1 to PDom 0 Hardware DCU PDom 0 PDom 1 Hardware 173 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 174. DCU DCU Create Guests Hardware DCU Hardware Logical Domain A Logical Domain B Logical Domain A Logical Domain B Logical Domain C Hypervisor Hypervisor 174 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PDom 0 PDom 1
  • 175. DCU DCU Create Zones In Each Guest Hardware DCU Hardware Kernel Hypervisor Hypervisor App App App App App App Solaris 11 Solaris 11 Solaris 10U9 + Patches Solaris 10U11Solaris 11 Logical Domain A Zone A App Logical Domain B Logical Domain A Logical Domain B App App Zone A App Logical Domain C Zone B User/Services App App 175 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PDom 0 PDom 1
  • 176. Configuring The Hypervisors 176 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Each Physical Domain can have its hypervisor configured differently than the others - Config 1 - All devices are virtualized and run through the Control Domain, or Service Domain. - Config 2 - All CPU and memory are virtualized and run through the Control Domain.All I/O is provided through the I/O Domain, which owns a PCIe root complex, or offers direct I/O to a specific PCIe device. - Config 3 – No devices are virtualized. All devices are directly accessed from the Control Domain, in which the Logical Domain Manager is optional. - Config 1 and Config 2 can be combined if needed.
  • 177. VM External Shared Storage Oracle VM Server Pool Secure Live Migration Eliminates Application Downtime  Live migration available on SPARC systems (w/o encryption) – SPARC M5/M6 – SPARC T5 – SPARC T4 – SPARC T3 – UltraSPARC T2 Plus – UltraSPARC T2  On-chip crypto accelerators deliver secure, wire speed encryption for live migration – No additional hardware required – Eliminates requirement for dedicated network  More secure, more flexible Secure Live Migration (SSL)VM VM SPARC servers 177 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 178. Cross CPU Migration - Architecture 178 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Allows migration of domains across sun4v architecture platforms • Supports migration among platforms • Will be extended to support new platforms as they are introduced – new platforms might not be migration-compatible with all previous platforms • Allows migration among same CPU architecture with different system clocks frequencies • Dependent on guest domain having Solaris 11 – Solaris introduces a generic sun4v CPU module, simulated 1GHz system clock if HW not available, other changes. • LDoms Manager introduces domain cpu-arch property
  • 179. Cross CPU Migration - Solaris 179 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Supported in guests running – Solaris 11.1 – Solaris 10 1/13 • Introduces new generic CPU module: sun4v-cpu – Domain service extension to identify CPU module capabilities – CPU Module has a major/minor version number, used by domain manager to determine capabilities of the guest • Simulates 1GHz system clock if needed – Kernel routines for read tick/stick modified to emulate clock rate: emulate 1GHz in generic mode; emulate boot frequency after migration in native mode to system with different clock frequency
  • 180. Cross CPU Migration – Generic Domains 180 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • LDom Manager/Firmware/Solaris must be of sufficient revision to support Cross CPU Migration – Firmware must support LDom Live Migration on both source and target domains – Guest domain must be Solaris 11 FCS or newer – Migration is for the most part unchanged • At the start of the migration, domain capabilities and generic CPU module version are retrieved and sent to the target • Check on target ensures that the target processor supports the generic CPU module version
  • 181. Live Migration Requirements 181 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Live Migration with encryption – Only support when source and target systems are the same type (i.e., both T3, both T4, etc…) using cpu-arch=generic – Under OVM for SPARC 3.1, live migration is supported across S3 core based systems (i.e. T4 to M5, or T5 to T4, etc….)
  • 182. Solaris Support 182 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 183. Software Support on M6-32 183 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal  Solaris 11.1 is required to run in the control domain. This means M5 only supports Solaris 11.1 on bare metal.  Solaris 10 1/13 is supported in guest, I/O, root, and service domains.  S10U9, S10U10, and S11 + appropriate Patch Bundle is supported in guest, I/O, root, and service domains.  M6 platform supports Dynamic Domains  OVM for SPARC 3.1
  • 184. Power Management 184 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
  • 185. Power Management Features 185 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Existing M Series (M3000-M9000) have no active power management features Feature T4 T5 M5/M6 Comments Dynamic Voltage & Frequency Scaling (DVFS) No   New to SPARC, already exists on x86 Cycle Skipping    T4 whole socket granularity T5/M5 sub socket granularity Coherency Link Scaling No * No T5-2 only Power Supplies Gold+ A261A A254 T5 PS (A261): Goal Platinum M5/M6 (A254): 3 Phase goal similar to platinum IFS (Intelligent Fan Control)    (Technically not CPU feature)