Enviar búsqueda
Cargar
Presentation sparc m6 m5-32 server technical overview
•
4 recomendaciones
•
2,421 vistas
S
solarisyougood
Seguir
Presentation sparc m6 m5-32 server technical overview
Leer menos
Leer más
Tecnología
Denunciar
Compartir
Denunciar
Compartir
1 de 232
Recomendados
Hit Refresh with Oracle GoldenGate Microservices
Hit Refresh with Oracle GoldenGate Microservices
Bobby Curtis
Oracle licensing on VMware, Cloud, Hyper V and other virtualised platforms
Oracle licensing on VMware, Cloud, Hyper V and other virtualised platforms
Onomi
Oracle GoldenGate
Oracle GoldenGate
oracleonthebrain
Adop and maintenance task presentation 151015
Adop and maintenance task presentation 151015
andreas kuncoro
Oracle GoldenGate Demo and Data Integration Concepts
Oracle GoldenGate Demo and Data Integration Concepts
Fumiko Yamashita
Oracle Managed File Transfer
Oracle Managed File Transfer
Johan Louwers
What's new in Oracle Trace File Analyzer version 12.2.1.1.0
What's new in Oracle Trace File Analyzer version 12.2.1.1.0
Sandesh Rao
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#2
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#2
オラクルエンジニア通信
Más contenido relacionado
La actualidad más candente
Oracle RAC features on Exadata
Oracle RAC features on Exadata
Anil Nair
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
オラクルエンジニア通信
Simple Solutions for Complex Problems
Simple Solutions for Complex Problems
Apcera
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
オラクルエンジニア通信
The DDS Tutorial - Part I
The DDS Tutorial - Part I
Angelo Corsaro
OpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design Solutions
Shane Gibson
Architecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPU
GlobalLogic Ukraine
Securefile LOBs
Securefile LOBs
Martin Berger
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
Allan Cantle
Oracle Security Presentation
Oracle Security Presentation
Francisco Alvarez
Expanding Asterisk with Kamailio
Expanding Asterisk with Kamailio
Fred Posner
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
Best Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open Source
NGINX, Inc.
Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001
jucaab
OCI GoldenGate Overview 2021年4月版
OCI GoldenGate Overview 2021年4月版
オラクルエンジニア通信
Decoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMI
Allan Cantle
Oracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12c
Trivadis
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
Edge AI and Vision Alliance
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
オラクルエンジニア通信
La actualidad más candente
(20)
Oracle RAC features on Exadata
Oracle RAC features on Exadata
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Oracle Database / Exadata Cloud 技術情報(Oracle Cloudウェビナーシリーズ: 2020年7月9日)
Simple Solutions for Complex Problems
Simple Solutions for Complex Problems
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
はじめてのOracle Cloud Infrastructure (Oracle Cloudウェビナーシリーズ: 2021年9月22日)
The DDS Tutorial - Part I
The DDS Tutorial - Part I
OpenStack Control Plane Architectures - Design Solutions
OpenStack Control Plane Architectures - Design Solutions
Architecture of TPU, GPU and CPU
Architecture of TPU, GPU and CPU
Securefile LOBs
Securefile LOBs
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
OMI - The Missing Piece of a Modular, Flexible and Composable Computing World
Oracle Security Presentation
Oracle Security Presentation
Expanding Asterisk with Kamailio
Expanding Asterisk with Kamailio
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Best Practices for Getting Started with NGINX Open Source
Best Practices for Getting Started with NGINX Open Source
Ebs 12.2 con9021_pdf_9021_0001
Ebs 12.2 con9021_pdf_9021_0001
OCI GoldenGate Overview 2021年4月版
OCI GoldenGate Overview 2021年4月版
Decoupling Compute from Memory, Storage and IO with OMI
Decoupling Compute from Memory, Storage and IO with OMI
Oracle Failover Database Cluster with Grid Infrastructure 12c
Oracle Failover Database Cluster with Grid Infrastructure 12c
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
“Making Edge AI Inference Programming Easier and Flexible,” a Presentation fr...
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
しばちょう先生が語る!オラクルデータベースの進化の歴史と最新技術動向#3
Similar a Presentation sparc m6 m5-32 server technical overview
Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprise
solarisyougood
The Cell Processor
The Cell Processor
Heiko Joerg Schick
Sparc t3 2 technical presentation
Sparc t3 2 technical presentation
solarisyougood
Fujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilities
solarisyougood
Computer Systems And Networks Configuration
Computer Systems And Networks Configuration
Tara Daly
Power 7 Overview
Power 7 Overview
lambertt
Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached
Fuad Arshad
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
SnehaLatha68
April 2014 IBM announcement webcast
April 2014 IBM announcement webcast
HELP400
Phytium 64 core cpu preview
Phytium 64 core cpu preview
inside-BigData.com
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
Arti Parab Academics
SUN主机产品介绍.ppt
SUN主机产品介绍.ppt
PencilData
Sparc t4 4 system technical overview
Sparc t4 4 system technical overview
solarisyougood
Overview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit Microcontrollers
Premier Farnell
AMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor Architecture
AMD
system unit and Motherboard
system unit and Motherboard
romeodait
IBM HPC Transformation with AI
IBM HPC Transformation with AI
Ganesan Narayanasamy
Sparc t4 1 system technical overview
Sparc t4 1 system technical overview
solarisyougood
Sparc t3 4 system technical overview
Sparc t3 4 system technical overview
solarisyougood
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard Baker
Similar a Presentation sparc m6 m5-32 server technical overview
(20)
Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprise
The Cell Processor
The Cell Processor
Sparc t3 2 technical presentation
Sparc t3 2 technical presentation
Fujitsu m10 server features and capabilities
Fujitsu m10 server features and capabilities
Computer Systems And Networks Configuration
Computer Systems And Networks Configuration
Power 7 Overview
Power 7 Overview
Oracle Database Appliance - RAC in a box Some strings attached
Oracle Database Appliance - RAC in a box Some strings attached
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
Softcore processor.pptxSoftcore processor.pptxSoftcore processor.pptx
April 2014 IBM announcement webcast
April 2014 IBM announcement webcast
Phytium 64 core cpu preview
Phytium 64 core cpu preview
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
SYBSC IT SEM IV EMBEDDED SYSTEMS UNIT I Core of Embedded Systems
SUN主机产品介绍.ppt
SUN主机产品介绍.ppt
Sparc t4 4 system technical overview
Sparc t4 4 system technical overview
Overview of ST7 8-bit Microcontrollers
Overview of ST7 8-bit Microcontrollers
AMD EPYC™ Microprocessor Architecture
AMD EPYC™ Microprocessor Architecture
system unit and Motherboard
system unit and Motherboard
IBM HPC Transformation with AI
IBM HPC Transformation with AI
Sparc t4 1 system technical overview
Sparc t4 1 system technical overview
Sparc t3 4 system technical overview
Sparc t3 4 system technical overview
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Richard_Baker-Intel_I-32_Processor_Architecture_Overview
Más de solarisyougood
Emc vipr srm workshop
Emc vipr srm workshop
solarisyougood
Emc recoverpoint technical
Emc recoverpoint technical
solarisyougood
Emc vmax3 technical deep workshop
Emc vmax3 technical deep workshop
solarisyougood
EMC Atmos for service providers
EMC Atmos for service providers
solarisyougood
Cisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overview
solarisyougood
Designing your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guide
solarisyougood
Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...
solarisyougood
Ibm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshop
solarisyougood
Power8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshop
solarisyougood
Power systems virtualization with power kvm
Power systems virtualization with power kvm
solarisyougood
Power vc for powervm deep dive tips & tricks
Power vc for powervm deep dive tips & tricks
solarisyougood
Emc data domain technical deep dive workshop
Emc data domain technical deep dive workshop
solarisyougood
Ibm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshop
solarisyougood
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshop
solarisyougood
Emc isilon technical deep dive workshop
Emc isilon technical deep dive workshop
solarisyougood
Emc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshop
solarisyougood
Emc vplex deep dive
Emc vplex deep dive
solarisyougood
Cisco mds 9148 s training workshop
Cisco mds 9148 s training workshop
solarisyougood
Cisco cloud computing deploying openstack
Cisco cloud computing deploying openstack
solarisyougood
Se training storage grid webscale technical overview
Se training storage grid webscale technical overview
solarisyougood
Más de solarisyougood
(20)
Emc vipr srm workshop
Emc vipr srm workshop
Emc recoverpoint technical
Emc recoverpoint technical
Emc vmax3 technical deep workshop
Emc vmax3 technical deep workshop
EMC Atmos for service providers
EMC Atmos for service providers
Cisco prime network 4.1 technical overview
Cisco prime network 4.1 technical overview
Designing your xen desktop 7.5 environment with training guide
Designing your xen desktop 7.5 environment with training guide
Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm aix technical deep dive workshop advanced administration and problem dete...
Ibm power ha v7 technical deep dive workshop
Ibm power ha v7 technical deep dive workshop
Power8 hardware technical deep dive workshop
Power8 hardware technical deep dive workshop
Power systems virtualization with power kvm
Power systems virtualization with power kvm
Power vc for powervm deep dive tips & tricks
Power vc for powervm deep dive tips & tricks
Emc data domain technical deep dive workshop
Emc data domain technical deep dive workshop
Ibm flash system v9000 technical deep dive workshop
Ibm flash system v9000 technical deep dive workshop
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshop
Emc isilon technical deep dive workshop
Emc isilon technical deep dive workshop
Emc ecs 2 technical deep dive workshop
Emc ecs 2 technical deep dive workshop
Emc vplex deep dive
Emc vplex deep dive
Cisco mds 9148 s training workshop
Cisco mds 9148 s training workshop
Cisco cloud computing deploying openstack
Cisco cloud computing deploying openstack
Se training storage grid webscale technical overview
Se training storage grid webscale technical overview
Último
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
DianaGray10
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
Object Automation
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
shyamraj55
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
DianaGray10
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
Matt Ray
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
SkyPlanner
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IES VE
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
Anna Loughnan Colquhoun
20200723_insight_release_plan
20200723_insight_release_plan
Jamie (Taka) Wang
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
DianaGray10
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
Udaiappa Ramachandran
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
D Cloud Solutions
Nanopower In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
Pedro Manuel
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
DianaGray10
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
DianaGray10
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
Mahmoud Rabie
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
Adam Moalla
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
YounusS2
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
francesco barbera
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
bruanjhuli
Último
(20)
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
20200723_insight_release_plan
20200723_insight_release_plan
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
Nanopower In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
Presentation sparc m6 m5-32 server technical overview
1.
1 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
2.
Download this slide http://ouo.io/MsNQB
3.
SPARC M6/M5-32 Server
– Technical Overview Insert Presenter’s Name Here Insert Presenter’s Title Here 3 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
4.
The following is
intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 4 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
5.
SPARC M6/M5-32 Server 5
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
6.
New l SPARC
M6 Servers 32-Socket High-End Datacenter Server Compute – Up to 32x SPARC M6 12 core 3.6GHz CPUs – Allows mixing with SPARC M5 6 core processors – Up to 1024x DDR3 DIMMs for max memory of up to 32TB I/O and storage – 32x 2.5” SAS-2 internal drives – 64x PCIe Gen3 low profile internal slots Scalability and investment protection – Upgradable with M6 processor Availability and management – Advanced RAS with redundant and hot swap of key components – Extensive virtualization with Oracle VM Server for SPARC + hard partitions – Integrated Oracle ILOM system management 6 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
7.
High-end Product Data
Sheet 7 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 Processor M6 3.6 GHz, 48 MB L3$ Max 32 (384 cores) Memory Max 32TB (32GB x 1024 DIMMs) Internal Disks Max 32 (2.5” HDD or SSD) RemovableMedia rKVMS CMU Max 16 IOU Max 4 PCIe Slots Max 64 PCIe Domains Max 4 physical Max 512 logical Power Options 3-phase/Dual Grid Power Consumption 25 KW System Management ILOM (main service processor)
8.
High-end Product Comparison 8
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 2 TB DDR3 DIMMs) CIe Gen 3 Dual Grid Max= 25KW vice processor) .1”Dx78.7”H (1677 kg) M9000-32 M5-32/M6-32 Processor SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88GHz, 6 MB L2$ Max 64 (256 cores) SPARC64 VII+ M5 3.6 GHz, 48 MB L3$ 3.0 GHz,12 MB L2$ Max 32 (192 cores) Max 64 (256 cores) M6 3.6 GHz, 48 MB L3$ Max 32 (384 cores) Memory Max 4 TB Max 3 (8GB x 512 DDR2 DIMMs) (32GB x 1024 Internal Disks Max 64 (2.5” SAS) Max 32 (2.5” HDD or SSD) RemovableMedia DVD, DAT rKVMS CMU and IOU Max 16 each Max 16 CMU and 4 IOU PCI Slots Max 64 PCIe Gen 1 Max 64 P Domains Max 24 physical Max 4 physical, Max 512 logical Power Options 1-phase/3-phase/DualGrid 3-phase/ Power Consumption Min= 12KW, Max= 14.6KW Min=23KW, System Management XSCF (service processor) ILOM (main ser MemoryBandwidth 737 GB/s 1,980 GB/s I/O Bandwidth 244 GB/s 1,024 GB/s Dimensions 33.5”Wx49.6”Dx70.9”H 35.6”Wx57 Weight (Max) Base cab: 2068 lbs.(940kg),Power cab: 770 lbs.(350 kg) 3697 lbs.
9.
Key Differences from
Earlier M-series 9 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M8000/M9000 M6-32 SP XSCF, running XCP SP, running ILOM Each DCU has it’s own SP Proxy (SPP) Domains Hard Domains: up to 24 Hard Domains: up to 4, Logical Domains: up to 512 Memory Clock DDR2 at 480MHz DDR3 at 1066MHz Lowest Latency 387ns 158 ns (Bounded Domain), 160 ns (Regular Domain) IO PCIe Gen 1 PCIe Gen 3 Smallest Hard Domain 1 CPU + 8 DIMMs (Quad-XSB) 4 CPUs + 128 DIMMs (2 CMU boards) Internal network PPP 10/100 Ethernet Clock and Interconnect Cold service for failed clock board and crossbar board Hot-plug replacement of failed clock and SBB boards Power Feed Single phase, or 3-phase. DPF optional Customer provides own cable for 3-phase 3-phase, DPF standard Cable is orderable Airflow Bottom to top Front to back Base IO Card Must use existing PCIe slot Dedicated slots just for Base IO card DVD Built-in. Can be assigned to any Base IO card N/A. Must use remote media of rKVMS. Cable Routing Bottom only Top or bottom
10.
Term Chart 10 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Term Definition CPU memoryboard Unit (CMU) Board equipped with two M6 CPU modules and up to 64 DIMMs of memory Chip Multi-Processor(CMP) Another term for a multi-core processor Buffer on Board (BoB) An ASIC that interfaces between the DIMMs and the MCU on the M6 Memory Controller Unit (MCU) Each M6 has two MCUs in order to communicatewith each BoB. IO board unit (IOU) Unit equipped with 16 PCIe Gen 3, 8 disk drives (HDD or SSD), and 8 10GbE ports DomainConfigurationUnit (DCU) A grouping of up to 4 CMUs, 1 IOU controlled by its own SP Proxy Physical Domain (PDom) An electrically fault-isolatedhard domain comprised of one or more DCUs Hypervisor The SPARC hypervisor is a small firmware layer that provides a stable virtualized machine architectureto which an operating system can be written.
11.
Front Cabinet Components 1.
System status indicators, key switch, and ESD grounding connector 2. 12 Power Supplies (PS) 3. 18 Fan Modules (FM) 4. Leveling Feet 5. 2 Power Supply Distribution Boards (PSDB) 6. 2 Service Processors (SP) 7. 2 Clock Boards 8. 12 Scalability Switch Boards (SSB) 9. Mounting Brackets 11 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
12.
Rear Cabinet Components 1.
Left cable management bracket 2. 4 IO Units (IOU) 3. 4 Service Processor Proxies (SPP) 4. 16 CPU Memory Units (CMU) 5. Leveling feet 6. 6 AC input filters (power cord connectors)(3+3) 7. Right cable management bracket 8. 16 PCIe hot-plug carrier for low-profile PCIe cards 9. 4 Express Module SAS for disks (EMS) 10. 32 Hard disk drives 11. System status indicators and ESD grounding connector 12. Mounting brackets 12 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
13.
M6-32 Front and
Rear Isometric Views 13 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
14.
CPU Memory Unit
(CMU) Two M6 CPU sockets with 12 cores each Each CPU runs at 3.6GHz Memory is type DDR3L RDIMM at 1066MHz 32 DIMM slots per M6 socket 16 and 32 GB DIMMs supported Min of 8 DIMMs per CPU M6 #1MEM #1 MEM #0 M6 #0 14 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 also supports CMUs with M5 processors !
15.
CMU Numbering CMU0 15
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal CMU15 As seen from rear of chassis MEM1 MEM0 CMP1 CMP0
16.
Zakim (ZK) –
Systems Memory Interface 16 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Also referred to as a BoB (Buffer-on-Board) Features and Technology: – 4 ZKs per M5/M6 memory controller – 2 ZKs per T5 memory controller – Supports DDR3 DIMMs – Memory Link to DDR Interface – 2 Memory Link Ports, 2 DDR ports – 8 Write FIFOs – Pass through commands to DIMMs – Core frequency: 1066 MHz
17.
Elements of a
CMU The BoB is really the Zakim ASIC 4 DDR3 DIMMs per BoB 4 BoBs per M6 memory controller 32 DIMMs per M6 processor The DIMMs for each processor are mounted on a memory module 2 memory modules per CMU Each M6 has 7 ports to allow direct communication with other M6s in the domain configuration unit (DCU) CMU Mem Mod 0 BoB BoB BoB BoB BoB BoB BoB BoB M6-0 Mem Ctlr Mem Ctlr Mem Mod 0 BoB BoB BoB BoB BoB BoB BoB BoB Mem Ctlr Mem Ctlr 17 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-1
18.
Internode Coherency Overview
Glueless 1-hop scaling to eight sockets – Glueless means no hub required to connect all 8 CPUs Aprecise directory tracks all L3s in the system – striped across all processors – stored in on-chip SRAMs – flexible for different socket counts Higher BW efficiency than snoop-based protocols enables better scaling – 50% more effective bandwidth than comparable snoopy implementation M5/ M6 M5/ M6 18 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M5/ M6 M5/ M6 M5/ M6 M5/ M6 M5/ M6 M5/ M6
19.
8 Socket Local
Coherency & Data Interconnect DIMMS M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 M5/M6/T5 DIMMSDIMMS DIMMS POINT-TO-POINT LOCAL INTERCONNECT DIMMS PCIe Gen3 Bandwidth 8 diff pairs per ports At 8Gb/sec ~8GB/sec/direction 19 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal DIMMS DIMMS DIMMS All-to-All Interconnect Where the Node-to-Node Fabric is 12 diff pairs per link in each direction. T5 interconnect bandwidth= 157.5 GB/sec T5-8 interconnect bandwidth= 1260 GB/sec DDR3-1066 MemoryBandwidth T5 is 133 GB/sec T5-8 is 1064 GB/s ~8Gb/sec/lane X 8 lanes = 64 Gb/s X 2 directions X 2 Ports/chip = 256Gb/s/chip = 32 GB/s per chip
20.
M5/M6 Global Coherency
& Data Interconnect DIMM 32 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 M5/M6 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 DIMM 32 POINT-TO-POINT LOCAL INTERCONNECT Memory Bandwidth Per M5/M6 = 68GB/sec Or 544GB/sec for 8 M5/M6 All-to-All Local Interconnect Where the Node-to-Node Fabric is 7x12 per link in each direction At 16Gb/sec/direction Bisection Bandwidth 16 Links x 24 GB/sec/Link X 2 directions = 768GB/sec Each CPU connects to 6 BXs Wherethe M5/M6-to-BX Fabric is 6 x 4 Links in each direction At 16Gb/sec ~8GB/sec/direction/link Global interconnect To and from the BXs For 8 CPUs is 48 x 4 Links ~768GB/sec/DCU 32 DIMMs per M5/M6 128 DIMMs / DCU ~4TB with 32GB DIMMs 20 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
21.
CMU: Root Complexes 21
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Each /SYS/CMU has 2 CMPs (CPUs), and each /SYS/CMU/CMP has 2 IOS (2 root complexes). NAC Format for root complex: /SYS/CMU[0-15]/CMP[0- 1]/IOS[0-1] PCIe DevPath for a root complex: dev:////pci@<x>/pci@1, where <x> is the Base address shown in this table CMP0 IOS0 CMP0 IOS1 CMP1 IOS0 CMP1 IOS1 CMU0 300 340 380 3c0 CMU1 500 540 580 5c0 CMU2 400 440 480 4c0 CMU3 600 640 680 6c0 CMU4 700 740 780 7c0 CMU5 900 940 980 9c0 CMU6 800 840 880 8c0 CMU7 a00 a40 a80 ac0 CMU8 b00 b40 b80 bc0 CMU9 d00 d40 d80 dc0 CMU10 c00 c40 c80 cc0 CMU11 e00 e40 e80 ec0 CMU12 f00 f40 f80 fc0 CMU13 1100 1140 1180 11c0 CMU14 1000 1040 1080 10c0 CMU15 1200 1240 1280 12c0
22.
Memory Population Rules 22
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal There are two channels from each BoB Each channel talks with 1 or 2 DIMMs CMUs are either ¼ or ½ populated with DIMMs Populate each CMU with the same size/rank DIMMs It is allowable to mix different DIMM sizes on different CMU boards DIMMs in the same position (0,1,2, or 3) within a MCU, must be homogeneous – must be of the same type (x4 or x8) – same dram device capacity (2Gb or 4Gb) – same number of ranks (SR, DR, or QR) DIMMs on the same channel must have the same rank capacity
23.
M5/M6 Memory Configuration
¼ Populated Best performance for eight DIMMs 0 1 2 3 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 23 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
24.
M5/M6 Memory Configuration
½ Populated Best performance for 16 DIMMs 0 1 2 3 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 24 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
25.
0 1 2 3 3 2 1 0 3 2 1 0 0 1 2 3 M5/M6 Memory Configuration
Fully Populated Best performance for 32 DIMMs 0 1 2 3 3 2 1 0 3 2 1 0 0 1 2 3 BoB0BoB1 BoB2BoB3 BoB5 BoB4BoB6BoB7 M6 fsr2 fsr0fsr1 fsr3 25 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
26.
Memory Module 26 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
27.
27 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
28.
IO Switch Board PCIe Switch 28
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
29.
I/O Unit (IOU) 29
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 1x IOU per physical domain 16 PCIe Gen3 Low Profile slots 8 2.5” HDD or SSD disk drives Four Base I/O Cards (used to access the disk drives) 8 10GbE per IOU PCIe carrier used to allow hot-plug of PCIe cards
30.
M5/M6 IOU LP
PCIe Carriers (8) (Hot Plug) Slots 9-16 Base IO Cards (2) EMS3, EMS4 Base IO Cards (2) EMS1, EMS2 LP PCIe Carriers (8) (Hot Plug) Slots 1-8 Two independent IO bays. 30 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Switch Board 1 (Hot Plug) 10 GbE (8) Hot Plug HDD Drives (4) 0-3 Hot Plug HDD Drives (4) 4-7 Switch Board 0 (Hot Plug)
31.
Elements of a
IOU 2 I/O bays per IOU (0 & 1) 8 PCIe Gen 3 slots per bay 8 SAS 2 disk drives Disk drives are in 2 groups – Group 0: Disks 0-3 – Group 1: Disks 4-7 I/O Bay 0 is controlled by I/O Switch Board 0 – Disk group 0 is controlled via HD Backplane 0 I/O Bay 1 is controlled by I/O Switch Board 1 – Disk group 1 is controlled via HD Backplane 1 IOU Bay 0 Bay 1 Disks 0-3 Disks 4-7 9 10 11 12 Base 3 13 14 15 16 Base 4 Base 1 1 2 3 4 31 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Base 2 5 6 7 8
32.
Base IO Card 32
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal This board is required in order to access the internal disk drives Each board provides 2 10GbE interfaces
33.
M6-32 2 Socket
DCU PCIe Block Diagram (Bounded PDom) IOB1 CMU2 CMU3 CMP0 CMP1 CMP0 CMP1 pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15 33 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal EMS3 EMS4 9 10 11 12 13 14 15 16
34.
M6-32 4 Socket
DCU PCIe Block Diagram 34 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
35.
M6-32 4 Socket
DCU PCIe Block Diagram (Bounded PDom) IOB1 CMU2 CMU3 CMP0 CMP1 CMP0 CMP1 pci_4 pci_5 pci_6 pci_7 pci_12pci_13 pci_14pci_15 35 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal EMS3 EMS4 9 10 11 12 13 14 15 16
36.
M6-32 6 Socket
DCU PCIe Block Diagram (Bounded PDom) 36 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
37.
M6-32 8 Socket
DCU PCIe Block Diagram 37 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
38.
M6-32 Base IO Block
Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 38 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
39.
M6-32 Base IO Block
Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 39 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
40.
M6-32 Base IO Block
Diagram 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 Disk Drives Are Dual Ported! 40 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
41.
24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T
24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 M6-32 Base IO Block Diagram Disk Drives Are Dual Ported! 41 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
42.
24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T
24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T 24p Gen2 Switch SAS2 x8 LSI 2008 Intel X540 Dual 10GBA SE-T DiscDrive3 DiscDrive2 DiscDrive1 DiscDriveo SAS-BP A B A B A B A B DiscDrive7 DiscDrive6 DiscDrive5 DiscDrive4 SAS-BP A B A B A B A B To Switch Board 1 To Switch Board 1To Switch Board 0 To Switch Board 0 Base IO Board 1 Base IO Board 2 Base IO Board 3 Base IO Board 4 x8 x8 x8 x8 X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe X4 SAS X8 PCIe RJ RJ 45 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 RJ 45 M6-32 Base IO Block Diagram Disk Drives Are Dual Ported! 42 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
43.
EMS to Disk
Mapping 43 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
44.
Redundant Disk Paths
In a Domain EMS1 and EMS3 provide SAS paths to the four drives Access all of the drives if one EMS fails Configure them for redundancy using the Oracle Solaris I/O multipathing feature 44 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
45.
PCIe Links 45 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal An IOU contains: 2 IOBs (I/O Switch Boards) 16 slots for LP-carriers (low-profile PCIe card carriers for user-selected add-in I/O) 4 slots for Base IO cards (on-board fixed I/O in a removable form-factor) The LP-carrier slots are labeled PCIE1-PCIE16, and the Base IO slots are labeled EMS1-EMS4. The PCIe data path originates from a root complex (in a CMU), connects to a PCIE_SWITCH (in an IOB) which fans out the signal to PCIe or EMS slots (in the IOU) or to KVMS targets on an SPP (Service Processor Proxy) CMUs IOB Slots CMU[0-1] IOU0/IOB0 IOU0/EMS[1-2], IOU0/PCIE[1-8], SPP0 CMU[2-3] IOU0/IOB1 IOU0/EMS[3-4], IOU0/PCIE[9-16] CMU[4-5] IOU1/IOB0 IOU1/EMS[1-2], IOU1/PCIE[1-8], SPP1 CMU[6-7] IOU1/IOB1 IOU1/EMS[3-4], IOU1/PCIE[9-16] CMU[8-9] IOU2/IOB0 IOU2/EMS[1-2], IOU2/PCIE[1-8], SPP2 CMU[10-11] IOU2/IOB1 IOU2/EMS[3-4], IOU2/PCIE[9-16] CMU[12-13] IOU3/IOB0 IOU3/EMS[1-2], IOU3/PCIE[1-8], SPP3 CMU[14-15] IOU3/IOB1 IOU3/EMS[3-4], IOU3/PCIE[9-16]
46.
Root Complex 4 CMUs
in DCU 46 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
47.
Root Complex 2 CMUs
in DCU 47 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
48.
Disk Device Paths 4
CMUs in DCU 48 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
49.
Disk Device Paths 2
CMUs in DCU 49 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
50.
T5 and M5/M6
PCIe Carrier Card Supports standard low-profile PCIe cards Air Flow 50 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PCIe Retimer x16 Connector (x8 electrical)
51.
About the F40
Flash Card (Aura2) 51 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Supported on T5-2, T5-4, T5-8, and M5-32/M6-32 Hot-plug supported Can not be placed in Slot 8 of IOU on M5-32/M6-32 due to thermal issues.
52.
Base IO Card SAS Controller PCIe Switch Intel 10GbE RJ45 Ports 52
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
53.
Bixby (BX)– M5/M6
Scalability ASIC 53 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Features and Technology: – 16-port Coherency switch router for M5/M6 CPUs > 8 Sockets – 8Address Serialization Pipelines – Supports 4 directory lookups per cycle – Each SSB contains one BX 12 switches in a M6-32 – For unbounded physical domains, each BX has a portion of the L3$ system directory: 6 BXs hold 50% of the system directory for a M6-32 – For bounded physical domains, the BX is not used
54.
Scalability Switch Board
(SSB) 54 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal BX
55.
Coherency Switch Connectivity 55
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 12 Switch cards (SSB), each with a BX ASIC M6-32 - Even numbered CMUs connect to odd numbered BXs - Odd numbered CMUs connect to even numbered BXs Remember, each M6 only has 6 scalability links. All 6 go to either even or odd SSBs. The need to separate BX boards in to even or odd, is due to the limited number of pin outs on the BX chip Each BX uses a 26-bit tag + 7-bit ECC to track one L3$. To track the 32P system, each BX has 8*32*12*424*2 tags (7.41MB)
56.
Additional BX Details 56
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal The 12 SSBs are grouped into two sets of 6. One set for even numbered CMU boards, and the other set for odd numbered CMU boards. The directory is oversized so that if 1 BX fails in a set, the system can be re- configured to operate with any 5 of the 6 BX's in a set. This allows for redundancy in the system directory. A DCU can only communicate data with other DCUs via the BX ASICs on the SSB. Even CMUs can only communicate with even CMUs in other DCUs. Same for odd. For an even CMU to communicate with an odd CMU in another DCU, it must hop at the end.
57.
Scalability Switch Assembly SP0
Connector SP1 Connector Clock0 Connector Clock1 Connector SSB Connectors 57 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
58.
Central Coherency Directory And Data Switch 12x BX ASICs I/O Connectivity DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU I/O Connectivity I/O Connectivity I/O Connectivity 58
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 Block Diagram
59.
DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And
Data Switch IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 Even numbered CMU boards connect to odd numbered SSB boards. IO Switch Board 0 IO Switch Board 0 IO Switch 59 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 M6-32 Block Diagram
60.
DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And
Data Switch BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 IO Switch Board 0 IO Switch Board 0 IO Switch 60 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 Even numbered CMU boards connect to odd numbered SSB boards. M6-32 Block Diagram
61.
DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Central Coherency Directory And
Data Switch BX 0 BX 1 BX 2 BX 3 BX 4 BX 5 BX 6 BX 7 BX 8 BX 9 BX 10 BX 11 IO Switch Board 1 IO Switch Board 0 IO Switch Board 1 IO Switch Board 1 IO Switch Board 1 IO Switch Board 0 IO Switch Board 0 IO Switch 61 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Board 0 M6-32 Block Diagram
62.
61 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Central Coherency Directory And Data Switches 12 13 14 15 3 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 4 Disks Disks 0-3 4-7 0 1 2 3 0 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 1 Disks Disks 0-3 4-7 4 5 6 7 1 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 2 Disks Disks 0-3 4-7 8 9 10 11 2 IO Switch Board 1 IO Switch Board 0 0 1 0 1 0 1 0 1 DCU 3 Disks Disks 0-3 4-7 M6-32 Block Diagram
63.
62 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Central Coherency Directory And Data Switches DCU 1 DCU 2 DCU 3 DCU 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch IO Switch Board 0 Board 1 Board 0 Board 1 Board 0 Board 1 Board 0 Board 1 Disks Disks Disks Disks Disks Disks Disks Disks 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0 1 2 3 Data Path Example CMU0-CPU0 to PCIe slot 61
64.
Clock Board 64 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 2 clock boards, in a redundant configuration Each clock board has dual clock sources with a dynamic multiplexer on each pair. There are 3 clock types distributed through out the system. System core clock 133MHz, Memory clock 133MHz and PCIe clock 100MHz. If one of the clock source fails (in a pair), the dynamic multiplexer automatically switches to secondary synthesizer without bring the system down. The domain keeps running. If more than one clock source in a pair fails, the system will reboot and the alternate clock board will become the active source. A failed clock board that has been marked inactive, it can be replaced while the system is running as there are 2 clock boards, in a redundant configuration.
65.
Service Processor (SP) 65
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Also known as the Integrated Lights Out Management, or ILOM Two redundant SP modules per M6-32 Provides primary platform configuration and management Works with the SPP in each DCU to configure and monitor the components in each DCU
66.
SP Board Serial Network Pilot3 Processor 66 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
67.
SSP Board Pilot3 Module sits
here 67 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
68.
SP and SPP
Connectivity Corporate LAN SP 0 SP 1 SPP 0 SPP 1 SPP 2 SPP 3 Ethernet 16 Port Switch SP Mezzanine 100 Base-T 100 Base-T 1 GbE1 GbE 100 Base-T SerialSerial Ethernet 16 Port Switch SP Mezzanine 68 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
69.
rKVMS 69 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal rKVMS is remote keyboard/mouse/storage connectivity The domain creates a USB keyboard, mouse and storage, as well as a graphics console window that is sent to the SPP The SP opens a port to initiate a graphical console interface so a user can manage the domain through this remote console Only the Golden SPP will have the path to the rKVMS
70.
Front/Rear Status Panels 70
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal No . LED Description 1 System Locator LED (white) The Locator LED can be turned on to identify the server. When on, it blinks rapidly. There are two methods for turning on the LocatorLED: • Issuing the Oracle ILOM commandset /SYS/LOCATE value=Fast_Blink • Pressing the Locator button on the front of the server. 2 System Service Required LED (amber) Indicates that service is required. • The Oracle ILOM show faulty commandprovides details about any faults that cause this indicator to light. • Under some fault conditions, individual component fault LEDs light in addition to the Service Required LED. 3 System Power OK LED (green) Indicates the followingconditions: • Off – System is not running in its normal state. System power might be off. The SPs might still be running. • Steady on – System is powered on and is running in its normal operating state. No service actions are required. • Fast blink – System is running in standby mode and can be quickly returned to full function. • Slow blink – A normal but transitoryactivity is taking place. Slow blinking might indicate that system diagnostics are running or that the system is booting. 4 Power On/Standby button The recessed Power On/Standby button toggles the system on or off. • Press once to power the server on and return the server to its previously configured operating configuration. • If the key switch is set to service mode, press once to shut the server down in a normal manner. • If the key switch is set to service mode, press and hold for at least 4 seconds to perform an emergency shut down. Caution - After using the Power On/Standby button, you must switch off the circuit breakers on your facilitypower grid in order to power down the server completely. The server will remain on standby power until you switch the circuit breakers off.
71.
Front/Rear Status Panels 71
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal No . LED Description 5 SP LED Indicates the following conditions: • Off – Indicates the AC power mighthave been connected to the power supplies. • Steady on, green – SPs are running in their normal operating state. • Blink, green – SPs are initializingthe Oracle ILOM firmware. • Steady on, amber – A SP error has occurred and service is required. 6 Front Component LEDs (amber) The front panel componentrequires service. 7 Rear Component LEDs (amber) The rear panel component requires service. 8 System key switch Using a supplied key, you can switch between normal and service modes. Normal operation mode: • You can power on the server by pressing the Power On/Standby button. • You cannot shut the server down using the Power On/Standbybutton. Service mode: • Server is placed in service mode • You can shut down the server using the Power On/Standby button. • The key cannot be removed from the key switch while in the Service position. 9 Antistatic wrist strap connector The server has two 10-mm connectors where you can attach an antistaticwrist strap prior to installing or servicing the server.
72.
M5 / M6
Differences, Configuration rules and PCIe Slot Guidelines 72 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
73.
M5 / M6
Configuration and Replacement Rules 73 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Mixing of M5 and M6 CMUs within a DCU is not allowed. All CMUs in a DCU must be either M5 or M6. However DCUs with M5 CMUs can co-exist with M6 CMUs in a domain in different DCUs Mixed (M5 and M6) clock boards in M5 chassis are allowed Mixed (M5 and M6) clock boards in a M6 chassis will not inhibit boot but will generate an alert to the user. Anytime the system detects mixed (M5 and M6) clock boards, and both boards are usable, it will use the M6 clock board as the active Note : The M6 clock board in mixed configurations being favored and detection of the mix in an M6 system as well as generating an alert, these changes will be post RR. FW will not stop the user from mixing clock boards.
74.
M5 / M6
Configuration Guidelines 74 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Fill Dark Green Slots First Fill Light Green Slots Second Fill Yellow Slots Last Factory configuration guidelines PCIe IO Slot Priority HDD Slot HDD/SSD EMS Slot BASE IO CARD CMU/IOB IO Slot HALFDCU 0 X 1 X CMU0 – IOB0 1* 2 1 X 3 4* 2 2 5 6* 3 7* 8 4 X 3 X CMU3 – IOB1 9 10* 5 X 11* 12 6 4 13* 14 7 15 16* FULLDCU 0 X 1 X CMU0/1 – IOB0 1 2 1 X 3 4 2 X 2 X 5 6 3 X 7 8 4 X 3 X CMU2/3 – IOB1 9 10 5 X 11 12 6 X 4 X 13 14 7 X 15 16
75.
A PCIE
card marked with an asterisk ( * ) is removed or added, the OBP device path may change, dependent on the state of the ILOM property /HOSTx/ioreconfigure and CMUs present. We recommend prioritizing the installation of PCIE cards ( grouped functionally ) based on the color scheme in the table : 1)Infiniband and Storage PCIE cards starting with dark green first, followed by light green slots without the asterisks. 2)10GigE Network PCIE cards on available dark and light green marked slots with the asterisks. 3) 10/100/1GigE Network cards on the light green and yellow marked with the asterisks. Due to thermal requirements F40/Aura2 cards must not use PCIe slot 8. 75 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M5 / M6 Configuration Guidelines
76.
The SPARC M6
Processor 76 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
77.
25+ Years of
SPARC Processors 1987 1992 1996 2000 20051988 1995 2002 Sunrise: 1st SPARC Processor SUNRAY UltraSPARC I SuperSPARC I UltraSPARC II UltraSPARC III UltraSPARC IIIi UltraSPARC IV+ UltraSPARC T1 SPARC T4 SPARC M5 2007 2010 UltraSPARC T3 UltraSPARC T2 SUNRAY 2011 2013 SPARC T5 77 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
78.
1987 1988 1992
1995 1996 2000 2002 2005 2007 2010 2011 2013 Sunrise: 1st SPARC Processor UltraSPARC IV+ UltraSPARC II SPARC T5 SuperSPARC I UltraSPARC IIIi UltraSPARC T2 SPARC T4 SUNRAY SUNRAY UltraSPARC I UltraSPARC III UltraSPARC T1 UltraSPARC T3 SPARC M5 SPARC S3 Core Processor Family 78 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
79.
M6 Processor 12
S3 cores @ 3.6Ghz – Single or multi-threaded operation per core – 6-48 total strands SPARC S3 Core – 1-8 Strand Dynamically Threaded Pipeline – Vertically Threaded 2-decode, Out-of-Order (OoO) 2-Issue – 16KB 4-way L1 I$ – 16KB 4-way L1 D$ – 128KB 8-way L2$ – ISA-based Crypto-acceleration Shared 48MB L3$ Memory Controllers – 2x Dual Channel DDR3L MCU's – Cascadable Buffer-on-Board (BoB) Integrated I/O – 2 x8 Lane PCIe Gen3 @ 8GT/s – 32GB/s IO Bandwidth per processor System Scalability – Up to 8-socket glueless – 32-socket Systems via scalability links M6 79 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
80.
SPARC M6: Processor
Overview 12 SPARC S3 cores, 96 threads 48MB shared L3 cache 4 DDR3 schedulers, maximum of 1TB of memory per socket 2 PCIe 3.0 x8 lanes Up to 8 sockets glue-less scaling Up to 32 sockets glued scaling 4.1 Tbps total link bandwidth 4.27 billion transistors Coherence&Scalability MIO Crossbar L3 L3 L3 L3 SPARC SPARC L3 Core Core Ctl SPARC SPARC L3 Core Core Ctl L3 SPARC SPARC Ctl Core Core SerDes SerDes SerDes SerDesMCU SPARC Core SPARC Core L3 SPARC SPARC Ctl Core Core SerDes SerDesMCU SPARC Core SPARC Core PCIe PCIe 80 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
81.
SPARC M6 CPU
Block Diagram 2x8PCIe3.0@8GBps 16GBpseachdirection Crypto 8 threads per Core IO Subsystem Memory Control Coherency2x4Switch Memory Control Coherency Links 12.8 Gbps per lane - 12 lanes per link (1075 Gbps) 12 x 5 Crossbar (~620GBps bandwidth) C0 C1 L3$ B0L3$ B0 1MB,16-way 128 KB L2$ SLC 16 KB L1I$ 16 KB L1D$ FGU SPARC S3 Core BoB BoB BoBBoB Coherence Unit Coherence Unit SLC SLC SLC SLC SLC Link 0 Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 C2 C3 C4 C5 C6 C7 C8 DDR3 – 1066 MHz DDR3 – 1066 MHz BoB BoB BoBBoB DDR3 – 1066 MHz DDR3 – 1066 MHz Scalability Links 12 Gbps per lane – 4 lanes per link (144 Gbps) C9 81 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal C10 C11 L3$ B0 12MB,16-way L3$ B2 12MB,16-way L3$ B1 12MB,16-way L3$ B3 12MB,16-way
82.
SPARC M6: Processor
Key Features 82 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPARC M6 based on SPARC S3 core: Same core as used in T4, T5 and M5 – 12 S3 cores, dual PCI Express 3.0 root complexes – Up to 32 M6 processors per system, 12 cores x 8 threads → 96 per M6, or up to 3072 threads – Clock frequency is 3.6 GHz – Each SPARC S3 core contains: 2 Integer pipelines 1 FGU pipeline (consisting of 3 physical sub-pipelines): – FPX pipeline – FGX pipeline – FPD pipeline 1 Load-Store (Memory) pipeline
83.
S3 Core Recap 83
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Out-of-order, dual-issue High frequency achieved with 3.6GHz 16 stage integer pipeline Dynamically threaded, one to eight strands Accelerates 16 encryption algorithms and random number generation
84.
SPARC S3 Core
Dual-issue, out-of-order Integrated encryption acceleration instructions Enhanced instruction set to accelerate Oracle SW stack Dynamically threaded pipeline – 1 to 8 threads 84 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
85.
SPARC Core Roadmap T3
Servers S2 Core M-Series SPARC64 VII/VII+ Core M5 Servers T5 Servers T4 Servers S3 Core M6 Servers S4 Core 85 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
86.
S3 Core Overview 86
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal 8-way threaded, dual-issue, OoO execution, in order commit Dynamically threaded with hardware-optimized resource sharing Support for Critical Threads Deep pipeline for high frequency operation (3 GHz in 40 nm) Balanced single-thread and multi-thread performance Enhanced instruction set to accelerate Oracle SW stack - PAUSE, fused compare-branch Integrated user-level cryptographic acceleration - DES/3DES,AES, Kasumi, Camellia, MD5, SHA-1, SHA-224/256/384/512, RSA, DSA, CRC32c Foundation core for future technology / product nodes
87.
Oracle SPARC S3
Core 87 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Enables T4, T5, M5, M6 • Design Goals - Develop a common replacement core for T-series processors - Significantly improve the single strand performance of the T3 processor - Significantly improve the throughput performance of the M3 processor - Improve the RAS and power management capabilities - Maintain backward ISA, Solaris and OVM for SPARC compatibility Core S1: M2, M3 Core S2: T2, T2+, T3 Core S3: T4, T5, M5, M6 Frequency 2.4 – 3.0 GHz 1.4 – 1.65 GHz T4: 2.85 – 3.0 GHz T5, M5, M6: 3.6 GHz L1 InstructionCache 64KB 16KB 16KB L1 Data Cache 64KB 8KB 16KB L2 Cache - - 128KB # of Pipelines 1 2 1 # of Threads per Pipeline 2 4 1-8 Dynamic Instructionsper Thread 4 per cycle 1 per cycle 2 per cycle Out of Order Issue Yes No Yes (36 instr window) CryptographyAcceleration None SPU ISA Based OVM for SPARC Compatible No Yes Yes SPARC V9 ISA Compatible Yes Yes Yes
88.
SPARC Security Securing All
the Stack Layers Secure database queries 43% faster; Low overhead for encryption WebLogic SSL & WS-Security 2x – 3x faster ZFS Filesystem Crypto 3x faster than x86 Java crypto 2x – 3x faster than x86 OpenSSL4.3x faster for single-thread security versus Power7 Results: 88 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
89.
SPARC T4/T5/M5/M6 Leads
in On-Chip Encryption Acceleration 89 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal On-Chip Accelerators Common Use Examples Mainly Used SPARC T4/T5/M5/M6 IBM Power7+ Only 3 cryptos for 8 cores Intel Westmere /SandyBridge Asymmetric /Public Key Encryption Web Browsers, phone calls, VPN, Secure FTP To protect data or files in transit RSA, DH, DSA, ECC RSA, ECC RSA, ECC Symmetric Key / Bulk Encryption Databases, credit card and social security numbers, private info To protect data or files at rest or stored: disks, backup tapes, etc. AES, DES, 3DES, Camellia, Kasumi AES (Modes: ECB, CBC, CTR, CCM, CCA, GCM, GCA, GMAC, CM, F8, XBC-MAC-96; Key lengths: 128b,192b, 256b) (Lacks AES-CFB used by Oracle database!!) AES Message Digest / Hash Functions Data lookup & authentication, digital signatures, message authentication codes (MAC) To compress/create a short summary from a data chunk and not expose it; To detect duplicate or corrupt data CRC32c, MD5, SHA-1, SHA-2 (SHA-224, SHA- 256, SHA-384, SHA-512) MD5, SHA-1, Partial SHA-2 (SHA-256, SHA- 512), HMAC supported for SHA (?) none Random Number Generation Ubiquitous To generate keys Supported Supported none till Ivy Bridge
90.
Next Generation l
SPARC Processors 90 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal T5 M5 M6 Processor 16 S3 cores, 128 threads @ 3.6GHz 6 S3 cores, 48 threads @ 3.6GHz 12 S3 cores, 96 threads @ 3.6GHz S3 Core Features 1-8 Thread, Dynamically threaded pipeline ISA-based Crypto-acceleration Shared L3$ 8MB 48MB Integrated I/O 2x8 Lane PCIe 3.0 @ 8GT/s System Scalability 7 Coherence Ports for scalability to 8S Additional 6 Scalability Ports for scaling to 32S Power Management Dynamic Voltage Frequency Scaling Enterprise Features OVM Server for SPARC (LDoms), Solaris Zones Scalability Port Redundancy, Dynamic Domains, Hot-Plug SP
91.
S3 Core: Dynamic
Threading Page <#>91 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal While software can activate up to 8 strands on each core at a time, hardware dynamically and seamlessly allocates core resources such as instruction, data, and L2 caches and TLBs, and out-of-order execution resources such as the 128-entry re-order buffer in the core among the active strands. Software activates strands by sending an interrupt to a HALTed strand. Software deactivates strands by executing a HALT instruction on each strand it wants to deactivate. No strand has special hardware characteristics; all strands have identical hardware capabilities.
92.
S3 Core: Dynamic
Threading Page <#>92 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate. If software effectively halts all strands except one on a core via Critical Thread Optimization, the core devotes all of its resources to the sole running strand. Thus, that strand will run as quickly as possible
93.
S3 Core: Critical
Thread Optimization 93 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal The S3 core, starting with Oracle Solaris 10 8/11, can optimize performance by assigning one software thread exclusive access to all of a core's hardware resources. That software thread is considered to be a "critical thread." Solaris automatically detects opportunities to perform this assignment: when one software thread has high CPU utilization and there are more cores than runnable threads. We recommend that users allow Solaris to automatically perform Critical Thread assignment. A privileged user can tell Solaris that a particular software thread should be a critical thread, via the nice(1) command. Solaris will then assign that thread to a core, even if there are more runnable threads than cores.
94.
S3 Core: Critical
Thread Optimization 94 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Solaris Critical Threads optimization for S3 core, tries to provide exclusive access of certain hardware resources to certain application threads Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate There is no new API for declaring threads as ‘critical’; that would require significant changes to source code Rather, to invoke Critical Thread Optimization, use the following CLI or system calls to flag a thread as critical by raising its priority to 60: – priocntl(1) – priocntl(2) – priocntlset(2)
95.
S3 Core: Critical
Thread Optimization 95 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Starting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised to priority 60; the thread can be in any scheduling class In Oracle Solaris 11, to be considered critical by the scheduler, a thread must be: – in the FX (Fixed Priority) or RT (Real-Time) scheduling classes – be raised to priority 60 by one of the previously mentioned mechanisms In either of the above instances, this one thread will run as quickly as possible as it has exclusive access to all core resources
96.
Critical Threads for
key applications 96 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Applicability Opportunity Current Status Database Logwriter, LMS Up to 30% improvement in efficiency LMS is already CT ready. LGWR planned for 12c JAVA (JVM) Compiler threads, GC and priority mapping support Up to 2x improvement for app startup, Smooth GC Support for JVM and JAVA apps to be CT aware is integrated in JDK7U4 Coherence Packet writer, service thread Up to 20% improvement in throughput Integrated in Coherence version 3.7.1 Patch 1 Solaris S11U1 / S10U11 Improve CT perf to be within 10% of best case (hand optimized) Optimizations for decayed PG util and stealing being integrated in S11U1
97.
Reliability, Availability, Serviceability (RAS) 97 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
98.
Definition of Terms 98
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Hot-plug: • refers to the fact that a component can be plugged and unplugged without powering down the platform. It applies to both hot swap and hot service. • Hot service: • refers to the ability to perform hot-plug operations, with the additional necessity of some operator actions (invocation of a CLI or actuating a hot service button on the component to be removed). • The system will notify the user when it is safe to remove the component. • Typical examples would be PCIe Express modules. • Hot swap: • refers to an operation where a component is unplugged and plugged in with no interaction with the ILOM or domain required. • Typical examples here are a single RAID disk or a power supplies.
99.
SPARC M5/M6/T5 System
RAS Overview 99 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Designed to minimize part count and operating temperature to enhance reliability • End-to-end data protection detecting and correcting errors throughout server – ECC everywhere • Processor and Memory protection • CPU core and thread off-lining • Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failover • Major components redundant & hot-pluggable • Fan, Power Supply, and internal disks • RAID capability for internal disks • Fault Management Architecture (FMA) support on ILOM
100.
End-to-End RAS Built
RAS from the inside out Start with the processor, then memory, system and IO, virtualization layer, and the OS Add Oracle Solaris Cluster software for additional service availability Fault Management Architecture (FMA) binds all the layers together 100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
101.
T5/M5/M6 Systems RAS •
Diagnosis engine on SP • Auto reconfigure on failure • Soft Error Rate Discrimination(SERD) • Bad page retirement • OS and SP watchdogs • FMA Component hot-upgradeable T5/M5 Processor • L1$ Tag, Status $ Data • Parity protection • Retry on error • L2$/L3$ Data • SEC/DED protection • Cache-line Sparing • L2$/L3$ Tags • SEC/DED protection • Inline Correction • Cache-line Sparing • L2$/L3$ Status & Directory • SEC/DED protection • Inline Correction • Architectural RegistersL2 Cache • SEC/DED protection • Precise Trap and Hypervisor Correction and Retry Power and Cooling • Advanced Power Management • Redundant hot-swap fans • Redundant hot-swap AC/DC • Dual grid power System I/O S11 FMA Hypervisor System • Redundant SPs with automatic failover • Redundant clock boards • Dual synthesizers per clock board • Diagnosis to the FRU level on first fault •PCI-Express end-to-end CRC • PCI Express link retry • Hot-plug low profile PCI Express cards • Redundant, hot-plug boot disks •Alternate connections between M5 and IO controllers Memory • SDRAM Soft Errors • ECC Protection and Correction • Extended ECC Protection • 4-bit Correction • Pin Steering • Channel Interconnect • CRC protection/Message Retry • Lane Sparing •Enables software partitioning (LDoms) virtualization and failure containment •Processor support for error clearing, correction and collection Central Directory and Switch •SEC/DED protection with in line correction • Physical domain isolation •CRC protected System Interconnect with message retry and lane sparing • Deconfigurable directory chips, no loss of functionality, minimized bandwidth loss • Redundant ScalabilitySwitch Boards 100 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Unique to M5/M6
102.
SPARC M6 RAS:
End-to-End Protection Internal Logic: parity and ECC – Architectural Registers – Cache structures – Internal networks Links: CRC retry CRC ECC with line retire Data-ECC,Address-parity Other (Parity, Retry etc) DFT, Debug etc. 102 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
103.
SPARC M6 RAS:
General Error Handling Flow Hardware Detection and Clean-up Gather Signature De-configure Resources Hypervisor Assist with clean-up Collect Hardware Data Generate Report for SP Manage De-configuration Service Processor Analyze Hypervisor Report Update Error History Initiate Service Call Initiate De-configuration Solaris De-configure User-Visible Resource Offline SerDes Lanes Retire Cache Lines Activate DIMM Spare Column Retire Threads Retire Cores Retire Pages 103 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
104.
SPARC M6 RAS:
Memory Errors ECC optimized for device failures Inline correction and auto write-back “Scrubber” prevents accumulation of upsets “E-retry” characterizes soft vs persistent Cell or word-line fail: Solaris retires page(s) Bit-line or pin fail: firmware deploys DIMM spare column Device fail: inline correction M6 Scheduler Normal RD/WR Scrubber RD Eretry RD/WR DIMM ECC Gen ECC Chk/Corr Err 104 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
105.
HW Aspects of
RAS – New and Interesting Features (1 of 3) 105 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Isolation of memory UE’s to a single DIMM - Cachelines (the unit of memory access) are fully contained within a single DIMM - Most (all?) past Oracle/Sun systems had cachelines that straddled two or four DIMMs - This means UE’s can be diagnosed to a single DIMM rather than generating a list of possibilities Memory pin sparing - ADIMM has 72 data pins (DQ’s) and we use a burst length of 8 when accessing DIMMs => 572 total bits of data - Only 564 bits used for data + ECC + control - Remaining 8 unused bits sourced from a single DQ - MCU can dynamically remap the layout to change which DQ is unused
106.
HW Aspects of
RAS – New and Interesting Features (2 of 3) 106 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Deconfigurability of Scalability Switch Boards - The 12 SSBs logically function as two groups of 6 - Agroup is capable of functioning with just 5 - Requires a system restart because it completely changes the mapping of SSBs to physical addresses that are tracked - In the case of a fatal error internal to an SSB, it will automatically be deconfigured and the system restarted - Replacement can happen while system is running but a manual restart will be required to re-integrate - In the case of non-fatal errors leading to service required, can be serviced live but requires 2 manual restarts - Note: single DCU physical domains can be configured to not use the SSBs
107.
HW Aspects of
RAS – New and Interesting Features (3 of 3) 107 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Dual clock synthesizers per clock board Reconfigurability of PCI fabric - All PCI slots are accessible by 2 different CMUs - Primary or alternate path programmed at the time the physical domain is started - Allows for full IO accessibility in the case of a faulty or missing CMU - Necessary because the PCI root complexes are on the M5/M6 chip - Note: complications arise when multiple logical domains are configured, so reconfiguration is suppressed once the system is “virtualized”
108.
SW Aspects of
RAS 108 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Fault diagnosis done on SP => covered under FMA - Failures typically isolated to a single FRU, with the exception of interconnect failures that span FRU boundaries - Automatic reconfiguration of resources Configuring for Availability => covered separately - Primarily addresses IO configuration using Logical Domains, and how to account for multipathing - LDoms also provide a level of isolation for some classes of uncorrectable errors, since the impact will only be to that LDom
109.
Fault Management 109 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Knowledge Articles in MOS ILOM fdd Diagnosis Faults and Alerts NoALOM Compatibility ILOM FMACaptive Shell Sideband Service Processor Network Connection New ILOM Fault Notification (SNMP Trap) ASR Support
110.
FMA – restricted
shell in SPSH 110 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal To get FMA details in SPSH – start -script /SP/faultmgmt/shell – Returns a faultmgmtsp> prompt Available build-in commands: – echo - Display information to user. Typical use: echo $? – help – Exit – exits restircted shell External commands: – fmadm - Administers the fault management service – fmdump - Displays contents of the fault and ereport/error logs – fmstat - Displays statistics on fault management operations – etcd - ereport injector
111.
ASR Support 111 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPARC M5/M6 servers will be supported byASR (Automatic Service Request) Continues use of sunHwTrapFaultDiagnosed SNMP notification Telemetry for ILOM fdd diagnosis Supports platform and FRU identity Supports multi-suspect list
112.
M6-32 FRU Serviceability * Reboot
required to activate new board. M6-32 112 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
113.
ILOM 113 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
114.
Service Processor Software
(ILOM) on M6 Systems 114 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal ILOM looks/behaves just like ILOM on other platforms Simple (user-visible) set of extensions to support Physical Domains Extensions to support Service Processor Proxies and redundant Service Processors - Minimal impact on user experience
115.
Oracle ILOM Key
Functions 115 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Management Interfaces • CLI, BUI, IPMI, SNMP • Firmware Updates • Remote Host Management • Inventory and Component Management • System Monitoring and Alert/Fault Management • UserAccount Management • Power Consumption Management
116.
ILOM on M6
– Enterprise Features 116 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal ILOM extensions to support Enterprise Features – Some of these are conceptually leveraged from XSCF SW SP Tracing Facility – Very useful for tracing inter-process activity – Reliable performance (elapsed time) measurements – Allows for tracing interactions between SPs and SPPs etc – Enterprise systems have lower volume, more complex configurations and high RAS expectations We cannot expect customers to reproduce bugs Need to collect as much debug info as possible on live system as it occurs on customer site – Coredump compression and snapshot collection – Unified snapshot from all SPs and SPPs
117.
ILOM on M6
– Enterprise Features 117 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Confstore distributed config database – Exploitable by CMM/Blades System Identity is maintained across FRU / SP replacement – aka TLI (Top-Level-Identifier) Flash Images are signed to avoid compromised images “hardened” edits of config files – Transaction oriented – before or after, no intermediate results Tunables framework for MAX_USERS etc Sensor Broadcast (SSBCAST) enhanced to support
118.
ILOM – New
Enterprise Features 118 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Creation of Dynamic Domains (i.e. PDoms) Native FMA support Redundant SPs SPP support
119.
ILOM User Roles 119
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal ILOM supports a maximum of 61 user accounts (configured under /SP/users) on the M6-32 SP M6-32 platforms introduce the concept of separately managed Physical Domains (aka PDom). Each PDom can be separately controlled. Each user account can be optionally configured with specific roles for an individual PDom. The user roles are additive. For example, a user account is permitted to do reset operations on a PDom if: 1. the user has 'r' role for the platform (in /SP/users/username/role) _OR_ 2. the user has 'r' role for the specific PDom (in /SP/users/username/host_roles/hostX_role)
120.
Roles and Capabilities 120
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Role Capabilities none user has no user roles, and prevents the user’s roles from being looked up in LDAP user (u) • Create, delete, disable, and enable user accounts. • Change a user’s password and password properties. • Change a user’s roles. • View all platform states. administrator(a) • Perform all Service Processor configurationother than the useradm and auditadm tasks. • Assign and unassign hardware to or from domains. • Perform Service Processor power operations. • Perform Service Processor failover operations on systems with more than one Service Processor. • Perform all operations on domainhardware. • View all platform and physical domain states. console (c) • Access the Oracle ILOM Remote Console and the SP console. • View and change the state of the Oracle ILOM console configurationvariables. reset (r) • View all states of the hardware assigned to the domain(s)on which this role is held. • View all states of the domain(s)on which this role is held. • Operate the system, which includes performingpower operations, resetting the system, hot-pluggingdevices, enabling and disablingcomponents,and fault managementfor the specified domain. read-only operator (o) • View all platform states. • Change the password and the Session Time-Out setting for their own user account. Whenthis role is defined at the domainlevel, users can: • View all states of the hardware assigned to the domain(s)on which this role is held. • View all states of the domain(s)on which this role is held. field engineering (s) perform all operations reserved for field engineers.
121.
Required User Roles 121
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Aspecific role is required for certain tasks: Task Required User Role Assign and unassign DCUs to a PDomain admin (a) Manage ILOM passwords user (u) Connect to a PDomain console (c) Perform power operations (start, stop, and reset) reset (r) Configure user accounts user (u) Configure host groups user (u)
122.
New to ILOM
3.1 122 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Simplified Data Model (SDM) Three-Level Model – Level One: System Summary Info – Level Two: Subsystem Summary – Level Three: Logical Topology Subsystems: Cooling, Power, CPUs, Memory, Storage, and Networking Also Blades, DCUs, CMUs, CPU Modules, I/O Modules on some platforms Open Problems unifies fault management with SDM
123.
ILOM 3.2: New
Linux distro and compilers 123 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6 will release with ILOM 3.2 Linux version 2.6.27.43, “SQUEEZE” – was 2.6.16.4, “SARGE” gcc version 4.4.5 – was 3.3.6 Why: – Old distro no longer supported – Security and bug fixes – Posix threads instead of Linux threads
124.
WEB CLI LUMAIN SDM BACKEND Platform
xml LIBHDL Hw service CAPI SSM API SDM Architecture 124 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
125.
SDM CLI 125 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal CLI is reorganized. /System target(tree) is introduced Different components of the system are grouped and organized into sub targets of /System At every level of the tree, the critical properties are shown along with any sub targets The applicable cli commands are supported at different levels of the /System tree. All targets and properties under /System are case insensitive
126.
SDM CLI (cont.) 126
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Health and health details are two of the common properties shown at every level to indicate the over all health of that sub tree. Open_Problems target shows the detailed descriptions of the faults in the system /SYS and /Storage targets are made legacy Continue to exist but hidden by default The legacy targets can be made visible by enabling /SP/cli/legacy_targets property.
127.
SDM CLI (Summary
level targets) 127 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System -d targets /System Targets: Open_Problems (0) Processors Memory Power Cooling Storage Networking PCI_Devices Firmware BIOS
128.
SDM CLI (Summary
level properties) 128 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System -d properties health = OK health_details = - open_problems_count = 0 type = Rack Mount model = Exadata X2-3 part_number = 8124854 serial_number = 2229CNL124 component_model = SUN FIRE X4170 M3 component_part_number = 7013743 component_serial_number = 1118CNL013 system_identifier = sysidentifier system_fw_version = 3.1.0.10 primary_operating_system = Not Available host_primary_mac_address = 00:21:28:d5:c0:b2 ilom_address = 10.153.55.201 ilom_mac_address = 00:21:28:D5:C0:B6 locator_indicator = Off power_state = Off actual_power_consumption = 5 watts action = (none)
129.
SDM CLI (Processors
Subsystem) 129 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal -> show /System/Processors/ Targets: CPUs Properties: health = OK health_details = - architecture = x86 64-bit summary_description = Two Intel Xeon Processor E5 Series installed_cpus = 2 max_cpus = 2 -> show /System/Processors/CPUs/ Targets: CPU_0 CPU_1
130.
ILOM on M6
– SP-Proxy Design 130 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6-32 is a large system with 32 CPU chips, 1024 DIMMs, 2160+ sensors – Too large for one SP to manage – I2C, JTAG traces etc – For comparison, T5-8 has 8 CPU chips, 128 DIMMs, 400+ sensors Divide-and-conquer: each SPP is responsible for one DCU (8 CPUs/256 DIMMs) – Re-use ILOM functionality on a per-DCU basis POD, Poller, rKVMS, USB-over-Ethernet – Offloads work from Main-SP to SPPs. Main-SP (SP0/SP1) “aggregates” all SPs/SPPs and presents a unified system view. SP administration can be done on the Main-SP or drilldown to the SPP for PDom-specific management
131.
ILOM on M5
– SP-Proxy Design 131 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal One SPP-per-PDom is chosen as “Golden-SPP” – Provides RKVMS service – Serves as the “Management-SP” for the Pdom Externally accessible target for IPMI, SNMP, spsh operations – Some functions from Main-SP are forwarded to Golden-SPP for service e.g. /HOSTx/SP/services/kvms, /HOSTx/SP/network/interconnect Configuration data from Golden-SPPs is backed to Main-SP – Confstore is a distributed extension of ILOM's memstore database – Maintains unique config data subset for every HOSTx – As SPPs are moved between HOSTx, each SPP will obtain the correct config data subset for its current HOSTx Audit logs, faults, etc forwarded from SPPs to Main-SP (and replicated)
132.
ILOM on M6
– Redundant SP Support 132 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Redundant pair of Main-SPs Active-SP will be the management SP for the platform. – Communicate all configuration data to the Standby – Standby ready to take over if the Active fails Heartbeat is maintained between SPs and SPPs – Active-SP → Standby-SP (SP0 or SP1) – Active-SP → SP-Proxies (SPP0, .. SPP3) Implemented checkpointing support – Use of checkpoint to recover from inconsistent state due to poweron/DR operations interrupted by SP Failover/SPP failures
133.
SP 133 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
134.
Service Processors (SP) SP0
SP1 134 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
135.
M6 M6 Service Processor
Proxy (SPP) The SPP is a local SP dedicated to the DCU Each DCU has its own SPP It purpose is to – Initialize, manage the CPU, Memory controller, DIMMs with the DCU – Environmental monitoring such as temp sensors and adjust 8 Fans In a multi-DCU domain scenario, the SPP with the lowest number available in the domain will automatically become the Golden SPP that will manage the domain for all multiple DCUs In case the Golden SPP fails, the next higher numbers SPP takes over. DCU SP To other SPP Units Disks 0-3 Disks 4-7 M6 M6 M6 M6 M6 M6 135 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPP
136.
SPP Function 136 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPPs are commanded by the main System SPs The communication is ethernet between the SP and SPPs Ethernet communication in SP used to manage the SPPs as well as rKVMS
137.
SPP Features 137 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Xilinx FPGA to provide the SSI boot interface to the CMU RKVMS (Remote Keyboard, Mouse and storage): - This feature allow a domain console to be shown at a remote display. It is able to send keyboard, mouse, and storage commands through Ethernet Packets. - The graphical pixel scraper as contained in a Matrox PCIE VGA inside the Pilot3 Controller. - Avirtual USB hub (not a USB controller) that can transmit and receive USB connections from an external USB UPD727200 controller. The Hub resides in the Pilot3 controller. - The Pilot3 converts these devices in to Ethernet packets and provides remote console functionality for the domain. - Logical Domain Control communication will use Ethernet over USB to share the global SRAM needed for CMU’s to communicate between each other. 100 Mhz PCIE reference clock distribution to CMU’s and IO boards. Ethernet interfaces to the SP board via the capacitive coupled scheme.
138.
Physical Domains 138 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
139.
M6 Systems Terminology 139
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6 Systems consist of CPU, Memory and IO - CPU and Memory are on CPU/Memory boards (CMUs) Each CMU contains 64 DIMM slots Domain Configuration Units (DCUs) are groups of 4 CMUs, and one IOU, connected by the local coherency interconnect ADynamic Domain consists of one or more DCUs ADynamic Domain is also referred to as a Physical Domain (or PDom)
140.
Domain Configuration Unit
(DCU) Since the M6-32 forms groups of 8-way bounded domains, 4 CMU boards and a single IOU form a single configuration unit, or DCU. All communication to other components in the DCU do not need to pass through a BX ASIC. ABX is only needed when data from one CMU needs to go to a CMU in a different DCU. Aphysical domain (PDom) can be made from one or more DCUs. Each DCU is controlled and managed by its own SP Proxy The admin manages the DCU via the main SP M6 1 M6 2 DCU SP To other SPP Units Disks 0-3 Disks 4-7 M6 7 M6 8 M6 5 M6 6 M6 3 M6 4 140 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal SPP
141.
Similar DCUs On
Other Systems 141 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal E25K M9000-32 M5-32/M6-32 1 Uniboard + 1 I/O Board 1) 1 CPU + 8 DIMMs + 2 PCIe slots 2) 1 CMU + 1 IOU 4 CMU + 1 IOU
142.
4 DCUs ADCU
has: – 1 IOU – 1 SPP – Up to 4 CMUs DCUs communicate using the SSB DCU0 142 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal DCU1 DCU2 DCU3
143.
DCU Population Rules 143
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Must install minimum of 4 CMU boards – minimum 2 CMU per DCU Additional CMU boards must be installed in pairs – DCU slots 0 & 3 first, then slots 1 & 2 – Need both even & odd in DCU for proper SSB communication HDD or SSD requirement per PDom – Minimum of 2 HDD – Minimum of 2 Base IO cards for the initial pair of HDD – Additional Base IO cards depend on placement of subsequent pairs of HDD
144.
DCU Memory Population
Rules 144 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Memory on CMUs are either ¼, ½, or fully populated – All the memory on the CMU must be the same density and rank – All the CMU boards in a DCU must be populated exactly the same way, and with the same exact DIMMs – CMU boards in different DCUs can be populated differently
145.
M5/M6 Population Rules 145
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Both processors on a CMU must be the same (both M5 or both M6) All CMUs in a DCU must be the same (all M5 or all M6 based CMUs) APDom can be made from a combination of M5 and M6 based DCUs – If DCU0 has four M5 cpus and DCU1 has eight M6 cpus, they can both be in the same PDom If a DCU has empty CMU slots, they can only be filled with CMUs exactly the same as the existing CMUs in that DCU
146.
Mixing M5 and
M6 CPUs Rules 146 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal All the CMUs in a DCU must be identical No mixing of M5 and M6 CMUs in a DCU APDom can be made from multiple DCUs, – some DCUs are M5 populated – other DCUs are M6 populated
147.
Dynamic Domains 147 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal There can be up to four physical domains (PDoms or PDomain) in the M6-32 Each PDom operates like an independent server that has full hardware isolation from other PDoms on the server Domain configurable units (DCUs) are the building blocks of PDoms Each PDom is represented as /Servers/PDomains/PDomain_x/HOST in Oracle ILOM where x ranges from 0 to one less than the maximum number of possible PDoms in the system (PDomain_0, PDomain_1, PDomain_2, PDomain_3).
148.
Expandable Dynamic Domains 148
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal A PDom has an attribute of expandability (expandable flag) By default, a Physical Domain consisting of a single DCU will run as a regular Physical Domain (i.e. expandable=true) A non-expandable PDom is called a Bounded PDom (or Bounded Physical Domain) (i.e. expandable=false) A PDom can be set to either a Bounded Physical Domain, or a Regular Physical Domain However, a Bounded Physical Domain is not affected by the loss of an SSB
149.
Bounded Dynamic Domains 149
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal ABounded Physical Domain (or Bounded PDom) – does not use the SSB for L3$ directory storage – for CMU to CMU access within the DCU Lower latency No impacted by loss of SSB Can never grow beyond a DCU Changing a domain to Bounded, or from Bounded to regular, requires the domain be halted before changing the “expandable” variable.
150.
Tradeoffs 150 Copyright ©
2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Pros Cons Dynamic Domain (PDom) • Expandable beyond 8 M6 CPUs • SSB stores L3$ directory for faster lookup (benefits remote DCU) • higher latency • loss of any SSB causes PDom reset Bounded Dynamic Domain (PDom) • Lower latency • loss of any SSB has no effect • Can not expand beyond 8 M6 CPUs
151.
PDoms Examples = Bounded
DD = DD PDom 0 PDom 0 PDom 1 PDom 0 PDom 1 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3 PDom 0 PDom 1 PDom 2 PDom 3 DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU 151 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
152.
DCU and Dynamic
Reconfiguration 152 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Individual CMU/IOU boards can not be logically moved between PDoms Only virtual CPUs within a PDom can be moved between logical domains in the same PDom A Dynamic Domain (i.e. PDom) can have its resources changed as a cold- service event.
153.
Latency for M6
and M8000/M9000 Systems 153 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Latency (ns) M8000 M9000-32 M6-32 PDom 8-socket M6-32 “Bounded PDom” 8-socket Within CMU 342 387 160 158 Within XB Group/DCU 402 447 222 221 Within Cabinet 402 464 329 -
154.
Domain Boot Sequence 154
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M6 starts executing Hostconfig code from flash prom Hostconfig does the following – Initialization and configuration of CPUs, Memory, etc. – Generates MDs and PRI for Hypervisor and Guest – Invokes diagnostics and applies platform policies to configure system around failed components – Jumps to Hypervisor on master CPU; others parked Hypervisor proceeds to – Copy itself from ROM to RAM – Initializes itself based on HV MD – Starts the guest (OpenBoot is the first guest) OpenBoot probes I/O devices based on MD and sets up the device tree for Solaris. Starts Solaris boot.
155.
Internal VLAN 155 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Internal VLAN network is not exposed to the customer network. End users should not have the need to login directly to the SPPs. Target IP: 169.254.10.xx SP 1 SP0 2 SP1 3 SSP0 17 SSP1 18 SSP2 19 SSP3 20
156.
Update Firmware 156 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal The SP firmware can be updated for all SPs at once The firmware image includes the Oracle ILOM firmware, OpenBoot PROM firmware, POST firmware, and miscellaneous files The firmware image is installed in the SP flash memory A user must have the administrator (a) or field engineering (s) role to update the firmware
157.
Relative Domain Boot
Times Boot Times In Minutes 157 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal M9000-64w/ 1TB M6-32 w/ 4TB No Post No Post Max Post AC Power On -> ILOM Shell 12:22 12:15 12:15 Start Host -> OBP OK 25:50 27:04 76:40 OBP OK -> Solaris Login 02:11 04:02 04:02 Total Power On to Solaris 40:23 43:21 92:57 Solaris Halt -> OBP 00:56 04:15 04:15 OBP -> Power Off 03:58 03:23 03:23 Total Solaris Halt to Power Off 04:54 07:38 07:38
158.
SPARC Virtualization Technologies for
The M5 158 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
159.
Oracle Solaris and
SPARC Virtualization Better Resource Utilization for a More Efficient Datacenter Dynamic Domains Oracle VM Server for SPARC M-Series T-Series, M5/M6 Oracle Solaris Zones Oracle Solaris DW DB Domain A OLTP DB OLTP DB Domain B Domain A App Domain B App Domain C Web Web Web 159 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal OracleSolaris8Zone OracleSolaris9Zone OracleSolarisZone OracleSolarisZone Web DB App Web
160.
Virtualization on M6
Systems 160 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal High degree of virtualization 1. Physical domains - Granularity of physical domains is 8 CPU sockets 2. Logical domains - Includes support of LDoms within physical domains 3. Oracle Solaris Zones for OS virtualization Oracle Enterprise Manager Ops Center provides an administrator- friendly integration of these different virtualization levels
161.
SPARC Physical Domain Oracle Solaris
11 Oracle Solaris 10 Oracle VM Server for SPARC Solaris 10 Zone Physical Domain Oracle Solaris 11 Oracle Solaris 10 Oracle VM Server for SPARC Solaris 11 Zone Most Extensive Virtualization Infrastructure Virtualization Can Be Layered Solaris 11 Zone Solaris Legacy Zone Solaris 10 Zone Solaris Legacy Zone 161 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
162.
Oracle Solaris Zones Built-in
Virtualization on Any Oracle Solaris System 162 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Same virtualization technology for all SPARC, x86 systems Simple; lowest overhead; highest performance - Ideally suited to leverage multithreading hardware Mission-critical deployments - Largest Sun financial and Telco customers all run Oracle Solaris Zones - In production on 25+% of installed Oracle Solaris systems Ideal for a variety of scenarios - Lightweight test environments - Dynamic environments with resource sharing - Rapid prototyping test beds on same hardware and OS - Zones cloning/migration/instant restart
163.
Built-in Virtualization Oracle Solaris
11 Zones • Secure, light-weight virtualization • Scales to 100s of zones/ node • Delegated administration • ZFS datasets, boot environments • Observability via zonestat • Solaris 10 Zones • NFS Server • Network stack isolation and resource management Co-engineered with installation, security, ZFS, networking, IPS, SPARC and x86 hypervisors 163 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
164.
Cloud-Scale Networking Virtualize,
consolidate network infrastructure – Increase performance and reduce costs – Secure Isolation Integrated functionality – Routing, Firewalling, Load Balancing, Bridging, High Availability • Parallel networking stack. Built to scale. • Hardware assisted Network Resource Management • Optimized for performance at every level •Ease of Use • Automatic Networking mode • Fine grained observability • VLAN isolation, dynamic VLAN provisioning 164 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
165.
Oracle VM Server
for SPARC The Virtualization Platform combining the best of Oracle Solaris and SPARC for Your Enterprise Server Workloads Isolated OS and applications in each logical (or virtual) domain Firmware-based hypervisor Each logical domain runs in dedicated CPU thread(s) M6-32 Server Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain GP Domain GP DomainGP Domain GP Domain SPARC Hypervisor 165 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
166.
Alignment with SPARC–
designed for Threads 166 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Traditional VM based on assumption “CPUs are scarce, so we must over-commit and time-slice them” • Overhead for time-slicing different contexts • Intercept for “privileged operations” • Latency servicing every interrupt • T5/M5/M6 systems are “thread-rich” - so we can dedicate CPU threads to each domain for native CPU performance • Eliminates CPU latency and overhead • Context switches in a single clock on cache miss or interval • Some VM systems also over-commit RAM, • Causes overhead and requires complex memory management • Ok for lightweight, occasional workloads, very bad for enterprise apps
167.
Roles of Domains 167
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Control domain • Creates and manages other logical domains and services • Control domain usually also a service and I/O domain • I/O domains • own physical I/O bus or devices. May run apps using physical I/O for native performance • Service domains • provide virtual network and disk devices. Typically an I/O domain • Guest domain: • run applications on virtual I/O devices provided by service domain
168.
Domain Components 168 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
169.
Hypervisor and Logical
Domains Oracle VM for SPARC 169 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal A logical domain (LDom) is a virtual machine comprised of a discrete logical grouping of resources Each LDom runs its own instance of Solaris Each LDom can be created, destroyed, reconfigured, and rebooted independently The hypervisor enforces the partitioning of the server's resources, and the OS and applications running in those partitions (i.e. LDoms) The hypervisor allocates a subset of the overall CPU, memory, and I/O resources of a server to a given logical domain Up to 128 Logical Domains per hypervisor Minimum version supported is 3.1
170.
Oracle VM Server
for SPARC: M6 Servers A physical domain is made of one or more Domain Configuration Units (DCU) Each physical domain must have a hypervisor Maximum of 128 logical domains per physical domain Isolated OS and applications in each logical (or virtual) domain Firmware-based hypervisor Each logical domain runs in dedicated CPU thread(s), and memory Physical Domain Hypervisor DCU IOU CMU CMU CMU CMU DCU IOU CMU CMU CMU CMU Up to 4 Domain DomainDomain Domain SOA DB 170 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
171.
LDom Example 171 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal This shows the following layers that make up the Logical Domains functionality: - User/services, or applications - Kernel, or operating systems - Firmware, or hypervisor - Hardware, including CPU, memory, and I/O
172.
Combinations of Physical
Domains 172 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Max LDoms: 128 Max LDoms: 256 SOA DB SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor Hypervisor DCU DCU DCU DCU DCU DCU DCU DCU CMU CMU CMU CMU CMU CMU CMU CMU Domain Max LDoms: 256 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU DCU DCU IOU IOU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU Domain Max LDoms: 384 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU DCU IOU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU CMU Domain SOA DB Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Domain SOA DB Domain Domain Domain Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Max LDoms: 512 SOA DB SOA DB Domain Domain Domain Domain Domain Domain Domain Hypervisor Hypervisor DCU DCU IOU IOU CMU CMU CMU CMU CMU CMU CMU CMU SOA DB Domain Domain Domain Hypervisor DCU IOU CMU CMU CMU CMU Domain
173.
DCU DCU Assign PDUs
to the PDom PDom 0 can be either a Bounded PDom, or a regular PDom PDom 1 is a regular PDom If PDom 0 is regular, then a DCU from PDom 1 can be logically moved from PDom 1 to PDom 0 Hardware DCU PDom 0 PDom 1 Hardware 173 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
174.
DCU DCU Create Guests Hardware DCU Hardware Logical Domain
A Logical Domain B Logical Domain A Logical Domain B Logical Domain C Hypervisor Hypervisor 174 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PDom 0 PDom 1
175.
DCU DCU Create Zones
In Each Guest Hardware DCU Hardware Kernel Hypervisor Hypervisor App App App App App App Solaris 11 Solaris 11 Solaris 10U9 + Patches Solaris 10U11Solaris 11 Logical Domain A Zone A App Logical Domain B Logical Domain A Logical Domain B App App Zone A App Logical Domain C Zone B User/Services App App 175 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal PDom 0 PDom 1
176.
Configuring The Hypervisors 176
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Each Physical Domain can have its hypervisor configured differently than the others - Config 1 - All devices are virtualized and run through the Control Domain, or Service Domain. - Config 2 - All CPU and memory are virtualized and run through the Control Domain.All I/O is provided through the I/O Domain, which owns a PCIe root complex, or offers direct I/O to a specific PCIe device. - Config 3 – No devices are virtualized. All devices are directly accessed from the Control Domain, in which the Logical Domain Manager is optional. - Config 1 and Config 2 can be combined if needed.
177.
VM External Shared Storage Oracle
VM Server Pool Secure Live Migration Eliminates Application Downtime Live migration available on SPARC systems (w/o encryption) – SPARC M5/M6 – SPARC T5 – SPARC T4 – SPARC T3 – UltraSPARC T2 Plus – UltraSPARC T2 On-chip crypto accelerators deliver secure, wire speed encryption for live migration – No additional hardware required – Eliminates requirement for dedicated network More secure, more flexible Secure Live Migration (SSL)VM VM SPARC servers 177 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
178.
Cross CPU Migration
- Architecture 178 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Allows migration of domains across sun4v architecture platforms • Supports migration among platforms • Will be extended to support new platforms as they are introduced – new platforms might not be migration-compatible with all previous platforms • Allows migration among same CPU architecture with different system clocks frequencies • Dependent on guest domain having Solaris 11 – Solaris introduces a generic sun4v CPU module, simulated 1GHz system clock if HW not available, other changes. • LDoms Manager introduces domain cpu-arch property
179.
Cross CPU Migration
- Solaris 179 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Supported in guests running – Solaris 11.1 – Solaris 10 1/13 • Introduces new generic CPU module: sun4v-cpu – Domain service extension to identify CPU module capabilities – CPU Module has a major/minor version number, used by domain manager to determine capabilities of the guest • Simulates 1GHz system clock if needed – Kernel routines for read tick/stick modified to emulate clock rate: emulate 1GHz in generic mode; emulate boot frequency after migration in native mode to system with different clock frequency
180.
Cross CPU Migration
– Generic Domains 180 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • LDom Manager/Firmware/Solaris must be of sufficient revision to support Cross CPU Migration – Firmware must support LDom Live Migration on both source and target domains – Guest domain must be Solaris 11 FCS or newer – Migration is for the most part unchanged • At the start of the migration, domain capabilities and generic CPU module version are retrieved and sent to the target • Check on target ensures that the target processor supports the generic CPU module version
181.
Live Migration Requirements 181
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal • Live Migration with encryption – Only support when source and target systems are the same type (i.e., both T3, both T4, etc…) using cpu-arch=generic – Under OVM for SPARC 3.1, live migration is supported across S3 core based systems (i.e. T4 to M5, or T5 to T4, etc….)
182.
Solaris Support 182 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
183.
Software Support on
M6-32 183 Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Solaris 11.1 is required to run in the control domain. This means M5 only supports Solaris 11.1 on bare metal. Solaris 10 1/13 is supported in guest, I/O, root, and service domains. S10U9, S10U10, and S11 + appropriate Patch Bundle is supported in guest, I/O, root, and service domains. M6 platform supports Dynamic Domains OVM for SPARC 3.1
184.
Power Management 184 Copyright
© 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal
185.
Power Management Features 185
Copyright © 2012, Oracle and/or its affiliates.All rights reserved. Confidential – Oracle Internal Existing M Series (M3000-M9000) have no active power management features Feature T4 T5 M5/M6 Comments Dynamic Voltage & Frequency Scaling (DVFS) No New to SPARC, already exists on x86 Cycle Skipping T4 whole socket granularity T5/M5 sub socket granularity Coherency Link Scaling No * No T5-2 only Power Supplies Gold+ A261A A254 T5 PS (A261): Goal Platinum M5/M6 (A254): 3 Phase goal similar to platinum IFS (Intelligent Fan Control) (Technically not CPU feature)