SlideShare una empresa de Scribd logo
1 de 61
Descargar para leer sin conexión
Advanced Root Cause Analysis,[object Object],Nathan Small,[object Object],Staff Engineer,[object Object],Global Support Services,[object Object],Rev B – September 13, 2010,[object Object]
Today we will learn how to fish,[object Object]
Advanced Root Cause Analysis,[object Object],Gathering Information,[object Object],Log Analysis,[object Object],Further Analysis,[object Object],Comparative Analysis,[object Object]
Logging Information,[object Object],VMkernel Logging: ,[object Object],Location: /var/log/vmkernel (ESX Classic) or /var/log/messages (ESXi),[object Object],Purpose: This log file contains informational messages, alerts, and warnings for various pieces of code that execute via the vmkernel. It also contains log entries dumped from module logging (Qlogic, Emulex, S/W iSCSI, etc),[object Object],Iterations: By default, this log has 36 rotations excluding the base log (vmkernel to vmkernel.36),[object Object],Related logs: Alert and warning VMkernel events are copied to /var/log/vmkwarning,[object Object],Service Console Logging (ESX Classic),[object Object],Location: Various logs under /var/log/,[object Object],Purpose: These logs would also appear in RHEL and contain the same type of log information you would expect from that OS (aside from vprobs in ESX 4.0),[object Object],Log files: boot, secure, messages, rpm, etc,[object Object]
Logging Information,[object Object],Hostd Logging: ,[object Object],Location: /var/log/vmware,[object Object],Purpose: This log contains entries from hostd operations including NFC (network file copy) operations.,[object Object],Iterations: By default, this log has 10 rotations which wrap (hostd-0 to hostd-9). Pay attention to the timestamp of the log to determine which log you wish to review,[object Object],Vpxa Logging,[object Object],Location: Various logs under /var/log/vmware/vpx,[object Object],Purpose: This log contains requests/communication between the host and vCenter or vCenter and the host,[object Object],Iterations: By default, this log has 10 rotations which wrap (vpxa-0 to vpxa-9). Pay attention to the timestamp of the log to determine which log you wish to review,[object Object]
Logging Information,[object Object],Esxcfg-boot Logging: ,[object Object],Location: /var/log/vmware,[object Object],Purpose: This log contains esxcfg-boot command information and results from the esxcfg-boot command when it is run.,[object Object],Iterations: There are 4 log iterations,[object Object]
HBA driver logging options,[object Object],By default, the HBA driver logging levels are not verbose. Increasing the logging levels can make a significant difference in finding root cause as well as resolution time for a case:,[object Object],Default logging:,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410001103280) to NMP device "naa.600508b40006f6930000a000021b0000" failed on physical path "vmhba1:C0:T10:L54" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x00x0.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508b40006f6930000a000021b0000" state in doubt; requested fast path state update...,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)ScsiDeviceIO: 747: Command 0x2a to device "naa.600508b40006f6930000a000021b0000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x00x0.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41000112bc80) to NMP device "naa.600508b40006f6930000a000021b0000" failed on physical path "vmhba1:C0:T10:L54" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x00x0.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)ScsiDeviceIO: 747: Command 0x2a to device "naa.600508b40006f6930000a000021b0000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x00x0.,[object Object]
HBA driver logging options,[object Object],Enhanced Qlogic driver logging:,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)<6>scsi(1:10:54) UNDERRUN status detected 0x15-0x18. resid=0x0 fw_resid=0x10000 cdb=0x2a os_underflow=0x10000,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)scsi(1:0:10:54) Dropped frame(s) detected (10000 of 10000 bytes)...retrying command.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)<6>scsi(1:10:54) UNDERRUN status detected 0x15-0x18. resid=0x0 fw_resid=0x10000 cdb=0x2a os_underflow=0x10000,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)scsi(1:0:10:54) Dropped frame(s) detected (10000 of 10000 bytes)...retrying command.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410001103280) to NMP device "naa.600508b40006f6930000a000021b0000" failed on physical path "vmhba1:C0:T10:L54" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x00x0.,[object Object],vmkernel: 0:00:22:39.107 cpu1:4270)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508b40006f6930000a000021b0000" state in doubt; requested fast path state update...,[object Object]
HBA driver logging options,[object Object],A review of /proc/scsi/qla2xxx/X:,[object Object],QLogic PCI to Fibre Channel Host Adapter for QLE2460:,[object Object],        Firmware version 4.04.09 [IP] [Multi-ID] [84XX] , Driver version 8.02.01-k1-vmw39,[object Object],BIOS version 2.02,[object Object],FCODE version 2.00,[object Object],EFI version 2.00,[object Object],Flash FW version 4.03.01,[object Object],ISP: ISP2432,[object Object],Login retry count =   008,[object Object],Execution throttle = 2048,[object Object],ZIO mode = 0x6, ZIO timer = 1,[object Object],Commands retried with dropped frame(s) = 40541,[object Object]
HBA driver logging options,[object Object],Here are the instructions to increase HBA logging levels for ESX 4:,[object Object],To enable enhanced logging for Qlogic FC (qla2xxx driver):,[object Object],# esxcfg-module -s ql2xextended_error_logging=1 qla2xxx,[object Object],To enable enhanced logging for Emulex FC (lpfc840 driver) ** :,[object Object],# esxcfg-module -s lpfc_log_verbose=1043,[object Object],To enable enhanced logging for QlogiciSCSI (qla4xxx driver):,[object Object],# esxcfg-module -s extended_error_logging=1 qla4xxx,[object Object],** Emulex logging options can be tricky. Please refer to KB 1005576,[object Object]
List/Load Module Parameters,[object Object],To list all loaded modules on an ESX host, use the vmkload_mod command:,[object Object],# vmkload_mod -l,[object Object],Name                R/O Addr          Length      R/W Addr          Length        ID Loaded,[object Object],vmklinux            0x880000          0x20000     0x28a9b80         0x4d000       1  Yes,[object Object],ioat                0x8a0000          0x3000      0x28f6ba0         0x3000        2  Yes,[object Object],ata_piix            0x8a3000          0xb000      0x28f9bc0         0x4000        3  Yes,[object Object],bnx2                0x8ae000          0x10000     0x28fdbe0         0x17000       4  Yes,[object Object],aacraid_esx30       0x8be000          0x10000     0x2914c00         0x9000        5  Yes,[object Object],e1000               0x8ce000          0x2a000     0x291dc20         0xd000        6  Yes,[object Object],qla2300_707_vmw     0x8f8000          0x5c000     0x292ac80         0xb3000       7  Yes,[object Object],<Snip>,[object Object]
List/Load Module Parameters,[object Object],To list all module parameters for a specific module, use vmkload_mod with the '-s' flag:,[object Object],# vmkload_mod -s qla4xxx,[object Object],vmkload_mod module information,[object Object], input file: /usr/lib/vmware/vmkmod/qla4xxx.o,[object Object], Version: Version 5.01.00-k8_rh5.2-01_vmw_2009_03_30, Build: 208167, Interface: 9.0, Built on: Nov  8 2009,[object Object], Parameters:,[object Object],heap_max: int,[object Object],    Maximum attainable heap size for the driver.,[object Object],heap_initial: int,[object Object],    Initial heap size allocated for the driver.,[object Object],ka_timeout: int,[object Object],    Keep Alive Timeout,[object Object],recovery_tmo: int,[object Object],    Recovery Timeout,[object Object],cmd_timeout: int,[object Object],    Command Timeout,[object Object],extended_error_logging: int,[object Object],    Option to enable extended error logging, Default is 0 - no logging, 1 - debug logging,[object Object]
List/Load Module Parameters,[object Object],To set a loadable module parameter, use esxcfg-module (Persistent across reboots):,[object Object],# esxcfg-module –s extended_error_logging=1 qla4xxx,[object Object],*Note: Ensure you enter the module parameter correctly otherwise the module will fail to load on boot. ,[object Object],This action will append a line to the bottom of /etc/vmware/esx.conf in the form of the following:,[object Object],<Snip> ,[object Object],/upgrades/complete[0000]/name = "depricatePrettyName",[object Object],/upgrades/complete[0001]/name = "moduleLineReformat",[object Object],/upgrades/complete[0002]/name = "enableTSO310",[object Object],/upgrades/complete[0003]/name = "persistVmkNicName",[object Object],/vmkernel/module/qla4xxx.o/options = "extended_error_logging=1“,[object Object]
List/Load Module Parameters,[object Object],After the loadable module parameter is set, the boot image needs to be rebuilt (ESX Classic only) and the host needs to be rebooted for the changes to take effect (or the module can be reloaded, however we do not support this action):,[object Object],# esxcfg-boot –b,[object Object],# reboot,[object Object],To enable an option immediately without rebooting (non-persistent across reboots), you can echo the same parameter to the proc nodes. This may not work for all modules however it has been proven to work for FC modules:,[object Object],# echo "ql2xextended_error_logging=1" > /proc/scsi/qla2xxx/z,[object Object],z = HBA #,[object Object],Note: This would be particularly useful if you are troubleshooting an issue live and need more information without rebooting the host which may clear the condition. ,[object Object]
Serial line logging/Remote Syslog/vMA,[object Object],While logging options for modules are plentiful, it may be necessary to setup serial line logging or remote syslog for an ESX host in the event that logging is missing or inconsistent. ,[object Object],Three good examples of when this would be useful would be: 1. If the ESX host hangs unexpectedly and no logs are generated for the event, 2. The service console goes into a read-only state, 3. The local raid controller or hardware experiences an issue causing logging to not be written down to disk.,[object Object],The vMA appliance can be used for remote syslog purposes but is more useful with an ESXi environment in which logs are not preserved on a reboot. Setting up the vMA appliance should be mandatory for any and all ESXi hosts. To do this, each ESXi host needs to be setup as a vi-fastpass target on the vMA appliance.,[object Object]
Serial line logging/Remote Syslog/vMA,[object Object],Instructions on how to setup serial line logging:,[object Object],http://kb.vmware.com/kb/1003900,[object Object],Instructions on how to setup remote syslog:,[object Object],http://articles.techrepublic.com.com/5100-22_11-5285872.html,[object Object],Instructions on how to setup ESXi host logging with vMA:http://www.simonlong.co.uk/blog/2010/05/28/using-vma-as-your-esxi-syslog-server/,[object Object]
Force crash of VM/ESX host,[object Object],When enhancing logging levels isn’t providing enough information or we need a deeper look at what the driver is doing in memory, it is sometimes necessary to crash a VM or the ESX host to review that memory dump.,[object Object],There are multiple options to capture a memory dump however it will depend on what level the memory dump needs to be seen:,[object Object],Memory inside the Guest OS: Taking a snapshot of the VM with memory state saved or force the OS to crash (E.g.: use the ctrl+scroll+scroll function for Windows),[object Object],Memory dump of the VMM: Use vm-support to list the WID and force crash the VM with the “-X” option. This will generate a vmx-dump file for consumption.,[object Object],Memory dump of the ESX host: Issue an NMI from a remote administrator adapter (ie: HP iLO) which will panic the host if the host is setup correctly.,[object Object]
Force crash of VM/ESX host continued,[object Object],Run the following commands to immediately enable the NMI trap: Note: This does not make the change in behavior persist across a reboot.For ESX 3.x:echo 1 > /proc/sys/kernel/unknown_nmi_panicecho 1 > /proc/sys/kernel/mem_nmi_panic  For ESX 4.x:echo 1 > /proc/sys/kernel/panic_on_unrecovered_nmiecho 1 > /proc/sys/kernel/unknown_nmi_panic,[object Object]
Force crash of VM/ESX host continued,[object Object],In order to have this change persistent across reboots, edit the file /etc/sysctl.conf and add the following lines to persist across reboots:For ESX 3.x:kernel.unknown_nmi_panic = 1kernel.mem_nmi_panic = 1For ESX 4.x:kernel.panic_on_unrecovered_nmi = 1kernel.unknown_nmi_panic = 1,[object Object]
Force crash of VM/ESX host continued,[object Object],VMware ESXi 3.xThere is no configurable option for ESXi 3.x to change the behaviour of ESXi when receiving an NMI. To observe the hang/crash event within the logs, prior to the failure, press Alt+F12 at the console to display the VMkernel log.,[object Object],VMware ESXi 4.xRun the following command followed by a reboot of the host:esxcfg-advcfg -k 2 nmiAction,[object Object]
Corruption messages in vmkernel log,[object Object],When corruption occurs it can be useful to review the logs from the host that saw the corruption occur. These messages will usually indicate what volume saw corruption, what type of corruption was seen, and what part of the VMFS structure experienced corruption (offset):,[object Object],Heartbeat Region Corruption:WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/foo/foo-560e1410.vswp': Invalid metadataFSS: 390: Failed with status Invalid metadata for f530 28 1 46ee2036 61d5698d 4004b12 f4c3b923 0 0 0 0 0 0 0 FS3: 6710: Reclaiming timed out heartbeat [HB state abcdef02 offset 3313664 gen 3 stamp 21824288493247 uuid 4a2ff95d-7967268a-db5c-001a64ca3e46 jrnl <FB 59001> drv 7.33] failed: Invalid metadata,[object Object]
Corruption messages in vmkernel log,[object Object],File Lock Corruption:vmkernel: Invalid lock address 0[lockAddr 0] Invalid lock type 0x0[lockAddr 496217088] Invalid lock addrWARNING: FS3: 556: Volume 4bef2afb-b8226400-2f20-0019b9b5a27b (“vmfs1") may be damaged on disk. Corrupt lock detected at offset 1d93ac00: [type 0 offset 0 v 0, hb offset 0WARNING: FS3: 7544: Volume 4beeef00-3222e0e8-c25f-0019b9b5a27b (“storevmdk") may be damaged on disk. Corrupt lock detected at offset ad419e4ead419e4d: [type a88c4fa2 offset 12484433702799121997 v 12484433870302846580, h,[object Object]
Corruption messages in vmkernel log,[object Object],Cluster/Resource Group Corruption:WARNING: Fil3: 4165: Unknown object type 0 WARNING: Fil3: 4165: Unknown object type 1314280013WARNING: Fil3: 9613: Found invalid object on 49e752ba-4d3c56e8-a7fd-0015177af4b7 <FD c0 r0> expected <FD c92 r125>,[object Object]
Corruption messages in vmkernel log,[object Object],The code still relies on some sanity when pasting these types of corruption messages. As such, there are instances where the logged message will state corruption offsets that are completely out of range:WARNING: FS3: 7544: Volume 4beeef00-3222e0e8-c25f-0019b9b5a27b (“storevmdk") may be damaged on disk. Corrupt lock detected at offset ad419e4ead419e4d: [type a88c4fa2 offset 12484433702799121997 v 12484433870302846580, h,[object Object],As you can see, these ranges do not conform to the expected value ranges.,[object Object]
VMFS Corruption (volume dump for analysis),[object Object],There are varying degrees of data required to successfully troubleshoot/resolve corruption in the VMFS structure depending on what has gotten corrupt. To simply address the HeartBeat region, 25M will suffice. To address the file lock regions, up to 1.2GB would be required.,[object Object],To gather a disk dump for review with VMware Support, please refer to the instructions in KB 1009565:http://kb.vmware.com/kb/1009565,[object Object]
Advanced Root Cause Analysis,[object Object],Gathering Information,[object Object],Log Analysis,[object Object],Further Analysis,[object Object],Comparative Analysis,[object Object]
Log format,[object Object],Logging in vSphere is quite verbose as is but it is important to know what you are looking at when doing a root cause analysis. In this section we will review the logging format for:,[object Object],/var/log/vmkernel and /var/log/vmkwarning,[object Object],/var/log/vmksummary,[object Object],/var/log/vmkiscsid.log,[object Object],/var/log/messages,[object Object]
vmkernel/vmkwarning,[object Object],The vmkernel log is your primary resource for logging messages when trying to determine root cause. By default this log will have 36 rotated iterations plus the base vmkernel log (vmkernel to vmkernel.36) with the exception of ESXi logging, which places all messages into /var/log/messages. ,[object Object],The best way to quickly review the vmkernel log messages for an ESXi host would be to run the following command:,[object Object],# cat messages* |grepvmkernel|less,[object Object],There is a secondary log file known as vmkwarning which has an iteration of 4 plus the base log file (vmkwarning to vmkwarning.4). This log file parses the vmkernel log for any messages with a status of WARNING or ALERT. Here would be an example of each:,[object Object],WARNING: SCSI: 4623: Manual switchover to vmhba2:1:30 completed unsuccessfully.,[object Object],ALERT: APIC: 1150: Lint1 interrupt on pcpu 0 (port x61 contains 0x91),[object Object]
vmkernel/vmkwarning,[object Object],Here is a breakdown of all fields in a standard vmkernel/vmkwarning log message:,[object Object],Nov 30 16:04:17 esx04vmkernel:28:02:20:33.356cpu4:1586)StorageMonitor:196:vmhba2:0:0:0 status = 0/7 0x0 0x00x0,[object Object],Nov 30 16:04:17 = Date and time,[object Object],esx04 = server name,[object Object],vmkernel: = logging type,[object Object],28:02:20:33.356 = uptime of host (days:hours:minutes:seconds:milliseconds),[object Object],cpu4: = cpu/core that trapped the message,[object Object],1586) = World ID or WID of process,[object Object],StorageMonitor: = Piece of code reporting message,[object Object],196: = line of code reporting the message,[object Object],vmhba2:0:0:0 status = 0/7 0x0 0x00x0= message content,[object Object]
vmkernel/vmkwarning,[object Object],Not all vmkernel log messages appear exactly in this fashion. When a driver dumps its logging output to the vmkernel log, there is less uniform formatting involved:,[object Object],Nov 30 16:04:17 esx04 vmkernel: 28:02:20:33.356 cpu4:1720)<4>lpfc2:0749:FPe:Completed Abort Task Set Data: x0 x0 x128,[object Object],Nov 30 16:04:17 = Date and time,[object Object],esx04 = server name,[object Object],vmkernel: = logging type,[object Object],28:02:20:33.356 = host uptime,[object Object],cpu4: = cpu that trapped the message,[object Object],1720) = WID of process,[object Object],<4>lpfc2:0749:FPe:Completed Abort Task Set Data: x0 x0 x128 = driver logging (non-uniform),[object Object]
vmkernel/vmkwarning,[object Object],Here are another two driver logging examples (both are from Qlogic FC driver):,[object Object],May 13 02:02:44 esx02 vmkernel: 0:01:11:59.660 cpu1:1064)scsi(0): Waiting for LIP to complete... ,[object Object],May 13 02:02:44 esx02 vmkernel: 0:01:11:59.660 cpu0:1064)<6>qla2x00_fw_ready ha_dev_f=0xc,[object Object]
vmksummary,[object Object],The vmksummary log file is quite useful since it will log the top 3 processes running in memory at the first minute of every hour but it will also indicate if there was a bad host shutdown as well as if a PSOD occurred. This log will show if a kernel (COS or vmkernel) stops responding.,[object Object],Here is a logging example of when a simple user initiated host reboot:,[object Object],Nov  2 11:01:06 rtpesx04 logger: (1257177666) hb: vmk loaded, 11302248.49, 11302235.731, 27, 153875, 153875, 0, ftAgent-89872, vmware-h-80764, webAcces-58600,[object Object],Nov  2 11:13:50 rtpesx04 logger: (1257178430) unloaded VMkernel,[object Object],Nov  2 11:14:27 rtpesx04 vmkhalt: (1257178467) Rebooting system...,[object Object],Nov  2 13:46:13 rtpesx04 vmkhalt: (1257187573) Starting system...,[object Object],Nov  2 13:46:19 rtpesx04 logger: (1257187579) loaded VMkernel,[object Object],Nov  2 14:01:03 rtpesx04 logger: (1257188463) hb: vmk loaded, 976.32, 963.584, 16, 153875, 153875, 0, vmware-h-71508, webAcces-69084, snmpd-30204,[object Object]
vmkiscsid.log,[object Object],The vmkiscsid.log log file is a new log file as of vSphere and will only be logged to if the software initiator is used.,[object Object],2010-01-11-06:59:44: iscsid: Nop-out timedout after 10 seconds on connection 42:0 state (3). Dropping session.,[object Object],2010-01-11-06:59:47: iscsid: Kernel reported iSCSI connection 46:0 error (1008) state (3),[object Object],2010-01-11-06:59:47: iscsid: connection42:0 is operational after recovery (2 attempts),[object Object]
messages,[object Object],The format for messages is no different than that of standard logging for any Linux distribution:,[object Object],Jan 24 00:01:01 esx6 syslogd 1.4.1: restart.,[object Object],It is important to know what information we populate in this log. One such object would be the vprobs logging, a new feature introduce in vSphere:,[object Object],Jan 24 00:11:21 esx6 vobd: Jan 24 00:11:21.656: 3552646292992us: [vprob.vmfs.heartbeat.timedout] 49fdca7e-4d680d70-51f7-0015c5f29bb6 SAN006-T3-PC2-001-RP-V5.,[object Object],Jan 24 00:11:23 esx6 vobd: Jan 24 00:11:23.592: 3552648228889us: [vprob.vmfs.heartbeat.recovered] 49fdca7e-4d680d70-51f7-0015c5f29bb6 SAN006-T3-PC2-001-RP-V5.,[object Object]
Tracing a command,[object Object],Over the years we have added layers of management to our product. As a result, a single operation changes hands several times from start to finish. It is important to understand this process flow when troubleshooting why an operation fails or times out.,[object Object],The main components involved in a single operation could be the following:,[object Object],VI Client,[object Object],Virtual Center (vpxd),[object Object],SQL Database,[object Object],Host connect agent for VC (vpxa),[object Object],Hostd,[object Object],Vmkernel,[object Object],ESX Service Console,[object Object],HBAs/NICs/Physical Components of the Host,[object Object]
Tracing a command,[object Object],Here is how the process flows for a simple rescan:1. User initiates rescan in VI Client2. VI Client sends rescan request to ESX host (vpxa) 3. vpxa sends rescan request to hostd4. hostd sends request to vmkernel5. vmkernel sends rescan to HBA driver 6. HBA driver updates vmkernel with new/existing LUN information 7. vmkernel updates hostd8. hostd hands LUN information to vpxa9. vpxa updates VI Client,[object Object]
Tracing a command,[object Object],VI Client Log (C:ocuments and SettingsSERNAMEocal Settingspplication DataMwarepxiclient-#.log):,[object Object],[viclient:SoapTran] 2010-06-23 10:21:39.929  Invoke 82 Start RescanAllHba on HostStorageSystem:storageSystem-19961 [bs-tse-vc40.bsl.vmware.com]. [Caller: VpxClient.HostConfig.StorageRescanRequestManager.RescanAllHba],[object Object],[viclient:SoapTran] 2010-06-23 10:21:44.460  Invoke 82 Finish RescanAllHba on HostStorageSystem:storageSystem-19961 [bs-tse-vc40.bsl.vmware.com] - Serial:0.001, Server:004.528,[object Object],[viclient:SoapTran] 2010-06-23 10:21:44.460  Invoke 85 Start RescanVmfs on HostStorageSystem:storageSystem-19961 [bs-tse-vc40.bsl.vmware.com]. [Caller: VpxClient.HostConfig.StorageRescanRequestManager.OnSingleRescanComplete],[object Object],[viclient:SoapTran] 2010-06-23 10:21:46.241  Invoke 85 Finish RescanVmfs on HostStorageSystem:storageSystem-19961 [bs-tse-vc40.bsl.vmware.com] - Serial:0.000, Server:001.735,[object Object]
Tracing a command,[object Object],Host VC agent Log (/var/log/vmware/vpxa/vpxa.log):,[object Object],[2010-06-23 10:36:48.794 0x134cab90 info 'App'] [VpxLRO] -- BEGIN task-internal-6871 --  -- vim.host.StorageSystem.rescanAllHba -- 52dc67f5-a2d1-af98-67f1-6bdf9f335997,[object Object],[2010-06-23 10:36:50.055 0x134cab90 info 'App'] [VpxLRO] -- FINISH task-internal-6871 --  -- vim.host.StorageSystem.rescanAllHba -- 52dc67f5-a2d1-af98-67f1-6bdf9f335997,[object Object],[2010-06-23 10:36:53.354 0x13446b90 info 'App'] [VpxLRO] -- BEGIN task-internal-6873 --  -- vim.host.StorageSystem.rescanVmfs -- 52dc67f5-a2d1-af98-67f1-6bdf9f335997,[object Object],[2010-06-23 10:36:53.764 0x13446b90 info 'App'] [VpxLRO] -- FINISH task-internal-6873 --  -- vim.host.StorageSystem.rescanVmfs -- 52dc67f5-a2d1-af98-67f1-6bdf9f335997,[object Object]
Tracing a command,[object Object],Hostd Log (/var/log/vmware/hostd.log):,[object Object],[2010-06-23 10:36:48.795 1A6C2B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.host.StorageSystem.rescanAllHba-258139,[object Object],[2010-06-23 10:36:48.949 1A6C2B90 verbose 'StorageSystem'] SendStorageInfoEvent() called,[object Object],[2010-06-23 10:36:48.950 1A6C2B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores called: refresh = true, rescan = false,[object Object],[2010-06-23 10:36:48.950 1A6C2B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called,[object Object],<Snip>,[object Object],[2010-06-23 10:36:50.047 1A6C2B90 info 'TaskManager'] Task Completed : haTask-ha-host-vim.host.StorageSystem.rescanAllHba-258139 Status success,[object Object]
Tracing a command,[object Object],Hostd Log (/var/log/vmware/hostd.log) continued:,[object Object],[2010-06-23 10:36:53.355 1A6C2B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.host.StorageSystem.rescanVmfs-258143,[object Object],[2010-06-23 10:36:53.355 1A6C2B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores called: refresh = true, rescan = true,[object Object],[2010-06-23 10:36:53.355 1A6C2B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called,[object Object],[2010-06-23 10:36:53.355 1A6C2B90 verbose 'FSVolumeProvider'] RescanVmfs called,[object Object],<Snip>,[object Object],[2010-06-23 10:36:53.763 1A6C2B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores: Done discovering new filesystem volumes.,[object Object],[2010-06-23 10:36:53.764 1A6C2B90 info 'TaskManager'] Task Completed : haTask-ha-host-vim.host.StorageSystem.rescanVmfs-258143 Status success,[object Object]
Tracing a command,[object Object],VMkernel Log (/var/log/vmkernel.log):,[object Object],Jun 23 10:36:48 vmkernel: 38:01:50:35.036 cpu0:5221)ScsiScan: 846: Path 'vmhba2:C1:T9:L0': Type: 0x0, ANSI rev: 2, TPGS: 0 (none),[object Object],Jun 23 10:36:48 vmkernel: 38:01:50:35.056 cpu0:5221)ScsiScan: 843: Path 'vmhba3:C0:T1:L0': Vendor: 'DGC     '  Model: 'RAID 5          '  Rev: '0226',[object Object],<Snip>,[object Object],Jun 23 10:36:53 vmkernel: 38:01:50:39.663 cpu0:5221)Vol3: 1488: Could not open device '4bb2464a-b108d7a3-d785-000cfc0089f3' for probing: No such target on adapter,[object Object],Jun 23 10:36:53 vmkernel: 38:01:50:39.663 cpu0:5221)Vol3: 608: Could not open device '4bb2464a-b108d7a3-d785-000cfc0089f3' for volume open: No such target on adapter,[object Object],Jun 23 10:36:53 vmkernel: 38:01:50:39.663 cpu0:5221)FSS: 3702: No FS driver claimed device '4bb2464a-b108d7a3-d785-000cfc0089f3': Not supported,[object Object]
Advanced Root Cause Analysis,[object Object],Gathering Information,[object Object],Log Analysis,[object Object],Further Analysis,[object Object],Comparative Analysis,[object Object]
Qlogic FC driver messages,[object Object],Qlogic logs rather user friendly and human readable error messages. There is very little translation required when decoding these messages:,[object Object],vmkernel: 7:12:52:12.942 cpu1:1114)<6>qla2xxx_eh_abort(0): aborting sp 0x3e704e80 from RISC. pid=7417334 sp->state=2,[object Object],vmkernel: 7:12:52:12.942 cpu1:1114)<6>qla2xxx_eh_abort(0): aborting sp 0x3e704e80 from RISC. pid=7417334 sp->state=2,[object Object],vmkernel: 7:12:52:12.942 cpu1:1114)qla24xx_abort_command(0): handle to abort=735,[object Object],vmkernel: 7:12:52:12.942 cpu1:1114)<6>qla24xx_abort_command(0): handle to abort=735,[object Object],vmkernel: 7:12:52:50.315 cpu7:1066)qla2x00_mailbox_command(1): timeout calling abort_isp,[object Object],vmkernel: 7:12:52:50.315 cpu7:1066)<6>qla2x00(1): Performing ISP error recovery - ha= 0x29c3b00.,[object Object],vmkernel: 7:12:52:50.325 cpu7:1066)qla24xx_nvram_config(1) setting 24XX operation mode to =0x6 timer delay =0x1 us,[object Object]
Emulex FC driver messages,[object Object],Emulex does not take the user friendly approach however it still maintains a very high level of verbosity. It also employs a standard format that makes it easy to read and understand once you are familiar with it. ,[object Object],Emulex publishes their error codes and how to decode them online:http://www-dl.emulex.com/support/vmware/732/vmware.pdf,[object Object]
Emulex FC driver messages,[object Object],VMkernel log message example:<4>lpfc2:0749:FPe:Completed Abort Task Set Data: x0x0x128,[object Object],HBA = lpfc2,[object Object],Emulex message ID = 0749,[object Object],Driver Preamble string = FPe,[object Object],Message Description = Completed Abort Task Set,[object Object],Data field:,[object Object],SCSI ID = x0,[object Object],LUN ID = x0,[object Object],Complete time (in mS) = x128,[object Object]
Emulex FC driver messages,[object Object],Here is the same error when referenced against Emulex documentation<4>lpfc2:0749:FPe:Completed Abort Task Set Data: x0x0x128,[object Object],elx_mes0749: Cmpl abort task set,[object Object],DESCRIPTION: Abort task set completed.,[object Object],DATA: (1) scsi_id(2) lun_id(3) cmpl time mS,[object Object],SEVERITY: Information,[object Object],LOG: LOG_FCP verbose,[object Object],ACTION: None required.,[object Object],FPe = FCP traffic history (See message log table in pdf),[object Object]
Emulex FC driver messages,[object Object],Here are some other Emulex logging examples:,[object Object],<4>lpfc0:1305:LKe:Link Down Event x70 received Data: x70 x20 x20010200,[object Object],<4>lpfc1:0250:DIe:EXPIRED nodev timer Data: x10c00 x0 xb,[object Object]
Emulex FC driver messages,[object Object],Let’s review each message in the Emulex documentation:,[object Object],<4>lpfc0:1305:LKe:Link Down Event x70 received Data: x70 x20 x20010200,[object Object],Message 1305:,[object Object],elx_mes1305: Link Down Event <eventTag> received,[object Object],DESCRIPTION: A link down event was received.,[object Object],DATA: (1) fc_eventTag (2) hba_state (3) fc_flag,[object Object],SEVERITY: Error,[object Object],LOG: Always,[object Object],ACTION: If numerous link events are occurring, check the physical connections to the Fibre Channel network.,[object Object]
Emulex FC driver messages,[object Object],<4>lpfc0:0250:DIe:EXPIRED nodev timer Data: x10c00 x0 xb,[object Object],Message 0250:,[object Object],elx_mes0250: EXPIRED nodev timer,[object Object],DESCRIPTION: A device disappeared for greater than the configuration parameter,[object Object],(lpfc_nodev_tmo) seconds. All I/O associated with this device will fail.,[object Object],DATA: (1) dev_did (2) scsi_id (3) rpi,[object Object],SEVERITY: Error,[object Object],LOG: Always,[object Object],ACTION: Check physical connections to Fibre Channel network and the state of the remote PortID.,[object Object]
HBA Driver Source Code,[object Object],It is not always clear why a particular message is thrown by the driver and it may be difficult to research what the condition means either because it is not documented well or even at all.,[object Object],As the drivers we use in our kernel are based on the Linux open source code versions, we can download this source and manually search for a message/error. The Emulex errors we just reviewed are available in the source code under lpfc_logmsg.c,[object Object],The source code is available here:http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4#open_source,[object Object],* Note: The link you want is under ESX/ESXi -> OSS Source Code and is a 600M download that contains all open source packages.,[object Object]
NMP messages,[object Object],NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100010ead00) to NMP device "naa.6006048cb94fa67564932bcf676a406a" failed on physical path "vmhba33:C0:T0:L2" H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x0 0x6.,[object Object],NMP = Code Module,[object Object],nmp_CompleteCommandForPath = Code Instruction,[object Object],Command 0x2a = SCSI Command Issued,[object Object],0x4100010ead00 = Command Index,[object Object],naa.6006048cb94fa67564932bcf676a406a = LUN command issued to,[object Object],vmhba33:C0:T0:L2 = path used,[object Object],H:0x0 D:0x2 P:0x0 = Component Status,[object Object],Valid sense data: 0x3 0x0 0x6. = SCSI sense key, ASC & ASCQ info,[object Object]
NMP messages,[object Object],Let’s take a closer look at the SCSI information for that last error:,[object Object],“… failed on physical path "vmhba33:C0:T0:L2" H:0x0D:0x2P:0x0 Valid sense data: 0x30x0 0x6.”,[object Object],Host status = H:0x0 = Ok,[object Object],Device Status = D:0x2 = Check Condition,[object Object],Plugin status = P:0x0 = Ok,[object Object],SCSI Sense Key = 0x3 = MEDIUM ERROR ,[object Object],Additional Sense Code, ASC Qualifier = 0x0/0x6 =  I/O Process Terminated,[object Object]
NMP messages,[object Object],This information can be obtained from t10.org:,[object Object]
Advanced Root Cause Analysis,[object Object],Gathering Information,[object Object],Log Analysis,[object Object],Further Analysis,[object Object],Comparative Analysis,[object Object]
Log Field Data,[object Object],In the log analysis section we talked about what each field in the vmkernel log meant. Now we are going to focus on why this information is important and how you can use these values to your advantage.,[object Object],Knowing each value can help you with the following:,[object Object],Determine World ID of VM ,[object Object],How frequently events are being logged (all the time vs. every 5 minutes),[object Object],Identifying any pattern of behavior (random VMs crashing on same pcpu/core),[object Object],Which code module the message came from,[object Object],Which exact line of code the message was generated from,[object Object],If subsequent messages are directly related to each other (timestamp),[object Object]
Log Field Data: Example 1,[object Object],vmkernel.log,[object Object],Apr  8 06:09:27 esxvmkernel: 7:12:07:20.454 cpu2:1274)VSCSI: 2803: Reset request on handle 8322 (0 outstanding commands),[object Object],Apr  8 06:09:27 esxvmkernel: 7:12:07:20.454 cpu4:1061)VSCSI: 3019: Resetting handle 8322 [0/0],[object Object],Apr  8 06:09:27 esxvmkernel: 7:12:07:20.454 cpu4:1061)VSCSI: 2871: Completing reset on handle 8322 (0 outstanding commands),[object Object]
Log Field Data: Example 1,[object Object],cat /proc/vmware/vm/1274/names,[object Object],vmid=1274   pid=-1     cfgFile="/vmfs/volumes/49bec690-6c6a8788-0b1b-0019b9d670ae/NEUBOS3ES328/NEUBOS3ES328.vmx"  uuid="50 06 73 c1 c3 48 cf 28-47 ea af 1b f0 67 8e 30"  displayName="NEUBOS3ES328“,[object Object],vmware.log,[object Object],Apr 08 06:09:27.257: vcpu-0| BUSLOGIC: Soft reset 0x6cff6,[object Object],Apr 08 06:09:27.257: vcpu-0| BUSLOGIC: Bus reset 0x6cff6 (0 cif),[object Object],Apr 08 06:09:27.257: vcpu-0| BUSLOGIC: Sync reset target 0, handle 8322,[object Object],Apr 08 06:09:27.258: vcpu-0| BUSLOGIC: Adapter reset complete 0x6cff6,[object Object]
Many Components, Many Factors,[object Object],When investigating an issue in the environment, it is paramount to review the logs from multiple host or even all hosts to determine if each host saw the issue the same or differently.,[object Object],In the event of an “all hosts except one” experienced an issue scenario, reviewing the single host that saw things different is paramount however only a cross section of the other impact hosts would be required. The reversal of this is also true for a one host experienced an issue and all other hosts were Ok.,[object Object]
Time Frame,[object Object],The time frame in which an event occurred is usually critical to root cause analysis. Once that time frame has be isolated, exploration into the logs of other related components (vmkiscsi.log, array controller log, hostd, etc) should be considered a next step if the conclusions in the vmkernel log aren’t conclusive enough.,[object Object],If multiple hosts were affected by this issue, verify this time frame against the logs from other host.,[object Object],If similar log entries appear for all hosts however the time is not exact (off by well over a minute), ensure that NTP is configured on the ESX hosts and is running correctly. This applies to all components of the infrastructure (switches, array, etc),[object Object]
Conclusion,[object Object],This presentation was designed to give you insight into how a VMware Technical Support Engineer reviews logs, gathers data, and performs an in-depth analysis.,[object Object],Our hope is to show you the skills that we use every day to help you determine root cause for an issue in your environment. ,[object Object],With this core knowledge, we hope that you will become more self sufficient within your own environment and be able to diagnose an issue as it is occurring rather than after the fact.,[object Object]
Download Link,[object Object],This slide deck is available from the following link for your reference:http://ftpsite.vmware.com/download/RCA.pptx,[object Object],Contact information:Nathan SmallStaff EngineerGlobal Support ServicesVMware Incnsmall@vmware.com,[object Object]

Más contenido relacionado

La actualidad más candente

Linux Timer device driver
Linux Timer device driverLinux Timer device driver
Linux Timer device driver艾鍗科技
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringNETWAYS
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2Aero Plane
 
How to assign unowned disk in the netapp cluster 8.3
How to assign unowned disk in the netapp cluster 8.3 How to assign unowned disk in the netapp cluster 8.3
How to assign unowned disk in the netapp cluster 8.3 Saroj Sahu
 
Ak13 upgrade
Ak13 upgradeAk13 upgrade
Ak13 upgradeAccenture
 
Tegra 186のu-boot & Linux
Tegra 186のu-boot & LinuxTegra 186のu-boot & Linux
Tegra 186のu-boot & LinuxMr. Vengineer
 
Kernel_Crash_Dump_Analysis
Kernel_Crash_Dump_AnalysisKernel_Crash_Dump_Analysis
Kernel_Crash_Dump_AnalysisBuland Singh
 
Linux PCI device driver
Linux PCI device driverLinux PCI device driver
Linux PCI device driver艾鍗科技
 
CLP S7 300 E S7 400
CLP S7 300 E S7 400 CLP S7 300 E S7 400
CLP S7 300 E S7 400 confidencial
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogicAleem Shariff
 
370410176 moshell-commands
370410176 moshell-commands370410176 moshell-commands
370410176 moshell-commandsnanker phelge
 
Linux : The Common Mailbox Framework
Linux : The Common Mailbox FrameworkLinux : The Common Mailbox Framework
Linux : The Common Mailbox FrameworkMr. Vengineer
 
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...juet-y
 
Evaluation of OpenFlow in RB750GL
Evaluation of OpenFlow in RB750GLEvaluation of OpenFlow in RB750GL
Evaluation of OpenFlow in RB750GLToshiki Tsuboi
 
My old security advisories on HMI/SCADA and industrial software released betw...
My old security advisories on HMI/SCADA and industrial software released betw...My old security advisories on HMI/SCADA and industrial software released betw...
My old security advisories on HMI/SCADA and industrial software released betw...Luigi Auriemma
 
Kernel Features for Reducing Power Consumption on Embedded Devices
Kernel Features for Reducing Power Consumption on Embedded DevicesKernel Features for Reducing Power Consumption on Embedded Devices
Kernel Features for Reducing Power Consumption on Embedded DevicesRyo Jin
 

La actualidad más candente (18)

Linux Timer device driver
Linux Timer device driverLinux Timer device driver
Linux Timer device driver
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2
 
Debugging linux
Debugging linuxDebugging linux
Debugging linux
 
How to assign unowned disk in the netapp cluster 8.3
How to assign unowned disk in the netapp cluster 8.3 How to assign unowned disk in the netapp cluster 8.3
How to assign unowned disk in the netapp cluster 8.3
 
Ak13 upgrade
Ak13 upgradeAk13 upgrade
Ak13 upgrade
 
Tegra 186のu-boot & Linux
Tegra 186のu-boot & LinuxTegra 186のu-boot & Linux
Tegra 186のu-boot & Linux
 
Kernel_Crash_Dump_Analysis
Kernel_Crash_Dump_AnalysisKernel_Crash_Dump_Analysis
Kernel_Crash_Dump_Analysis
 
Linux PCI device driver
Linux PCI device driverLinux PCI device driver
Linux PCI device driver
 
CLP S7 300 E S7 400
CLP S7 300 E S7 400 CLP S7 300 E S7 400
CLP S7 300 E S7 400
 
Debugging 2013- Jesper Brouer
Debugging 2013- Jesper BrouerDebugging 2013- Jesper Brouer
Debugging 2013- Jesper Brouer
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogic
 
370410176 moshell-commands
370410176 moshell-commands370410176 moshell-commands
370410176 moshell-commands
 
Linux : The Common Mailbox Framework
Linux : The Common Mailbox FrameworkLinux : The Common Mailbox Framework
Linux : The Common Mailbox Framework
 
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...
Building a Converged Infrastructure based on FCoE, Dell Blades and Force10 sw...
 
Evaluation of OpenFlow in RB750GL
Evaluation of OpenFlow in RB750GLEvaluation of OpenFlow in RB750GL
Evaluation of OpenFlow in RB750GL
 
My old security advisories on HMI/SCADA and industrial software released betw...
My old security advisories on HMI/SCADA and industrial software released betw...My old security advisories on HMI/SCADA and industrial software released betw...
My old security advisories on HMI/SCADA and industrial software released betw...
 
Kernel Features for Reducing Power Consumption on Embedded Devices
Kernel Features for Reducing Power Consumption on Embedded DevicesKernel Features for Reducing Power Consumption on Embedded Devices
Kernel Features for Reducing Power Consumption on Embedded Devices
 

Similar a Analisis_avanzado_vmware

HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightLinaro
 
Crash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_TizenCrash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_TizenLex Yu
 
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Jagadisha Maiya
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversSatpal Parmar
 
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilitiesBlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilitiesBlueHat Security Conference
 
Writing Metasploit Plugins
Writing Metasploit PluginsWriting Metasploit Plugins
Writing Metasploit Pluginsamiable_indian
 
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...abdenour boussioud
 
Important cisco-chow-commands
Important cisco-chow-commandsImportant cisco-chow-commands
Important cisco-chow-commandsssusere31b5c
 
JomaSoft VDCF - Solaris Private Cloud
JomaSoft VDCF - Solaris Private CloudJomaSoft VDCF - Solaris Private Cloud
JomaSoft VDCF - Solaris Private CloudJomaSoft
 
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerNETWAYS
 
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopEmc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopsolarisyougood
 
Kernel debug log and console on openSUSE
Kernel debug log and console on openSUSEKernel debug log and console on openSUSE
Kernel debug log and console on openSUSESUSE Labs Taipei
 
Network Automation Tools
Network Automation ToolsNetwork Automation Tools
Network Automation ToolsEdwin Beekman
 
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335x
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335xMoksha - HTML5/CSS with Qt5+Snowshoe on AM335x
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335xPrabindh Sundareson
 
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020Eric Lin
 
How to debug ocfs2 hang problem
How to debug ocfs2 hang problemHow to debug ocfs2 hang problem
How to debug ocfs2 hang problemGang He
 

Similar a Analisis_avanzado_vmware (20)

HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with Coresight
 
Crash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_TizenCrash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_Tizen
 
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
 
Install oracle11gr2 rhel5
Install oracle11gr2 rhel5Install oracle11gr2 rhel5
Install oracle11gr2 rhel5
 
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilitiesBlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
 
Writing Metasploit Plugins
Writing Metasploit PluginsWriting Metasploit Plugins
Writing Metasploit Plugins
 
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
 
Important cisco-chow-commands
Important cisco-chow-commandsImportant cisco-chow-commands
Important cisco-chow-commands
 
Emc
EmcEmc
Emc
 
x86_1.ppt
x86_1.pptx86_1.ppt
x86_1.ppt
 
JomaSoft VDCF - Solaris Private Cloud
JomaSoft VDCF - Solaris Private CloudJomaSoft VDCF - Solaris Private Cloud
JomaSoft VDCF - Solaris Private Cloud
 
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
 
Emc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshopEmc vnx2 technical deep dive workshop
Emc vnx2 technical deep dive workshop
 
Kernel debug log and console on openSUSE
Kernel debug log and console on openSUSEKernel debug log and console on openSUSE
Kernel debug log and console on openSUSE
 
Network Automation Tools
Network Automation ToolsNetwork Automation Tools
Network Automation Tools
 
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335x
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335xMoksha - HTML5/CSS with Qt5+Snowshoe on AM335x
Moksha - HTML5/CSS with Qt5+Snowshoe on AM335x
 
C programming session10
C programming  session10C programming  session10
C programming session10
 
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020
Experience on porting HIGHMEM and KASAN to RISC-V at COSCUP 2020
 
How to debug ocfs2 hang problem
How to debug ocfs2 hang problemHow to debug ocfs2 hang problem
How to debug ocfs2 hang problem
 

Último

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 

Último (20)

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 

Analisis_avanzado_vmware

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.

Notas del editor

  1. Taken from http://www.zamaanonline.com/funny-fishing-cartoon-4026
  2. For information on the state in doubt messages, please see KB 1022026
  3. Emulex logging options can be tricky. Please refer to KB 1005576
  4. Trying to echo these options has not always proven to be successful. It may depend on driver type, version, or other factor.
  5. The Invalid metadata status indicates that the content of the heartbeat region is not correct.
  6. The Invalid metadata status indicates that the content of the heartbeat region is not correct.
  7. The Invalid metadata status indicates that the content of the heartbeat region is not correct.
  8. The exception to this type of this standard vmkernel log would be the addition of ALERT or WARNING
  9. The lock at offset 4292608 gets stolen incorrectly by another other host, thus we PSOD
  10. VI Client logs are found under C:\\Documents and Settings\\USERNAME\\Local Settings\\Application Data\\VMware\\vpx
  11. In the previous slide we saw a message that contained a value of “FPe”. When referencing it in this table we can see the description for this event is “FCP traffic history”. FCP traffic history messages would related to anything traffic related like aborts, timeouts, etc.
  12. I have downloaded this package, moved it to my home directory on scripts, and extracted it. This allows me to use tools such as grep to search for strings in the driver code.
  13. A popular Vmware blog website known as VMProfessional.com has a SCSI sense data decode utility