UCS PXE Boot

Pieces Needed to successfully PXE boot on UCS (DHCPd, TFTPd, Installation Media, PXE client):

1: DHCP server: This doc assumes the DHCP server is on same VLAN as the PXE client (else use ip-helper address config)
Option A = IOS dhcp server config:

ip dhcp pool VLAN903
network 172.23.182.0 255.255.255.0
bootfile linux-install/pxelinux.0
next-server 172.23.182.59
dns-server 171.70.168.183
domain-name cisco.com
default-router 172.23.182.1

service dhcp

Option B = RHEL (or other linux) dhcpd config:

[root@localhost etc]# more dhcpd.conf
#
# DHCP Server Configuration file.
# see /usr/share/doc/dhcp*/dhcpd.conf.sample
#
ddns-update-style interim;
ignore client-updates;

allow booting;
allow bootp;

class "pxeclients" {
match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
next-server 172.23.182.59;
filename "linux-install/pxelinux.0";
}

subnet 172.23.182.0 netmask 255.255.255.0 {

option routers 172.23.182.1;
option subnet-mask 255.255.255.0;
option domain-name "cisco.com";
option domain-name-servers 171.70.168.183;

range dynamic-bootp 172.23.182.51 172.23.182.250;
default-lease-time 21600;
max-lease-time 43200;
}
[root@localhost etc]#

Troubleshooting DHCP:

1) You might want to test this with a non-PXE DHCP client to make sure that DHCP is indeed working as expected.

2) A successful DHCP negotiation should look something like this (DHCPDISCOVER, DHCPOFFER, DHCPREQUEST, DHCPACK ('debug ip dhcp server packet' from IOS):

Feb 2 06:34:48.536 PDT: DHCPD: DHCPDISCOVER received from client 0025.b500.038a on interface Vlan903.
Feb 2 06:34:48.536 PDT: DHCPD: Sending DHCPOFFER to client 0025.b500.038a (172.23.182.136).
Feb 2 06:34:48.536 PDT: DHCPD: broadcasting BOOTREPLY to client 0025.b500.038a.
Feb 2 06:34:50.568 PDT: DHCPD: DHCPREQUEST received from client 0025.b500.038a.
Feb 2 06:34:50.568 PDT: DHCPD: Sending DHCPACK to client 0025.b500.038a (172.23.182.136).
Feb 2 06:34:50.568 PDT: DHCPD: broadcasting BOOTREPLY to client 0025.b500.038a.

2) Installation Source (http server):
http://172.23.182.59/Linux/

a. mount -o loop

/root/RHEL5.4-Server-20090819.0-x86_64-DVD.iso on /mnt/disk type iso9660 (rw,loop=/dev/loop0)

b.httpd.conf


Options Indexes
AllowOverride None

Alias /Linux /mnt/disk

c. restart httpd.
d.

[root@localhost pxeboot]# pwd
/mnt/disk/images/pxeboot

boot.iso minstg2.img README TRANS.TBL
diskboot.img pxeboot stage2.img xen


3) TFTP server:(RHEL)

http://syslinux.zytor.com/wiki/index.php/PXELINUX (sample)

This could run on the same server as the DHCP (i.e if using Linux) or not, but it should be in the same VLAN as the PXE cleints.

Set up TFTP server: We'll be PXE booting to RHEL so we'll need the RHEL files on the tftp server.

1. Verify if tftp server is installed: rpm -q tftp-server
tftp-server-0.49-2
2. tftp config file : /etc/xinetd.d/tftp

3. chmod 777 /tftpboot

4. sbin/chkconfig --level 345 xinetd on /sbin/chkconfig --level 345 tftp on (set it to boot at startup)

5. [root@localhost tftpboot]# ls -al
total 44
drwxr-xr-x 3 root root 4096 Jan 29 16:47 .
drwxr-xr-x 26 root root 4096 Jan 29 15:28 ..
drwxr-xr-x 4 root root 4096 Feb 1 13:36 linux-install
-rw-r-r- 1 root root 13148 Jan 28 14:33 pxelinux.0

6. [root@localhost tftpboot]# cd linux-install/
[root@localhost linux-install]# ls -al (these files need to be here for the linux kernel boot are copied from /mnt/disk/images/pxeboot)
total 9192
drwxr-xr-x 4 root root 4096 Feb 1 13:36 .
drwxr-xr-x 3 root root 4096 Jan 29 16:47 ..
-r-rr- 1 root root 7399936 Jan 28 14:32 initrd.img
drwxr-xr-x 2 root root 4096 Jan 29 17:02 msgs
-rw-r-r- 1 root root 13100 Dec 19 2005 pxelinux.0
drwxr-xr-x 2 root root 4096 Feb 1 11:22 pxelinux.cfg
-r-rr- 1 root root 1932316 Jan 28 14:32 vmlinuz
[root@localhost linux-install]#

7. [root@localhost linux-install]# cd pxelinux.cfg (location of pxe client config files)
[root@localhost pxelinux.cfg]# ls -al
total 100
drwxr-xr-x 2 root root 4096 Jan 30 22:35 .
drwxr-xr-x 4 root root 4096 Jan 29 16:42 ..
-rw-r-r- 1 root root 0 Jan 28 15:36 01-00-25-B5-00-03-1A (need to 'touch' empty files named as mac-address of clients with 01- prefix)
-rw-r-r- 1 root root 0 Jan 28 15:36 01-00-25-B5-00-03-1B
-rw-r-r- 1 root root 0 Jan 30 19:22 01-00-25-B5-00-03-2A
-rw-r-r- 1 root root 0 Jan 29 16:40 01-00-25-B5-00-18-FE
-rw-r-r- 1 root root 0 Jan 30 22:35 01-00-25-B5-55-03-1A
-rw-r-r- 1 root root 0 Jan 28 14:43 A
-rw-r-r- 1 root root 0 Jan 28 14:43 AC
-rw-r-r- 1 root root 0 Jan 28 14:44 AC1
-rw-r-r- 1 root root 0 Jan 28 14:44 AC17
-rw-r-r- 1 root root 0 Jan 28 14:44 AC17B
-rw-r-r- 1 root root 0 Jan 28 14:44 AC17B6
-rw-r-r- 1 root root 0 Jan 28 14:44 AC17B6F
-rw-r-r- 1 root root 0 Jan 28 14:44 AC17B6FE
-rw-r-r- 1 root root 122 Jan 28 14:36 default <<

4) PXE client:

a couple options with UCS here, LAN boot on bare metal blade/rack server, or a newly built stock ESX client will try to PXE boot

Service Profile should have LAN as first boot option.

Client should boot, get ip address, then tftp kernel and begin install. You'll then need to point to an installation source.

a. Service Profile Config - (vnic configuration - select Native VLAN if the PXE/DHCP servers are not on 6120's default vlan,

we selected native vlan for 182 because our 6120/N5k switches' native vlan was 1 and our DHCP server was in vlan 182)


Boot Order: (selected vnic)




KVM output of the server PXE booting:





Please verify the pinning to confirm the VETH is taking right path northbound.

UCS3-FI-A(nxos)# show run int veth730
version 4.0(1a)N2(1.2j)

interface vethernet730
no cdp enable
switchport trunk native vlan 182
switchport trunk allowed vlan 182
bind interface Ethernet1/1/7
pinning server sticky border-interface Ethernet1/2
pinning server pinning-failure link-down

UCS3-FI-A(nxos)# show pinning server-interfaces (Portchannel 30 is our Test lab network, which does not access to vlan 182, so the server was failing to PXE boot (veth 731 is veth of server1/8 and we have to use manual pin group))

------------------------------------+-------------
SIF Interface Sticky Pinned Border Interface Pinned Duration
------------------------------------+-------------
veth730 Yes(hard-pinned) Eth1/2 0:27:17
veth731 No Po30 0:12:35
veth732 No Po30 0:5:54
veth733 No Eth1/2 0:4:16
Eth1/20 Yes - -
Eth1/1/1 Yes - -
Eth1/1/2 Yes - -
Eth1/1/3 Yes - -
Eth1/1/4 Yes - -
Eth1/1/5 Yes - -
Eth1/1/6 Yes - -
Eth1/1/7 Yes - -
Eth1/1/8 Yes - -
Eth1/1/9 Yes - -

Total Interfaces : 14

Comments (Hide Comments)

After we pinned the interfaces,
UCS3-FI-A(nxos)# show pinning server-interfaces

---------------------------------------------------+----------------
SIF Interface Sticky Pinned Border Interface Pinned Duration
---------------------------------------------------+----------------
veth736 Yes(hard-pinned) Eth1/2 0:22:27
veth737 Yes(hard-pinned) Eth1/2 0:20:11
veth738 Yes(hard-pinned) Eth1/2 0:17:56
veth739 Yes(hard-pinned) Eth1/2 0:19:16
Eth1/20 Yes - -
Eth1/1/1 Yes - -
Eth1/1/2 Yes - -
Eth1/1/3 Yes - -
Eth1/1/4 Yes - -
Eth1/1/5 Yes - -
Eth1/1/6 Yes - -
Eth1/1/7 Yes - -
Eth1/1/8 Yes - -

UCS3-FI-A(nxos)# show run int eth 1/2
version 4.0(1a)N2(1.1e)

interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan 1,55-56,182,998
pinning border
no shutdown

interface vethernet736
switchport trunk native vlan 182
switchport trunk allowed vlan 182
bind interface Ethernet1/1/2
pinning server sticky border-interface Ethernet1/2
pinning server pinning-failure link-down

---------------------
enter service-profile pxe-server7 instance
associate server 1/7
enter boot-definition
enter lan
enter path primary
set vnic vnic1
exit
set order 1
exit
set descr ""
set reboot-on-update no
exit

enter vnic vnic1
enter eth-if vlan182
set default-net yes
exit
set adaptor-profile ""
set identity dynamic-mac 00:25:B5:00:03:7A
set identity mac-pool ""
set order 1
set pin-group vlan1
set qos-policy ""
set stats-policy default
set template-name ""
exit
power up
set boot-policy ""
set descr ""
set dynamic-vnic-conn-policy ""
set host-fw-policy 2j-2027
set identity dynamic-uuid derived
set identity uuid-suffix-pool ""
set identity wwnn-pool node-default
set ipmi-access-profile ""
set local-disk-policy default
set mgmt-fw-policy ""
set scrub-policy ""
set sol-policy default
set src-templ-name ""
set stats-policy default
exit

UCS-tech-support

Show Tech:

 

 

BMC – Bios Info

 

BIOS Complete – smbios is sent

• blade memory, boot order, cpu, memory

 

 

Collect show tech:

 

UCS1-FI-A(local-mgmt)# show tech-support ucsm detail

UCS1-FI-A(local-mgmt)# show tech chassis 1 ALL DETAIL

 

copy workspace:///techsupport/20091105202812_UCS1-FI_BC001_all.tar scp:

 

 

IBMC3:

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IBMC3/tmp}1071)ls

IBMC3_TechSupport_bios_post_results.txt

IBMC3_TechSupport_smbios.bin

IBMC3_TechSupport.txt

 

BIOS Complete Status:

1

Means it is complete.

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IBMC3/tmp}1078)more IBMC3_TechSupport.txt

 

BMC info:

ver: 1.0(1e)

process that are running

 

smbios.bin. we need to check the decode utility: dmidecode (it is on the switch on linux)

 

tar ball decode:

 

OBFL:

 

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IBMC3/obfl}1079)ls

obfl-log    obfl-log.1  obfl-log.2  obfl-log.3  obfl-log.4  obfl-log.5

 

l

ved situation

5:2009 Nov  4 22:48:15:BMC:mctool@127.6.254.1:1616: mcserver_ipmi_extensions.c:2

99:[mcserver_set_bios_boot_order]Setting BIOS Boot Order

5:2009 Nov  4 22:48:15:BMC:mctool@127.6.254.1:1616: mcserver_ipmi_extensions.c:5

45:[mcserver_set_uuid]Setting Soft UUID: e7 a6 ce d6 7d 2e 11 de ae 4b 00 0b ab

01 c0 fb

5:2009 Nov  4 22:48:15:BMC:mctool@127.6.254.1:1616: mcserver_ipmi_extensions.c:2

12:[mcserver_set_vdd_power]"Power Off"

 

:2009 Nov  5 16:20:58:BMC:IPMI:525: mcddI2CDrv.c:836:PI2CWriteRead: ioctl to dr

iver failed to read Bus[f8].Dev[30]!  ErrorStatus[fd]

0:2009 Nov  5 16:20:58:BMC:IPMI:525: mcddI2CDrv.c:836:PI2CWriteRead: ioctl to dr

iver failed to read Bus[f8].Dev[32]!  ErrorStatus[fd]

 

Who rebooted ?

IPMI command

Auto booted – Poweroff /Power On

 

/var/log/messages

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/var/log}1097)ls – messages are gone if blade reboots:

avct_server        critical           messages.1

avct_server.1      critical.1         virtual_media

avct_server.first  messages           x-remserial.log

 

 

tail x-remserial.log

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x8

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x10

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x8

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x8

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x8

Wed Nov  4 23:11:47 2009: 53de: Serial Read: 0x5

Wed Nov  4 23:11:47 2009: 53de: Net Send: 0xcd bytes

Wed Nov  4 23:12:03 2009: 53de: x-remserial is closing.

Wed Nov  4 23:41:57 2009: 5a83: x-remserial started.

Thu Nov  5 00:24:47 2009: 5a83: x-remserial is closing.

 

tail messages

6:2009 Nov  5 20:22:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:596:[mcserver_get_smbios_table]Getting SMBIOS

6:2009 Nov  5 20:22:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:647:SMBIOS Table version match. No need to send SMBIOS table back.

6:2009 Nov  5 20:24:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:596:[mcserver_get_smbios_table]Getting SMBIOS

6:2009 Nov  5 20:24:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:647:SMBIOS Table version match. No need to send SMBIOS table back.

6:2009 Nov  5 20:26:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:596:[mcserver_get_smbios_table]Getting SMBIOS

6:2009 Nov  5 20:26:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:647:SMBIOS Table version match. No need to send SMBIOS table back.

6:2009 Nov  5 20:28:23:BMC:mctool@127.5.254.1:19798: mcserver_net.c:110:New Connection from [127.5.254.1]

6:2009 Nov  5 20:28:24:BMC:mctool@127.5.254.1:19798: mcserver_show_techsupport.c:48:Received a show techsupport detailed cmd

6:2009 Nov  5 20:28:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:596:[mcserver_get_smbios_table]Getting SMBIOS

6:2009 Nov  5 20:28:37:BMC:mctool@127.5.254.1:27309: mcserver_ipmi_extensions.c:647:SMBIOS Table version match. No need to send SMBIOS table back.

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/var/log}1105)cd virtual_media/

 

Monitor

 

more avct_server

 

6:2009 Nov  5 17:51:33:BMC:avct_server:593: Video Signal detected: 720 x 400, co

lor depth is: 4 bits

6:2009 Nov  5 17:51:33:BMC:avct_server:593: VGA Text mode

6:2009 Nov  5 17:51:33:BMC:avct_server:593: AvspResolutionChange: 720 x 400

6:2009 Nov  5 17:51:33:BMC:avct_server:593: avct_setVGAPaletteMsg reports new pa

lette with 16 colors

6:2009 Nov  5 17:51:35:BMC:avct_server:593: avct_setVGAPaletteMsg reports new pa

lette with 16 colors

6:2009 Nov  5 18:01:47:BMC:avct_server:593: avct_setVGAPaletteMsg reports new pa

lette with 16 colors

 

 

CMC Tech Support

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1}1125)ls

cmc                     ipc_cm_redwood_events   proc_slabinfo.out

cores                   ipc_fru                 ps_det.out

df.out                  ipc_ohms                ps.out

dmclient_test.frus      ipc_sdr                 psreadings.out

dmclient_test.out       ipc_seeprom             redwood

fancontrol.out          ipc_sel                 release_info

free.out                ipc_sensor              root_cmd_history

fru                     ipc_thermal             sdr

fsl-i2c.1-counters.out  ipc_updated             sel

fsl-i2c.2-counters.out  lsof.out                sensor

ifconfig_a.out          netstat.out             top.out

ipc_chassis_info        proc_cpuinfo.out        uname.out

ipc_cm_local_cluster    proc_interrupts.out     uptime.out

ipc_cm_redwood          proc_meminfo.out

 

 

 

S1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/cmc}1129)ls

cms.info.atu     cms.info.info    cms.info.vtu

cms.info.global  cms.info.port    log

 

• added new blade/power changes
• why the fan changes – Thermal logs

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/cmc/log}1131)ls

• chassis_info            dmserver.5.gz           pmon
• chassis-status.xml      iom_serial_num          pmon_pstate
• cluster_manager         ipmiserver              pwrmgr
• cluster_manager.log     krphy.log               sam_connected
• cmc_manager             mctools_params          thermal
• cmc_manager.log         messages                thermal.1.gz
• cmc_ohms_errors         messages.1.gz           thermal.2.gz
• cmc_ohms_status         obfl-cmc.log            thermal.3.gz
• cmc_post_status         obfl-swupdate.log       thermal.4.gz
• critical                ohms.log                thermal.5.gz
• dmserver                peer_cmc_ohms_status    thermal.log
• dmserver.1.gz           peer_cmc_post_status    uboot_console.log
• dmserver.2.gz           peer_uboot_post_status  uboot_post_status
• dmserver.3.gz           platform_ohms
• dmserver.4.gz           platform_post

Transport to BMC And adapter,

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/redwood}1135)ls

logs                show_elog           show_oper

show_ctx            show_ilog           show_post

show_debug_satctrl  show_ints           show_sts

 

rw> Redwood POST Results:

  legend:

        '.' PASSED

        'X' FAILED

        ' ' Not Run

+-------------------+-+----+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                   |A|    | | | | | | | | | | | | | | |

|                   |S|ASIC| | |H|H|H|H|H|H|H|H|N|N|N|N|

|                   |I|LVL |C|B|I|I|I|I|I|I|I|I|I|I|I|I|

| POST Test         |C|RSLT|I|I|0|1|2|3|4|5|6|7|0|1|2|3|

+-------------------+-+----+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| 0. Register Test  |0|  . | | | | | | | | | | | | | | |

| 1. MBIST          |0|  . | | | | | | | | | | | | | | |

| 2. CI Loopback    |0|  . | | | | | | | | | | | | | | |

| 3. Serdes         |0|    | | |.|.|.|.|.|.|.|.|.|.|.|.|

| 4. PHY BIST       |0|    | | |.|.|.|.|.|.|.|.|.|.|.|.|

| 5. PRBS           |0|    | | |.|.|.|.|.|.|.|.|.|.|.|.|

| 6. PCS Loopback   |0|    | | |.|.|.|.|.|.|.|.|.|.|.|.|

| 7. IIF PRBS       |0|    | | | | | | | | | | | | | | |

| 8. Runtime Failure|0|    | | | | | | | | | | | | | | |

+-------------------+-+----+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

rw>

 

r

rw> CI /BI – CMC facing ports

ASIC 0:

+--+----+-+----+-----+-------------------------+-+

|  |    | |    |MAC  |        PHY              | |

|P | N  |A|    |-+-+-+----+-+-+-+--------+-----+ +

|o | a  |d|    | | |A|    |X| | |        |     | |

|r | m  |m|    |L|R|L|    |G|P|P|        |     |S|

|t | e  |i|Oper|C|M|G|MDIO|X|C|M|        |     |F|

|  |    |n| St |L|T|N|adr |S|S|D| u-code | Ver |P|

+--+----+-+----+-+-+-+----+-+-+-+--------+-----+-+

| 0| CI |E| Up | | | |  0 |0|0|0| n/a    | 0.00| |

| 1| BI |E| Up | | | |  0 |0|0|0| n/a    | 0.00| |

| 2| HI0|-| Dn | | |1| 18 |0|0|0| Ok     | 1.09| |

| 3| HI1|-| Dn | | |1| 19 |0|0|0| Ok     | 1.09| |

| 4| HI2|E| Up | | | | 16 |1|1|1| Ok     | 1.09| |  -- blade 5.6 is up.

| 5| HI3|E| Up | | | | 17 |1|1|1| Ok     | 1.09| |

| 6| HI4|E| Up | | | | 14 |1|1|1| Ok     | 1.09| |

| 7| HI5|E| Up | | | | 15 |1|1|1| Ok     | 1.09| | - blade 3.4

| 8| HI6|-| Dn | | |1| 12 |0|0|0| Ok     | 1.09| |

| 9| HI7|-| Dn | | |1| 13 |0|0|0| Ok     | 1.09| |  -- blade 0

|10| NI0|E| Dn |1| | | 23 |1|0|0| Ok     | 1.39|*|

|11| NI1|E| Dn |1| | | 22 |1|0|0| Ok     | 1.39|*|

|12| NI2|E| Dn |1| | | 21 |1|0|0| Ok     | 1.39|*|

|13| NI3|E| Up | | | | 20 |1|1|1| Ok     | 1.39|*|

+--+----+-+----+-+-+-+----+-+-+-+--------+-----+-+

 

 

        +---+----+----+----+

  SFP:  |[$]| [$]| [$]| [$]|

        +---+----+----+----+

          :    :    :    |

        +-+----+----+----+-+

        | 0    1    2    3 |

        | I    I    I    I |

        | N    N    N    N |

        |                  |

        |      ASIC 0      |

        |                  |

        | H H H H H H H H  |

        | I I I I I I I I  |

        | 0 1 2 3 4 5 6 7  |

        +-+-+-+-+-+-+-+-+--+

          - - | | | | - -

         +-+-+-+-+-+-+-+-+

         |-|-|v|v|v|v|-|-|

         +-+-+-+-+-+-+-+-+

Blade:    8 7 6 5 4 3 2 1

HI0 – blade 8

HI1- blade 7

 

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/redwood}1142)more show_elog

rw> 380422.431 - 0-HI4: Admin state changed to Dsbl

380422.458 - 0-HI4: Oper state changed to Dn

380425.282 - 0-HI4: Admin state changed to Enbl

380425.339 - 0-HI4: Admin state changed to Dsbl

380428.257 - 0-HI4: Admin state changed to Enbl

380429.379 - 0-HI4: Oper state changed to Up

380438.419 - 0-HI4: Oper state changed to Dn

380438.458 - 0-HI4: Admin state changed to Dsbl

380441.193 - 0-HI4: Admin state changed to Enbl

 

Detailed States

 

rw> ASIC: 0:

+-------+--------------------------+--------------+-----------+-----------+-----

------+

| ASIC  | Interrupt Bit Field      |    Count1    |  Thresh1  |  Count2   |  Thr

esh2  |

| Port  |                          |              |           |           |

      |

+-------+--------------------------+--------------+-----------+-----------+-----

------+

| 0-HI2 | not_synced_lane_3        |            9 |         0 |         0 |

    1 |

| 0-HI2 | not_synced_lane_2        |           26 |         0 |         0 |

    1 |

| 0-HI2 | not_synced_lane_1        |           15 |         0 |         0 |

    1 |

| 0-HI2 | not_synced_lane_0        |           11 |         0 |         0 |

    1 |

| 0-HI2 | synced_lane_3            |           28 |         0 |         0 |

    1 |

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/cmc/log}1153)ls

chassis_info            dmserver.5.gz           pmon

chassis-status.xml      iom_serial_num          pmon_pstate

cluster_manager         ipmiserver              pwrmgr

cluster_manager.log     krphy.log               sam_connected

cmc_manager             mctools_params          thermal

cmc_manager.log         messages                thermal.1.gz

cmc_ohms_errors         messages.1.gz           thermal.2.gz

cmc_ohms_status         obfl-cmc.log            thermal.3.gz

cmc_post_status         obfl-swupdate.log       thermal.4.gz

critical                ohms.log                thermal.5.gz

dmserver                peer_cmc_ohms_status    thermal.log

dmserver.1.gz           peer_cmc_post_status    uboot_console.log

dmserver.2.gz           peer_uboot_post_status  uboot_post_status

dmserver.3.gz           platform_ohms

dmserver.4.gz           platform_post

 

Each process has the status, u need to look specifically

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/cmc/log}1154)more ohms.log

Logfile: /var/cmc/log/ohms.log

 

EEPROM Log:

 

cmc slot number 0

signature                      : 0xa5a50000

version                        : 1

uboot post status              : 0x20000000

cmc post status                : 0x01000000

cmc ohms status                : 0x00000040

chassis ok led status          : ON

chassis fault led status       : OFF

cpu single ecc errors          : 0

sysinfo ok                     : 1

uptime                         : 6 days, 22:36

total memory                   : 256632 kB

free memory                    : 148904 kB

process count                  : 76

load average                   : 1.89, 1.52, 1.42

 

peer signature                 : 0xb5b50001

peer version                   : 1

peer uboot post status seeprom : 0x00000000

peer uboot post status         : 0x00000000

peer cmc post status seeprom   : 0x00000000

peer cmc post status           : 0x00000000

peer cmc ohms status           : 0x00000000

peer chassis ok led status     : ON

peer chassis fault led status  : OFF

peer cpu single ecc errors     : 0

peer sysinfo ok                : 1

peer uptime                    : 9 days,  2:04

peer total memory              : 256632 kB

peer free memory               : 155048 kB

peer process count             : 76

peer load average              : 2.03, 1.37, 1.27

 

Chassis OHMS status          cmc0       cmc1

master                     : -          1

cmc ohms status            : 0x00000040 0x00000000

cpu error                  : 0          0

memory error               : 0          0

memory controller error    : 0          0

selected image error       : 0          0

alternate image error      : 0          0

i2c bus 0 error            : 0          0

i2c bus 1 error            : 1          0

i2c bus 1 master error     : 0          0

cpu mdio bus error         : 0          0

cpu interrupt error        : 0          0

cpu kernel crash           : 0          0

user process restart       : 0          0

cpu low memory             : 0          0

obfl error                 : 0          0

serial link error          : 0          0

cpu tsec1 eth error        : 0          0

cpu tsec2 eth error        : 0          0

inlet 1 temp sensor error  : 0          0

inlet 2 temp sensor error  : 0          0

redwood temp sensor error  : 0          0

pca9539 hub error          : 0          0

iom fru error              : 0          0

chassis fru error          : 0          0

chassis seeprom error      : 0          0

fan error                  : 0          0

minor thermal error        : 0          0

minor power error          : 0          0

cms error                  : 0          0

bmc error                  : 0          0

sam error                  : 0          0

major thermal error        : 0          0

major power error          : 0          0

 

Error Counts:

Error   Count   Description

  274       2   error reading fan device id

  275       2   error reading fan speed

 

Peer Error Counts:

Error   Count   Description

 

I2C device driver statistics:

Name             Addr     Success  IO error  NotFound   Timeout      Busy Interr

upt   Refused  TooLarge

iom-fru          0-0050         c         0         0         0         0

  0         0         0

c.fru            1-0012         2         0         0         0         0

  0         0         0

c.seeprom        1-0014      2607         0         0         0         0

  0         0         0

c.ms             1-0013         0         0     

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1/cmc/log}1158)more thermal

2009 Nov  5 20:20:01 UCS1-FI BC01_IOM01_thermal-3-QCI1326002U  0:dm_get_fan_duty

cycle:Error send/recv req, error: Operation not permitted

2009 Nov  5 20:20:01 UCS1-FI BC01_IOM01_thermal-3-QCI1326002U  0:dm_get_fan_duty

cycle:Error send/recv req, error: Operation not permitted

2009 Nov  5 20:20:01 UCS1-FI BC01_IOM01_thermal-5-QCI1326002U  0:saratoga_fan_po

licy:F5 no tach reading from ipmi, trying dmserver

2009 Nov  5 20:20:01 UCS1-FI BC01_IOM01_thermal-3-QCI1

 

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1}1162)ls

cmc                     ipc_cm_redwood_events   proc_slabinfo.out

cores                   ipc_fru                 ps_det.out

df.out                  ipc_ohms                ps.out

dmclient_test.frus      ipc_sdr                 psreadings.out

dmclient_test.out       ipc_seeprom             redwood

fancontrol.out          ipc_sel                 release_info

free.out                ipc_sensor              root_cmd_history

fru                     ipc_thermal             sdr

fsl-i2c.1-counters.out  ipc_updated             sel

fsl-i2c.2-counters.out  lsof.out                sensor

ifconfig_a.out          netstat.out             top.out

ipc_chassis_info        proc_cpuinfo.out        uname.out

ipc_cm_local_cluster    proc_interrupts.out     uptime.out

ipc_cm_redwood          proc_meminfo.out

 

Internal Chassis Ethernet status

 

more ifconfig_a.out

 

Summary system:

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1}1167)more dmclient_test.out

Last scan time                          : 599803

Chassis-id                              : 1

Fabric-id                               : 2

Cluster-id                              : 335c22fe-b901-11de-9f0d-000decd53a04

Slot id                                 : 0

Amber LED status                        : ON

Green LED status                        : OFF

Chassis ok LED status                   : ON

Chassis fault LED status                : ON

Locate LED status                       : OFF

Locate buttion status                   : 0

Backplane status                        : 1

Blades present                          : 2 3 4 5

Blades powered on                       : 0 1 2 3 4 5 6 7

Blades alerted                          : 2 3 4 5

Fans present                            : 0 1 2 3 4 5 6 7

Fans alerted                            : 0 1 2 3 4 5 6 7

PSs present                             : 0 2

PSs RMT on                              : 0 1 2 3

PS DC ok                                : 0 2

PS AC ok                                : 0 2

SFPs present                            :

Redwood pll status                      : 0 1 2

Redwood status                          : 0

IBMC Link status                        : 2 3 4 5

uBoot post status                       : 0x20000000

Peer uBoot post status                  : 0x00000000

CMC post status                         : 0x01000000

Peer CMC post status                    : 0x00000000

Health status                           : 0x00000040

Peer Health status                      : 0x00000000

Number of ethernet addresses            : 10

Start IOM ethernet address              : 00:26:51:08:37:be

Eth0 MAC:                               : 00:26:51:08:37:be

Eth1 MAC                                : 00:26:51:08:37:bf

HIF state                               : 2 3 4 5   ------ blade 3,4,5,6 are up..

dmserver init status                    : 1

gilroy status (is hung)                 : 0

 

 

Current Fan status:

S1-FI_BC001_all/IOCard1/techsupport_detailed_iocard1}1168) more fancontrol.out

Fan 0 dutycycle: 30

Fan 1 dutycycle: 30

Fan 2 dutycycle: 30

Fan 3 dutycycle: 30

Fan 6 dutycycle: 30

Fan 7 dutycycle: 30

MEZZ Card status:

• Menlo limited space.

Palo Card sends a lot more data.

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105202812_UCS1-FI_BC001_all}1176)ls

IBMC3                            MEZZ31_TechSupport.txt

IBMC3_TechSupport.tar            MEZZ31_TechSupport.txt.done

IBMC3_TechSupport.tar.gz.done    MEZZ41_TechSupport.txt

IBMC4_TechSupport.tar.gz         MEZZ41_TechSupport.txt.done

IBMC4_TechSupport.tar.gz.done    MEZZ51_TechSupport.txt

IBMC5_TechSupport.tar.gz         MEZZ51_TechSupport.txt.done

IBMC5_TechSupport.tar.gz.done    MEZZ61_TechSupport.txt

IBMC6_TechSupport.tar.gz         MEZZ61_TechSupport.txt.done

IBMC6_TechSupport.tar.gz.done    mnt

IOCard1                          obfl

IOCard1_TechSupport.tar          tmp

IOCard1_TechSupport.tar.gz.done  usr

IOCard2_TechSupport.tar.gz       var

IOCard2_TechSupport.tar.gz.done

 

 

S1-FI_BC001_all}1177)more MEZZ31_TechSupport.txt

Current Firmware Image Verion 1.0(1e)

Backup Firmware Image (GOOD) Version 1.0(1e)

 

Menlo Configuration:

        Uplink Port 0 Mac Address: 00:26:51:08:83:d6

        Uplink Port 1 Mac Address: 00:26:51:08:83:d7

 

 

 

Link States:

        Port Physical Logical

        uif0        1       1

        uif1        1       1

        eth0        1       1

        eth1        1       1

         fc0        2       2

         fc1        2       2

 

 

VIF Information:

Log VIC

        [0000] 00000:01:17:09:810 vif[2]: veth785: s:STANDBY_UP_ACTIVATE(e:ACTIV

ATE_SUCCESS)->s:A

        [0001] 00000:01:17:09:810 active_vif_up: port 0 veth785 primary 1

 

        [0002] 00000:01:17:09:810 vic_set_ratelimit: samindex 785 pif 0 rate 0xf

fffffff burst 0xf

        [0003] 00000:01:17:09:810 failover-rec-complete: port 0 uif 0 p veth785

s veth786

        [0004] 00000:01:17:04:000 vif[7]: vfc761: s:INIT(e:CREATE)->s:CREATE

 

        [0005] 00000:01:17:04:000 create: port 0 vfc761 primary 1

 

        [0006] 00000:01:17:04:000 vif[2]: veth785:

 

[0105] 00000:01:17:03:500 Egress FLOGI: port 1, wwpn 2000000000010130, o

xid 51e

        [0106] 00000:01:17:03:550 FLOGI LS_ACC: port 1, ox_id 0x51e, d_id 0xef00

13

 

s:CREATE(e:ENABLE_STANDBY)->s

:ENABLE_STANDBY

 

Palo Card has more data: as it has a linux running there.

Config, debugdump and obfl/var

 

Config – Fruid/vnic.cfg

 

 

 

SAM tech support: show tech ucsm detail

 

With show tech support of N5k.

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM}1203)

 

df_a.out                        sam_process_state

dhcpd.leases                    sam_techsupportinfo

dmesg.out                       sw_techsupportinfo

ifconfig_a.out                  UCSM_A_TechSupport.tar

iptables.out                    UCSM_A_TechSupport.tar.gz.done

isan                            UCSM_B_TechSupport.tar.gz

ls_l.out                        UCSM_B_TechSupport.tar.gz.done

opt                             var

sam_cluster_state

 

• Switch tech support – see if we can talk to the nic. Vnic up ?

(sw-tech-support)

 

Sam tech support – config of UCS

 

Var/sysmgr/sam_logs

 

Need to look for primary log – connection info – managing instance – A or B.

 

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM/var/sysmgr/sam_logs}1212)ls

auto_update.log               svc_sam_controller.log

error_log.1257380923          svc_sam_dcosAG.log

httpd.log                     svc_sam_dme.log

load_warnings.log             svc_sam_extvmmAG.log

svc_sam_bladeAG.log           svc_sam_hostagentAG.log

svc_sam_bladeAG.log.1.gz      svc_sam_nicAG.log

svc_sam_bladeAG.log.2.gz      svc_sam_nicAG.log.1.gz

svc_sam_bladeAG.log.3.gz      svc_sam_nicAG.log.first.gz

svc_sam_bladeAG.log.4.gz      svc_sam_pamProxy.log

svc_sam_bladeAG.log.first.gz  svc_sam_portAG.log

svc_sam_cliD.log

 

dhcp.leases - each blades ip address for pnuos

iptables.out –kvm

 

Chain FORWARD (policy DROP 0 packets, 0 bytes)

pkts bytes target     prot opt in     out     source               destination

   57  5244 ACCEPT     icmp --  *      *       0.0.0.0/0            127.5.1.3

9541  551K ACCEPT     tcp  --  *      *       0.0.0.0/0            127.5.1.3           tcp dpt:2068  << kvm

    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            127.5.1.3           udp dpt:623 - ipmi

   54  5295 ACCEPT     tcp  --  *      *       0.0.0.0/0            127.5.1.3           tcp dpt:22 -- ssh

   57  5244 ACCEPT     icmp --  *      *       0.0.0.0/0            12

 

These three ports should be up in northbound ound.

 

SAM Cluster state

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM}1224)more sam_cluster_state

Cluster Id: 0x335c22feb90111de-0x9f0d000decd53a04

 

Start time: Thu Nov  5 00:28:36 2009

Last election time: Thu Nov  5 00:28:43 2009

 

A: UP, PRIMARY

B: UP, SUBORDINATE

 

A: memb state UP, lead state PRIMARY, mgmt services state: UP

B: memb state UP, lead state SUBORDINATE, mgmt services state: UP

   heartbeat state PRIMARY_OK

 

INTERNAL NETWORK INTERFACES:

eth1, UP

eth2, UP

 

HA READY

Chassis detailed state:

Chassis, serial: FOX1325G5F5, state: active

 

Process state

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM}1225)more sam_process_state

 

SERVICE NAME             STATE     RETRY(MAX)    EXITCODE    SIGNAL    CORE

------------             -----     ----------    --------    ------    ----

svc_sam_controller     running           0(4)           0         0      no

svc_sam_dme            running           0(4)           0         0      no

svc_sam_dcosAG         running           0(4)           0         0      no

svc_sam_bladeAG        running           0(4)           0         0      no

svc_sam_portAG         running           0(4)           0         0      no

svc_sam_hostagentAG    running           0(4)           0         0      no

svc_sam_nicAG          running           0(4)           0         0      no

svc_sam_extvmmAG       running           0(4)           0         0      no

httpd                  running           0(4)           0         0      no

svc_sam_cliD           running           0(4)           0         0      no

svc_sam_pamProxy       running           0(4)           0         0      no

sfcbd                  running           0(4)           0         0      no

dhcpd                  running           0(4)           0         0      no

sam_core_mon           running           0(4)           0         0      no

 

httpd can;’t service if the cluster is not in good state.

last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM/isan/apache/logs}1235)more httpd.pid

3108

SW_tech_support

 

 

SAM_techsupport

Server 1/3:

            Equipped Product Name: Cisco B200-M1

            Equipped PID: N20-B6620-1

            Equipped Serial (SN): QCI133000NM

            Slot Status: Equipped

            Acknowledged Product Name: Cisco B200-M1

            Acknowledged PID: N20-B6620-1

            Acknowledged Serial (SN): QCI133000NM

            Acknowledged Memory (MB): 49152

            Acknowledged Effective Memory (MB): 49152

            Acknowledged Cores: 8

            Acknowledged Adapters: 1

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM}1243)more sw_techsupportinfo

 

`show tech-support details`

`show switchname`

UCS1-FI-A

 

 

Nexus 5k tech support details of 6120

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM/UCSM_A/var/sysmgr/sam_logs}1264)ls

auto_update.log               svc_sam_bladeAG.log.2.gz      svc_sam_dcosAG.log            svc_sam_nicAG.log.first.gz

error_log.1257380923          svc_sam_bladeAG.log.3.gz      svc_sam_dme.log               svc_sam_pamProxy.log

httpd.log                     svc_sam_bladeAG.log.4.gz      svc_sam_extvmmAG.log          svc_sam_portAG.log

load_warnings.log             svc_sam_bladeAG.log.first.gz  svc_sam_hostagentAG.log

svc_sam_bladeAG.log           svc_sam_cliD.log              svc_sam_nicAG.log

svc_sam_bladeAG.log.1.gz      svc_sam_controller.log        svc_sam_nicAG.log.1.gz

 

 

• host agent logs  - deployment activity of all the blades will be there.

[INFO][0xb6537bb0][Nov  5 19:41:17.063][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'POWER_ON_FAIL'

[INFO][0xb6537bb0][Nov  5 19:41:17.064][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got POWER_ON_FAIL sensorNumber : 95

[INFO][0xb6537bb0][Nov  5 19:41:17.065][mcAG:debugSensorBits]   [mc: 7F050103 19] Sensor state:value 15:1 16:0

[INFO][0xb6537bb0][Nov  5 19:41:17.067][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'IOH_THERMTRIP_N'

[INFO][0xb6537bb0][Nov  5 19:41:17.067][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got IOH_THERMTRIP_N sensorNumber : 24

[INFO][0xb6537bb0][Nov  5 19:41:17.069][mcAG:debugSensorBits]   [mc: 7F050103 19] Sensor state:value 66:1 67:0

[INFO][0xb6537bb0][Nov  5 19:41:17.071][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'CPUS_PRCHT_N'

[INFO][0xb6537bb0][Nov  5 19:41:17.071][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got CPUS_PRCHT_N sensorNumber : 31

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:readingCb]         [mc: 7F050105 1B] readingCb: mcHandle = 0x1b, type = 3, le

n = 95

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:readingCb]         [mc: 7F050105 1B] mc_reading_cb_t request from MC client,

sending McEvent instance 0x8586618/231

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:getDeviceAddress]  getDeviceAddress for McEvent instance 0x8586618 returning

a device address instance (nil) for IP 7F050105

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:getDeviceAddress]  getDeviceAddress for McEvent instance 0x8586618 returning

a device address instance 0x8209b70 for IP 7F050105

 

what blade AG and dme sent Boot order is ?

 

 

keywords in dmelog: Critical is the keyword.

 

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM/UCSM_A/var/sysmgr/sam_logs}1274)grep CRIT svc_sam_dme.log

jejoseph@last-call-5{/home/jejoseph/personal_html/SAN/UCS/tech/20091105201437_UCS1-FI_UCSM/UCSM_A/var/sysmgr/sam_logs}1275)

 

O][0xb6537bb0][Nov  5 19:41:17.063][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'POWER_ON_FAIL'

[INFO][0xb6537bb0][Nov  5 19:41:17.064][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got POWER_ON_FAIL sensorNumber : 95

[INFO][0xb6537bb0][Nov  5 19:41:17.065][mcAG:debugSensorBits]   [mc: 7F050103 19] Sensor state:value 15:1 16:0

[INFO][0xb6537bb0][Nov  5 19:41:17.067][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'IOH_THERMTRIP_N'

[INFO][0xb6537bb0][Nov  5 19:41:17.067][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got IOH_THERMTRIP_N sensorNumber : 24

[INFO][0xb6537bb0][Nov  5 19:41:17.069][mcAG:debugSensorBits]   [mc: 7F050103 19] Sensor state:value 66:1 67:0

[INFO][0xb6537bb0][Nov  5 19:41:17.071][mcAG:fetchSensorValue]  [mc: 7F050103 19] 'CPUS_PRCHT_N'

[INFO][0xb6537bb0][Nov  5 19:41:17.071][mcAG:fetchSensorValue]  [mc: 7F050103 19] Got CPUS_PRCHT_N sensorNumber : 31

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:readingCb]         [mc: 7F050105 1B] readingCb: mcHandle = 0x1b, type = 3, le

n = 95

[INFO][0xb5a37bb0][Nov  5 19:41:17.075][mcAG:readingCb]         [mc: 7F050105 1B] mc_reading_cb_t request from MC client,

sending McEvent instance 0x8586618/231

 

 

7F050103 --- ip address

 

Port ag manages only the switch:

grep Executing svc_sam_portAG.log|more – which commands are executed in the switch:

 

[INFO][0xb68b0080][Nov  5 00:28:41.971][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:28:41.973][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:28:52.043][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:28:52.100][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:29:02.144][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:29:02.150][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:29:12.614][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:29:12.647][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:29:22.793][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:29:22.807][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:29:32.855][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68b0080][Nov  5 00:29:32.864][app_sam_portAG:SC_Exec] Executing command: interface ethernet 1/1

[INFO][0xb68b0080][Nov  5 00:29:32.889][app_sam_portAG:SC_Exec] Executing command: exit

[INFO][0xb68b0080][Nov  5 00:29:37.936][app_sam_portAG:SC_Exec] Executing command: configure terminal

[INFO][0xb68afbb0][Nov  5 00:29:58.002][app_sam_portAG:SC_Exec] Executing command: feature port-security

[INFO][0xb68afbb0][Nov  5 00:29:59.778][app_sam_portAG:SC_Exec] Executing command: feature fcoe

 

 

 

Steps in debugging:

 

1. operational setup
2. Datapath-Issues

 

Debugging SAM/setup issues

• FSM stuck at
• SAM talks to single end-point
• Look at dme logs
• Look at AG Logs – no end point connection/ always look at switch and redwood

(are the ports are up)

• Always at end-point
• -deployment failure (invalid config)

 

• configured vnic failed – creation of vnic failed, few lines before failure on port AG

 

Given object there can be only one FSM operational in an object at the same time.

Rest gets scheduled.

 

 --- show tech debugging

 

 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

BMC Cut Thro Interfaces:

BMC main feature – Server monitoring/interaction BIOS

Server Management – power/boot-order/Serial over lan, kvm/Virt Media services

Int. platform management Interface – IPMI

BMC runs ipmi server – IPMI Device interface – host OS can talk to BMC

KCS interface – keyboard interface- for BIOS

BMC-IP is there.

 

Ipmi v 2.0

 

LIVE TROUBLESHOOTING:

 

Mezz connection:

UCS1-FI-A# connect adapter 1/3/1

 

Available commands:

  exit                - Exit from subshell

  help                - List available commands

  history             - Show command history

  show-asic-stats     - Show adapter's asic stats

  show-cfg            - Show adapter's configuration

  show-debug-log      - Show adapter's debug log

  show-fwlist         - Show firmware versions on the adapter

  show-identity       - Show adapter identity

  show-memory         - Show adapter's memory

  show-panic-log      - Show adapter's panic log

  show-phyinfo        - Show adapter phy info

  show-port-stats     - Show adapter's port stats

  show-systemstatus   - Show adapter status

  show-vif-stats      - Show adapter's vif stats

  show-vifs           - Show adapter's vifs

adapter 1/3/1 #show-vifs

-------------+

| veth785        |  Up | 0 | 0 |     1 |     0 |   3 | 0 | 00:25:b5:01:01:3b                 |

| veth783        |  Up | 1 | 1 |     1 |     1 |   3 | 0 | 00:25:b5:11:01:3a                 |

| veth784        |  Sb | 1 | 0 |     1 |     1 |   3 | 0 | 00:25:b5:11:01:3a                 |

| veth786        |  Sb | 0 | 1 |     1 |     0 |   3 | 0 | 00:25:b5:01:01:3b                 |

|  vfc762        |  Up | 3 | 1 |     1 |     3 |   3 | 3 | 2000000001010130:2000000000010130 |

|  vfc761        |  Up | 2 | 0 |     1 |     2 |   3 | 3 | 2000000001010130:2000000000010131 |

 

ASIC stats:

 

|  NET EG STAT               |  CURRENT             |DIFF             |  NET IG STAT                |  CURRENT             |DIFF             | +----------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | EG0_LEARN_REQ_DROP         |                     0|                0| IG0_FWD_LOOKUP_NO_HIT       |                     0|                0| | EG0_PKT_DROP_CMD           |                     0|                0| IG0_PKT_DROP_FC_MCAST       |                     0|                0| | EG0_PKT_DROP_LIFCFG_INVALID|                     0|                0| IG0_PKT_DROP_INVALID_FC_LIF |                     0|                0| | EG0_PKT_DROP_LIFMAP_NO_HIT |                     0|                0| IG0_PKT_NULL_PIF            |                  2158|                0| | EG0_PKT_DROP_SRC_BIND      |                     0|                0|                             |                      |                 | +----------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | EG1_LEARN_REQ_DROP         |                     0|                0| IG1_FWD_LOOKUP_NO_HIT       |                     0|                0| | EG1_PKT_DROP_CMD           |                     0|                0| IG1_PKT_DROP_FC_MCAST       |                     0|                0| | EG1_PKT_DROP_LIFCFG_INVALID|                     0|                0| IG1_PKT_DROP_INVALID_FC_LIF |                     0|                0| | EG1_PKT_DROP_LIFMAP_NO_HIT |                     0|                0| IG1_PKT_NULL_PIF            |                   221|                0| | EG1_PKT_DROP_SRC_BIND      |                     0|                0|                             |                      |                 | +----------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ +----------------------------------------------------------------------------------------------------------------------------------------------+ |                                                   PAUSE STATS                                                                                | +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ |  PAUSE TX STAT               |  CURRENT             | DIFF            |  PAUSE RX STAT              |  CURRENT             | DIFF            | +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | HOST10GBE_PORT0_TX_PAUSE_CFC |                     0|                0| HOST10GBE_PORT0_RX_PAUSE_CFC|                     0|                0| | HOST10GBE_PORT0_TX_PAUSE_PFC |                     0|                0| HOST10GBE_PORT0_RX_PAUSE_PFC|                     0|                0| +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | HOST10GBE_PORT1_TX_PAUSE_CFC |                     0|                0| HOST10GBE_PORT1_RX_PAUSE_CFC|                     0|                0| | HOST10GBE_PORT1_TX_PAUSE_PFC |                     0|                0| HOST10GBE_PORT1_RX_PAUSE_PFC|                     0|                0| +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | DCE_PORT0_TX_PAUSE_CFC       |                     0|                0| DCE_PORT0_RX_PAUSE_CFC      |                     0|                0| | DCE_PORT0_TX_PAUSE_PFC       |                     0|                0| DCE_PORT0_TX_PAUSE_PFC      |                     0|                0| +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ | DCE_PORT1_TX_PAUSE_CFC       |                     0|                0| DCE_PORT1_RX_PAUSE_CFC      |                     0|                0| | DCE_PORT1_TX_PAUSE_PFC       |                     0|                0| DCE_PORT1_TX_PAUSE_PFC      |                     0|                0| +------------------------------+----------------------+-----------------+-----------------------------+----------------------+-----------------+ adapter 1/3/1 # bv

 

 

Connect iom 1

 

fex-1# show platform software redwood rate

+-------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+

| Port  || Tx Packets |  Tx Rate  |   Tx Bit   || Rx Packets |  Rx Rate  |   Rx Bit   |Avg Pkt|Avg Pkt|   |

|       ||            |  (pkts/s) |    Rate    ||            |  (pkts/s) |    Rate    | (Tx)  | (Rx)  |Err|

+-------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+

| 0-NI3 ||        211 |        42 | 414.94Kbps ||        148 |        29 |  25.38Kbps |  1229 |   107 |   |

| 0-HI5 ||         16 |         3 |   2.40Kbps ||          0 |         0 |   0.00 bps |    93 |     0 |   |

| 0-HI4 ||         90 |        18 |  13.89Kbps ||        151 |        30 | 393.39Kbps |    96 |  1628 |   |

| 0-HI3 ||         18 |         3 |   2.63Kbps ||          0 |         0 |   0.00 bps |    91 |     0 |   |

| 0-HI2 ||         22 |         4 |   3.54Kbps ||          1 |         0 | 240.00 bps |   100 |   150 |   |

| 0-BI  ||         36 |         7 |   5.37Kbps ||         41 |         8 |   8.05Kbps |    93 |   122 |   |

| 0-CI  ||         31 |         6 |   6.68Kbps ||         28 |         5 |  17.97Kbps |   134 |   401 |   |

+-------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+

 

 

fex-1# show platform software redwood sts

Board Status Overview:

legend:

        ' '= no-connect

        X  = Failed

        -  = Disabled

        :  = Dn

        |  = Up

        $  = SFP+ present

        v  = Blade Present

------------------------------

 

        +---+----+----+----+

  SFP:  |[$]| [$]| [$]| [$]|

        +---+----+----+----+

          :    :    :    |

        +-+----+----+----+-+

        | 0    1    2    3 |

        | I    I    I    I |

        | N    N    N    N |

        |                  |

        |      ASIC 0      |

        |                  |

        | H H H H H H H H  |

        | I I I I I I I I  |

        | 0 1 2 3 4 5 6 7  |

        +-+-+-+-+-+-+-+-+--+

• - | | | | - -
• +-+-+-+-+-+-+-+-+
•          |-|-|v|v|v|v|-|-|
•          +-+-+-+-+-+-+-+-+
• Blade:    8 7 6 5 4 3 2 1

 

 

fex-1# show platform software redwood phy list 5221

BCM5221_CONTROL               : [0x00000000]

BCM5221_STATUS                : [0x00000001]

BCM5221_PHY_ID_HI             : [0x00000002]

BCM5221_PHY_ID_LO             : [0x00000003]

BCM5221_AUTO_NEG_ADV          : [0x00000004]

BCM5221_LINK_PARTNER_ABILITY  : [0x00000005]

BCM5221_AUTO_NEG_EXPANSION    : [0x00000006]

BCM5221_NEXT_PAGE             : [0x00000007]

BCM5221_NEXT_PAGE             : [0x00000008]

BCM5221_100BX_AUX_CONTROL     : [0x00000010]

BCM5221_100BX_AUX_STATUS      : [0x00000011]

BCM5221_100BX_RCV_ERROR_CNT   : [0x00000012]

BCM5221_100BX_RCV_ERROR_CNT   : [0x00000013]

BCM5221_100BX_DISCONNECT_CNT  : [0x00000014]

BCM5221_PTEST                 : [0x00000017]

BCM5221_AUX_CONTROL_STATUS    : [0x00000018]

BCM5221_AUX_STATUS_SUMMARY    : [0x00000019]

BCM5221_INTERRUPT             : [0x0000001a]

BCM5221_AUX_MODE2             : [0x0000001b]

BCM5221_10BT_AUX_GEN_STATUS   : [0x0000001c]

BCM5221_AUX_MODE              : [0x0000001d]

BCM5221_AUX_MULTI_PHY         : [0x0000001e]

BCM5221_BROADCOM_TEST         : [0x0000001f]

BCM5221_AUX_MODE4             : [0x0000001a]

BCM5221_AUX_STATUS2           : [0x0000001b]

BCM5221_AUX_STATUS3           : [0x0000001c]

BCM5221_AUX_MODE3             : [0x0000001d]

 

 

fex-1# show platform software redwood phy list 54980

CM54980_PHY_EXT_CONTROL       : [0x00000010]           <cfg>

BCM54980_PHY_EXT_STATUS        : [0x00000011]           <cfg>

BCM54980_RX_ERR_CNT            : [0x00000012]           <cfg>

BCM54980_TX_ERROR_CODE_CNT     : [0x00000013]           <cfg>

BCM54980_RX_ERROR_CODE_CNT     : [0x00000014]           <cfg>

BCM54980_EXP_ACCESS_REG        : [0x00000017]           <cfg>

BCM54980_MISC_CONTROL          : [0x00000018]           <cfg>

 

 

BMC sel view.