Total Pageviews

MDS Blogs

Please visit
http://mds9000.blogspot.com for
MDS Config and Troubleshooting Info.

-jerome.jsph@gmail.com

UCS Config -Disclaimer

Please note that this is just a lab recreation and documentation, this is no way replaces official manual and best pratice documentations

Tuesday, November 3, 2009

ESX server_NIC_teaming.doc

ESX server /UCS network Config:

 

 

 

UCS LAN config:

 

 

SAN Config

 

 

 

 

  • Uses network bonding(teaming and multiple vlans portgroups) for ESX Server
  • [root@esx4-143 sbin]# esxcfg-vswitch -l
  • Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
  • vSwitch0       32          9           32                1500    vmnic0,vmnic1
  •  
  •   PortGroup Name      VLAN ID  Used Ports  Uplinks
  •   VM Network          0        2           vmnic0,vmnic1
  •   vlan153             153      1           vmnic0,vmnic1
  •   vlan143             143      1           vmnic0,vmnic1
  •   Service Console     0        1           vmnic0,vmnic1
  •   VMkernel            0        1           vmnic0,vmnic1

 

[root@esx4-143 sbin]# esxcfg-vswif -l

Name     Port Group/DVPort   IP Family IP Address                              Netmask                                 Broadcast        Enabled   TYPE

vswif0   Service Console     IPv4      172.20.2.143                            255.255.254.0                           172.20.3.255     true      STATIC

 

On VM win XP , I added two more network adapters for vlan 143/153

   

 

 

On the winxp, configure the interfaces appropriately:

 

It seems pretty straight forward on ESX server.

 

 

Troubleshooting:

  • mac-tables
  • vnics
  • hbas
  • teaming options
  • n1kv install

 

MAC-tables:

UCS1-FI-A(nxos)# show mac-address-table

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

1         0050.5651.0130    dynamic 0         veth760

143       0050.5651.0130    dynamic 0         veth760

153       0050.5651.0130    dynamic 0         veth760

 

--------------------------------------------------------------------------------

Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port

Interface                                                                   Ch #

--------------------------------------------------------------------------------

veth760       1      eth  trunk  up      none                        10G(S) --

 

UCS1-FI-A(nxos)# show run int veth760

version 4.0(1a)N2(1.1e)

 

interface vethernet760

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

                  Mode   Trunk                          Mode  Speed  Channel

                         Mode                                 (Gbps)

-------------------------------------------------------------------------------

vfc762     1      F      --      up               --     F       auto –

 

UCS1-FI-A(nxos)# show run int vfc 762

version 4.0(1a)N2(1.1e)

 

interface vfc762

  description server 1/3, VHBA

  no shutdown

  bind interface vethernet8954

 

UCS1-FI-A(nxos)# show run int  vethernet8954

version 4.0(1a)N2(1.1e)

 

interface vethernet8954

  switchport access vlan 100

  bind interface Ethernet1/1/3

  pinning server

 

UCS1-FI-A(nxos)# show VLAN FCOE

VLAN      VSAN      Status

--------  --------  --------

100       1         Operational

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

1         0050.5651.0130    dynamic 0         veth760

143       0050.5651.0130    dynamic 0         veth760

153       0050.5651.0130    dynamic 0         veth760

153       0050.56a2.2aba    dynamic 0         veth760 < winxp

4042      0023.04c6.c075    dynamic 0         Eth1/1/4

4042      0026.5108.83d7    dynamic 0         Eth1/1/3

4043      0023.04c6.c075    static  0         Eth1/1/4

4043      0026.5108.83d7    static  0         Eth1/1/3

4044      0026.5108.120c    dynamic 0         Eth1/1/9

4044      0203.f724.741b    dynamic 0         Eth1/1/9

4044      0244.83c4.4c08    dynamic 0         Eth1/1/9

1         0100.5e7f.fffa    igmp    0         veth760

153       0100.5e7f.fffa    igmp    0         veth760

 

UCS1-FI-B(nxos)# SHOW RUN INT VETH 759

version 4.0(1a)N2(1.1e)

 

interface vethernet759

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

MAC 0131 IS THE VNIC and  0050.567b.b055

[root@esx4-143 sbin]# esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type

vmk0       VMkernel            IPv4      172.20.3.62                             255.255.254.0   172.20.3.255    00:50:56:7b:b0:55 1500    65535     true    STATIC

 

UCS1-FI-B(nxos)# SHOW MAC-Address-table

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

1         0000.0000.3d52    dynamic 0         veth759

1         000c.297e.8c3d    dynamic 0         veth759

1         0050.564a.4c97    dynamic 0         veth759

1         0050.5651.0131    dynamic 0         veth759

1         0050.567b.b055    dynamic 0         veth759

1         0050.56a2.3d52    dynamic 0         veth759

143       0050.5651.0131    dynamic 0         veth759

143       0050.56a2.68ac    dynamic 0         veth759

153       0050.5651.0131    dynamic 0         veth759

4042      0023.04c6.c074    dynamic 0         Eth1/1/4

4042      0026.5108.83d6    dynamic 0         Eth1/1/3

4043      0023.04c6.c074    static  0         Eth1/1/4

4043      0026.5108.83d6    static  0         Eth1/1/3

4044      0026.5108.37be    dynamic 0         Eth1/1/9

4044      0203.f724.741a    dynamic 0         Eth1/1/9

4044      0244.83c4.4c07    dynamic 0         Eth1/1/9

1         0100.5e7f.fffa    igmp    0         veth759

143       0100.5e7f.fffa    igmp    0         veth759

Total MAC Addresses: 18

 

So mac address of vnics, xp,esx server show up in the mac-address table.

 

 

 

 

Disabled port 16 (network uplink port)

 

Esx server works fine as nic teaming is there

 

[root@esx4-143 sbin]# ping 172.20.2.1

PING 172.20.2.1 (172.20.2.1) 56(84) bytes of data.

64 bytes from 172.20.2.1: icmp_seq=1 ttl=255 time=0.255 ms

64 bytes from 172.20.2.1: icmp_seq=2 ttl=255 time=0.246 ms

 

--- 172.20.2.1 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.246/0.250/0.255/0.016 ms

 

Winxp running in ESX server can ping all three networks clean net ,153 and 143 net.

 

Verified the vnics are down

 

 

UCS1-FI-A(nxos)# show mac-address-table

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

4042      0023.04c6.c075    dynamic 0         Eth1/1/4

4042      0026.5108.83d7    dynamic 0         Eth1/1/3

4043      0023.04c6.c075    static  0         Eth1/1/4

4043      0026.5108.83d7    static  0         Eth1/1/3

4044      0026.5108.120c    dynamic 0         Eth1/1/9

4044      0203.f724.741b    dynamic 0         Eth1/1/9

4044      0244.83c4.4c08    dynamic 0         Eth1/1/9

Total MAC Addresses: 7

Eth1/16       1      eth  trunk  down    Administratively down       10G(D)

--------------------------------------------------------------------------------

Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port

Interface                                                                   Ch #

--------------------------------------------------------------------------------

veth760       1      eth  trunk  down    inactive                    10G(S) --

 

Got 1/16 up:

 

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Eth1/16       1,100,143,153

veth760       1,143,153

veth764       1,143,153

Eth1/1/9      4044

 

Now will shutdown eth 1/16 in fabric B, to verify it

 

On Fab A

 

UCS1-FI-A(nxos)# show mac-address-table

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

1         000c.297e.8c3d    dynamic 0         veth760

1         0050.564a.4c97    dynamic 0         veth760

1         0050.5651.0130    dynamic 0         veth760

1         0050.567b.b055    dynamic 0         veth760

1         0050.56a2.3d52    dynamic 0         veth760

143       0050.5651.0130    dynamic 0         veth760

143       0050.56a2.68ac    dynamic 0         veth760

153       0050.5651.0130    dynamic 0         veth760

153       0050.56a2.2aba    dynamic 0         veth760

4042      0023.04c6.c075    dynamic 0         Eth1/1/4

4042      0026.5108.83d7    dynamic 0         Eth1/1/3

4043      0023.04c6.c075    static  0         Eth1/1/4

4043      0026.5108.83d7    static  0         Eth1/1/3

4044      0026.5108.120c    dynamic 0         Eth1/1/9

4044      0203.f724.741b    dynamic 0         Eth1/1/9

4044      0244.83c4.4c08    dynamic 0         Eth1/1/9

1         0100.5e7f.fffa    igmp    0         veth760

Total MAC Addresses: 17

 

veth760       1      eth  trunk  up      none                        10G(S)

 

But on Fabric B

 

UCS1-FI-B(nxos)# show mac-address-table

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

4042      0023.04c6.c074    dynamic 0         Eth1/1/4

4042      0026.5108.83d6    dynamic 0         Eth1/1/3

4043      0023.04c6.c074    static  0         Eth1/1/4

4043      0026.5108.83d6    static  0         Eth1/1/3

4044      0026.5108.37be    dynamic 0         Eth1/1/9

4044      0203.f724.741a    dynamic 0         Eth1/1/9

4044      0244.83c4.4c07    dynamic 0         Eth1/1/9

Total MAC Addresses: 7

 

Eth1/16       1      eth  trunk  down    Administratively down       10G(D)

 

veth759       1      eth  trunk  down    inactive                    10G(S) --

veth763       1      eth  trunk  down    inactive                    10G(S) –

 

 

 

 

 

 

 

 

 

 

 


VLAN_BU_ESC.doc

 

VLAN topology in BU-ESCALATION Lab

 

Vlan 1 – clean net

Vlan 143, 153 and 163 – dirty net. Not routable outside this network.

Config on 3750 :

Port 1/0/1 – goes southbound to N5k, it trunks 1,143,153 and 163 and has vlan interface configured.

ucs-lab-sw-1#show run int gig 1/0/1

interface GigabitEthernet1/0/1

switchport trunk encapsulation dot1q

switchport mode trunk

 

ucs-lab-sw-1#show int trun

 

Port        Mode         Encapsulation  Status        Native vlan

Gi1/0/1     on           802.1q         trunking      1

 

Port        Vlans allowed on trunk

Gi1/0/1     1-4094

 

Port        Vlans allowed and active in management domain

Gi1/0/1     1,10,50-136,143,153,163,590,802

 

Port        Vlans in spanning tree forwarding state and not pruned

Gi1/0/1     1,10,50-136,143,153,163,590,802

ucs-lab-sw-1#

interface Vlan1

ip address 172.20.3.176 255.255.254.0

interface Vlan143

ip address 10.6.143.254 255.255.255.0

interface Vlan153

ip address 10.6.153.254 255.255.255.0

interface Vlan163

ip address 10.6.163.254 255.255.255.0

 

ucs-lab-sw-1#show vlan br

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Gi1/0/2, Gi1/0/3, Gi1/0/4

                                                Gi1/0/5, Gi1/0/6, Gi1/0/7

                                                Gi1/0/8, Gi1/0/9, Gi1/0/10

                                                Gi1/0/11, Gi1/0/12, Gi1/0/13

                                                Gi1/0/14, Gi1/0/15, Gi1/0/16

                                                Gi1/0/17, Gi1/0/18, Gi1/0/19

                                                Gi1/0/20, Gi1/0/21, Gi1/0/22

                                                Gi1/0/23, Gi1/0/24, Gi1/0/25

                                                Gi1/0/26, Gi1/0/27, Gi1/0/28

                                                Gi1/0/29, Gi1/0/30, Gi1/0/31

                                                Gi1/0/32, Gi1/0/33, Gi1/0/49

                                                Gi1/0/50, Gi1/0/51, Gi1/0/52

 

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

143  VLAN0143                         active    Gi1/0/34, Gi1/0/35, Gi1/0/36

                                                Gi1/0/37, Gi1/0/38

153  VLAN0153                         active    Gi1/0/39, Gi1/0/40, Gi1/0/41

                                                Gi1/0/42, Gi1/0/43

163  VLAN0163                         active    Gi1/0/44, Gi1/0/45, Gi1/0/46

                                                Gi1/0/47, Gi1/0/48

590  VLAN0590                         active

802  VLAN0802                         active

 

interface GigabitEthernet1/0/33

switchport mode access

!

interface GigabitEthernet1/0/34

switchport access vlan 143

!

interface GigabitEthernet1/0/35

switchport access vlan 143

!

interface GigabitEthernet1/0/36

switchport access vlan 143

!

interface GigabitEthernet1/0/37

switchport access vlan 143

!

interface GigabitEthernet1/0/38

switchport access vlan 143

!

interface GigabitEthernet1/0/39

switchport access vlan 153

!

interface GigabitEthernet1/0/40

switchport access vlan 153

!

interface GigabitEthernet1/0/41

switchport access vlan 153

!

interface GigabitEthernet1/0/42

switchport access vlan 153

!

interface GigabitEthernet1/0/43

switchport access vlan 153

!

interface GigabitEthernet1/0/44

switchport access vlan 163

!

interface GigabitEthernet1/0/45

switchport access vlan 163

!

interface GigabitEthernet1/0/46

switchport access vlan 163

!

interface GigabitEthernet1/0/47

switchport access vlan 163

!

interface GigabitEthernet1/0/48

switchport access vlan 163

!

interface GigabitEthernet1/0/49

!

interface GigabitEthernet1/0/50

!

interface GigabitEthernet1/0/51

!

interface GigabitEthernet1/0/52

!

interface Vlan1

ip address 172.20.3.176 255.255.254.0

!

interface Vlan143

ip address 10.6.143.254 255.255.255.0

!

interface Vlan153

ip address 10.6.153.254 255.255.255.0

!

interface Vlan163

ip address 10.6.163.254 255.255.255.0

 

N5K config:

n5k-1# show cdp neig

Device-ID            Local Intrfce Hldtme Capability  Platform      Port ID

 

jpannick-3560G         mgmt0         153   S I       WS-C3560G-48P Gig0/15

jerry_firewall         Eth1/2        139   R S I     WS-C6509-E    Gig9/26

ucs-lab-sw-1           Eth1/8        131   S I       WS-C3750G-48T Gig1/0/1  <<<North

UCS1-FI-A(SSI132107EQ) Eth1/16       134   S I s     N10-S6100     Eth1/16 <<< South bound

 

 

 

Northbound config:

n5k-1# show run int eth 1/8

version 4.0(1a)N2(1)

 

interface Ethernet1/8

  switchport mode trunk

  speed 1000

  switchport trunk allowed vlan 1,10,143,153,163

 

n5k-1# show vlan

 

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Eth1/1, Eth1/2, Eth1/3, Eth1/4

                                                Eth1/5, Eth1/6, Eth1/7, Eth1/9

                                                Eth1/10, Eth1/11, Eth1/12

                                                Eth1/13, Eth1/14, Eth1/15

                                                Eth1/17, Eth1/18, Eth1/19

                                                Eth1/20

10   VLAN0010                         active

143  VLAN0143                         active

153  VLAN0153                         active

163  VLAN0163                         active

 

n5k-1# show int trun

 

--------------------------------------------------------------------------------

Port          Native  Status        Port

              Vlan                  Channel

--------------------------------------------------------------------------------

Eth1/8        1       trunking      --

Eth1/15       1       trunking      --

Eth1/16       1       trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Eth1/8        1,10,143,153,163  <<North

Eth1/15       1-3967,4048-4093

Eth1/16       1,10,143,153,163 <<< South

 

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Eth1/8        1,10,143,153,163  <<<North

Eth1/15       none

Eth1/16       1,10,143,153,163 >>> South

n5k-1# show run

version 4.0(1a)N2(1)

feature interface-vlan

ssh key rsa 768 force

ip host n5k-1 0.0.0.0

vrf context management

  ip route 0.0.0.0/0 172.20.2.1

switchname n5k-1

vlan 1,10,143,153,163

interface Vlan1

  no shutdown

interface Ethernet1/1

  speed 1000

interface Ethernet1/2

  speed 1000

interface Ethernet1/3

  speed 1000

interface Ethernet1/4

interface Ethernet1/5

interface Ethernet1/6

interface Ethernet1/7

interface Ethernet1/8

  switchport mode trunk

  speed 1000

  switchport trunk allowed vlan 1,10,143,153,163

interface Ethernet1/9—1/13 – same config

interface Ethernet1/14

interface Ethernet1/15

  shutdown

  switchport mode trunk

interface Ethernet1/16

  switchport mode trunk

  switchport trunk allowed vlan 1,10,143,153,163

interface Ethernet1/17

interface Ethernet1/18

interface Ethernet1/19

interface Ethernet1/20

  switchport monitor

monitor session 1

  source interface Ethernet1/16 both

  destination interface Ethernet1/20

  no shut

interface mgmt0

  ip address 172.20.2.126/23

 

 

Config on UCS (GUI)

 

  • vlan 1 is there by default – we are using that for clean, though on northbound out of 3560 is 801.
  • Vlan 153 and 143 are created on both fabrics,  153 , I created separately on both fabrics and 143 , I created like this.

 

 

Once this is created., you will see trunking enabled on 6120s.

Connect nxos A:

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Eth1/1, Eth1/2, Eth1/3, Eth1/4

                                                Eth1/5, Eth1/6, Eth1/7, Eth1/8

                                                Eth1/9, Eth1/10, Eth1/11

                                                Eth1/12, Eth1/13, Eth1/14

                                                Eth1/15, Eth1/17, Eth1/18

                                                Eth1/19, Eth1/20, Eth2/1, Eth2/2

                                                Eth2/3, Eth2/4, Eth1/1/1

                                                Eth1/1/2, Eth1/1/3, Eth1/1/4

                                                Eth1/1/5, Eth1/1/6, Eth1/1/7

                                                Eth1/1/8

143  VLAN0143                         active

153  VLAN0153                         active

4044 SAM-vlan-management              active

4047 SAM-vlan-boot                    active

 

UCS1-FI-A(nxos)# show run int eth 1/16

version 4.0(1a)N2(1.1e)

 

interface Ethernet1/16

  switchport mode trunk

  switchport trunk allowed vlan 1,143,153

  pinning border

  no shutdown

 

 

That’s abt setting uplinks and vlans.

 

Service Profile:

enter service-profile test instance

         associate server 1/3

        

        

         enter vnic vnic1

             enter eth-if default – VLAN 1

                 set default-net yes

             exit

             set adaptor-profile ""

             set identity dynamic-mac 00:25:B5:11:01:3A

             set identity mac-pool ""

             set order 2

set pin-group ""

             set qos-policy ""

             set stats-policy default

             set template-name ""

         exit

         enter vnic vnic2

             enter eth-if dirty153 – vlan 153

                 set default-net yes – native vlan

             exit

             set adaptor-profile ""

             set identity dynamic-mac 00:25:B5:01:01:3B

             set identity mac-pool ""

             set order 3

             set pin-group ""

             set qos-policy ""

             set stats-policy default

             set template-name ""

         exit

         power up

         set boot-policy ""

         set descr ""

         set dynamic-vnic-conn-policy ""

         set host-fw-policy ""

         set identity dynamic-uuid derived

         set identity uuid-suffix-pool ""

         set identity wwnn-pool node-default

         set ipmi-access-profile ""

         set local-disk-policy ""

         set mgmt-fw-policy ""

         set scrub-policy ""

         set sol-policy default

         set src-templ-name ""

         set stats-policy default

     exit

 

On the GUI:

 

 

 

If the Native VLAN is not set, we have seen ping packets getting lost for vlans other than 1.

 

If Native vlan is set, then each vlan is tagged.

 

Here is ethereal output or wireshark output , we captured thro’ NAM.

 

You see VLAN—is 153 in ID: where we get reply back , this where we do

Select native vlan

 

If you unselect Native Vlan : VLAN id becomes 1, and it gets dropped in northbound switches as it reaches vlan 1 instead of dirty-net vlan 153.

 

 

 

Redhat Config on 1/3 service profile test.

 

[root@lnx-143 ~]# ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:25:B5:01:01:3B --- Fabric B

          inet addr:10.6.153.100  Bcast:10.6.153.255  Mask:255.255.255.0

          inet6 addr: fe80::225:b5ff:fe01:13b/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:20887 errors:0 dropped:0 overruns:0 frame:0

          TX packets:20323 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:2058155 (1.9 MiB)  TX bytes:2002255 (1.9 MiB)

 

[root@lnx-143 ~]# ifconfig eth1

eth1      Link encap:Ethernet  HWaddr 00:25:B5:11:01:3A

          inet addr:172.20.2.143  Bcast:172.20.3.255  Mask:255.255.254.0

          inet6 addr: fe80::225:b5ff:fe11:13a/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:43054 errors:0 dropped:0 overruns:0 frame:0

          TX packets:48736 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:4283364 (4.0 MiB)  TX bytes:5319399 (5.0 MiB)

 

 

[root@lnx-143 ~]# ping 172.20.2.1

PING 172.20.2.1 (172.20.2.1) 56(84) bytes of data.

64 bytes from 172.20.2.1: icmp_seq=1 ttl=255 time=0.442 ms

64 bytes from 172.20.2.1: icmp_seq=2 ttl=255 time=0.335 ms

 

--- 172.20.2.1 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1000ms

rtt min/avg/max/mdev = 0.335/0.388/0.442/0.057 ms

[root@lnx-143 ~]# ping 10.6.153.1

PING 10.6.153.1 (10.6.153.1) 56(84) bytes of data.

64 bytes from 10.6.153.1: icmp_seq=1 ttl=255 time=0.484 ms

64 bytes from 10.6.153.1: icmp_seq=2 ttl=255 time=0.341 ms

64 bytes from 10.6.153.1: icmp_seq=3 ttl=255 time=0.536 ms

 

We put the server1/4 in 143 network.

From server 4(10.6.143.100) we were able to ping 10.6.153.100

(u need to route add –net 10.6.0.0/16 –gw 10.6.143.1).

 

 

Troubleshooting Guide:

[root@lnx-143 ~]# arp -a

sjc2-2lab-clean-2net.cisco.com (172.20.2.1) at 00:22:0D:F5:20:00 [ether] on eth1

? (10.6.153.1) at 00:22:0D:F4:40:00 [ether] on eth0

? (10.6.153.253) at 00:22:56:02:16:C3 [ether] on eth0

 

UCS1-FI-B(nxos)# show mac-address-table vl 153

VLAN      MAC Address       Type    Age       Port

---------+-----------------+-------+---------+------------------------------

153       0025.b501.013b    dynamic 0         veth719

 

UCS1-FI-B(nxos)# show run int vethernet 719

version 4.0(1a)N2(1.1e)

 

interface vethernet719

  switchport trunk native vlan 153

  switchport trunk allowed vlan 153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

 

UCS1-FI-B(nxos)# show run int ethernet 1/1/3

version 4.0(1a)N2(1.1e)

 

interface Ethernet1/1/3

  no shutdown

  vntag max-vifs 15

  pinning server

  fabric-interface Eth1/17 – which server port on FI, it is going to

 

UCS1-FI-B(nxos)# show int br | in up

fc2/1      1      NP     --      up               swl    NP      2    --

Eth1/16       1      eth  trunk  up      none                        10G(D) --

Eth1/17       1      eth  fabric up      none                        10G(D) --

veth719       153    eth  trunk  up      none                        10G(S) --

veth720       143    eth  trunk  up      none                        10G(S) --

Eth1/1/3      1      eth  access up      none                        10G(D) --

Eth1/1/4      1      eth  access up      none                        10G(D) --

Eth1/1/9      4044   eth  trunk  up      none                        10G(D) –

 

--- More troubleshooting tips on how packets traverse inside UCS –

6120/IOM/NIF/VIF/CIF ports to continue…

 

[root@lnx-144 ~]# modinfo ixgbe

filename:       /lib/modules/2.6.18-128.el5/weak-updates/ixgbe/ixgbe.ko

version:        1.3.18-k5

license:        GPL

description:    Intel(R) 10 Gigabit PCI Express Network Driver

author:         Intel Corporation, <linux.nics@intel.com>

srcversion:     47AB2007BA0E144CBA56643

alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*

alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*

alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*

alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*

alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*

depends:

vermagic:       2.6.18-128.1.1.el5 SMP mod_unload gcc-4.1

 

Create Vlan tagged interface

 

On linux

 

Vconfig add eth0 153

Ifup eth0.153

 

ifconfig eth0.153 10.6.153.100 netmask 255.255.255.0 broadcast 10.6.153.255 up

 

Then u don’t need native vlan tag.

 

 

Failover config:

 

We need to have vlan tagging done on OS.

 

 

  • Configured trunking on vnic – added 3 vlans.
  • Remove any native vlan tagging.
  • [root@lnx-143 ~]# ifconfig eth0.153 – fabric B
  • eth0.153  Link encap:Ethernet  HWaddr 00:25:B5:01:01:3B
  •           inet addr:10.6.153.100  Bcast:10.6.153.255  Mask:255.255.255.0
  •           inet6 addr: fe80::225:b5ff:fe01:13b/64 Scope:Link
  •           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  •           RX packets:81 errors:0 dropped:0 overruns:0 frame:0
  •           TX packets:140 errors:0 dropped:0 overruns:0 carrier:0
  •           collisions:0 txqueuelen:0
  •           RX bytes:7052 (6.8 KiB)  TX bytes:23671 (23.1 KiB)

 

   Ifconfig eth1:

[root@lnx-143 ~]# ifconfig eth1

eth1      Link encap:Ethernet  HWaddr 00:25:B5:11:01:3A

          inet addr:172.20.2.143  Bcast:172.20.3.255  Mask:255.255.254.0

          inet6 addr: fe80::225:b5ff:fe11:13a/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:13404 errors:0 dropped:0 overruns:0 frame:0

          TX packets:2310 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:1051625 (1.0 MiB)  TX bytes:225518 (220.2 KiB)

 

http://www.cyberciti.biz/tips/howto-configure-linux-virtual-local-area-network-vlan.html

 

[root@lnx-143 network-scripts]# more ifcfg-eth*

::::::::::::::

ifcfg-eth0

::::::::::::::

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=eth0

BOOTPROTO=none

HWADDR=00:25:b5:01:01:3b

IPV6INIT=no

IPV6_AUTOCONF=yes

ONBOOT=yes

TYPE=Ethernet

PEERDNS=yes

USERCTL=no

 

ifcfg-eth0.153

::::::::::::::

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=eth0.153

BOOTPROTO=none

HWADDR=00:25:b5:01:01:3b

IPADDR=10.6.153.100

NETMASK=255.255.255.0

NETWORK=10.6.153.0

GATEWAY=10.6.153.1

IPV6INIT=no

IPV6_AUTOCONF=yes

ONBOOT=yes

TYPE=Ethernet

VLAN=yes

 

ifcfg-eth1

::::::::::::::

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=eth1

BOOTPROTO=none

BROADCAST=172.20.3.255

HWADDR=00:25:B5:11:01:3A

IPADDR=172.20.2.143

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.254.0

NETWORK=172.20.2.0

ONBOOT=yes

GATEWAY=172.20.2.1

TYPE=Ethernet

 

[root@lnx-143 devices]# pwd

/etc/sysconfig/networking/devices

[root@lnx-143 devices]# ls

ifcfg-eth0  ifcfg-eth0.153  ifcfg-eth1

[root@lnx-143 default]# pwd

/etc/sysconfig/networking/profiles/default

[root@lnx-143 default]# ls

hosts  ifcfg-eth0  ifcfg-eth0.153  ifcfg-eth1  network  resolv.conf

 

http://www.cisco.com/en/US/products/ps10281/products_configuration_example09186a0080af4119.shtml

 

 

UCS1-FI-B(nxos)# show int br | inc veth

veth720       143    eth  trunk  up      none                        10G(S) --

veth732       1      eth  trunk  up      none                        10G(S) --

veth735       1      eth  trunk  up      none                        10G(S) –

 

UCS1-FI-B(nxos)# show run int veth732

version 4.0(1a)N2(1.1e)

 

interface vethernet732

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

UCS1-FI-B(nxos)# show run int veth735

version 4.0(1a)N2(1.1e)

 

interface vethernet735

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

UCS1-FI-A(nxos)# show int br | inc veth

veth721       1      eth  trunk  up      none                        10G(S) --

veth733       1      eth  trunk  up      none                        10G(S) --

veth734       1      eth  trunk  up      none                        10G(S) –

UCS1-FI-A(nxos)# show run int veth733

version 4.0(1a)N2(1.1e)

 

interface vethernet733

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

UCS1-FI-A(nxos)# show run int veth734

version 4.0(1a)N2(1.1e)

 

interface vethernet734

  switchport trunk allowed vlan 1,143,153

  bind interface Ethernet1/1/3

  no pinning server sticky

  pinning server pinning-failure link-down

 

So when Failover is enabled , two interface (veths) for both Fabric a and B

For same vlan.

 

We have total 4 – 2 for vlan 1 and 2 for vlan 153.

 

 

We disabled 1/16 on Fabric A, so now after little time, I should have

Connectivity to linux server up.

 

UCS1-FI-A(nxos)# show int br | inc veth

veth733       1      eth  trunk  down    inactive                    10G(S) --

veth734       1      eth  trunk  down    inactive                    10G(S) –

 

down on fabric A because the pinned link is down.

 

 

But fabric B , the interfaces are up

 

UCS1-FI-B(nxos)# show int br | inc veth

veth732       1      eth  trunk  up      none                        10G(S) --

veth735       1      eth  trunk  up      none                        10G(S) –

 

and my linux putty disconnected, but I was able to connect it back

 

[root@lnx-143 ~]# ping 172.20.2.1

PING 172.20.2.1 (172.20.2.1) 56(84) bytes of data.

64 bytes from 172.20.2.1: icmp_seq=1 ttl=255 time=1.23 ms

64 bytes from 172.20.2.1: icmp_seq=2 ttl=255 time=0.662 ms

 

--- 172.20.2.1 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.662/0.947/1.232/0.285 ms

[root@lnx-143 ~]# ping 10.6.153.1

PING 10.6.153.1 (10.6.153.1) 56(84) bytes of data.

64 bytes from 10.6.153.1: icmp_seq=1 ttl=255 time=1001 ms

64 bytes from 10.6.153.1: icmp_seq=2 ttl=255 time=1.99 ms

64 bytes from 10.6.153.1: icmp_seq=3 ttl=255 time=0.386 ms

--- 10.6.153.1 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 0.386/334.583/1001.366/471.487 ms, pipe 2

 

 

Onto Bonding:

 

http://www.linux-corner.info/bonding.html

 

 

[root@lnx-143 ~]# more /etc/modprobe.conf

alias eth1 ixgbe

alias scsi_hostadapter mptbase

alias scsi_hostadapter1 mptsas

alias scsi_hostadapter2 qla2xxx

alias scsi_hostadapter3 usb-storage

alias eth0 ixgbe

alias eth0.0 ixgbe

#added by jerome

alias bond0 bonding

alias bond1 bonding

options bond0 max_bonds=2 miimon=100 mode=1

options bond1 miimon=100 mode=1

# 100 ms failover times Mode 1 active/backup

 

 

 

[root@lnx-143 network-scripts]# pwd

/etc/sysconfig/network-scripts

[root@lnx-143 network-scripts]# ls ifcfg-*

ifcfg-bond0  ifcfg-eth0  ifcfg-eth1  ifcfg-lo

 

ifcfg-bond0

::::::::::::::

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=bond0

BOOTPROTO=none

BROADCAST=172.20.3.255

IPADDR=172.20.2.143

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.254.0

NETWORK=172.20.2.0

ONBOOT=yes

GATEWAY=172.20.2.1

TYPE=Ethernet

MII_NOT_SUPPORTED=yes

 

ifcfg-eth0

::::::::::::::

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=eth0

BOOTPROTO=none

HWADDR=00:25:b5:01:01:3b

IPV6INIT=no

IPV6_AUTOCONF=yes

ONBOOT=yes

TYPE=Ethernet

PEERDNS=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

MII_NOT_SUPPORTED=yes

 

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=eth1

BOOTPROTO=none

#BROADCAST=172.20.3.255

HWADDR=00:25:B5:11:01:3A

#IPADDR=172.20.2.143

IPV6INIT=yes

IPV6_AUTOCONF=yes

#NETMASK=255.255.254.0

#NETWORK=172.20.2.0

ONBOOT=yes

#GATEWAY=172.20.2.1

TYPE=Ethernet

MASTER=bond0

SLAVE=yes

MII_NOT_SUPPORTED=yes

 

 

Removed other network config related to eth0.153

 

Rebooted the linux.

 

bond0     Link encap:Ethernet  HWaddr 00:25:B5:01:01:3B

          inet addr:172.20.2.143  Bcast:172.20.3.255  Mask:255.255.254.0

          inet6 addr: fe80::225:b5ff:fe01:13b/64 Scope:Link

          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

          RX packets:1673 errors:0 dropped:0 overruns:0 frame:0

          TX packets:767 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:145182 (141.7 KiB)  TX bytes:72785 (71.0 KiB)

 

bond1     Link encap:Ethernet  HWaddr 00:00:00:00:00:00

          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

 

eth0      Link encap:Ethernet  HWaddr 00:25:B5:01:01:3B

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:1145 errors:0 dropped:0 overruns:0 frame:0

          TX packets:767 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:107326 (104.8 KiB)  TX bytes:72785 (71.0 KiB)

 

eth1      Link encap:Ethernet  HWaddr 00:25:B5:01:01:3B

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:528 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:37856 (36.9 KiB)  TX bytes:0 (0.0 b)

(other unrelated interface loopback and ipv6 removed)

 

Bond0 using eth0 ‘s mac address. (eth0 and eth1 = same mac address, as we

Had failedover)

 

PING 172.20.2.1 (172.20.2.1) 56(84) bytes of data.

64 bytes from 172.20.2.1: icmp_seq=1 ttl=255 time=0.521 ms

64 bytes from 172.20.2.1: icmp_seq=2 ttl=255 time=0.668 ms

64 bytes from 172.20.2.1: icmp_seq=3 ttl=255 time=0.731 ms

64 bytes from 172.20.2.1: icmp_seq=4 ttl=255 time=0.552 ms

64 bytes from 172.20.2.1: icmp_seq=5 ttl=255 time=0.409 ms

64 bytes from 172.20.2.1: icmp_seq=6 ttl=255 time=0.366 ms

 

Now I disabled failover on the vnics in UCS

 

 

 

Configured bond0.153

[root@lnx-143 network-scripts]# ls -la ifcfg*

-rw-r--r-- 3 root root 280 Oct 30 11:11 ifcfg-bond0

-rw-r--r-- 1 root root 293 Oct 30 11:55 ifcfg-bond0.153

-rw-r--r-- 3 root root 246 Oct 30 11:08 ifcfg-eth0

-rw-r--r-- 3 root root 332 Oct 30 11:41 ifcfg-eth1

-rw-r--r-- 1 root root 254 Jul  4 02:10 ifcfg-lo

[root@lnx-143 network-scripts]# more ifcfg-bond0.153

# Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection

DEVICE=bond0.153

BOOTPROTO=none

BROADCAST=10.6.153.255

IPADDR=10.6.153.100

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=10.6.153.0

ONBOOT=yes

GATEWAY=10.6.153.1

TYPE=Ethernet

MII_NOT_SUPPORTED=yes

VLAN=yes

[root@lnx-143 devices]# ls

ifcfg-bond0  ifcfg-bond0.153  ifcfg-eth0  ifcfg-eth1

 

[root@lnx-143 default]# ls

hosts        ifcfg-bond0.153  ifcfg-eth1  resolv.conf

ifcfg-bond0  ifcfg-eth0       network

 

Rebooted the server and ping worked fine even fabric A or fabric was down.

 

I tried adding Fabric A-B and B-A failover, still the MAC address of the :3B

Showed up (bonding nic) in Fabric A or Fabric B depending on which fabric failed.

We failed 1/16 which is uplink port.

 


Followers