How to install Dell MD Storage Array vCenter Plug-in.

The Dell Modular Disk Storage Array vCenter Management Plug-in is a plug-in for your vCenter (doh! ) that provides integrated management of Dell MD series of storage arrays through the vSphere Client. This plug-in allows you to configure the hosts with the storage arrays and to create, map, and delete virtual disks from the storage arrays. It also allows you to create hardware snapshots, virtual disk copies, and remote replication but these are premium features and need to be purchased separately. I must say, I have been working with this plug-in for several days now and I really start to like it! Before you start the installation, make sure you have downloaded the latest version of the plug-in from Dell. The installation is really easy and straight forward so I won’t go in to detail here but there are some caveats you should consider during and after the installation.

SSL and Non-SSL Jetty Port Numbers

After the initial installation you will be presented with a configuration window for the port of the Jetty Web Service. If you have VMware Update Manager running on your vCenter Server, change the port number to something else because port 8084 is used by VMware Update Manager.

If you don’t change it the plug-in will not be enabled in the vCenter and you’ll get the following error.

Continue with the configuration wizard and provide the IP address of the vCenter Server as well as the password for the Administrator account and optionally the e-mail address. When the registration with the vCenter is complete, open your vSphere Client and login to the vCenter Server. Install the SSL certificate and ignore the warning. You should now have the Dell MD Storage Array vCenter Plug-in icon in your vCenter under Solutions and Applications.

Assign Storage Administrator role to the users

If you go straight to the plug-in after the installation, you’ll receive the following error.In order to use the plug-in you have to assign the Storage Administrator role to appropriate users. Go to Roles under Administration on the vCenter Server and add a new role. Name it Storage Administrators and assign Read/Write permission under Storage Administrator privilege.

Close the window and go to Hosts and Clusters. Select the vCenter object, right click it and select Add Permissions. Under Assigned Role pane, pull down the menu and select Storage Administrators. Under Users and Groups, click the Add button and add appropriate users to the Storage Administrators group.

Close and reopen the vSphere Client for the permissions to take effect.

Add the storage array to the plug-in

Open the plug-in once again, install SSL certificate to Trusted Root Certification Authorities and accept the SSL warning. Next, click Add Array and enter the IP addresses for both controllers and the password for the array. The plug-in should successfully connect to the array and you can now manage your storage array from the vCenter Server. Have fun!

Cheers!

- Marek.Z

Lessons Learned: Dell PowerConnect 5524 switches and vSphere 5.

Some time ago, I configured a pair of Dell PowerConnect 6224 switches for iSCSI storage network and wrote a small blog post about the configuration. This time I had a chance to work with the Dell PowerConnect 5524 switches which were also used for an iSCSI storage network. These switches are cheaper and a bit less powerful than the 6224 series but still good for a small, dedicated iSCSI network. Before you begin with the configuration, update the firmware if applicable. Connect the stack cables, run the configuration wizard, set the (enable) password etc. The rest of the configuration is quite straightforward, just like on the 6224 series but there are some settings that should be considered.

• Create a dedicated iSCSI VLAN and add appropriate ports to the VLAN.
• Turn on the iSCSI Auto-Configuration feature this will enable Jumbo Frames, set the Spanning-Tree Port-Fast feature, disable the Unicast Storm Control and enable Flow Control.
• Set the Speed of the ports in the iSCSI VLAN to gigabit connection.

Here is a quick how-to of the configuration.

Create a dedicated VLAN for iSCSI traffic

1. Login to the switch en go to the configuration mode.
2. Enter VLAN database: Switch(config)#  vlan database
3. Create VLAN: Switch(config-vlan)#  vlan 2
4. Back to config mode: Switch(config-vlan)#  exit
5. Enter VLAN 2 interface config: Switch(config)#  interface vlan 2
6. Name the VLAN: Switch(config-if)# name iSCSI
7. Back to enable mode: Switch(config-if)# end
8. Verify: Switch# show vlan

Enable iSCSI Auto-Configuration

1. Enter the configuration mode and type: Switch(config): iscsi enable
2. You will be asked if you want to continue and the Flow Control will be enabled on all interfaces. Answer with Yes.
3. Now you need save your settings and reload the switch: Switch# write memory
4. Reload the switch: Switch# reload
5. After the reload, verify iSCSI settings with Switch# show iscsi

Assign the interfaces to the iSCSI VLAN

In this case port 1 to 10 on switch 1.

1. Select multiple interfaces: Switch(config)# interface range gigabitethernet 1/0/1-10
2. Add interfaces to VLAN 2: Switch(config-if-range)# switchport access vlan 2
3. Force gigabit connection for all iSCSI ports:
4. Switch(config-if-range)# no negotiation
5. Switch(config-if-range)# speed 1000
6. Save your settings: Switch# write memory
7. You view the configuration with: Switch# show run

Repeat the steps above for the interfaces on switch 2, use interface range gigabitethernet 2/0/1-10 command to select port 1 to 10 on switch 2. It’s also a good idea to shutdown the unused ports on the switches for security reasons.

That’s it, you’re done!

All you have to do now is connect the cables to the switch, storage array and servers and you’re good to go.

Cheers!

- Marek.Z

Preparing Internal Flash Cards on Dell R710 for ESXi 5.x Installation.

On my last project I worked once again with the Dell PowerEdge R710 servers but this time the customer followed our advice and purchased the servers with internal 2 GB flash cards. Auto deploy would of course be more awesome but due to the limited knowledge of vSphere at our customer, we decided to go with the flash cards. During the installation of ESXi 5.0 I noticed something unusual. The flash cards were detected correctly in the BIOS but the ESXi installer failed to install the software. Apparently the flash cards were not prepared for the installation.

Here is a quick guide on how to prepare the flash cards for the installation of vSphere 5 Hypervizor.

1. Place the flash card in the card reader of your laptop or PC.
2. Windows will detect the card and will ask you to format it. In my case it failed.
3. Fire up Diskpart and create a new partition.
4. First, list the disks in your system: DISKPART> list disk
5. Select the correct disk (in my case disk 2): DISKPART>  select disk 2
6. Create new partition: DISKPART> create partition primary
7. Format the disk with FAT32 as you normally would
8. Place the flash card back in the server, power on and go to BIOS.
9. Make sure the flash card is the first boot device in the Boot Sequence settings.
10. Verify that the USB Flash Drive Emulation Type is set to Hard Disk.
11. Save your settings and reboot the server.

Wait for the ESXi installer to start and follow the default procedure. The vSphere 5 Hypervizor should now install correctly.

Cheers!

- Marek.Z

P.S. Don’t forget to wear an antistatic wrist strap when you remove and install the flash cards in the server

Troubleshooting IP-Hash load balancing on a NFS storage network.

When configuring a NFS storage network at one of our customers some time ago, I noticed that the ESXi host wasn’t utilizing all NICs assigned to the NIC team for the VMkernel traffic. After some research, I have found this article written by Frank Denneman a while ago and this VMware KB document. According to the blog post and the article mentioned above, this issue may occur if the calculated hash returns the same result based on the source IP and both destination IP’s. Before we jump in to troubleshooting, let’s take a look at what is exactly going wrong.

Setup

The setup consisted of 4 Dell R710 ESXi hosts connected through 2 stacked Cisco 2960 switches to a NetApp FAS3210 filer. Four NICs per server have been dedicated to NFS storage and cabled in a redundant configuration (2 per switch in EtherChannel). See the drawing for more details.

Solution

To see what is going wrong, we need to calculate the IP-Hash manually. The formula is:

Source IP XOR Destination IP = x MOD y = z where:

Source IP = VMkernel IP address in Hexadecimal
Destination IP = IP address of the NFS filer in Hexadecimal
x = Exclusive OR operation output
y = Number of physical NICs
z = Modulo operation output

First, let’s calculate the IP-Hash value of the IP addresses in the current setup. To do this, we need to convert the IP addresses from decimal to hexadecimal. I used the BitCricket IP Calculator to do the conversion.

Next, calculate the IP-Hash with the formula specified earlier and take a look at the outcome. You can use Windows Calculator to do this, just set the view to Programmer and make sure it is set to Hex and Qword.

1.    C0A86465 XOR C0A86478 = 1D MOD 4 = 1
2.    C0A86465 XOR C0A86482 = E7 MOD 4 = 3
3.    C0A86465 XOR C0A8648C = E9 MOD 4 = 1
4.    C0A86465 XOR C0A86496 = F3 MOD 4 = 3

As you can see, the values are not unique. That’s what causes the problem. The IP-Hash calculation only returns 2 different values instead of 4. To correct this, we need to reconfigure the destination IP addresses (on the NFS Filer) so that every IP-Hash calculations return a unique value. The IP addresses have been reconfigured as follow:

Let’s have a look at the IP-Hash calculations now.

1.    C0A86465 XOR C0A8646F = A MOD 4 = 2
2.    C0A86465 XOR C0A86470 = 15 MOD 4 = 1
3.    C0A86465 XOR C0A86471 = 14 MOD 4 = 0
4.    C0A86465 XOR C0A86472 = 17 MOD 4 = 3

As you can see, the IP-Hash calculation now returns unique values in all four cases. This will now allow utilization of all four NICs from the ESXi host to the NFS Filer.

Cheers!

- Marek.Z

Book review: VMware vSphere 5 Clustering Technical Deepdive.

The VMware vSphere 5 Clustering Technical Deepdive book has been published a while ago and probably most of you already read it. I purchased the book right after the release and I was reading the book really slowly due to some time management issues. Today, I have finally finished reading this awesome book that was written by Duncan Epping (@DuncanYB) and Frank Denneman (@FrankDenneman) and now it’s time for a little review.

This book consists of more than 340 pages about the most important key features of VMware vSphere 5: the VMware High Availability (HA), VMware Distributed Resource Scheduler (DRS) including Distributed Power Management (DPM) and new in this book is the vSphere Storage DRS (SDRS) which was released in vSphere 5.

The first part of the book concentrates on the vSphere HA and includes topics about the redesigned HA stack, new components like the Fault Domain Manager (FDM) and the new Master/Slave node concept. The new “Datastore Heartbeating” feature and the state where a cluster can be isolated or partitioned are also explained in this chapter.

The second part of the book gives you detailed information about the vSphere DRS. It includes topics about some new features of DRS in vSphere 5 as well as vMotion and EVC. The DRS and DPM calculations and recommendations are extensively explained with good examples and of course the resource pools and it’s control.

Part three of the book is about the new vSphere Storage DRS feature in vSphere 5. In my opinion, this is one of the coolest features of vSphere 5. The chapters in this part of the book will discuss all key features of SDRS like resource aggregation, load balancing, datastore maintenance mode but also how it is calculated and when it is invoked. The Storage vMotion (including the new Mirror Mode), Storage I/O Control and some new enhancement in vSphere 5 are also described in this part.

Part four, called “Integration” is an interesting chapter. It describes the integration points of HA, DRS and SDRS. For example, the sub-topic HA and Stateless ESXi describe what happens with the HA configuration when a host reboots. The topics are small, to the point and give you good understanding about the integration points in vSphere 5.

This is just a small summary of the main topics of the VMware vSphere 5 Clustering Technical Deepdive book, there is a lot more. Just like in the previous version of the book, the basic design principle quotes are also included but not indexed which was the case in the previous book. The technical level is high but the topics are interesting for all VMware enthusiasts. The figures and tables are clear and make some topics easier to understand.

Summary

Although the old HA and DRS Technical Deepdive book is great, the new vSphere 5 Clustering book is even better. I know from twitter that Duncan and Frank worked really hard on this book and managed to complete it just in time the vSphere 5 was released. I guess they really have raised the bar for themselves this time. Just like I said earlier, this book is great and this is the bible you must have read if you are a VMware System Administrator, Engineer, Consultant or Architect.

You can order your copy here or here.

Cheers!

- Marek.Z