Table of Contents
Advertisement
Quick Links
IBM BladeCenter Deployment Guide
IBM
BladeCenter
Deployment Guide
®
™
Part 3 - Blades
Chris Gillen, IBM
Sam Litenatsky, IBM
Kevin Conover, IBM
http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564
Version 3.0, 3/10/2006
1
The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for
guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM.
Advertisement
Table of Contents
Related Manuals for IBM 8832
Summary of Contents for IBM 8832
- Page 1 IBM BladeCenter Deployment Guide BladeCenter Deployment Guide ® ™ Part 3 - Blades Chris Gillen, IBM Sam Litenatsky, IBM Kevin Conover, IBM http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 Version 3.0, 3/10/2006 The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind.
- Page 2 IBM BladeCenter Deployment Guide http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 © 2006 IBM Version 3.0, 3/10/2006 The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM.
-
Page 3: Table Of Contents
Remote Control Hints and Tips ... 9 Embedded Fibre Channel Switches... 12 2.4.1 SAN Configuration... 12 2.4.2 Disabling ports on the QLogic switch ... 13 Blade Configuration and Installation ... 16 Configuring the Blades for Remote Boot ... 16 3.1.1 Disabling IDE devices ... 16 3.1.2... - Page 4 Table 1 Recommended settings for the QLogic QLA23xx adapter card for Windows 2000 or 2003 ... 15 Table 2 Management Module BIOS Information... 50 Table 3 HS20 – Model 8832 Firmware Information ... 50 Table 4 HS20 – Model 8843 Firmware Information ... 51 Table 5 HS40 BIOS and ISMP Information...
- Page 5 IBM BladeCenter Deployment Guide Figure 30 linux nofb... 26 Figure 31 Unmount the Diskette Image ... 26 Figure 32 Boot Failure for RH 3.0 ... 28 Figure 33 Kickstart Modifications ... 28 Figure 34 fdisk Output... 29 Figure 35 Invalid Partition Table ... 29 Figure 36 RPM Install of Broadcom Device Drivers ...
-
Page 6: Introduction
HS20 or HS40 blade servers. The blade servers utilize the IBM BladeCenter Fibre Channel Expansion Card connected to the BladeCenter Fibre Channel Switch Modules via the BladeCenter backplane. The switch modules in turn are connected to the core fibre channel switches that host the SAN Fabric. -
Page 7: Notes
Module slots 3 and 4 are used for Fibre Channel or other switch types and connect to the Server Blades as well as external devices. • Power - Four modular power supplies provide primary and backup power to all BladeCenter subsystems. -
Page 8: Installation And Configuration
This process is described in detail later in this document. 2.1 Network Configuration The two switch modules provide redundant paths to each of the server blades for reaching other server blades within the same chassis or for communication to external devices via the network infrastructure. -
Page 9: Bladecenter Management Module
Note: For switch communication through the management module external Ethernet port, the switch module internal network interface and the management module internal and external interfaces must be on the same subnet. 2.2 BladeCenter Management Module To use the Management Module to perform tasks in the coming sections remotely, first do the following: Open a browser and input the IP address of the MM. -
Page 10: Figure 4 Disable Key Repeats
2. Repeating Keystrokes - in order to remove the repeating keystroke problem, you need to check the "Disable key repeats" box and click Save. Figure 4 Disable key repeats 3. Start a Remote Control of the target blade by clicking on Remote Control under Blade Tasks. Ensure that the media tray and KVM selections are for the target blade) Figure 5 KVM and Media Ownership http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564... -
Page 11: Figure 6 Remote Control Window
4. The Remote Control window is a pop up window. If you have a popup blocker installed, Remote Control will be blocked, so make sure popups are allowed for the management module IP. Note: Remote Control does work with Firefox, so if you are having trouble with the popup blocker in Windows XP SP2, Firefox may be a viable alternative. -
Page 12: Embedded Fibre Channel Switches
Shown in Figure 8 is an example of a dual path SAN configuration that we will be using as a reference in this document. Two McData directors steer traffic through separate fabrics to the IBM BladeCenters, which use Qlogic HBAs (Host Bus Adapters). Separate fabrics allow full redundancy in the storage path. -
Page 13: Disabling Ports On The Qlogic Switch
Figure 8 Storage Connection Diagram 2.4.2 Disabling ports on the QLogic switch Here are the steps necessary to disable ports on the QLogic switch. This can be done either via a telnet session or a browser session. 2.4.2.1 Using Telnet 1. -
Page 14: Figure 11 Port Status Offline
IP address of the target QLogic switch in the window that will pop up - That should open SAN Browser. 2. Click the handle in the left pane to drop down the switch - highlight it and the faceplate will appear in the right pane. -
Page 15: Figure 13 Port Properties
3. Right click on the desired port and choose "port properties" from the menu. 4. At the top of port properties put a bullet next to "offline" or "down". 5. Repeat steps 1-4 and reverse step 5 to online the port. Figure 12 Port Properties http://knowledge.storage.ibm.com/servers/storage/support/hbasearch/interop/hbaSearch.do Parameters... -
Page 16: Blade Configuration And Installation
Blade Configuration and Installation 3 Configuring the Blades for Remote Boot The next few steps require BIOS configuration. To enter the BIOS, follow these steps: 1. Logon to the Management Module and select Power/Restart under Blade Tasks. 2. Select the blade to be restarted, then at the bottom of the page click Restart Blade and “Ok” when prompted. -
Page 17: Figure 15 Hs20 Boot Sequence
Figure 14 HS20 Boot Sequence 3. Adjusting the boot sequence for HS40 blades is slightly different. Use the arrow keys to scroll to each startup device, then hit enter and select the appropriate value in the popup window as shown in Figure 15 HS40 Boot Sequence. -
Page 18: Configuring The Qlogic Hba
Figure 16 Changing boot order from Management Module for HS20 3.2 Configuring the Qlogic HBA 3.2.1 Acquiring HBA WWPNs (World Wide Port Names) Each HBA has a unique WWPN by which it is identified in a SAN network. Accurate tracking of WWPNs ensures that each server has the proper storage presented to the appropriate location. -
Page 19: Configuring The Path To The Boot Lun
Figure 17 Configuring the QLogic HBA Figure 18 Host Adapter Settings 3.2.2 Configuring the path to the boot LUN The HBA that the server will be booting from will need to be defined in the bios. At this point the HBA’s WWID has been zoned on the backend storage as the owner of the boot LUN. -
Page 20: Figure 20 Enable Selectable Boot Illustration
IBM BladeCenter Deployment Guide Figure 19 Enable Selectable Boot Illustration Figure 20 Bootable LUN address http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 © 2006 IBM Version 3.0, 3/10/2006 The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind. -
Page 21: Testing The Path To The Boot Lun
Critical Note: At this point, only one boot path is configured. During the INITIAL Operating System build, only one boot path should be active. After the initial OS build a secondary path can be added if this is supported by the storage vendor. 3.2.3 Testing the path to the boot LUN The path to the LUN that the server will be booting from can be tested at this point. -
Page 22: Ethernet Port Numbering
Ethernet switch in switch bay 1, and the connection named Local Area Connection 2 goes to the Ethernet switch in switch bay 2. We use the phrase most cases, because this is not always the case. For Windows 2000, the order of the Local Area Connection names assigned is based on the order in which the drivers for each NIC are installed. - Page 23 Linux (Red Hat Version 3 Update 6) and SLES 9 has shown that eth0 goes to the Ethernet switch in bay 1 and eth1 goes to the switch in bay 2. On an HS40 blade, 0 & 2 should be connected to top switch module and 1 &...
-
Page 24: Installation Process For Red Hat Linux
Red Hat AS 3.0 Update 3 and above.. Please see Section 4.4 for known issues related to this and other releases of Red Hat. 1. Recommended: From the secondary QLogic Fibre Channel switch module, DISABLE the port of the secondary HBA. (This HBA can be re-enabled after the initial OS load). This process is detailed in 2.4. -
Page 25: Figure 28 Loading The Remote Diskette
Figure 27 Loading the Remote Diskette c. Power on or restart the target blade. Figure 28 Power On/Restart Blade 5. From this point on, the Red Hat installation proceeds like a standard installation. Note: you must specify “linux nofb”, when installing via BladeCenter Remote Control, otherwise you will see the “No Video Available”... - Page 26 11. Once the OS has been installed, you can re-enable the port for the secondary HBA on the Qlogic switch. 12. Zone the McData switch for the WWPN of the second HBA http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind.
-
Page 27: Kickstart Information
13. Power on the server and enter the QLogic BIOS by pressing CTRL+Q. 14. From the Configurable Boot Settings, set the BOOT Device to the alternate controller's World Wide Name (that you gathered) and the BOOT LUN to 0. 15. From the Basic Settings panel, verify that the BIOS on this adapter is Enabled. 16. - Page 28 Attempting a linux rescue does not appear to work; there are no partitions found on /dev/sda. The problem appears to be with the autopartitioning code and/or the partitioning statements in the ks.cfg. If you are stuck and can’t get the second path disabled on the switch, then the following approach should work.
-
Page 29: Red Hat Linux As Broadcom Driver Installation
[root@fafrin root]# fdisk -l Disk /dev/sda: 18.1 GB, 18158714880 bytes 255 heads, 63 sectors/track, 2207 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start /dev/sda1 /dev/sda2 /dev/sda3 1544 /dev/sda4 1806 Figure 33 fdisk Output Note: If you see this message shown in Figure 34, answer no. In this situation, it may be possible to install successfully without the %pre script since it won’t attempt to partition the second path. -
Page 30: Issues Related To Red Hat As 2.1/3.0 And Blades
[root@localhost root]# [root@localhost root]# total 285 -rwxr-xr-x 1 root root 279887 Jan 6 20:06 basplnx-6.2.1-1.src.i386.rpm [root@localhost root]# [root@localhost root]# [root@localhost root]# [root@localhost root]# [root@localhost tmp]# rpm -ivh basplnx-6.2.1-1.src.i386.rpm 1:basplnx ########################################### [100%] [root@localhost tmp]# [root@localhost tmp]# cd /usr/src/redhat/ [root@localhost redhat]# [root@localhost redhat]# Execu ting(%prep): /bin/sh -e /var/tmp/rpm-tmp.4397 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.4397... - Page 31 Bladecenter to/from that blade, the USB "stuff" (KVM and media tray) get re-hotplugged, for lack of a better term, and the sdc errors are generated. They occur even after switch the media tray owner to a different blade. The result is too many spurious errors which is making the internal support team cranky.
- Page 32 One final note on this item: currently the media tray is required for installing Linux on the blades, but should only be needed in rare circumstances after that. Additionally, since there appear to be several combinations of events that will cause the EFI shell boot option to be restored in the boot sequence on a blade (see item 4), it is recommended that boot sequence be confirmed when an HS40 is rebooted until the f HS40 boots from the HBA during a restart.
- Page 33 If the host (blade server) resolution is not 1024 x following steps to provide the correct resolution data to the mouse handler: If you need to switch to b. To unload the mouse device driver module c. To notify the mouse device driver of the video resolution, add the /etc/modules.conf file:...
-
Page 34: Other Linux Distributions And Blades
At this point, the only new problem encountered was that our 8832 blade would not boot from the .iso image o It would boot from the actual CDROM mounted remotely, and the install. -
Page 35: Rdac Installation And Configuration
• If a host server has multiple HBA ports and each HBA port sees both controllers (via an un-zone switch), the Linux RDAC driver may return I/O errors during controller failover. • Linux SCSI device names have the possibility of changing when the host system reboots. We recommend using a utility such as devlabel to create user-defined device names that will map devices based on a unique identifier, called a UUID. -
Page 36: Summary Of Linux Changes For Sol
There are several well written white papers available that describe the steps required to configure Serial Over LAN for the BladeCe nter. The firmware sections of these documents can be skipped as long as you have followed the firmware recommendations in this document. For Linux, you must make the changes to both /etc/inittab and your boot loader configuration file (eith functionality under Linux. -
Page 37: Alt-Sysrq
4.8.3 alt-sysrq One of the reasons yo u may be configuring SOL is for utilizing the alt-sysrq “magic key sequence”. ter the command: > sysctl -w kernel.sysrq = 1 This will update that value in /etc/sysctl.conf and also set the value in oc/sys/kernel/sysrq to 1. -
Page 38: Netdump
IBM BladeCenter Deployment Guide Figure 37 alt-sysrq from PuTTY Note, PuTTY has a nice feature that will log all of your session output to a file. Specify that before opening the telnet session. In certain rare circumstances, the serial data of the 8843 Blade Server may stop flowing. This results in a blank SOL session with only a blinking cursor. - Page 39 Here is the link to the instructions from Red Hat that can be used to set up netdump: http://www.redhat.com/support/wpapers/redhat/netdump/setup.html The netdump server from the RHEL3U3 distribution is version 0.6.11-3. Basically, here is what needs to be done at the server: 1) rpm -i netdump-server-0.6.11-3.i386.rpm 2) passwd netdump 3) chkconfig netdump-server on...
-
Page 40: Install Process For Windows 2000 And 2003 Remote Boot
10. Once the OS has been installed, you can re-enable the port for the secondary HBA on the Qlogic switch. 11. If required, zone the external switch for the WWPN of the second HBA 12. Power on the server and enter the QLogic BIOS by pressing CTRL+Q. -
Page 41: Broadcom Advanced Control Suite Installation
8. Type the location of the Ethernet device drivers that you extracted in step 6. Select OK. 9. When the message "System found a driver for device" is displayed, select Next to install the device driver. 10. When you are prompted to restart the blade server, select Yes if you do not have more device drivers to install, or select No if you have another device driver to install. -
Page 42: Remote Update Of The Firmware On An Hs20 Or Hs40
1. In the navigation frame, click Firmware Update under Blade Tasks, select the Target blade slot number, then click Browse. 2. Switch to the drive and directory containing the unpackaged files. The package files are found on Diskette 2. Navigate to the PKT file you want to update, named as: BWBTxxy.PKT... - Page 43 Notes: 1. It is necessary to upgrade the BIOS on the HS40 to the 1.44 level and the ISMP to 1.24 if you are installing Linux on the blade. (See Table 5) There is a problem identified in the 1.32 version of the BIOS, related to EFI, which causes processors to be disabled.
-
Page 44: Flash The Qlogic Hba Bios
Flash the QLogic HBA BIOS Perform the following steps to flash the QLogic HBA BIOS: 1. Create a blank, bootable disk formatted with MS-DOS, PC-DOS, or DR-DOS. 2. Using a text editor, verify that the config.sys and autoexec.bat files are blank and contain no entries. 3. -
Page 45: Red Hat Boot Diskette Modification Process
Red Hat Boot Diskette Modification Process Copy the Network Boot Diskette (bootnet.img) and Supplemental Driver Diskette (drvnet.img) files from the installation media (CD or ISO) to a temporary staging area. # cp /mnt/cdrom/images/{bootnet.img,drvnet.img} . # ls bootnet.img drvnet.img Create the temporary mount point and two module directories. # mkdir -p mnt/{boot,drv,initrd} # ls mnt boot... - Page 46 # ls 2.4.9-e.40BOOT # ls 2.4.9-e.40BOOT/{tg3.o,e1000.o} tg3.o e1000.o Remove device drivers that will not be utilized from the driver modules extracted to the first module directory in step #7. Since the BladeCenter HS20 and BladeCenter HS40 servers do not support 3Com 3c59x, Intel 10/100, AMD PCnet32 or USB-OHCI devices, it is safe to remove the associated device drivers to free much needed space in the image.
- Page 47 # mv pcitable.new pcitable Remove driver entries from the module-info file that are associated with the device drivers removed in step #9. Remove all stanzas that begin with the name of a removed device driver (3c59x, eepro100, pcnet32, usb-ohci). For example: 3c59x "3Com 3c590/3c595/3c90x/3cx980"...
-
Page 48: Modify Or Create The Post-Install Driver Diskette Image
Copy the initrd.img file to the bootnet.img image. # mv initrd.img mnt/boot Unmount the bootnet.img. # umount mnt/boot At this point the bootnet.img file should be ready for use in remote deployments. Modify or Create the Post-install Driver Diskette Image Create the supplemental driver disk containing drivers that will be used in the post-install environment. - Page 49 gunzip -c < /tmp/mnt-$$/modules.cgz | cpio -id for moduleDir in *; do for moduleFile in `find $moduleDir -name "*.o"`; do if [ -d /lib/modules/$moduleDir ]; then targetModuleFiles=`find /lib/modules/$moduleDir -name "$moduleFile"` if [ ."$targetModuleFiles" = ."" ]; then cp $moduleFile /lib/modules/$moduleDir/kernel else for targetModuleFile in $targetModuleFiles;...
-
Page 50: Appendix A. Firmware Versions
Version Release Date Build Id Table 3 HS20 – Model 8832 Firmware Information http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM. -
Page 51: Table 4 Hs20 – Model 8843 Firmware Information
HS20 BIOS – Model 8843 Version Release Date Build Id Build Date HS20 BMC – (8843) Version Release Date Build Id HS20 Diagnostics (8843) Version Release Date Build Id Table 4 HS20 – Model 8843 Firmware Information HS40 BIOS Version Release Date Build Id Build Date... - Page 52 Release Date Build Id Table 6 LS20 BIOS and BMC Information QLogic HBA Firmware See 7 Flash the QLogic HBA BIOS for instructions on how to update the HBA BIOS. Version Table 7 QLogic Firmware Information Note: we are now recommending v1.43 for the QLogic BIOS. Old minimum was v1.38. http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 The BladeCenter Deployment Guide is intended as a personal productivity tool.
-
Page 53: Appendix B. Worksheets
User ID for BladeCenter Modules Password for BladeCenter Modules BladeCenter Components Management Module 1 Management Module 2 Ethernet Switch Module 1 Ethernet Switch Module 2 Fibre Channel Switch Module 1 Fibre Channel Switch Module 2 Blade Configuration Information Hostname IP Address SAN Configuration System... - Page 54 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM. IBM BladeCenter Deployment Guide Flash the BIOS... 44 Reset Default Settings ... 44 Testing boot LUN ... 21 WWPN ... 18 QLogic Switch Disabling ports ... 13 SANSurfer... 14 RDAC... 35 Red Hat anaconda ... 32 AS 2.1 Installation ...