It is important to change the MTU for IB network, since by default it is set as 1500 (default value for ethernet), which will not result in optimal performance for IB traffic. Infiniband specific configuration and commands Change MTU for IB card for the cluster Then cd /export/rocks/install rocks create distro, and ssh into each compute node and reinstall it by /boot/kickstart/cluster-kickstart. From head node, you may also do a rocks list host profile compute-0-0 > /dev/null to check if there is any error message in compiling this xml file in kickstart script. The direction of stderr and stdout is to help diagnose potential problems if installation fails.
etc/init.d/openibd restart 2> /root/temp5 > /root/temp5.5 Then edit /export/rocks/install/site-profiles/6.2/nodes/extend-compute.xml and add the lines to the section: Then do a mkdir -p /export/rocks/install/contrib/extra/install and copy this file to this directory.
#Centos 7 static ip speed duplex 100 download
To address this problem, I have to download OFED version 3.4-1.0.0.0 (which is suitable for CentOS 6.6) from. This occurs in some systems, but not other systems. I still get an error umad_init: can't read ABI version from /sys/class/infiniband_mad/abi_version if I run service openibd restart service opensmd restart followed by ibhosts.
#Centos 7 static ip speed duplex 100 install
I found that simply doing yum install openib ibutils infiniband-diags opensm does not solve the problem per se. However, occasionally this is not the case, and sometimes you may want to add IB to an existing Rocks installation. It seems that Rocks 7 automatically handles these types of issues.īy default, if an infiniband card is already present in the system, then IB-related packages (such as opensm and ibutils and drivers) will be installed automatically by Rocks. The following description only applies to version 6. Similarly, the configuration files below are also automatically generated by Rocks and you should not edit them, as any edit will not survive a refreshment of Rocks configuration or a ~]# cat ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0 I recommend that you put IP addresses for IPMI into this file (technically Rocks has the ability to handle IPMI by DHCP but it rarely works well, so it is best that you just set up static IP for IPMI ports and put these IP into the host file). Instead, to define new hosts, you must edit /etc/hosts.local file. In other words, you should never edit the /etc/hosts file yourself, because any edits will not be preserved as this file will be generated automatically by Rocks whenever the system refreshes configuration file or when the system restarts. However, the user-defined hosts are stored in /etc/hosts.local file, and rocks sync config and rocks sync network will copy this to all cluster. These files includ /etc/nf which gives name server information, /etc/hosts which gives host information (as they may not be in name server), /etc/sysconfig/network-scripts/ifcfg-eth0 for configuring network interface. In a typical Linux system, there are several configuration files for enabling network in a host machine. For example, setting up new network should be done by rocks list/set host network commands, and setting up firewall should be done by rocks list/set host firewall commands. So it is very important to always use rocks command to administrater networks, rather than using the typical Linux commands (again, because if you use typical Linux command, the changes are not stored in MySQL and will not survive compute node re-installation or system upgrades).
I learned this in a very hard way, because it is not really documented anywhere clearly. One of the most important things, that you cannot easily learn from other places such as Rocks manual, is that Rocks uses a MySQL database to store all information related to network configuration. I also give some tricks and tips for troubleshooting the system and for accomplishing common tasks. In this section, I mostly describe those administrative tasks that are important yet unique for Rocks cluster operating system. Network configuration and administration is a complex topic.