PMON (ospid: nnnn): terminating the instance due to error 481

We got a known issue with ASM not coming up in second and subsequently failed to start crs and other resources.

Review grid alert log & os Logs

• $GRID_HOME/log/<nodename>/alert<nodename>.log

oifcfg shows

$oifcfg –getif

eth0 192.168.2.10 global_clusterinterconnect

eth1 192.168.10.2 global

usb0 169.254.95.0

eth0:2 169.254.96.0

eth0:3 169.254.95.0

From 11g R2 (I believe 11.2.0.2 onwards) there is a cluster resource called HAIP which used to manage the cluster interconnects high availability. Prior to 11gr2 if cluster interconnect goes down there will be hang/node evictions depends on the situation. Where in from 11gr2 onwards we can specify up to 3 (as I known) cluster interconnects for a cluster which internally manages with this non-routable IP’s, Essentially, even if one of the physical interface is offline, private interconnect traffic can be routed through the other available physical interface. This leads to highly available architecture for private interconnect traffic.

Nice explanation from Riyaz’s Note:-

HAIP, High Availability IP, is the Oracle based solution for load balancing and failover for private interconnect traffic. Typically, Host based solutions such as Bonding (Linux), Trunking (Solaris) etc is used to implement high availability solutions for private interconnect traffic. But, HAIP is an Oracle solution for high availability. During initial start of clusterware, a non-routeable IP address is plumbed on the private subnet specified. That non-routable IP is used by the clusterware and the database for private interconnect traffic.

Now back to the issue:-

As you can see the usb0 is attached to the the internal IP (169.254.X.X is an non routable IP’s range internal to OS) , clusterware will confused with this and unable to start the crs resources promptly.

Clusterware picked two IP addresses on 169.254.x.x subnet on eth1 private interface as shown below. These two IP addresses will be used by the clusterware and RAC database for private interconnect traffic.

$ifconfig –a
...
eth0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 169.254.95.95 netmask ffff8000 broadcast 169.254.95.255
eth0:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 169.254.95.96 netmask ffff8000 broadcast 169.254.95.255
…

Review of database shows that these two IP addresses are used for private interconnects in node 1.

SQL> select * from gv$cluster_interconnects
INST_ID     NAME             IP_ADDRESS           IS_ SOURCE
---------- --------------- ---------------- --- -----------------------------1           eth0:3           169.254.95.95        NO
1           eth0:2           169.254.95.96        NO

Solution:-

For now we got to know that some intermediate device like USB0, (usb printers also possible) enabled in server causing this issue we have asked Unix team to disable the same, Further also request them to disable at bios level to not get repeated the same after a reboot.

Hope this helps!!!!