11gr2 gipcd connection refuse : one of the node not starting

Well , we have got an issue that one of the node is not starting up , its a two node GRID Infrastructure and the logs shown as below

in crs alert_rac01.log

CRSD not started – unable to locate OCR

in gipcd.log

2014-09-10 06:23:00.186: [    GIPC][1117619840] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 730], original from [clsCrsctlUtil.cpp : 2934]
2014-09-10 06:23:00.186: [  OCRMSG][1117619840]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2014-09-10 06:23:00.186: [  OCRMSG][1117619840]GIPC error [29] msg [gipcretConnectionRefused]
2014-09-10 06:23:00.186: [  OCRMSG][1117619840]prom_connect: error while waiting for connection complete [24]
2014-09-10 06:24:24.809: [  OCRMSG][240011008]GIPC error [29] msg [gipcretConnectionRefused]

When verified the private network and public networks are working fine

Another important thing is if you stop the running node and start the other node it works fine, so that means at any point of time only one node is working.

This is some what known issue and have already written something about back in 2012 and at that time instances were crashing due to cluster crash. But still old issue.

Just to make everyone aware , ensure there should be no internal devices like usb0 or something enabled in cluster since the private network uses haip with internal IP in 169.* format and the internal devices also uses same range to communicate hence, the crsd get confused and reach the usb device rather cluster node and does not found any OCR. Sounds funny but yes. Read here

http://db.geeksinsight.com/2012/09/24/pmon-ospid-nnnn-terminating-the-instance-due-to-error-481/

Hope this helps