Cluster not starting- cssd(12103)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

Hello All,

Long time !!! Its been very busy schedule and could not spend much on the blog at all.

Anyways back now, and here this time with gpnp profile.

Issue is , the node 2 not starting due to gpnp profile missing/mismatch. Following in the crs alert log in node 2

[ohasd(5347)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2014-11-16 10:14:54.881:
[gipcd(5858)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_COMM_ERR (Communication error).
[ohasd(11783)]CRS-2769:Unable to failover resource 'ora.gpnpd'.
2014-11-16 10:01:38.505:
[cssd(12103)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-11-16 10:01:38.506:
[cssd(12103)]CRS-1703:Initialization of the required component GPNP failed because the GPNP server daemon is not up; details at (:CSSSC00004:) in
2014-11-16 10:01:41.511:
[cssd(12103)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in
2014-11-16 10:01:44.344:
[ohasd(11783)]CRS-2771:Maximum restart attempts reached for resource 'ora.gpnpd'; will not restart.

Those who does not know about gpnp profile, gpnp profile is new in 11gr2 and used extensively by cluster when start and any modifications done to cluster, for example, any of the following tools used

    - srvctl
    - oifcfg
    - crsctl

Now upon investigation, understood that the following changes were made

    - Private network has been modified, 192.168.1 to 10.10.1 and 192.168.2 to 10.10.2
    - While doing modification using oifcfg the node 2 is down

Well, this is not good, since any changes to cluster configuration will have to populate in the gpnp profile in all nodes, located at 

    $ORACLE_HOME/gpnp/peer/profiles/profile.xml --> global profile

    $ORACLE_HOME/gpnp/rac02/profiles/peer/profile.xml --> local profile

So when the oifcfg used it updates the local profile and copies that profile to other nodes using gpnp agent.

Now, in our case the gpnp is not in sync across the nodes the node 2 got failed to start.

To investigate, first check the node 2 profile using gpnptool

Note: gpnptool is only way to update your profile, do not modify it manually, since this got signature and wallet update the profile may get corrupted.

Note: the profile is edited for clarity.

[root@rac02 gpnp]#  gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
         /u01/gi/app/oracle/12.1.0.1/grid/bin/gpnptool.bin get -o-

Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
GPnP service is not running on localhost. Found locally cached profile...
...

<gpnp:Network id="net1" IP="192.168.56.0" Adapter="eth0" Use="public"/><gpnp:Network id="net2" IP="192.168.1.0" Adapter="eth1" Use="cluster_interconnect"/><gpnp:Network id="net3" IP="192.168.2.0" Adapter="eth2" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+OCRDG/rac-cluster/ASMPARAMETERFILE/registry.253.862511131" Mode="legacy"/ Success.

Error CLSGPNP_NO_DAEMON getting profile.

Error is due to GPNP daemon not running and also if you look the network shows old address 192.168* rather 10.10.

I have tried to use rget which is remote profile get, and it is intact showing correct private network address. 10.10.

[root@rac01 peer]# gpnptool rget
Warning: some command line parameters were defaulted. Resulting command line:
         /u01/gi/app/oracle/12.1.0.1/grid/bin/gpnptool.bin rget -o-
Found 1 gpnp service instance(s) to rget profile from.

get-profile request to tcp://rac01:40082 (mdns:service:gpnp._tcp.local.://rac01:40082/agent=gpnpd,cname=rac-cluster,guid=01292fcb30feff20ff18cd0b98bb3adc,host=rac01,pid=6228/gpnpd h:rac01 c:rac-cluster u:01292fcb30feff20ff18cd0b98bb3adc):

<gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.56.0" Adapter="eth0" Use="public"/><gpnp:Network id="net3" Adapter="eth2" Use="cluster_interconnect" IP="10.10.2.0"/><gpnp:Network id="net2" Adapter="eth1" Use="cluster_interconnect" IP="10.10.1.0"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+OCRDG/rac-cluster/ASMPARAMETERFILE/registry.253.862511131" Mode="legacy"/

Success.

Lets proceed to modify the profile.xml in node 2, Navigate to local profile location in node2

[root@rac02 peer]# cd /u01/gi/app/oracle/12.1.0.1/grid/gpnp/rac02/profiles/peer

Copy the profile to profile.back.

[root@rac02 peer]# cp profile.xml profile.bak

Un-sign the profile using gpnptool unsign, this will remove the signature from profile.

[root@rac02 peer]# gpnptool unsign -p=profile.bak
Warning: some command line parameters were defaulted. Resulting command line:
         /u01/gi/app/oracle/12.1.0.1/grid/bin/gpnptool.bin unsign -p=profile.bak -o-

<gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.56.0" Adapter="eth0" Use="public"/><gpnp:Network id="net2" IP="192.168.1.0" Adapter="eth1" Use="cluster_interconnect"/><gpnp:Network id="net3" IP="192.168.2.0" Adapter="eth2" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+OCRDG/rac-cluster/ASMPARAMETERFILE/registry.253.862511131" Mode="legacy"/></gpnp:GPnP-Profile>
Success.

Edit the profile, overwrite and change the sequence as well.

<pre>
[root@rac02 peer]# gpnptool edit  -net2:net_ip="10.10.1.0" -net3:net_ip="10.10.2.0" -prf_sq=7 -p=profile.bak -o=profile.bak -ovr
Resulting profile written to "profile.bak".
Success.
[root@rac02 peer]#

</pre>

Sign back the profile, this will add the signature in profile and as well as updates the wallet, here take the wallet location.

[root@rac02 peer]# gpnptool sign -p=profile.bak -w=file:/u01/gi/app/oracle/12.1.0.1/grid/gpnp/rac02/wallets/peer -o=profile.new

Copy back the profile.bak to profile.xml

[root@rac02 peer]# cp profile.bak profile.xml

Verify the IP address changed to 10.10.

[root@rac02 peer]# cat profile.xml
<gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.56.0" Adapter="eth0" Use="public"/><gpnp:Network id="net2" IP="10.10.1.0" Adapter="eth1" Use="cluster_interconnect"/><gpnp:Network id="net3" IP="10.10.2.0" Adapter="eth2" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+OCRDG/rac-cluster/ASMPARAMETERFILE/registry.253.862511131" Mode="legacy"/>

[root@rac02 peer]#

Noticed? the private address got changed now, hope so this will work

stop the crs , use -f option since the HAS stack only running no crs is not running.

[root@rac02 peer]# crsctl stop crs -f

start the crs, normally

[root@rac02 peer]# crsctl start crs

In alert log,

[gpnpd(7017)]CRS-2328:GPNPD started on node rac02.
2014-11-16 10:58:27.978:

Check in the configuration in node 2 using oifcfg

[root@rac02 rac02]# oifcfg getif -global
eth0  192.168.56.0  global  public
eth2  10.10.2.0  global  cluster_interconnect
eth1  10.10.1.0  global  cluster_interconnect

Just in case if you want to edit the spfile the following command can be used,
<pre>
[root@rac02 peer]# gpnptool edit -asm:asm_spf='+OCRDG/rac-cluster/ASMPARAMETERFILE/registry.253.862511131' -asm_dis='' -p=profile.bak -o=profile.bak -ovr -prf_sq=8
Resulting profile written to "profile.new".
Success.
[root@rac02 peer]#
</pre>

-Thanks

Geek DBAGeek DBA