PROC-26: Error while accessing the physical storage, PROT-602: ,OCR/Voting Disk fail to mount, OCR corruption

Want to know new features in 12c, 18c, 19c, 20c ??? Follow this

More than 100+ features listed here : http://db.geeksinsight.com/category/12c-database/

More than 50+ features listed here: http://db.geeksinsight.com/category/18c-database/

More than 40 features listed here: http://db.geeksinsight.com/category/19c-database/

Upcoming 20c features : http://db.geeksinsight.com/category/20c-database/

 

Issue: CRSD not coming up, the other resource until CRS, (need CRS startup sequence, see here) CRSD is failing with error, Possible OCR corruption.

in $GI_HOME/log/<nodename>crsd.log

PROC-26: Error while accessing the physical storage,

Environment:-

GI Version: 11.2.0.2, RDBMS Version: 11.2.0.2 , Two Node RAC on Linux

We are not using ASMLIB, using block devices directly i.e /dev/mapper in asm_diskstring.

First lets check the environment,

Environment Checks:-

Verify OCR Diskgroup & its location

[root@racp001 bin]# ./ocrcheck
Errors in file :
ORA-27091: unable to queue I/O
ORA-15081: failed to submit an I/O operation to a disk
ORA-06512: at line 4
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
[root@racp001 bin]#

Verify VOTING DISK & its location

[root@racp001 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 693f8f3d88884f36bfb73dbb23a73a22 (/dev/mapper/disk1-1gbp1) [DATA]
Located 1 voting disk(s).

Note:- as its in 11gr2 , the crsd need not to come up.

Check the OCR location in /etc/oracle/ocr.log

[root@racp001 bin]# cat /etc/oracle/ocr.loc

ocrconfig_loc=+DATA/registry.255.71929292

local_only=FALSE

Try manually mounting the disk group +DATA, to see any issues appearing.

SQL> alter diskgroup data mount

Diskgroup Mounted.

SQL>

It seem that DATA diskgroup is mounted, but still CRS is unable to read due to some issue.

Okay, I am concluding this is physical corruption , (as I have no time , have to bring up the nodes), so decided to rebuild the OCR/Voting Disk. Here are the steps for the same.

Resolution Steps:-

Step #1:- First ensure you have valid backup of OCR.

[root@racp001 bin]# ./ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

racp001     2012/09/29 21:38:05     /home/11.2.0/grid/cdata/racp001-scan/backup00.ocr

racp001     2012/09/29 17:38:05     /home/11.2.0/grid/cdata/racp001-scan/backup01.ocr

racp001     2012/09/29 13:38:04     /home/11.2.0/grid/cdata/racp001-scan/backup02.ocr

racp001     2012/09/28 01:38:00     /home/11.2.0/grid/cdata/racp001-scan/day.ocr

racp001     2012/09/20 01:37:40     /home/11.2.0/grid/cdata/racp001-scan/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

I have a valid automatic backup, so I can restore the OCR. Now I proceed with the other steps. If you dont have backup, please do not proceed.

Step #2:- Login to ASM and drop the diskgroup DATA (my ocr/voting diskgroup name is DATA)

SQL> drop diskgroup data;
drop diskgroup data
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15276: ASM diskgroup DATA has cluster voting files

SQL>  drop diskgroup data force including contents;
drop diskgroup data
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15276: ASM diskgroup DATA has cluster voting files

Well its not allowing me to do the same, further the cluster is up so its not allowing me to do so.

Step #3: Stop CRS force

./crsctl stop crs -f
[root@racp001 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racp001'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racp001'
CRS-2673: Attempting to stop 'ora.asm' on 'racp001'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racp001'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racp001'
CRS-2677: Stop of 'ora.mdnsd' on 'racp001' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racp001' succeeded

.... ctssd waiting for asm to shut down (check in alert log, I have killed manually)

CRS-2677: Stop of 'ora.asm' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racp001' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racp001'
CRS-2677: Stop of 'ora.cssd' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racp001'
CRS-2677: Stop of 'ora.gipcd' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racp001'
CRS-2677: Stop of 'ora.gpnpd' on 'racp001' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racp001' has completed

Stop the HAS as well.

[root@racp001 bin]# ./crsctl stop has
CRS-4133: Oracle High Availability Services has been stopped.

Step #4:- Start CRS in exclusive mode with nocrs option, (11.2.0.2 got nocrs option)

crsctl start crs –excl –nocrs

[root@racp001 bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'racp001'
CRS-2676: Start of 'ora.mdnsd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racp001'
CRS-2676: Start of 'ora.gpnpd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racp001'
CRS-2672: Attempting to start 'ora.gipcd' on 'racp001'
CRS-2676: Start of 'ora.cssdmonitor' on 'racp001' succeeded
CRS-2676: Start of 'ora.gipcd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racp001'
CRS-2672: Attempting to start 'ora.diskmon' on 'racp001'
CRS-2676: Start of 'ora.diskmon' on 'racp001' succeeded
CRS-2676: Start of 'ora.cssd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'racp001'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2672: Attempting to start 'ora.ctssd' on 'racp001'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2676: Start of 'ora.drivers.acfs' on 'racp001' succeeded
CRS-2676: Start of 'ora.ctssd' on 'racp001' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racp001'
CRS-2676: Start of 'ora.asm' on 'racp001' succeeded
[root@racp001 bin]#

Step #5:- Now try dropping the disk group.

SQL>  drop diskgroup data force including contents;
drop diskgroup data
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15276: ASM diskgroup DATA has cluster voting files

Still not allowing,

Lets try in different way.

Note:- the diskgroup name is OCRDATA, the old group name is DATA

SQL> create diskgroup OCRDATA disk '/dev/mapper/disk1-1gbp1' disk '/dev/mapper/disk2-1gbp1' disk '/dev/mapper/disk3-1gbp1' attribute 'compatible.asm'='11.2';
create diskgroup DATA disk '/dev/mapper/disk1-1gbp1' disk '/dev/mapper/disk2-1gbp1' disk '/dev/mapper/disk3-1gbp1' attribute 'compatible.asm'='11.2'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15033: disk '/dev/mapper/disk2-1gbp1' belongs to diskgroup "DATA"
ORA-15033: disk '/dev/mapper/disk3-1gbp1' belongs to diskgroup "DATA"

Oops , I have bumped now, not allowing. Fortunately you can add disk forcibly.

SQL> create diskgroup OCRDATA force disk '/dev/mapper/disk1-1gbp1' force disk '/dev/mapper/disk2-1gbp1' force disk '/dev/mapper/disk3-1gbp1' attribute 'compatible.asm'='11.2';

Diskgroup created.

Whoa! its added.

Step #7 : Change the diskgroup location in ocr.loc in /etc/oracle/ocr.loc

[root@racp001 bin]# cat /etc/oracle/ocr.loc

ocrconfig_loc=+OCRDATA/*****

local_only=FALSE

Step #8:- Restore the OCR backup to newly created disk

./ocrconfig -restore /home/11.2.0/grid/cdata/racp001-scan/backup00.ocr

<Prompt returns with no errors or output>

Verify the OCR is back or not

./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2824
Available space (kbytes) : 259296
ID : 1423756622
Device/File Name : +OCRDATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured

Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user

Cool, its back now, Next replace the voting disk

Step #9:-  Replace the voting disk back

[root@racp001 bin]# ./crsctl replace votedisk +OCRDATA
Failed to create voting files on disk group OCRDATA.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.
[root@racp001 bin]#

Stumped again.

Again bounced the CRS and started with excl option. then tried

[root@racp001 bin]# ./crsctl replace votedisk +OCRDATA

Successful addition of voting disk 4e47edd858334f49bf2325c9c9dca437.
Successful addition of voting disk 0d2a508b89a54f5ebfb463d9cb699a99.
Successful addition of voting disk 314bb56f40574f30bf514e96e9b3f504.

[root@racp001 bin]#crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE 4e47edd858334f49bf2325c9c9dca437 (/dev/mapper/disk1-1gbp1' )
2. ONLINE 0d2a508b89a54f5ebfb463d9cb699a99 (/dev/mapper/disk2-1gbp1' )
3. ONLINE 314bb56f40574f30bf514e96e9b3f504 (/dev/mapper/disk3-1gbp1' )

Step 10#:- Restart entire cluster and bring it up normal.

[root@racp001 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racp001'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racp001'
CRS-2673: Attempting to stop 'ora.asm' on 'racp001'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racp001'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racp001'
CRS-2677: Stop of 'ora.mdnsd' on 'racp001' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racp001' succeeded

CRS-2677: Stop of 'ora.asm' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racp001' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racp001'
CRS-2677: Stop of 'ora.cssd' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racp001'
CRS-2677: Stop of 'ora.gipcd' on 'racp001' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racp001'
CRS-2677: Stop of 'ora.gpnpd' on 'racp001' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racp001' has completed

[root@racp001 bin]# ./crsctl start resource –all

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'racp001'
CRS-2676: Start of 'ora.mdnsd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racp001'
CRS-2676: Start of 'ora.gpnpd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racp001'
CRS-2672: Attempting to start 'ora.gipcd' on 'racp001'
CRS-2676: Start of 'ora.cssdmonitor' on 'racp001' succeeded
CRS-2676: Start of 'ora.gipcd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racp001'
CRS-2672: Attempting to start 'ora.diskmon' on 'racp001'
CRS-2676: Start of 'ora.diskmon' on 'racp001' succeeded
CRS-2676: Start of 'ora.cssd' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'racp001'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2672: Attempting to start 'ora.ctssd' on 'racp001'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racp001'
CRS-2676: Start of 'ora.drivers.acfs' on 'racp001' succeeded
CRS-2676: Start of 'ora.ctssd' on 'racp001' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racp001' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racp001'
CRS-2676: Start of 'ora.asm' on 'racp001' succeeded

… this should also start your database/listener/scanlisteners/instance/services as i have not specified init here.
[root@racp001 bin]#

Now all are back to normal.

-Hope this helps.

Geek DBA