ASM: ASM Disk status “_DROPPED_0001_****”

Hi,

Thanks to my colleague(Bhanu) fo raising this,

Issue with ASM disks not persistent in reboot (due to some mis configuration in creating devices or its assignments, thats a different story) causing ASM disks to be failed, fortunately the Voting Disk and the OCR were on the normal redundancy group which does not lid up the night mares for us.

Here you go, seeing the asm disk and status showing the status very wrong,

Point #1, the GRID disk group are part of Diskgroup 3 where the first two lines shows 0 as GN (group number)
Point #2, the line 4 shows the status of the diskgrup is dropped, (can be many reasons, hung after drop etc etc)
Point #3, Candidate disks are just candidates which need not to worry they have just allocated.

So we have problematic disks /dev/mapper/asmt1nrocr3,/dev/mapper/asmt1nrocr1 which are failgroups of GRID (normal redundancy)

col PATH for a50
col HEADER_STATUS for a15
col STATE for a15
col FAILGROUP for a20
col FAILGROUP_TYPE for a25
select group_number as gn, path, name, header_status, state,
failgroup,FAILGROUP_TYPE from v$asm_disk;

       GN PATH                       NAME                 HEADER_STATUS   STATE     FAILGROUP    FAILGROUP_TYPE
---------- -------------------------- -------------------- --------------- --------- ------------ -------------------------
         0 /dev/mapper/asmt1nrocr3                         MEMBER          NORMAL                 REGULAR
         0 /dev/mapper/asmt1nrocr1                         MEMBER          NORMAL                 REGULAR
         0 /dev/mapper/asmt1nrredo2                        CANDIDATE       NORMAL                 REGULAR
         3                            _DROPPED_0001_GRID   UNKNOWN         FORCING   GRID_0001    REGULAR
         0 /dev/mapper/asmt1nrredo1                        CANDIDATE       NORMAL                 REGULAR
         2 /dev/mapper/asmt1nr0002    FRA01_0000           MEMBER          NORMAL    FRA01_0000   REGULAR
         1 /dev/mapper/asmt1nr0001    DATA01_0000          MEMBER          NORMAL    DATA01_0000  REGULAR
         3 /dev/mapper/asmt1nrocr2    GRID_0000            MEMBER          NORMAL    GRID_0003    REGULAR

Lets check the voting disk status , to confirm which disk is now accessible

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   556186cec0944fafbf8d03cd09767110 (/dev/mapper/asmt1nrocr2) [GRID]
Located 1 voting disk(s).

We have only one working disk which is /dev/mapper/asmt1nrocr2 so the rest two above in Point#1 were just name sake there.

Another check for OCR, okay, OCRCHECK only shows the diskgroup name not the path of the disks, you can skip this step if you want.

$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2964
         Available space (kbytes) :     259156
         ID                       :   26755138
         Device/File Name         :      +GRID
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

To fix this, (ensure, double ensure you are adding right disks),

1) add to dummy diskgroup (use force option as their metadata is already part of somedisk as it knows, not we ;))
2) And drop the dummy diskgroup

SQL> create diskgroup DUMMY external redundancy 
        disk '/dev/mapper/asmt1nrocr3' force;

Diskgroup created.

SQL> drop diskgroup dummy;

Diskgroup dropped.

SQL> create diskgroup DUMMY1 external redundancy 
     disk '/dev/mapper/asmt1nrocr1'

Diskgroup created.

SQL> drop diskgroup dummy1;

Diskgroup dropped.

When you verify the disks become FORMER, as like below.

SQL> select group_number as gn, path, name, header_status, state,
        failgroup,FAILGROUP_TYPE from v$asm_disk;

        GN PATH                      NAME               HEADER_STATUS   STATE    FAILGROUP  FAILGROUP_TYPE
---------- ------------------------- ------------------ --------------- -------- -------------------------
         0 /dev/mapper/asmt1nrocr3                      FORMER          NORMAL              REGULAR
         0 /dev/mapper/asmt1nrocr1                      FORMER          NORMAL              REGULAR
         0 /dev/mapper/asmt1nrredo2                     CANDIDATE       NORMAL              REGULAR
         3                           _DROPPED_0001_GRID UNKNOWN         FORCING  GRID_0001  REGULAR
         0 /dev/mapper/asmt1nrredo1                     CANDIDATE       NORMAL              REGULAR
         2 /dev/mapper/asmt1nr0002   FRA01_0000         MEMBER          NORMAL   FRA01_0000 REGULAR
         1 /dev/mapper/asmt1nr0001   DATA01_0000        MEMBER          NORMAL   DATA01_0000REGULAR
         3 /dev/mapper/asmt1nrocr2   GRID_0000          MEMBER          NORMAL   GRID_0003  REGULAR

8 rows selected.

Now add back to the disks to failure groups of diskgroup GRID

SQL> alter diskgroup GRID add failgroup GRID_0001 
        disk '/dev/mapper/asmt1nrocr1';

Diskgroup altered.

SQL>  alter diskgroup GRID add failgroup GRID_0002 
        disk '/dev/mapper/asmt1nrocr3';

Diskgroup altered.

Verify the disks now, Whoa!!! the dropped disk also gone and the disks are part of GRID now,

SQL>  select group_number as gn, path, name, header_status, 
         state,failgroup,FAILGROUP_TYPE from v$asm_disk;

        GN PATH                     NAME        HEADER_STATUS   STATE  FAILGROUP   FAILGROUP_TYPE
---------- ------------------------ ----------- --------------- ------ ----------- --------------
         0 /dev/mapper/asmt1nrredo2             CANDIDATE       NORMAL             REGULAR
         0 /dev/mapper/asmt1nrredo1             CANDIDATE       NORMAL             REGULAR
         2 /dev/mapper/asmt1nr0002  FRA01_0000  MEMBER          NORMAL FRA01_0000  REGULAR
         1 /dev/mapper/asmt1nr0001  DATA01_0000 MEMBER          NORMAL DATA01_0000 REGULAR
         3 /dev/mapper/asmt1nrocr3  GRID_0001   MEMBER          NORMAL GRID_0002   REGULAR
         3 /dev/mapper/asmt1nrocr2  GRID_0000   MEMBER          NORMAL GRID_0003   REGULAR
         3 /dev/mapper/asmt1nrocr1  GRID_0002   MEMBER          NORMAL GRID_0001   REGULAR

7 rows selected.

Verify the voting disks, cool three disks were on the slide

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   556186cec0944fafbf8d03cd09767110 (/dev/mapper/asmt1nrocr2) [GRID]
 2. ONLINE   08922b1061924ffbbf147a7afa5aac83 (/dev/mapper/asmt1nrocr1) [GRID]
 3. ONLINE   6fd6e2184f544f0dbfa855a8812b9ca1 (/dev/mapper/asmt1nrocr3) [GRID]
Located 3 voting disk(s).

Last check if you want, ocrcheck

$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2964
         Available space (kbytes) :     259156
         ID                       :   26755138
         Device/File Name         :      +GRID
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

Hope this helps to you all as well