ORA-15036: disk ‘string’ is truncated, ASM disks group DATA is not mounting

Hello,

Another issue came up today!!!

Issue:- ORA-15036: disk 'string' is truncated The size of the disk, as reported by the operating system, was smaller than the size of the disk as recorded in the disk header block on the disk.

Apparently the concerned disk group is not mounting, unfortunately this contains our OCR Disks (okay, bigger problem)

Sequence of actions or the issues encountered:-

1) DBA found that disk partitions are not visible, passed to unix to fix the issue

2) Unix tried many alternatives and involved Novell (Suse Linux) as well , and Novell wisely rewritten the disk headers and they are visible now

3) Passed to DBA, now the DBA turn to mount the diskgroups

4) Got hit with ORA-15036, and CRSD failing to come up , parallely its resources as well.

5) Raised SR, Oracle suggested to recreate and restore the diskgroups data

6) Novell, said they can't do anything.

Strucked ....... and Corner end..... what else?

One of our charming DBA raised that can we modify the disk header?, so we asked Novell but they said, they cant, and moreover the issue is at ASM Lib looking and trying to mount with partition (size x), where in the asm header of the disk contains (size y), and fails to do so.

Surely that point hitted me and the tool KFED comes into rescue.

Now here you go, look at the sequence

1) how we identify the disk header and its by reading the disk contents through kfed read

2) Modified the disk header with exact size of the partition through kfed.

CAUTION:- DO AT YOUR OWN RISK (DO NOT TRY IN YOUR PRODUCTION SYSTEMS UNLESS YOU HAVE STRONG REASON TO DO SO)

Output edited for clarity:-

XXXXXXXXXXX:~ # /u01/gi/oragrid/grid/11.2.0/bin/kfed read /dev/sda5
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check: 3202865560 ; 0x00c: 0xbee7d998
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:ORCLDISKASMT1NROCR1 ; 0x000: length=19
kfdhdb.driver.reserved[0]: 1414353729 ; 0x008: 0x544d5341
kfdhdb.driver.reserved[1]: 1330794033 ; 0x00c: 0x4f524e31
kfdhdb.driver.reserved[2]: 3232323 ; 0x010: 0x00315243
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA ; 0x028: length=4 --> Disk Name
kfdhdb.grpname: DATA ; 0x048: length=4 --> Disk Group Name
kfdhdb.fgname: DATA ; 0x068: length=4 --> Failure Disk group Name
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32954477 ; 0x0a8: HOUR=0xd DAYS=0x3 MNTH=0x6 YEAR=0x7db
kfdhdb.crestmp.lo: 3094464512 ; 0x0ac: USEC=0x0 MSEC=0x72 SECS=0x7 MINS=0x2e
kfdhdb.mntstmp.hi: 32957329 ; 0x0b0: HOUR=0x11 DAYS=0x1c MNTH=0x8 YEAR=0x7db
kfdhdb.mntstmp.lo: 1886795776 ; 0x0b4: USEC=0x0 MSEC=0x18e SECS=0x7 MINS=0x1c
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 204790 ; 0x0c4: 0x00031ff6 ---> Disk size in KB
kfdhdb.pmcnt: 3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
........
kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000

As you can see the disk size (kfdhdb.dsksize: 204790 ) 199.99GB , let see the disk header size using fdisk

[root@rac2 ~]# fdisk –l /dev/sda

Disk /dev/sda: 199GB, 213674622976 bytes

255 heads, 63 Sectors/track, 652 Cylinders

Units = cylinder of *** x 512 = ******

Did you observed the difference, if not

ASM Disk size in KB in kfed report says = 204790 * 1024  * 1024 = 214737879040

Fdisk size reports = 213674622976

Difference of Bytes: (214737879040-213674622976) =  1063256064 Bytes  i.e 0.99 GB less than what asm is header contains.

Now, How to fix it? Again using same kfed but with write option now

kfed op=read dev=/dev/sda5 | sed -e '24,24s/ '204790' / '203776' /' -e '24,24s/length=.*/length='$newlength'/' > /tmp/$shortname.kfed

kfed op=write dev=$file text=/tmp/$shortname.kfed CHKSUM=YES

Note: My kfed report contains only one appearance of 204790 hence I used it directly, if you have multiple entries (just in case) you may have to wisely choose different approach for changing your disk size.

Now again fire kfed read command

XXXXXXXXXXX:~ # /u01/gi/oragrid/grid/11.2.0/bin/kfed read /dev/sda5
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check: 3202865560 ; 0x00c: 0xbee7d998
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:ORCLDISKASMT1NROCR1 ; 0x000: length=19
kfdhdb.driver.reserved[0]: 1414353729 ; 0x008: 0x544d5341
kfdhdb.driver.reserved[1]: 1330794033 ; 0x00c: 0x4f524e31
kfdhdb.driver.reserved[2]: 3232323 ; 0x010: 0x00315243

kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA ; 0x028: length=4 --> Disk Name
kfdhdb.grpname: DATA ; 0x048: length=4 --> Disk Group Name
kfdhdb.fgname: DATA ; 0x068: length=4 --> Failure Disk group Name
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32954477 ; 0x0a8: HOUR=0xd DAYS=0x3 MNTH=0x6 YEAR=0x7db
kfdhdb.crestmp.lo: 3094464512 ; 0x0ac: USEC=0x0 MSEC=0x72 SECS=0x7 MINS=0x2e
kfdhdb.mntstmp.hi: 32957329 ; 0x0b0: HOUR=0x11 DAYS=0x1c MNTH=0x8 YEAR=0x7db
kfdhdb.mntstmp.lo: 1886795776 ; 0x0b4: USEC=0x0 MSEC=0x18e SECS=0x7 MINS=0x1c
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 203776 ; 0x0c4: 0x00031ff6 ---> Disk size in KB, this time it got changed
kfdhdb.pmcnt: 3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
........

Now, tried to restart the ASM and VIOLA !!!! its worked , diskgroup was mounted without any issues and the as usual support saga continues  !!!!

(a big relief for me !!)

-Hope this helps

Geek DBA