RAC: Cluster Interconnect performance troubleshooting – Quick reference

A quick troubleshooting reference for cluster interconnect performance in Oracle RAC to Identify network and contention issues.

a) Check for "gc cr lost blocks" wait event in Automatic Workload Repository (AWR)/sysstats.

    If found, check for these errors on the NIC:

        Dropped packets/fragments
        Buffer overflows
        Packet reassembly failures or timeouts
        TX/RX errors

    Use these commands to find any errors:

        netstat -s
        Ifconfig -a
        ORADEBUG

 b) Identify Interconnect performance from AWR.

    Under Global Cache and Enqueue Services - Workload Characteristics

        Avg global cache cr block receive time (ms): should be <=15 ms

        Global Cache and Enqueue Services - Messaging Statistics

        Avg message sent queue time on ksxp (ms): should be <1 ms

        Under Interconnect Ping Latency Stats

        Avg Latency 8K msg should be close to Avg Latency 500B msg.

c) These wait events from AWR/sysstat can indicate contention related to RAC.

        GC current block busy
        GV cr block busy
        GC current buffer busy
        GC buffer busy acquire/release

d) These wait events in the AWR indicate that there might be a Hot Block that is causing these wait events. From the AWR Segment Statistics, you can find the objects.

        Enq:TX Index Contention
        Gc buffer busy
        Gc current block busy
        Gc current split

e) This issue will be noticed if multiple sessions are inserting into a single object or are using a sequence, and the indexed column is sequentially increasing. To address the specific issues:

        Identify the indexes and Global Hash Partition them.
        Increase the Sequence Cache if ordering is not a problem. –

Thanks

Geek DBA