RAC Performance: DFS Lock Handle + Row Cache Lock + ENQ: SQ Contention , hanged the rac instance.

Symptoms:

Alert log says WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!

Wait events in Node A: DFS Lock Handle and hanged

Wait events in Node B: row cache locks and library cache locks (sessions permitted but very slow)

AWR reports says for the interesting period: DFS Lock Handle

Background:- DFS Lock Handle

DFS stands for distributed file system is an ancient name, associated with cluster file system operations, in a Lock manager supplied by vendors in Oracle Parallel Server Environment (prior name for RAC). But, this wait event has morphed and is now associated with waits irrelevant to database files also.

This will occur in RAC environment, possible with sequences especially when you have sequences + cache + Ordered set.

Means created like this, create sequence s1 min value 1 cache 20 order;

As RAC is multi instance environment, the values of the sequences need to be synchronized as they are need to be ordered.

Simulation of the issue:-

For example consider this sequence of sessions and their possible waits while accessing sequence next value:-

Session 1 on node-A: nextval -> 101 (DFS Lock handle) (CR read)

Session 2 on node-A: nextval –> 102

Session 1 on node-B: nextval -> 103 (DFS Lock handle)

Session 1 on node-B: nextval –> 104

Session 1 on node-A: nextval -> 105 (DFS Lock handle)

Session 1 on node-A: nextval -> 106 (more selects)

Session 1 on node-A: nextval –> 998

Session 1 on node-B: nextval -> 999 (DFS Lock handle)

Session 1 on node-B: nextval -> 1000 (CR read)

If you look in the gv$session_wait_history it shows as “DFS lock handle” with the “p1″ parameter been the object_id of the sequence.

Ref:-

http://www.pythian.com/news/383/sequences-in-oracle-10g-rac/

Troubleshooting 'DFS lock handle' waits « Oracle database internals

Row cache lock

In order for DDL to execute, it must acquire a row cache lock to lock the Data Dictionary information. The shared pool contains a cache of rows from the data dictionary that helps reduce physical I/O to the data dictionary tables and allows locking of individual data dictionary rows. The locks on the data dictionary rows are called row cache enqueue locks. The enqueue lock structures are allocated from the shared pool as needed but when these requests wait and time out is when we see the row cache lock wait.

Each row cache lock will be on a specific data dictionary object. This is called the enqueue type and can be found in the v$rowcache view. In this sample select from v$rowcache you can find the enqueue types and the type of activity being performed within the dictionary cache.

Row cache lock will be associated with specific enqueue type (described in v$lock_type) on a dictionary row. Check the following view can provide you the fair idea of dictionary cache aka row cache activity. Check the note : 468334.1 for different types of rowcache locks

View: V$rowcache

In RAC Environments this can appear as the library and dictionary cache’s are global and has to be to synchronized. Further more, this event can appear if the sequences specified as nocache in RAC environments.

ENQ: SQ Contention

The SQ enqueue is the Sequence Cache enqueue (a.k.a. enq: SQ – contention) is used to serialize access to Oracle sequences . Especially when specified cache+noorder for sequences attribute in RAC environments

hence, Specify cache + order for sequences in RAC.