Is your SGA is > 10gb? Then probably you have to look at huge pages

In Theory:-

HugePages is a feature integrated into the Linux kernel 2.6. It is a method to have larger page size that is useful for working with very large memory. HugePages is useful for both 32-bit and 64-bit configurations. HugePage sizes vary from 2MB to 256MB, depending on the kernel version and the hardware architecture. For Oracle Databases, using HugePages reduces the operating system maintenance of page states, and increases Translation Lookaside Buffer (TLB) hit ratio.

Without HugePages, the operating system keeps each 4KB of memory as a page, and when it is allocated to the SGA, then the lifecycle of that page (dirty, free, mapped to a process, and so on) is kept up to date by the operating system kernel.

With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, since each page table entry is pointing to pages from 2MB to 256MB. Also, the kernel has fewer pages whose lifecyle must be monitored.

As the amount of memory available on systems grows and the amount of memory needed by the database grows the traditional 4k page size used in most Linux systems is becoming a bit too small. As the total memory allocated increases the number of pages that must be managed also increases – meaning more work for the kernel. With HugePages you can increase the typical 4KB page size to something like 2MB. This means that for the same amount of RAM being used your OS will have a multiple of 512 less pages to manage. In addition, with HugePages the pages are pinned in memory and can’t be swapped to disk, thus avoiding possible disk writes. Another key benefit I’ve read is that HugePages are managed via a global PageTable rather than every process having its own PageTable – this also reduces the amount of memory needed.

Ok enough about hugepages documentation, Steps to implement the hugepages

The basics steps are as follows:-

* Set the memlock ulimit for the oracle user.
* Disable Automatic Memory Managment if necesary as it is incompatible with HugePages.
* Run the Oracle supplied hugepages_settings.sh script to calculate the recommended value for the vm.nr_hugepages kernel parameter.
* Edit /etc/sysctl.conf with the vm.nr_hugepages with the recommeneded setting.
* Reboot the server

OS Level Settings Considerations:-

/etc/sysctl.conf

• kernel.shmmax – set to the largest SGA on your server plus 1G
• kernel.shmall – set to sum of all SGAs on the server divided by page size – ‘getconf PAGESIZE’

/etc/security/limits.conf

• oracle soft memlock – set to slightly less than total RAM on server (in KB)
• oracle hard memlock – set to slightly less than total RAM on server (in KB)

So, for my system for example,

• RAM = 128GB = 132093152 kB
• SGA = 48GB – however, to allow for possible growth and given I have 128GB total, I’m going to use 64G for my numbers
• PGA = 16GB
• shmmax = 64GB+1GB = 65GB= 69793218560
• shmall = 1SGA @ 64GB = 64G/4096 = 16,777,216
• oracle soft memlock = slightly less than 132093152 = 130000000
• oracle hard memlock = oracle soft memlock = 130000000

2) Oracle has a script (in Note 401749.10 that will determine what they recommend for your HugePages configuration. Run this script:

->./hugepage_settings.sh ...

Recommended setting: vm.nr_hugepages = 24580

Then add it to

Next add the following to /etc/sysctl.conf

vm.nr_hugepages=24580

Reboot the server

3) Verify the hugepages settings

cat /proc/sys/vm/nr_hugepages

->grep Huge /proc/meminfo

HugePages_Total: 24580

HugePages_Free: 16212

HugePages_Rsvd: 16209

Hugepagesize: 2048 kB

Now your Hugepages is configured, you can use them when your database instance is started.

But hold the problem is if there is inadequate huge pages left in OS and to startup your instance oracle will silently fall back to normal paging of memory.

To overcome this oracle 11gr2 (11.2.0.2 & 3) has a parameter called use_large_pages which has two values “true” and “only”

With use_large_pages=true – if there are less hugepages then total allocated SGA – Oracle will write a warning message to alert.log and will go on with normal pages means do not use huge pages at all.

But again, in 11.2.0.3 the default behavior has changed – now with use_large_pages=true and less then SGA hugepages Oracle will allocate part of the SGA with them and the resting part with normal 4k pages. In alert.log it will look like

Specified value of sga_max_size is too small, bumping to 94220845056

****************** Large Pages Information *****************

Large Pages configured system wide = 43000 (84 GB) Large Page size = 2048 KB

RECOMMENDATION: Total Shared Global Region size is 88 GB. For optimal performance, prior to the next instance restart increase the number of unused Large Pages by atleast 1929 2048 KB Large Pages (3858 MB) system wide to get 100% of the Shared Global Region allocated with Large pages

***********************************************************

Did you observe part it has allocated from the huge pages and rest in normal OS pages.

With use_large_pages=only Oracle will check during the startup if there’s enough preallocated large pages and if there isn’t – will not proceed starting up with a message like

Specified value of sga_max_size is too small, bumping to 94220845056

****************** Large Pages Information *****************

References:-