Linux Understanding SHMMAX and SHMALL Settings

Linux Understanding SHMMAX and SHMALL Settings

Linux and understanding SHMMAX and SHMALL kernel paramaters are the key to your oracle databases performance. These two parameters are pretty important paramaters and they affect how Oracle creates and manages SGA’s.  Your operating system has to provide an application access to a certain type of memory with certain types of controls.  It’s the Unix kernel that allows applications to access shared memory (IPC), and how much memory, in multiple big chunks or small chunks etc…
Sometimes, portions of memory get used leaving memory segments out of order.  So these chunks of memory, from now on called segments, can get used inefficiently leaving unused memory segments.  This can be caused by many reasons which I won’t get into now.  So when Orale wants to access some shared memory on startup, it knows how much memory it needs for it’s SGA and has to figure out the best and most efficient way to access the resource.  If Oracle needs 8GB or shared memory but can only find 4 seperate 2GB segments that are out of order, it will then have no choice but to use this memory model.  This non-contiguous memory model isn’t the best.  Oracle will use shared memory in 3 different ways or memory models.  The first memory model as just described, the non-contiguous model where the memory segments are scattered over the place.  Another memory model being multiple memory segments that are in fact contiguous, so this is better, less overhead than the non-contiguous model.  But the ultimate is one big segment that is the size of the SGA. So in this case an 8 GB segment.  And that is where the tuning comes into play.  How to tune the kernel and it’s paramaters to meet your Oracle applications shared memory needs.  As I mentioned in the opening paragraph, its the SHMMAX that is the key kernel paramater in Linux to tune the SGA shared memory requirement.

Shared memory and SHMMAX and SHMALL

Just to clarify a couple of definitions, I will go over some terminology.  The term shared memory is a term used to describe a type of memory management in the Unix kernel.  It is a memory region that can shared between different processes.  Oracle uses this shared memory when using and implementing the SGA, which has to be visible to all database sessions.  Using a shared memory model, this avoids creating duplicate copies of memory.  So programs can eaily share data.  These regions of memory, because they are shared need to have controls on who can access them and who can’t. A locking mechanism is required. This is where semaphores come into play. Basically, flags that are either on or off. If the memory is being used, the process turns on the flag and other processes have to wait until the semaphore is freed or the flag is turned off.  So shared memory needs semaphores, they go hand in hand.  I should probably mention that an application can access shared memory programatically through a set of POSIX C routines, common on most Unix like operating systems including Linux.  The IPC or shared memory API’s have common IPC calls like shmget() and shmat() which allow the programmer to get a shared memory segment, or attach to a segment.  There are many more.  Now the kernel paramaters SHMMAX and SHMALL need to be defined now. SHMMAX is really just the maximum size of a single shared memory segment.  It’s size is represented in bytes”.  And SHMALL is the sum of all shared memory segments on the whole system.  But it is measured in number of pages.  Will leave it at that for now.
At times on very busy transactional databases, you can run out of semaphores. This seriously limits the performance of a database. You can look at your semaphores as follows:

# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 65537      root       600        1         
0x00000000 229378     oracle     660        300       
0x00000000 265667     oracle     660        300      
0x00000000 247100     oracle     660        360

Enough of a semaphore tangent. Linux being a type of Unix, there are 300 different ways of finding the value of a system paramaters. The traditional way is calling ipcs to list the shared memory limits:

# ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096       <--- this is SHMMNI
max seg size (kbytes) = 67108864    <--- this is SHMMAX
max total shared memory (kbytes) = 17179869184<- this is SHMALL
min seg size (bytes) = 1

Or cat’ing the /proc/* data structures

#  cat /proc/sys/kernel/shmall
4294967296
#  cat /proc/sys/kernel/shmmax
68719476736

Now, how do you configure SHMALL for an Oracle database

Right off the top, since SHMALL is the sum of all the shared memory segments on your system, you had better make it smaller than your total system memory.  If I remember correctly, back in the 90′s, when memory was more expensive than the US space program,  on HP-UX systems we used to define shared memory SHMALL equivalent to much larger than physical memory. Swap disks were busy then. I don’t know if it’s possible to do this now, but it’s not recommended. I’ll have to test that some day for fun. SHMALL should also be less than the sum of all the SGA’s configured on the system.  If you allow more shared memory than the physical memory on the system, then your asking for a world of hurt.  Obviously your system has to be sized appropriately for the work to be done.  If you create a new database instance and start Oracle which will then want to create a new SGA, shmget() -> shmat() etc… it will fail with the much feared “out of memory” etc…  This just means that SHMALL is doing it’s job properly and preventing Oracle from taking what it isn’t allowed to take.
A typical error message caused by there being insufficient memory when an Oracle instance is starting would be somehing like:

ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device

This means SHMALL is doing it’s job. As shared memory is a device, ie. /dev/shm, the error message above makes sense.

The logic behind finding the SHMALL value for your system

Configuring the kernel parameter SHMALL is mostly just basic math.  It’s a decision that involves finding the amount of memory you want to give to Oracle, or how much you can give away.  This memory I am talking about can be talked about in terms of physical memory.  Physical memory shouldn’t be confused with virtual memory which has swap space included.  From the physical memory we take away the memory that the operating system needs.  Linux will use a lot of memory for disk caching and other assorted data structures.  So lets be moderately generous and say we will give the Unix kernel 1GB, this is a good rule.

Working example

System memory 12GB
Linux Kernel  1GB
Memory for Oracle  11GB (System memory – Linux Kernel)
So we can assign SHMALL the value of 11GB. But… SHMALL is measured in memory pages not bytes.  We need to divide the 11GB by the systems page size value.  It’s almost always 4K which is the recommended size, but to be safe, you run the following command:

# getconf PAGE_SIZE
4096

We have to convert the 11GB byte value into a page value because as I mentioned above, SHMALL is measured in page size.  I don’t make the rules, sorry.  I use bc, the unix calculator and I don’t need decimal places.

# bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
11*1024*1024*1024   <--- 11 x 1K x 1K x 1K gives me 11GB value in bytes
11811160064
11811160064/4096    <--- 11GB value in bytes divided by 4k (page size)
2883584             <--- page size value to assign to SHMALL

This is a tad conveluted, but easy math.  Lets make this SHMALL kernel paramater permanent by adding it to the /etc/sysctl.conf file.

# vi /etc/sysctl.conf
kernel.shmall = 2883584

The logic behind finding the SHMMAX value for your system

If you read the release and install and configuration documentation for Oracle 11g, Oracle recommends you set shmmax to half of your physical memory or 4 GB less 1 byte. Choose whichever is lower.  The interesting 4 GB less 1byte memory setting is based on the virtual memory addressing limitations of the 32 bit (x86) processors. All 64 bit processors can address much larger memory addresse space and is not limited like it’s 32 bit brethren. So much larger shared memory segments can be permitted and created and used.
When Oracle first starts up, after having read it’s configuration files it will know how much memory it needs for it’s assorted SGA’s.  The best fit for an Oracle SGA shared memory segment is when the SGA fits into one shared memory segment.  As I mentioned earlier, Oracle can use non-contiguous shared memory segments, but its not Oracle’s chosen memory model.  Therefore by defenition, if the SGA is larger than the value of SHMMAX, then Oracle has to use one of the two other remaining shared memory models.  Contiguous or non-contiguous memory segments.
I try to set the SHMMAX to the size of the largest SGA defined on your database server.  If your system has 6 Oracle instances configured, 2 require 2 GB SGA’s, 2 require 1.5GB SGA’s and 2 require 1 GB SGA’s, you would set the SHMMAX to 2GB.
Edit your sysctl.conf and add the SHMMAX value to 2 GB in bytes

# vi /etc/sysctl.conf
kernel.shmmax = 2147483648

Or, as this is Unix and there are many ways of doing things, use sysctl -w argument=xyz which writes to the sysctl.conf file.

# sysctl -w kernel.shmmax=2147483648

Now reload the parameters by asking sysctl politely to to reread it’s configuration

# sysctl –p

Now that you have configured shmmax and shmall, and all the other steps required to configure you system for Oracle, I will jump ahead very fast here and install Oracle and create a test db instance.

# su - oracle
$ cd /tmp/2wherever/install/files/are
$ ./runInstaller

….

# service oracle start

Create a test databse

$ dbca

etc…..

# ipcs -a
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6c6c6536 0          root       600        4096       0
0x2270894c 2359297    oracle     660        1512046592 30
------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 294914     apache     600        1
0x00000000 327683     apache     600        1
0x396b9144 983044     oracle     660        154
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

The last few steps have been exceedingly oversimplied, but the point here is to show the shared memory use.  You will also see that user oracle has grabbed 154 semaphores.  This isn’t a huge number, and in this case it’s only a test db instance so no real traffic.