Finding Memory leaks Within Solaris Applications Using libumem

Debugging Methodology :

The malloc() and free() memory management methods are used by many application developers. An application can be written without a dependence on any particular memory management programming interface by using the standard memory management methods malloc() and free(). This section will outline the steps needed to take advantage of the libumem library to debug an application's memory transactions.

Library Interposition and libumem Flags

If the libumem library is interposed (by setting the LD_PRELOAD environment variable) when executing an application, the malloc() and free() methods defined within the libumem library will be used whenever the application calls malloc() or free(). In order to take advantage of the debugging infrastructure of the libumem library, one needs to set the UMEM_DEBUG and the UMEM_LOGGING flags in the environment where the application is being executed. The most common values for these flags are as follows: UMEM_DEBUG=default and UMEM_LOGGING=transaction. With these settings, a thread ID, high-resolution time stamp, and stack trace are recorded for each memory transaction initiated by the application.

The following are examples of the commands used to set the appropriate debug flags and interpose the libumem library when executing an application.


(csh)

%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;
setenv LD_PRELOAD libumem.so.1;)

or

(bash)

bash-2.04$UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1

More details about the debug flags (UMEM_DEBUG and UMEM_LOGGING) can be found in the umem_debug(3MALLOC) man page.

MDB Commands

The developer can view the debug information pertaining to an application's memory management transactions by using MDB. The following commands within MDB can be used to provide a great deal of information about the memory transactions that took place during the execution of the application.

::umem_status

* Prints the status of the umem indicating if the logging features have been turned on or off

::findleaks

* Prints a summary of the memory leaks found within the application
::umalog

* Prints the memory transactions initiated by the application and the correlated stack traces
::umem_cache

* Prints the details about each of the umem caches
[address]::umem_log

* Prints the umem transaction log for the application
[address]::umem_verify

* Prints the integrity of the umem caches which is useful in determining if a buffer has been corrupted

address$
* Prints the contents of the umem_bufctl_audit structure as defined in the /usr/include/umem_impl.h header file

Example :

Traditional Memory Leak

In order to examine if SunMC has a memory leak, one can execute the following steps to narrow down the section of the code which is causing the leak.

1. The libumem library is only available on systems which are running the Solaris 9 OS, Update 3 and above.
%uname -a
SunOS hostname 5.9 Generic_112233-05
(csh)

2. Execute the application with the libumem library interposed and the appropriate debug flags set.
%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;setenv LD_PRELOAD libumem.so.1; ./a.out)
(bash)
#UMEM_DEBUG=default;UMEM_LOGGING=transaction;LD_PRELOAD=libumem.so.1; ./a.out
3. Use the gcore (1) command to get an application core to analyze the application's memory transactions.

%ps -ef | grep esd
user1 970 714 0 10:42:42 pts/4 0:00 ./a.out

%gcore 970
gcore: core.970 dumped
4. Use MDB to analyze the core for memory leaks using the commands described in the previous section.

$mdb core.970

Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]



> ::umem_log

CPU ADDR     BUFADDR         TIMESTAMP THREAD 

 0 0002e0c8 00055fb8     159d27e121a0 00000001

 0 0002e064 00055fb8     159d27e0fce8 00000001

 0 0002e000 00049fc0     159d27da1748 00000001

   00034904 00000000                0 00000000

   00034968 00000000                0 00000000

   ... snip ...



Here we can see that there have been three transactions by thread #1 on cpu #0.



> ::umalog



T-0.000000000  addr=55fb8  umem_alloc_32

        libumem.so.1`umem_cache_free+0x4c

        libumem.so.1`process_free+0x68

        libumem.so.1`free+0x38

        main+0x18

        _start+0x108



T-0.000009400  addr=55fb8  umem_alloc_32

        libumem.so.1`umem_cache_alloc+0x13c

        libumem.so.1`umem_alloc+0x44

        libumem.so.1`malloc+0x2c

        main+0x10

        _start+0x108



T-0.000461400  addr=49fc0  umem_alloc_24

        libumem.so.1`umem_cache_alloc+0x13c

        libumem.so.1`umem_alloc+0x44

        libumem.so.1`malloc+0x2c

        main+4

        _start+0x108

 The three transactions consist of one allocation to the 24 byte umem

cache, and one memory allocation and release from the 32 byte umem

cache. Note that the high resolution timestamp output in the upper left

hand corner is relative to the last memory transaction initiated by the

application.

> ::findleaks

CACHE     LEAKED   BUFCTL CALLER

0003d888       1 00050000 libumem.so.1`malloc+0x0

----------------------------------------------------------------------

 Total       1 buffer, 24 bytes

This shows that there is one 24 byte buffer which has been leaked.

Solaris 10 Container Resource Management using Dynamic Resource Pool & FSS

Let’s think of a physical Solaris server with 2 CPUs which needs to host:

1. One Database server - one that needs a fully dedicated CPU and
2. Two Web servers -one that is more flexible and can share CPUs.

To accomplish these different levels of isolation we use a Solaris Container technology called Dynamic Resource Pools that enables CPU resources to be dedicated to specific applications. In this example, the database server needs a separate resource pool, while the Web servers can share another.

Login to the Global zone as super user and activate the resource pool facility:

#pooladm –e

Check the current status of resource pools using:

#pooladm

Save the current configuration to the default file (/etc/pooladm.conf) using:

#pooladm –s

Check the number of processors available to the system:

#psrinfo

Create a processor set containing one CPU:

#poolcfg –c ‘create pset db-pset (uint pset.min=1; uint pset.max=1)’

Update the configuration:

#pooladm –c

Check the configuration:

#pooladm

Create a pool for Database zone:

#poolcfg –c ‘cerate pool dbpool’

Link the pool with processor set:

#poolcfg –c ‘associate pool dbpool (pset db-pset)’

Commit the change:

#pooladm –c

Check the configuration:

#pooladm

***************************************************

Set the pool while creating the non-global zone for Database or to an existing zone by:

zonecfg:dbzone>set pool=dbpool

zonecfg:dbzone>verify

Now if you set the default pool for other 2 web server zones, they are going to use the default pool associated with default processor set with 1 CPU, while Database zone will start using another CPU dedicatedly using processor set “db-pset”.

zonecfg:webzone1>set pool= pool_default

zonecfg:webzone2>set pool= pool_default

Check the same by trying to disable any of the processor using;

#psradm –f [0,1]

Fair Share Scheduler:

While the two Web servers are capable of sharing the remaining CPUs on the system, they each need a minimum guarantee of CPU resources that will be available to them. This is made possible by another Solaris Container technology called the Fair Share Scheduler (FSS). This technology enables CPU resources to be allocated proportionally to applications. That is, each application gets assigned a number of the available “shares” of the total CPU.

Enable FSS for pool-default:

#poolcfg -c 'modify pool pool_default (string pool.scheduler="FSS")'

Commit:

#pooladm –c

Move all the processes in the default pool and its associated zones under the FSS.

Either by rebooting the system or by using:

#priocntl -s -c FSS -i class TS

#priocntl -s -c FSS -i pid 1

(valid classes are RT for real-time, TS for time-sharing, IA for inter-active, FSS for fair-share, or FX for Fixed-Priority)

Remember, the two Web servers share the CPU resources of the default pool with each other as well as the global zone, so you need to specify how those resources should be shared using the Fair Share Scheduler (FSS).

With FSS, the relative importance of applications is expressed by allocating CPU resources based on shares—a portion of the system's CPU resources assigned to an application. The larger the number of shares assigned to an application, the more CPU resources it receives from the FSS software relative to other applications. The number of shares an application receives is not absolute – important thing is how many shares it has relative to other applications, and whether they will compete with it for CPU resources. That is how it is being dynamic.

Assign three shares to the first web server container (more priority):

zonecfg:Web-zone1> add rctl

zonecfg:Web-zone1:rctl> set name=zone.cpu-shares

zonecfg:Web-zone1:rctl> add value (priv=privileged,limit=3,action=none)

zonecfg:Web-zone1:rctl> end

zonecfg:Web-zone1> exit

Assign two shares to the second web server container (less priority):

zonecfg:Web-zone2> add rctl

zonecfg:Web-zone2:rctl> set name=zone.cpu-shares

zonecfg:Web-zone2:rctl> add value (priv=privileged,limit=2,action=none)

zonecfg:Web-zone2:rctl> end

zonecfg:Web-zone2> exit

You now have 3 containers created; one with a fixed amount of CPU, and two dynamically sharing CPU with the global zone.

Now the DB server will run on its own guaranteed CPU, protected from the other applications on this system, while the Web servers share the remaining 1 CPU. To clarify the FSS share usage, the first Web server application has three out of the total six shares (3 – web-zone1, 2 – webzone2, 1 – global zone), entitling it to 0.5 CPU worth of the 1 CPU (1*3/6=0.5); the second (web-zone2) has two of the six shares, giving it 0.33 CPUs worth; and the global zone gets the remaining one of six shares – that is 0.17 CPU worth.

Just to comment, Oracle (when you involve Oracle DB like this setup) acknowledges these types of Solaris Containers as a valid license boundary. In their terminology they are known as Capped Containers, and are made by a combination of Dynamic Resource Pools and Solaris Zones - where the amount of CPUs in the pool determines the size of the license. So you need to consider the same when using this technology with Oracle Database server or any other oracle product.