Sunny Ahuwanya's Blog

Mostly notes on .NET and C#

Testing the BufferPool Project

One of the challenges I faced while working on the BufferPool project was verifying that it worked perfectly. Although I wrote a bunch of unit tests to assert the right behavior, I couldn’t conclude that it actually ensured that buffers were pinned within a pre-defined contiguous memory block.

To confirm this, I manually compared memory dumps from a process that used the library with dumps from another one that did not.

Two programs were used to perform this test. The first program is a variant of the asynchronous socket listener sample on MSDN, modified to call GC.Collect upon accepting a fifth connection. It also creates a long empty byte array on each new connection to simulate other memory allocations that typically take place during the read callback (which isn’t triggered during the test).

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;

namespace BufferPinningTester
{

    public class AsynchronousSocketListener
    {
        public static ManualResetEvent allDone = new ManualResetEvent(false);
        static int connectCount = 0;

        public static void StartListening()
        {
            // Establish the local endpoint for the socket.
            // The DNS name of the computer
            // running the listener is "host.contoso.com".
            IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
            IPAddress ipAddress = ipHostInfo.AddressList[0];
            IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);

            // Create a TCP/IP socket.
            Socket listener = new Socket(AddressFamily.InterNetwork,
                SocketType.Stream, ProtocolType.Tcp);

            // Bind the socket to the local endpoint and listen for incoming connections.
            try
            {
                listener.Bind(localEndPoint);
                listener.Listen(100);

                while (true)
                {
                    // Set the event to nonsignaled state.
                    allDone.Reset();

                    // Start an asynchronous socket to listen for connections.
                    Console.WriteLine("Waiting for a connection...");
                    listener.BeginAccept(
                        new AsyncCallback(AcceptCallback),
                        listener);

                    // Wait until a connection is made before continuing.
                    allDone.WaitOne();
                    connectCount++;

                    if (connectCount == 5)
                    {
                        GC.Collect();
                        Console.WriteLine("GC Collected");
                    }
                    
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }

            Console.WriteLine("\nPress ENTER to continue...");
            Console.Read();

        }

        public static void AcceptCallback(IAsyncResult ar)
        {

            Console.WriteLine("In AcceptCallback");
            // Signal the main thread to continue.
            allDone.Set();

            // Get the socket that handles the client request.
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);

            // Create the state object.
            StateObject state = new StateObject();
            //create some random array
            byte[] b = new byte[StateObject.BufferSize];
            state.workSocket = handler;
            handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                new AsyncCallback(ReadCallback), state);
        }

        public static void ReadCallback(IAsyncResult ar)
        {
            Console.WriteLine("In ReadCallback");

            String content = String.Empty;

            // Retrieve the state object and the handler socket
            // from the asynchronous state object.
            StateObject state = (StateObject)ar.AsyncState;
            Socket handler = state.workSocket;

            
            // Read data from the client socket. 
            int bytesRead = handler.EndReceive(ar);

            return;

        }

        public static int Main(String[] args)
        {
            StartListening();
            return 0;
        }
    }


// State object for receiving data from remote device.
public class StateObject {
    // Client socket.
    public Socket workSocket = null;
    // Size of receive buffer.
    public const int BufferSize = 16000;
    // Receive buffer.
    public byte[] buffer = new byte[BufferSize];
    // Received data string.
    public StringBuilder sb = new StringBuilder();
}



}

The second program is a version of the first program modified to use a buffer pool. In this program, The 16KB buffer allocations are managed by the pool.

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using ServerToolkit.BufferManagement;

namespace BufferPinningTester
{

    public class AsynchronousSocketListener
    {
        public static ManualResetEvent allDone = new ManualResetEvent(false);
        static int connectCount = 0;
        static BufferPool pooledBuffers = new BufferPool(80000, 1, 1);
        

        public static void StartListening()
        {
            // Establish the local endpoint for the socket.
            // The DNS name of the computer
            // running the listener is "host.contoso.com".
            IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
            IPAddress ipAddress = ipHostInfo.AddressList[0];
            IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);

            // Create a TCP/IP socket.
            Socket listener = new Socket(AddressFamily.InterNetwork,
                SocketType.Stream, ProtocolType.Tcp);

            // Bind the socket to the local endpoint and listen for incoming connections.
            try
            {
                listener.Bind(localEndPoint);
                listener.Listen(100);

                while (true)
                {
                    // Set the event to nonsignaled state.
                    allDone.Reset();

                    // Start an asynchronous socket to listen for connections.
                    Console.WriteLine("Waiting for a connection...");
                    listener.BeginAccept(
                        new AsyncCallback(AcceptCallback),
                        listener);

                    // Wait until a connection is made before continuing.
                    allDone.WaitOne();
                    connectCount++;

                    if (connectCount == 5)
                    {
                        GC.Collect();
                        Console.WriteLine("GC Collected");
                    }
                    
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }

            Console.WriteLine("\nPress ENTER to continue...");
            Console.Read();

        }

        public static void AcceptCallback(IAsyncResult ar)
        {

            Console.WriteLine("In AcceptCallback");
            // Signal the main thread to continue.
            allDone.Set();

            // Get the socket that handles the client request.
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);

            // Create the state object.
            StateObject state = new StateObject();
            //create some random array
            byte[] b = new byte[StateObject.BufferSize];
            state.workSocket = handler;
            IBuffer readBuffer = pooledBuffers.GetBuffer(StateObject.BufferSize);
            handler.BeginReceive(readBuffer.GetArraySegment(), 0,
                new AsyncCallback(ReadCallback), state);
        }

        public static void ReadCallback(IAsyncResult ar)
        {
            Console.WriteLine("In ReadCallback");

            String content = String.Empty;

            // Retrieve the state object and the handler socket
            // from the asynchronous state object.
            StateObject state = (StateObject)ar.AsyncState;
            Socket handler = state.workSocket;

            
            // Read data from the client socket. 
            int bytesRead = handler.EndReceive(ar);

            return;

        }

        public static int Main(String[] args)
        {
            StartListening();
            return 0;
        }
    }


// State object for receiving data from remote device.
public class StateObject {
    // Client socket.
    public Socket workSocket = null;
    // Size of receive buffer.
    public const int BufferSize = 16000;
    // Receive buffer.
    //public byte[] buffer = new byte[BufferSize];
    // Received data string.
    public StringBuilder sb = new StringBuilder();
}



}

The test procedure consists of running the socket listener program and connecting to it from five telnet clients simultaneously.
A memory snapshot is taken before the fifth connection is made and another snapshot is taken after it is made. Both snapshots are then compared.
The test is conducted twice, once for each socket listener program.
To take these snapshots, you’ll need to use WinDbg debugger tool coupled with the Son of Strike (SOS) debugging extension.

In Detail:

  • Run the first socket listener program.
  • Run WinDbg and attach the socket listener process.
  • Load Son of Strike (SOS) from WinDbg by executing .loadby sos clr (if you are working with .NET 2.0/3.5, execute .loadby sos mscorwks instead).
  • Execute g to continue running the program.
  • Connect four telnet clients to port 11000 (the socket listener port).
    While running telnet, it’s important not to hit any key after telnet makes the connection, so as not to transmit any data and keep the buffer pinned. If you see a "In ReadCallback" message in the socket listener console window, it means data was transmitted – we don’t want that.
  • Press ctrl-break in WinDbg to halt execution of the program.
  • Execute the !gchandles command to get information about asynchronous pinned handles.
  • Execute the !dumpheap command to get a memory snapshot of objects in the heap.
  • Execute g to continue running the program.
  • Connect the fifth telnet client.
  • You should see "GC Collected" in the socket listener program console window.
  • Press ctrl-break in WinDbg to halt execution of the program.
  • Execute the !gchandles command to get information about asynchronous pinned handles.
  • Execute the !dumpheap command to get a memory snapshot of objects in the heap.
  • Run the test again, this time for the second socket listener program.

A sample output from !gchandles is shown below. The number of asynchronous pins are highlighted.

0:005> !gchandles
*********************************************************************
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y <symbol_path> argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
*********************************************************************
PDB symbol for mscorwks.dll not loaded
GC Handle Statistics:
Strong Handles: 37
Pinned Handles: 6
Async Pinned Handles: 4
Ref Count Handles: 0
Weak Long Handles: 35
Weak Short Handles: 10
Other Handles: 0
Statistics:
              MT    Count    TotalSize Class Name
000007fef8ac7370        1           24 System.Object
000007fef92a2f88        1           32 System.Threading.RegisteredWaitHandle
000007fef8ac87c0        1           48 System.SharedStatics
000007fef8ac8078        1          136 System.ExecutionEngineException
000007fef8ac7f68        1          136 System.StackOverflowException
000007fef8ac7e58        1          136 System.OutOfMemoryException
000007fef8ac8980        1          192 System.AppDomain
000007fef92a3040        5          200 System.Threading._ThreadPoolWaitOrTimerCallback
000007fef8aca540        5          240 System.Reflection.Assembly
000007fef8ab5d68        5          240 System.Threading.ManualResetEvent
000007fef76be9b0        4          256 System.Net.Logging+NclTraceSource
000007fef8ac8188        2          272 System.Threading.ThreadAbortException
000007fef76d59a8        4          288 System.Diagnostics.SourceSwitch
000007fef8ac43f8        4          384 System.Reflection.Module
000007fef8ac8520        4          416 System.Threading.Thread
000007fef8ace270        7          448 System.Security.PermissionSet
000007fef8699058        4          480 System.Threading.OverlappedData
000007fef8ac4558       35         5040 System.RuntimeType+RuntimeTypeCache
000007fef8ab5870        6        33856 System.Object[]
Total 92 objects
  

You now have useful output from the !dumpheap and !gchandles commands for both programs.
You can download the !dumpheap and !gchandles output from my tests here.
You’ll notice that the output from dumpheap after garbage collection is much smaller than the output before garbage collection. That’s because garbage collection got rid of stale objects in memory.

If you compare the two !dumpheap output listings from the first (unpooled) test, you can easily verify that all memory addresses that hold byte arrays that are at least 16,000 bytes long were collected by the GC except for four addresses. You can tell that a memory address holds a byte array if its MT (MethodTable) address is the one for System.Byte[] in the statistics section at the bottom of the !dumpheap listings. On my system, the MT for System.Byte[] is 000007fef8acfac0.
The output from !gchandles shows that there were four asynchronous pins before garbage collection and five after garbage collection. This observation strongly suggests that those empty byte arrays that were created were collected while the pinned buffer byte arrays were not.

Now let’s take a look at the memory behavior of the pooled test by comparing its two !dumpheap output listings before and after garbage collection. The desired behavior with pooled buffers is that all pinned addresses are within one long byte array.
You’ll observe that all byte arrays in the heap are collected except one, and that one is 92 KB long.
The pooler creates a minimum pool size of 92 KB so as to force .NET to allocate the array in the large object heap.
The output from !gchandles shows that there were four asynchronous pins before garbage collection and five after garbage collection. Since only byte arrays were pinned, it means they all occurred within the 92k byte array block!
We have verified that the pooler is working correctly.

To see details of the 92KB block, run the !dumpobj command on the block’s memory address:

0:005> !dumpobj 00000000127b9060
Name: System.Byte[]
MethodTable: 000007fef8acfac0
EEClass: 000007fef86d2680
Size: 92184(0x16818) bytes
Array: Rank 1, Number of elements 92160, Type Byte
Element Type: System.Byte
Fields:
None