5.1 Chunk memory

Most of the memory used by Polipo is stored in chunks, fixed-size blocks of memory; the size of a chunk is defined by the compile-time constant CHUNK_SIZE, and defaults to 4096 bytes on 32-bit platforms, 8192 on 64-bit ones. Chunks are used for storing object data (bodies of instances) and for temporary I/O buffers. Increasing the chunk size increases performance somewhat, but at the cost of larger granularity of allocation and hence larger memory usage.

By default, Polipo uses a hand-crafted memory allocator based on mmap(2) (VirtualAlloc under Windows) for allocating chunks; while this is very slightly faster than the stock memory allocator, its main benefit is that it limits memory fragmentation. It is possible to disable the chunk allocator, and use malloc(3) for all memory allocation, by defining MALLOC_CHUNKS at compile time; this is probably only useful for debugging.

There is one assumption made about CHUNK_SIZE: CHUNK_SIZE multiplied by the number of bits in an unsigned long (actually in a ChunkBitmap — see chunk.c) must be a multiple of the page size, which is 4096 on most systems (8192 on Alpha, and apparently 65536 on Windows).

As all network I/O will be performed in units of one to two chunks, CHUNK_SIZE should be at least equal to your network interface’s MTU (typically 1500 bytes). Additionally, as much I/O will be done at CHUNK_SIZE-aligned addresses, CHUNK_SIZE should ideally be a multiple of the page size.

In summary, 2048, 4096, 8192 and 16384 are good choices for CHUNK_SIZE.