Testing Accumulo with G1

Background

In order to avoid long Java garbage collection pauses and avoid memory fragmentation Accumulo long ago resorted to using off heap memory to store and sort incoming data.   This was accomplished using JNI and C++ STL maps with custom allocators.

The main reason long GC pauses must be avoided is that it can cause Accumulo server processes to lose their lock in zookeeper and shut down.   A secondary reason to avoid GC pauses is to keep the system responsive to queries.

With the release of the new G1 garbage collector in Java 7, I wanted to see how Accumulo performed storing and sorting its incoming data on the Java heap.  This is accomplished by setting   tserver.memory.maps.native.enabled to false.

Test Environment

I ran a set of experiments on a single node Accumulo instance to get a sense of how well G1 would work with the following environment.

Variable	Value
Accumulo version	1.5.0-SNAPSHOT circa 10/28
Java version	7u9
Hadoop verions	1.0.3
OS	RHEL 6
Compression	snappy
Java GC options	-XX:+PrintCommandLineFlags -XX:+PrintGCDetails -XX:+UseG1GC
# tablets	64

Continuous ingest, an Accumulo test suite, was used to generate key value pairs for this experiment. Two continuous ingest processes were run for each test.

Test Results

Below are the results of running the test. A rate of zero means the test did not complete because a long Java GC pause caused the tablet server to die. No swapping occurred during these test.

Map Memory	Java Heap	Native map	Ingest	Rate	Max Pause
6G	1G	TRUE	100M	247,787	0.77s
6G	1G	TRUE	202M	232,894	0.98s
2G	1G	TRUE	202M	232,221	1.46s
2G	1G	TRUE	100M	230,562	0.08s
2G	6G	FALSE	100M	144,916	3.84s
6G	12G	FALSE	100M	144,790	10s
2G	6G	FALSE	202M	142,650	4.8s
6G	12G	FALSE	202M	141,658	10.54s
6G	8G	FALSE	100M	0	> 30s
2G	4G	FALSE	100M	0	> 30s

Conclusion

From the test results two things are apparent. First the native C++ map is faster. Second, if the difference between the Java Heap size and the amount of memory Accumulo uses is too small then a long pause occurs. For example when the Java heap was 8G and Accumulo used 6G for ingest, the test failed.

I think the failures occurred because there were no good G1 heap regions to collect. All test had 64 tablets. Continuous ingest generates uniform data that spreads evenly across all tablets. Since writes continually go to all tablets, each G1 heap region will have data for each tablet. When one tablet is minor compacted this frees a little bit of memory in each heap region. When Java attempts to free memory there are a lot of heap regions with a little bit of free memory. Therefore G1 is forced to pick a set of these and copy live objects to newer heap regions. I suspect its this copying that eventually takes too long and causes a long pause.

In the cases where G1 did not fail the Java heap was much larger than the memory Accumulo used to store incoming data. In this case, I think there were older heap regions that were mostly free and quickly collected. With more memory, as maps are compacted older heap regions have less live objects.

So it seems Accumulo can be made to work with G1, but its slower and uses more memory.