JVM 初探——使用堆外内存减少 Full GC（下）
（点击上方公众号，可快速关注）
来源：翡青，
blog.csdn.net/zjf280441589/article/details/54406665
第2组: off-heap、not affect by GC、need serialize
EhcacheApp
public class EhcacheApp extends AbstractAppInvoker {
private static Cache<String, FeedDO> cache;
static {
ResourcePools resourcePools = ResourcePoolsBuilder.newResourcePoolsBuilder()
.heap(1000, EntryUnit.ENTRIES)
.offheap(480, MemoryUnit.MB)
.build();
CacheConfiguration<String, FeedDO> configuration = CacheConfigurationBuilder
.newCacheConfigurationBuilder(String.class, FeedDO.class, resourcePools)
.build();
cache = CacheManagerBuilder.newCacheManagerBuilder()
.withCache("cacher", configuration)
.build(true)
.getCache("cacher", String.class, FeedDO.class);
}
@Test
@Override
public void invoke(Object... param) {
for (int i = 0; i < SIZE; ++i) {
String key = String.format("key_%s", i);
FeedDO feedDO = createFeed(i, key, System.currentTimeMillis());
cache.put(key, feedDO);
}
System.out.println("write down");
// read
for (int i = 0; i < SIZE; ++i) {
String key = String.format("key_%s", i);
Object o = cache.get(key);
checkValid(o);
if (i % 10000 == 0) {
System.out.println("read " + i);
}
}
}
}
MapDBApp与前同.
第3组: off-process、not affect by GC、serialize、affect by process communication
LocalRedisApp
public class LocalRedisApp extends AbstractAppInvoker {
private static final Jedis cache = new Jedis("localhost", 6379);
private static final IObjectSerializer serializer = new Hessian2Serializer();
@Test
@Override
public void invoke(Object... param) {
// write
for (int i = 0; i < SIZE; ++i) {
String key = String.format("key_%s", i);
FeedDO feedDO = createFeed(i, key, System.currentTimeMillis());
byte[] value = serializer.serialize(feedDO);
cache.set(key.getBytes(), value);
if (i % 10000 == 0) {
System.out.println("write " + i);
}
}
System.out.println("write down");
// read
for (int i = 0; i < SIZE; ++i) {
String key = String.format("key_%s", i);
byte[] value = cache.get(key.getBytes());
FeedDO feedDO = serializer.deserialize(value);
checkValid(feedDO);
if (i % 10000 == 0) {
System.out.println("read " + i);
}
}
}
}
RemoteRedisApp类似, 详细代码可参考下面完整项目.
备注:
- TTC: Total Time Cost 总共耗时
- C/T: Count/Time 次数/耗时(seconds)
结果分析
对比前面几组数据, 可以有如下总结:
将长生命周期的大对象(如cache)移出heap可大幅度降低Full GC次数与耗时;
使用off-heap存储对象需要付出serialize/deserialize成本;
将cache放入分布式缓存需要付出进程间通信/网络通信的成本(UNIX Domain/TCP IP)
附:
off-heap的Ehcache能够跑出比in-heap的HashMap/Guava更好的成绩确实是我始料未及的O(∩_∩)O~, 但确实这些数据和堆内存的搭配导致in-heap的Full GC太多了, 当heap堆开大之后就肯定不是这个结果了. 因此在使用堆外内存降低Full GC前, 可以先考虑是否可以将heap开的更大.
附: 性能测试框架
在main函数启动时, 扫描com.vdian.se.apps包下的所有继承了AbstractAppInvoker的类, 然后使用Javassist为每个类生成一个代理对象: 当invoke()方法执行时首先检查他是否标注了@Test注解(在此, 我们借用junit定义好了的注解), 并在执行的前后记录方法执行耗时, 并最终对比每个实现类耗时统计.
依赖
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-proxy</artifactId>
<version>${commons.proxy.version}</version>
</dependency>
<dependency>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
<version>${javassist.version}</version>
</dependency>
<dependency>
<groupId>com.caucho</groupId>
<artifactId>hessian</artifactId>
<version>${hessian.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
</dependency>
启动类: OffHeapStarter
/**
* @author jifang
* @since 2017/1/1 上午10:47.
*/
public class OffHeapStarter {
private static final Map<String, Long> STATISTICS_MAP = new HashMap<>();
public static void main(String[] args) throws IOException, IllegalAccessException, InstantiationException {
Set<Class<?>> classes = PackageScanUtil.scanPackage("com.vdian.se.apps");
for (Class<?> clazz : classes) {
AbstractAppInvoker invoker = createProxyInvoker(clazz.newInstance());
invoker.invoke();
//System.gc();
}
System.out.println("********************* statistics **********************");
for (Map.Entry<String, Long> entry : STATISTICS_MAP.entrySet()) {
System.out.println("method [" + entry.getKey() + "] total cost [" + entry.getValue() + "]ms");
}
}
private static AbstractAppInvoker createProxyInvoker(Object invoker) {
ProxyFactory factory = new JavassistProxyFactory();
Class<?> superclass = invoker.getClass().getSuperclass();
Object proxy = factory
.createInterceptorProxy(invoker, new ProfileInterceptor(), new Class[]{superclass});
return (AbstractAppInvoker) proxy;
}
private static class ProfileInterceptor implements Interceptor {
@Override
public Object intercept(Invocation invocation) throws Throwable {
Class<?> clazz = invocation.getProxy().getClass();
Method method = clazz.getMethod(invocation.getMethod().getName(), Object[].class);
Object result = null;
if (method.isAnnotationPresent(Test.class)
&& method.getName().equals("invoke")) {
String methodName = String.format("%s.%s", clazz.getSimpleName(), method.getName());
System.out.println("method [" + methodName + "] start invoke");
long start = System.currentTimeMillis();
result = invocation.proceed();
long cost = System.currentTimeMillis() - start;
System.out.println("method [" + methodName + "] total cost [" + cost + "]ms");
STATISTICS_MAP.put(methodName, cost);
}
return result;
}
}
}
包扫描工具: PackageScanUtil
public class PackageScanUtil {
private static final String CLASS_SUFFIX = ".class";
private static final String FILE_PROTOCOL = "file";
public static Set<Class<?>> scanPackage(String packageName) throws IOException {
Set<Class<?>> classes = new HashSet<>();
String packageDir = packageName.replace('.', '/');
Enumeration<URL> packageResources = Thread.currentThread().getContextClassLoader().getResources(packageDir);
while (packageResources.hasMoreElements()) {
URL packageResource = packageResources.nextElement();
String protocol = packageResource.getProtocol();
// 只扫描项目内class
if (FILE_PROTOCOL.equals(protocol)) {
String packageDirPath = URLDecoder.decode(packageResource.getPath(), "UTF-8");
scanProjectPackage(packageName, packageDirPath, classes);
}
}
return classes;
}
private static void scanProjectPackage(String packageName, String packageDirPath, Set<Class<?>> classes) {
File packageDirFile = new File(packageDirPath);
if (packageDirFile.exists() && packageDirFile.isDirectory()) {
File[] subFiles = packageDirFile.listFiles(new FileFilter() {
@Override
public boolean accept(File pathname) {
return pathname.isDirectory() || pathname.getName().endsWith(CLASS_SUFFIX);
}
});
for (File subFile : subFiles) {
if (!subFile.isDirectory()) {
String className = trimClassSuffix(subFile.getName());
String classNameWithPackage = packageName + "." + className;
Class<?> clazz = null;
try {
clazz = Class.forName(classNameWithPackage);
} catch (ClassNotFoundException e) {
// ignore
}
assert clazz != null;
Class<?> superclass = clazz.getSuperclass();
if (superclass == AbstractAppInvoker.class) {
classes.add(clazz);
}
}
}
}
}
// trim .class suffix
private static String trimClassSuffix(String classNameWithSuffix) {
int endIndex = classNameWithSuffix.length() - CLASS_SUFFIX.length();
return classNameWithSuffix.substring(0, endIndex);
}
}
注: 在此仅扫描项目目录下的单层目录的class文件, 功能更强大的包扫描工具可参考spring源代码或Touch源代码中的PackageScanUtil类.
AppInvoker基类: AbstractAppInvoker
提供通用测试参数 & 工具函数.
public abstract class AbstractAppInvoker {
protected static final int SIZE = 170_0000;
protected static final IObjectSerializer serializer = new Hessian2Serializer();
protected static FeedDO createFeed(long id, String userId, long createTime) {
return new FeedDO(id, userId, (int) id, userId + "_" + id, createTime);
}
protected static void free(ByteBuffer byteBuffer) {
if (byteBuffer.isDirect()) {
((DirectBuffer) byteBuffer).cleaner().clean();
}
}
protected static void checkValid(Object obj) {
if (obj == null) {
throw new RuntimeException("cache invalid");
}
}
protected static void sleep(int time, String beforeMsg) {
if (!Strings.isNullOrEmpty(beforeMsg)) {
System.out.println(beforeMsg);
}
try {
Thread.sleep(time);
} catch (InterruptedException ignored) {
// no op
}
}
/**
* 供子类继承 & 外界调用
*
* @param param
*/
public abstract void invoke(Object... param);
}
序列化/反序列化接口与实现
public interface IObjectSerializer {
<T> byte[] serialize(T obj);
<T> T deserialize(byte[] bytes);
}
public class Hessian2Serializer implements IObjectSerializer {
private static final Logger LOGGER = LoggerFactory.getLogger(Hessian2Serializer.class);
@Override
public <T> byte[] serialize(T obj) {
if (obj != null) {
try (ByteArrayOutputStream os = new ByteArrayOutputStream()) {
Hessian2Output out = new Hessian2Output(os);
out.writeObject(obj);
out.close();
return os.toByteArray();
} catch (IOException e) {
LOGGER.error("Hessian serialize error ", e);
throw new CacherException(e);
}
}
return null;
}
@SuppressWarnings("unchecked")
@Override
public <T> T deserialize(byte[] bytes) {
if (bytes != null) {
try (ByteArrayInputStream is = new ByteArrayInputStream(bytes)) {
Hessian2Input in = new Hessian2Input(is);
T obj = (T) in.readObject();
in.close();
return obj;
} catch (IOException e) {
LOGGER.error("Hessian deserialize error ", e);
throw new CacherException(e);
}
}
return null;
}
}
完整项目地址: https://github.com/feiqing/off-heap-tester.git.
GC统计工具
#!/bin/bash
pid=`jps | grep $1 | awk '{print $1}'`
jstat -gcutil ${pid} 400 10000
使用
sh jstat-uti.sh ${u-main-class}
附加: 为什么在实验中in-heap cache的Minor GC那么少?
现在我还不能给出一个确切地分析答案, 有的同学说是因为CMS Full GC会连带一次Minor GC, 而用jstat会直接计入Full GC, 但查看详细的GC日志也并未发现什么端倪. 希望有了解的同学可以在下面评论区可以给我留言, 再次先感谢了O(∩_∩)O~.
觉得本文对你有帮助？请分享给更多人
关注「ImportNew」，看技术干货