一次永久代泄漏（perm泄漏）排查
写在前面
网上关于perm区泄露的文章比较少，特别是对于动态类加载方面问题的分析比较少，在此记录下。html
perm区问题通常两种解决方案：java
- 启动时perm区问题，通常修改PermSize,MaxPermSize参数便可
- 运行时动态生成类加载问题，这种问题比较难搞，须要关心动态加载了哪些类
周末早晨收到几台机器告警(fullgc告警)(perm大于90%告警)，为快速解决问题，先把几台机器重启解决，留下了一台禁用端口保留现场进行问题分析。平时上线发版比较频繁，发版后jvm回收，若是较长时间没有发版可能会形成泄露，收到告警。python
吃完早饭后登上机器进行排查。git
排查过程
登入机器，查看内存使用高的进程：github
topweb
pid:15298apache
既然是perm区问题，查看永久代状况：json
jmap -permstat pid > 15298dump.permstatbootstrap
class_loader classes bytes parent_loader alive? type <bootstrap> 3630 21866152 null live <internal> 0x0000000705e4df00 1 2008 0x00000006c002e450 dead sun/reflect/DelegatingClassLoader@0x00000007c0050c30 0x00000007282e1040 4 23480 0x00000006c002e450 dead com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040
经过awk统计type类型，查看加载了哪一种类型的类：swift
awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2
ava/net/URLClassLoader@0x00000007c01c1598 0 java/util/ResourceBundle$RBClassLoader@0x00000007c0337380 0 type 0 sun/reflect/DelegatingClassLoader@0x00000007c0050c30 11361872 sun/reflect/misc/MethodUtil@0x00000007c777bf60 134528 <internal> 21866152 com/alibaba/fastjson/util/ASMClassLoader@0x00000007c44de0e0 4785232 N/A 5146 sun/misc/Launcher$AppClassLoader@0x00000007c021c2d8 5423216 com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040 5488288 sun/misc/Launcher$ExtClassLoader@0x00000007c01c1978 615216 org/eclipse/jetty/webapp/WebAppClassLoader@0x00000007c06e0da0 738107040
WebAppClassLoader加载最多，达到了738107040，近738m。
知道了加载的类最多，怎么分析具体加载了哪一种类呢？
以前通常经过Java -verbose查看启动类加载过程。
对于运行时貌似有两种方式：
- 基于ClassFilter写一个拦截器，dump特定的类，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)编译好类，在编译好的类目录下调用下面的命令进行dump
- 问了(政威老师)，准备试试arthas，https://alibaba.github.io/arthas/classloader.html
机器上安装好arthas-boot.jar，并启动：
java -jar arthas-boot.jar
选择进程，执行类后台导出，便后续分析：
classloader -a >> &
打开导出文件：
hash:null, BootstrapClassLoader [B [C [D [F [I [J [Lcom.sun.activation.registries.MimeTypeFile; [Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable; [Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer; [Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State; [Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State; [Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec; [Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec; [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry; [Lcom.sun.jmx.mbeanserver.MXBeanMapping; [Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature; [Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property; [Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl; [Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression; [Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar; [Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint; [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher; [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf; [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator; [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode; [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr; [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo; [Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator; [Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt; [Lcom.sun.org.apache.xerces.internal.util.Status; [Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry; [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry; [Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State; [Lcom.sun.org.apache.xerces.internal.xni.QName; [Lcom.sun.org.apache.xerces.internal.xni.XMLLocator; [Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar; [Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar; [Lcom.sun.org.apache.xerces.internal.xs.ShortList; [Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation; [Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse; [Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration; [Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem; [Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration; [Lcom.sun.org.apache.xerces.internal.xs.XSObject; [Lcom.sun.org.apache.xerces.internal.xs.XSParticle; [Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSTerm; [Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime; [Lcom.sun.org.apache.xml.internal.dtm.DTM; [Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser; [Lcom.sun.org.apache.xml.internal.dtm.DTMIterator; [Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry; [Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType; [Lcom.sun.org.apache.xpath.internal.Expression; [Lcom.sun.org.apache.xpath.internal.ExpressionNode; [Lcom.sun.org.apache.xpath.internal.XPathVisitable; [Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator; [Lcom.sun.org.apache.xpath.internal.axes.PathComponent; [Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest; [Lcom.sun.org.apache.xpath.internal.axes.SubContextList; [Lcom.sun.org.apache.xpath.internal.objects.XObject; [Lcom.sun.org.apache.xpath.internal.patterns.NodeTest; [Lcom.sun.xml.internal.ws.org.objectweb.asm.Item; [Lcom.sun.xml.internal.ws.org.objectweb.asm.Type; [Ljava.awt.AWTKeyStroke; [Ljava.awt.Dimension; [Ljava.awt.Queue; [Ljava.awt.event.ActionListener; [Ljava.awt.event.ComponentListener; [Ljava.awt.event.FocusListener; [Ljava.awt.event.HierarchyBoundsListener; ......
统计最多的类：
#!/usr/bin/python
from collections import Counter
package_name_count_dic = {}
for s in open('14.txt'):
full_qualified_name = s.strip()
if not full_qualified_name:
continue
class_name_index = full_qualified_name.rfind('.')
if class_name_index >= 0:
package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]
else:
package_name = class_name = full_qualified_name
if package_name in package_name_count_dic:
package_name_count_dic[package_name] += 1
else:
package_name_count_dic[package_name] = 1
k = Counter(package_name_count_dic)
high = k.most_common(5)
for p, c in high:
print p, c
最多的几个类：
真凶：ma.glasnost.orika.generated。
查看代码中谁使用了orika类库。
发现是闪购同窗，因为咱们的系统目前对接多方，闪购同窗还在咱们系统作代码开发，在codereview上存在一些问题。
@Component
public class SGMapperFactory implements FactoryBean<MapperFactory> {
@Override
public MapperFactory getObject() {
return new DefaultMapperFactory.Builder().build();
}
@Override
public Class<?> getObjectType() {
return MapperFactory.class;
}
@Override
public boolean isSingleton() {
return true;
}
}
每次调用getObject都会新建立DefaultMapperFactory对象。MapperGenerator 每次会动态产生类。
至此问题排查，推进闪购同窗改动。
后记
看代码是3月份代码写的，为什么到如今才发现问题呢？
首先是平时发版比较频繁，jvm发版后问题解决。 最近发版比较少，同时这些是扩容机器，在最近的几回发版中并无发版，因此形成类加载持续一段时间，最后形成永久代泄露。 为尽早发现问题须要在流程上进行控制，好比增长codereview细致程度，在灰度发版后对发版机器进行引流压测，尽早发现问题，解决问题。
相关文章
- 1. 一次永久代泄漏（perm泄漏）排查
- 2. Metaspace泄漏排查
- 3. 一次 Java 内存泄漏的排查
- 4. JS内存泄漏排查
- 5. jvisualvm内存泄漏排查
- 6. Java内存泄漏排查
- 7. 区块链项目：golang内存泄漏，goroutine泄漏排查
- 8. DFT泄漏 频谱泄漏
- 9. csdn 代码泄漏
- 10. 一次毕生难忘的 Java 内存泄漏排查经历
- 更多相关文章...
- • 一对一关联查询 - MyBatis教程
- • 一对多关联查询 - MyBatis教程
- • 算法总结-二分查找法
- • 算法总结-归并排序
相关标签/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
欢迎关注本站公众号,获取更多信息
相关文章
- 1. 一次永久代泄漏（perm泄漏）排查
- 2. Metaspace泄漏排查
- 3. 一次 Java 内存泄漏的排查
- 4. JS内存泄漏排查
- 5. jvisualvm内存泄漏排查
- 6. Java内存泄漏排查
- 7. 区块链项目：golang内存泄漏，goroutine泄漏排查
- 8. DFT泄漏 频谱泄漏
- 9. csdn 代码泄漏
- 10. 一次毕生难忘的 Java 内存泄漏排查经历