关于一次Weblogic活动线程的问题处理

时间 2019-11-16

原文原文链接

Weblogic控制台监控发现环境>>服务器>>你的服务器>>监控>>线程中活动执行线程居然是2000多。同一套系统在另外一套平台上，而且访问的人很多，也没有超过100。重启此应用，活动进程依然没有变化，真是奇怪。java

查看转储线程堆：linux

==== FULL THREAD DUMP===============

            Tue Aug 12 10:54:48 2014

            Oracle JRockit(R)R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64

            "Main Thread" id=1idx=0x4 tid=12399 prio=5 alive, waiting, native_blocked

                -- Waiting for notification on:weblogic/t3/srvr/T3Srvr@0xc70a6538[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)

                ^-- Lock released whilewaiting: weblogic/t3/srvr/T3Srvr@0xc70a6538[fat lock]

                atweblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)

                atweblogic/Server.main(Server.java:71)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "(Signal Handler)" id=2idx=0x8 tid=12400 prio=5 alive, native_blocked, daemon

            "(OC Main Thread)" id=3idx=0xc tid=12401 prio=5 alive, native_waiting, daemon

            "(GC Worker Thread 1)" id=?idx=0x10 tid=12402 prio=5 alive, daemon

            "(GC Worker Thread 2)"id=? idx=0x14 tid=12403 prio=5 alive, daemon

            "(GC Worker Thread 3)"id=? idx=0x18 tid=12404 prio=5 alive, daemon

            "(GC Worker Thread 4)"id=? idx=0x1c tid=12405 prio=5 alive, daemon

            "(Code Generation Thread1)" id=4 idx=0x20 tid=12406 prio=5 alive, native_waiting, daemon

            "(Code Optimization Thread1)" id=5 idx=0x24 tid=12407 prio=5 alive, native_waiting, daemon

            "(VM Periodic Task)" id=6idx=0x28 tid=12408 prio=10 alive, native_blocked, daemon

            "Finalizer" id=7 idx=0x2ctid=12409 prio=8 alive, native_waiting, daemon

                atjrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(NativeMethod)

                atjrockit/memory/Finalizer.access$700(Finalizer.java:12)

                atjrockit/memory/Finalizer$4.run(Finalizer.java:189)

                atjava/lang/Thread.run(Thread.java:619)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "Reference Handler" id=8idx=0x30 tid=12410 prio=10 alive, native_waiting, daemon

                atjava/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(NativeMethod)

                atjava/lang/ref/Reference.access$100(Reference.java:11)

                atjava/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "(Sensor Event Thread)"id=9 idx=0x34 tid=12411 prio=5 alive, native_blocked, daemon

            "VM JFR Buffer Thread"id=10 idx=0x38 tid=12412 prio=5 alive, in native, daemon

            "Timer-0" id=13 idx=0x3ctid=12415 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:java/util/TaskQueue@0xc504e198[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(NativeMethod)

                atjava/lang/Object.wait(Object.java:485)

                atjava/util/TimerThread.mainLoop(Timer.java:483)

                ^-- Lock released whilewaiting: java/util/TaskQueue@0xc504e198[fat lock]

                atjava/util/TimerThread.run(Timer.java:462)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "Timer-1" id=14 idx=0x40 tid=12416prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:java/util/TaskQueue@0xc504e548[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/util/TimerThread.mainLoop(Timer.java:509)

                ^-- Lock released whilewaiting: java/util/TaskQueue@0xc504e548[fat lock]

                atjava/util/TimerThread.run(Timer.java:462)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '0'for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x44tid=12417 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6e748b0[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(NativeMethod)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released whilewaiting: weblogic/work/ExecuteThread@0xc6e748b0[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '1'for queue: 'weblogic.kernel.Default (self-tuning)'" id=16 idx=0x48tid=12418 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6d42d00[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released while waiting:weblogic/work/ExecuteThread@0xc6d42d00[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '2'for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x4ctid=12419 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6e1c620[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released whilewaiting: weblogic/work/ExecuteThread@0xc6e1c620[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

…………此处省略大量重复的信息web

在下对weblogic了解甚少，对java开发也是丈二和尚，身边也没有懂weblogic的人，看来只能借助网络搜索了。百度、ITpub、CSDN全试了都没有相关问题。后来加了几个weblogic的qq群，在群里发问基本也是没人管。哎，这世界怎么就没有个好心人呢？不过好心人最终仍是出现了，他直接就找到了问题所在——这个问题的缘由就在启动参数上。（就是啊，从新启动都不行，那十有八九是启动参数的问题啊，我这笨脑子！）服务器

形成这种状况的参数在weblogic\user_projects\domains\base_domain\bin下的setDomainEnvNaNd中的Dweblogic.threadpool.MinPoolSize。个人配置是这样的：网络

JAVA_OPTIONS="${JAVA_OPTIONS}${JAVA_PROPERTIES} -Dwlw.iterativeDev=${iterativeDevFlag}-Dwlw.testConsole=${testConsoleFlag} -Dweblogic.threadpool.MinPoolSize=2000 -Dweblogic.threadpool.MaxPoolSize=4000-Dwlw.logErrorsToConsole=${logErrorsToConsoleFlag}"多线程

exportJAVA_OPTIONSdom

这里的-Dweblogic.threadpool.MinPoolSize=2000意思是默认线程池大小，这个参数设置多少合适我也不知道，具体设置于不设置有什么大的区别也不知道……额，反正是它在搞鬼。我将此参数去掉以后，从新启动，活动线程变少了。oop

网抄一些weblogic的知识，留做备查——————优化

检查线程数spa

经过weblogic控制台能够查看线程数的统计信息。weblogic9及以上的线程是自优化的。但应该查看系统的线程最大数是否过大，若是过大，就要注意系统为何会有这么大的压力。以下为示例截图

对应中文翻译：

ActiveExecute Threads：在活动的线程池内处理请求的线程个数

ExecuteThread Total Count：线程池内线程的总数

ExecuteThread Idle Count：池内的空闲线程数。它不包含stuck和standby的线程数。它是指等待接收新请求到来并处理的线程个数

queuelength ：在等待队列里的请求数，一般保留默认值 65536 ，队列长度代表了同时发来请求的最大数， 65536 个请求是个很大的数，即便达到这个最大数，也是不多见的。若是达到最大队列长度，WebLogic 会自动成倍增加队列大小，以处理额外的工做。注意：超过 65536 个请求预示队列中的线程有问题，不单单只是队列自己的长度问题，实践代表在队列中有堵塞线程或线程数不足的状况存在。

hoggingthread count ：线程处理一个请求时间超过必定值被视为hogging状态，若是继续处理请求超过必定时间将被视为stuck，或处理完请求后被放回线程池

standbythread count ：统计在standby（备用）线程池内的线程数。这些线程不须要处理当前请求被放入standby池内，当活动的线程池内须要更多线程时，这些线程将被激活。

Execute Thread Total Count= Active Execute Threads+ standbythread count