线上使用java作hive的任务调度,经过java
Runtime 类的exec方法来执行hive的job。最近发现有job卡死的状况。具体表现是调度的脚本挂起,并且还可能致使hive的lock不能正常释放。
使用jstack打印java的thread信息:
发现以下的lock,最终定位到waitFor函数。
"main" prio=10 tid=0x000000005b24c800 nid=0x280e in Object.wait() [0x00002b3dee8e7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eb6f88b8> (a java.lang.UNIXProcess)
at java.lang.Object.wait(Object.java:485)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
- locked <0x00000000eb6f88b8> (a java.lang.UNIXProcess)
at calltest.main(calltest.java:8)
在java的api中发现关于的Runtime类的描述,每一个jvm都有一个Runtime类的实例,Runtime类会起一个单独的进程来运行相关的命令
Every Java application has a single instance of class Runtimethat allows the application to interface with
the environment in which the application is running.
Executes the specified string command in a separate process with the specified environment and working directory.
同时,发现以下的解释:
即由exec生成的进程没有本身的console,会和父进程有IO操做联系,在输出数据比较大的时候,可能会致使缓冲区写满,从而致使进程死锁,而当前进程因为设置了waitFor会一直在等待子进程结束,从而当前子进程也进入阻塞状态。
The created subprocess does not have its own terminal or console. All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (getOutputStream(), getInputStream(), getErrorStream()). The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
waitFor方法:
causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.
解决方法:
能够在调用waitFor方法以前,为子进程建立两个线程,来读取标准输出和标准错误输出便可。