TL;DR:热更新问题
要增强的目标类ThreadPoolExecutor
,在增强前(bytebuddy.installOn → instrument.addTransformer / redefine / retransform)已经被classloader装载到jvm里
导致instrument对已加载的类增强有些问题,这个问题也许可以通过深入redefine或retransform的机制解决,快速且稳定的解决方案为在插件装载完成前不加载ThreadPoolExecutor
类
0.前置
instrument中的几个基本逻辑:
ClassFile
实际指的是java字节码,class文件格式,这个对象在内存中,和文件没有关系
Note the term class file is used as defined in section 3.1 of The Java™ Virtual Machine Specification, to mean a sequence of bytes in class file format, whether or not they reside in a file.
jdk8中接口只有一个transform方法,类在加载、redefined或retransformed时会被调用来增强(the transformer's transform method is invoked when classes are loaded, redefined, or retransformed.)
java 5+,已经加载的类重新进行转换处理,即会触发重新加载类定义,需要注意的是,新加载的类不能修改旧有的类声明,譬如不能增加属性、不能修改方法声明
java 6+,与如上类似,但不是重新进行转换处理,而是直接把处理结果(bytecode)直接给JVM
“Agents use these methods to retransform previously loaded classes without needing to access their class files.”
redefine和retransform的区别:https://stackoverflow.com/questions/19009583/difference-between-redefine-and-retransform-in-javaagent
1.原始问题
自定义executors插件未生效,调试发现onInstall
时抛UnsupportedOperationException
,且未打印出来,这里异常栈顶是sun.instrument.InstrumentationImpl#retransformClasses
,由SkyWalking调用ByteBuddy时指定了AgentBuilder.RedefinitionStrategy.RETRANSFORMATION
策略加载
对比相同使用bytebuddy的自定义agent生效:https://gitlab.sunyongfei.com/platform-basic/java-agents/tree/threadpool-qy/thread-agent
其中onInstall
前的RedefinitionStrategy
不同:
SkyWalking采用AgentBuilder.RedefinitionStrategy.RETRANSFORMATION
,自定义agent采用AgentBuilder.RedefinitionStrategy.REDEFINITION
net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy:
/**
* <p>
* A redefinition strategy regulates how already loaded classes are modified by a built agent.
* </p>
* <p>
* <b>Important</b>: Most JVMs do not support changes of a class's structure after a class was already
* loaded. Therefore, it is typically required that this class file transformer was built while enabling
* {@link AgentBuilder#disableClassFormatChanges()}.
* </p>
*/
enum RedefinitionStrategy {
/**
* Disables redefinition such that already loaded classes are not affected by the agent.
*/
DISABLED(false, false) {
@Override
public void apply(Instrumentation instrumentation,
AgentBuilder.Listener listener,
CircularityLock circularityLock,
PoolStrategy poolStrategy,
LocationStrategy locationStrategy,
DiscoveryStrategy discoveryStrategy,
BatchAllocator redefinitionBatchAllocator,
Listener redefinitionListener,
LambdaInstrumentationStrategy lambdaInstrumentationStrategy,
DescriptionStrategy descriptionStrategy,
FallbackStrategy fallbackStrategy,
RawMatcher matcher) {
/* do nothing */
}
@Override
protected void check(Instrumentation instrumentation) {
throw new IllegalStateException("Cannot apply redefinition on disabled strategy");
}
@Override
protected Collector make() {
throw new IllegalStateException("A disabled redefinition strategy cannot create a collector");
}
},
/**
* <p>
* Applies a <b>redefinition</b> to all classes that are already loaded and that would have been transformed if
* the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is <b>not</b>
* registered for applying retransformations.
* </p>
* <p>
* Using this strategy, a redefinition is applied as a single transformation request. This means that a single illegal
* redefinition of a class causes the entire redefinition attempt to fail.
* </p>
* <p>
* <b>Note</b>: When applying a redefinition, it is normally required to use a {@link TypeStrategy} that applies
* a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
* the constrains given by this type strategy.
* </p>
*/
REDEFINITION(true, false) {
@Override
protected void check(Instrumentation instrumentation) {
if (!instrumentation.isRedefineClassesSupported()) {
throw new IllegalStateException("Cannot apply redefinition on " + instrumentation);
}
}
@Override
protected Collector make() {
return new Collector.ForRedefinition();
}
},
/**
* <p>
* Applies a <b>retransformation</b> to all classes that are already loaded and that would have been transformed if
* the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is registered
* for applying retransformations.
* </p>
* <p>
* Using this strategy, a retransformation is applied as a single transformation request. This means that a single illegal
* retransformation of a class causes the entire retransformation attempt to fail.
* </p>
* <p>
* <b>Note</b>: When applying a retransformation, it is normally required to use a {@link TypeStrategy} that applies
* a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
* the constrains given by this type strategy.
* </p>
*/
RETRANSFORMATION(true, true) {
@Override
protected void check(Instrumentation instrumentation) {
if (!DISPATCHER.isRetransformClassesSupported(instrumentation)) {
throw new IllegalStateException("Cannot apply retransformation on " + instrumentation);
}
}
@Override
protected Collector make() {
return new Collector.ForRetransformation();
}
};
RedefinitionStrategy的存在是因为“Most JVMs do not support changes of a class's structure after a class was already loaded. Therefore, it is typically required that this class file transformer was built while enabling disableClassFormatChanges().”
即大部分JVM不支持在类被装载后修改,需要指定对这些已经被装载的类如何Redefine策略
open-jdk8 HotSpot VM instrumentation的支持:支持Redefine,不支持Retransform
最上面抛出异常的图调用方法:this.retransformClasses
为反射获取的sun.instrument.InstrumentationImpl#retransformClasses
方法:
打断点查看native方法的支持情况,原来这里才会懒加载,即上面instrumentation对retransform支持到这里才是正确的:
支持retransform,也就是说执行retransformClasses0(mNativeAgent, classes);原生方法抛了上述异常,此时ThreadPoolExecutor类应该是被加载了的:
验证:自定义agent也改为和SkyWalking一致的RETRAINSFORMATION策略,期望如果也变得不生效,则说明增强时ThreadPoolExecutor
类已被装载,且是策略选择问题
结果:自定义agent依旧生效
说明要么自定义agent没有装载ThreadPoolExecutor
类,这个策略自然也就对ThreadPoolExecutor
类无效;要么可能压根就不是这个问题
验证自定义agent在增强时是否加载了目标类:断点打在apply:4812, AgentBuilder$RedefinitionStrategy
stack:
apply:4812, AgentBuilder$RedefinitionStrategy (net.bytebuddy.agent.builder)
doInstall:9463, AgentBuilder$Default (net.bytebuddy.agent.builder)
installOn:9384, AgentBuilder$Default (net.bytebuddy.agent.builder)
instrumentation:58, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
premain:38, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
invoke0:-1, NativeMethodAccessorImpl (sun.reflect)
invoke:62, NativeMethodAccessorImpl (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
loadClassAndStartAgent:386, InstrumentationImpl (sun.instrument)
loadClassAndCallPremain:401, InstrumentationImpl (sun.instrument)
发现SkyWalking已经加载了ThreadPoolExecutor,自定义agent没有加载
2.问题定位
通过在ThreadPoolExecutor构造器方法上打断点,定位到加载ThreadPoolExecutor在日志组件中,获取单例FIleWriter时会通过ThreadPoolExecutor创建异步线程
org.apache.skywalking.apm.agent.core.logging.core.FileWriter#FileWriter
private FileWriter() {
logBuffer = new ArrayBlockingQueue(1024);
final ArrayList<String> outputLogs = new ArrayList<String>(200);
Executors.newSingleThreadScheduledExecutor(new DefaultNamedThreadFactory("LogFileWriter"))
.scheduleAtFixedRate(new RunnableWithExceptionProtection(new Runnable() {
@Override
public void run() {
try {
logBuffer.drainTo(outputLogs);
for (String log : outputLogs) {
writeToFile(log + Constants.LINE_SEPARATOR);
}
try {
fileOutputStream.flush();
} catch (IOException e) {
e.printStackTrace();
}
} finally {
outputLogs.clear();
}
}
}, new RunnableWithExceptionProtection.CallbackWhenException() {
@Override
public void handle(Throwable t) {
}
}), 0, 1, TimeUnit.SECONDS);
}
3.修复
org.apache.skywalking.apm.agent.core.logging.core.WriterFactory
,增加FILE_WRITTER_INIT_FLAG
开关,在所有插件和bytebuddy installlOn执行结束前不允许初始化FileWriter
,日志只能输出到STDOUT
executors-plugin插件本身在插桩时不能打印日志,否则会死循环递归调用interpretor逻辑,引发stackoverflow等问题