揭秘JDK 21 faketime演练CPU拉满的迷局

揭秘JDK 21 faketime演练CPU拉满的迷局

问题背景

业务团队需要进行跨年演练，主要通过faketime来模拟未来特定时间点，我们在演练过程中出现CPU上升异常，最终拉满CPU配额，触发CPU throttle现象。

上述CPU拉满现象在JDK 21版本(容器镜像为AlmaLinux 9)可以稳定复现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// 同事提供的复现代码如下所示
public class Main {
    public static void main(String[] args) {
        java.util.Date now = new java.util.Date();
        System.out.println("time:" + now);

        for (int k = 0; k < 10; k++) {
            Thread t = new Thread(() -> {
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException ignore) {
                }
            });
            t.setDaemon(false);
            t.start();
        }
    }
}

通过Tomcat环境中的自定义脚本/opt/tomcat/bin/extraenv.sh，用于设置额外的环境变量或运行自定义初始化逻辑，faketime（0.9.10）相关配置如下所示：

1
2
3
4
5
6
7
8
cat >>/opt/tomcat/bin/extraenv.sh <<EOF

export LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1
export FAKETIME="${FAKETIME}"
export DONT_FAKE_MONOTONIC=1
EOF
fi
supervisorctl restart tomcat

技术背景

LD_PRELOAD 简介

当启动一个动态链接的程序时，它并没有所有需要的函数代码。那么将会发生什么？

  1. 程序被加载到内存中
  2. 动态链接器会找出该程序运行所需的其他库（.so 文件）
  3. 它也会将这些库加载到内存中
  4. 然后将所有东西连接起来

LD_PRELOAD是一个系统环境变量，用于在运行程序时强制加载自定义的共享库。它是 Linux 环境中的动态链接功能的一部分，允许在程序运行时动态地影响共享库的加载行为。

当一个程序运行时，它会加载所需的共享库（如标准 C 库 libc）。LD_PRELOAD 允许在程序加载其正常依赖的共享库之前，优先加载用户指定的共享库。因此，用户可以用自定义的实现替换某些函数或修改程序的行为，而无需更改程序本身的代码。下述为主要的一些使用场景：

a. 函数拦截与重写

  • 通过创建一个共享库，重写某些系统调用或函数的实现（例如 malloc()、open()）。

  • 在目标程序运行时，强制优先使用自定义实现，从而修改程序行为。

    b. 性能调试

  • 拦截内存分配函数（如 malloc 和 free）以检测内存泄漏。

  • 记录函数调用和参数，帮助调试程序运行。

    c. 兼容性修复

  • 如果运行的程序依赖某些过时的库或函数，可以通过 LD_PRELOAD 动态提供兼容实现，而不需要更改系统环境。

    d. 时间伪造

  • 像 faketime 这样的工具使用 LD_PRELOAD 注入库，从而伪造程序感知到的系统时间。

    e. 安全性测试

  • 拦截和检查文件访问、网络调用等敏感操作，模拟安全攻击场景或保护特定资源。

使用方式

1
LD_PRELOAD=/PATH/DEST.so <command>

下述为三个主要场景示例：

a. 拦截并修改系统调用
假设我们有一个自定义库 myopen.so，重写了 open() 函数以记录文件访问：

1
2
3
4
5
6
7
#include <stdio.h>
#include <fcntl.h>

int open(const char *pathname, int flags, ...) {
    printf("File opened: %s\n", pathname);
    return open(pathname, flags);
}

编译为共享库：

1
gcc -shared -fPIC -o myopen.so myopen.c -ldl

使用 LD_PRELOAD 加载：

1
LD_PRELOAD=./myopen.so ls

输出将记录所有文件访问。

b. 模拟内存分配问题

​ 使用 libmalloc.so 来检测内存分配函数（如 malloc 和 free 的调用次数）：

1
LD_PRELOAD=/usr/lib/libmalloc.so my_program

c. 修复兼容性问题

​ 如果程序需要一个旧版本的 libc.so，可以使用：

1
LD_PRELOAD=/path/to/old_libc.so my_program

在使用LD_PRELOAD ，需要注意以下几点：

​ a. 优先级

  • LD_PRELOAD 的库会优先加载并覆盖系统默认的库，但仅对动态链接的程序生效。静态链接的程序不会被影响。

​ b. 权限限制

  • 出于安全原因，LD_PRELOAD 通常被忽略在 setuid 或 setgid 程序中，以防止恶意注入。

​ c. 影响范围

  • 仅影响当前终端会话或明确使用 LD_PRELOAD 的进程，不会改变全局环境。

​ d. 调试复杂性

  • 在覆盖核心系统函数时，使用 LD_PRELOAD 可能引入意外行为。

faketime简介

libfaketime 是一个与 faketime 工具配套的共享库，通过 LD_PRELOAD 环境变量，在程序载入时优先加载 libfaketime.so.1 共享库。它通过特定虚假时间的逻辑代替 libc 原生提供的时间函数逻辑，实现了为程序指定绝对日期或相对日期的能力，而不影响操作系统的全局时间。

  1. faketime 设置方式

libfaketime 接受五种提供 faketime 的方式，包括：

​ a. 通过设置环境变量 FAKETIME

​ b. 通过由环境变量 FAKETIME_TIMESTAMP_FILE 声明的文件

​ c. 通过家目录下的 .faketimerc 文件

​ d. 通过系统级别的 /etc/faketimerc 文件

​ e. 通过设置环境变量FAKETIME_UPDATE_TIMESTAMP_FILE 并执行命令 date -s ""

使用 b/d/d 方式，可以按照语法修改文件内容。由：

  • 环境变量 FAKETIME_CACHE_DURATION 确定的缓存时间，实现 faketime 的热更新

  • 或环境变量 FAKETIME_NO_CACHE=1 放弃缓存，实现 faketime 从文件的实时获取

  • 或未配置环境变量，使用默认缓存时间 10 秒

  1. 工作原理

libfaketime 拦截常见的时间系统调用，例如：

  • time()
  • gettimeofday()
  • clock_gettime()

通过注入自己的实现，它会返回伪造的时间值，而不是实际的系统时间。这种方式只对通过动态链接加载的程序有效，静态链接的程序不受影响。faketime有以下几点特性：

​ a. 固定时间
让程序始终感知为某个特定时间点，例如模拟 2025 年 1 月 1 日。

​ b. 时间偏移
以当前时间为基准，增加或减少一定的时间偏移量，例如提前一天或推迟一小时。

​ c. 加速或减速时间
模拟时间流逝的速度变化，例如每秒模拟为两秒（时间加速），或延缓时间流逝速度（时间减速）。

​ d. 动态时间调整
支持通过文件或环境变量动态调整伪造时间的规则，灵活应对测试需求。

  1. 安装与使用

    libfaketime 通常可以通过系统的包管理器安装：

  • 在 Debian系：

    1
    sudo apt install libfaketime

  • 在 RHEL系：

    1
    sudo yum install libfaketime

​ 通过设置 LD_PRELOAD，将 libfaketime 注入目标程序：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="<伪造时间>" <命令>

​ 主要有以下几种场景：

​ a. 固定时间点
​ 让程序感知到固定时间，例如 2025 年 1 月 1 日：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="2025-01-01 00:00:00" date

​ 输出结果：

1
Wed Jan  1 00:00:00 UTC 2025

​ b. 时间偏移
​ 将时间向前或向后偏移 1 天：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="+1d" date

​ 或：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="-1d" date

​ c. 时间加速
模拟时间加速，让程序运行时间每秒等效于两秒：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="@2x" sleep 5

本应需要 5 秒的任务，现在可能只需 2.5 秒完成。

​ d. 动态调整时间
使用文件配置时间伪造规则：

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="@" ./my_program

​ 通过配置文件（如 /etc/faketimerc）动态调整时间规则。

​ 使用faketime需要注意以下几点：

​ a. 仅影响动态链接程序
​ 静态链接的程序直接使用系统调用，不会被 libfaketime 拦截，比如Golang

​ b. 对多线程程序的影响
多线程程序可能对时间操作敏感，使用 libfaketime 时需要注意可能引发的并发问题。

​ c. 环境隔离
使用 LD_PRELOAD 仅影响当前终端或进程，不会对全局环境产生影响。

​ d. 安全问题
出于安全考虑，某些敏感程序可能会忽略 LD_PRELOAD（如 setuid 程序）。

寻根溯源

根据issue的说明，需要针对特定JDK版本，移除变量DONT_FAKE_MONOTONIC

  1. 定位JVM时钟

java -Xlog:all=info 是 Java 的一个命令行选项，用于控制 JVM 的日志输出行为, 在排查过程中使用该配置，打印关键信息（连半路出家都算不上）。

1
2
3
4
5
6
7
LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1 FAKETIME="@2024-12-22 19:44:32" java -Xlog:all=info Main.java



[0.006s][info][os       ] Use of pthread_condattr_setclock is supported
[0.006s][info][os       ] Relative timed-wait using pthread_cond_timedwait is associated with CLOCK_MONOTONIC
[0.006s][info][os,thread] Lookup of __pthread_get_minstack succeeded

可以发现JDK 21底层JVM判断可以使用CLOCK_MONOTONIC时钟是通过pthread_condattr_setclock判断，当 JVM 使用条件变量（如 pthread_cond_timedwait）实现超时等待时，可以选择使用单调时钟（CLOCK_MONOTONIC）来避免系统时间调整对超时行为的影响。

​ 2. 排查阻塞点

通过下述信息可以发现，Thread.Sleep行为调用链以及阻塞点

1
2
3
4
5
6
7
8
Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)):
#0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2  0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#3  0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#4  0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
#5  0x00007fb06fdf39c0 in ?? ()
#6  0x0000000000000000 in ?? ()

下述为详细DEBUG过程

  1. 执行gdb命令，attach java进程

    1
    gdb -p 15959

  2. 堆栈回溯

1
2
3
4
5
6
(gdb) bt
#0  0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
#2  0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
#3  0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so
#4  0x000055fa5ded6b0f in main ()

  1. 列出当前所有线程信息

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    (gdb) info threads
      Id   Target Id         Frame 
      25   Thread 0x7fb089023700 (LWP 15960) "java" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      24   Thread 0x7fb08553c700 (LWP 15962) "GC Thread#0" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      23   Thread 0x7fb08543b700 (LWP 15963) "G1 Main Marker" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      22   Thread 0x7fb08533a700 (LWP 15964) "G1 Conc#0" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      21   Thread 0x7fb084a31700 (LWP 15965) "G1 Refine#0" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      20   Thread 0x7fb084930700 (LWP 15966) "G1 Service" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      19   Thread 0x7fb084753700 (LWP 15967) "VM Periodic Tas" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      18   Thread 0x7fb084652700 (LWP 15968) "VM Thread" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      17   Thread 0x7fb0843c8700 (LWP 15969) "Reference Handl" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      16   Thread 0x7fb0842c7700 (LWP 15970) "Finalizer" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      15   Thread 0x7fb0841c6700 (LWP 15971) "Signal Dispatch" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      14   Thread 0x7fb0648af700 (LWP 15972) "Service Thread" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      13   Thread 0x7fb0647ae700 (LWP 15973) "Monitor Deflati" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      12   Thread 0x7fb0646ad700 (LWP 15974) "C2 CompilerThre" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      11   Thread 0x7fb0645ac700 (LWP 15975) "C1 CompilerThre" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      10   Thread 0x7fb0643db700 (LWP 15976) "Common-Cleaner" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      9    Thread 0x7fb0642da700 (LWP 15977) "Notification Th" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      8    Thread 0x7fb0641d9700 (LWP 15978) "C2 CompilerThre" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      7    Thread 0x7fb00efff700 (LWP 15979) "C2 CompilerThre" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      6    Thread 0x7fb00eefe700 (LWP 15980) "GC Thread#1" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      5    Thread 0x7fb00edfd700 (LWP 15981) "GC Thread#2" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      4    Thread 0x7fb00ecfc700 (LWP 15982) "GC Thread#3" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
      3    Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      2    Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    * 1    Thread 0x7fb089024740 (LWP 15959) "java" 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0

    其中关键信息

    1
    2
    3
    3    Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      2    Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

  2. 显示所有线程的堆栈回溯

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
    228
    229
    230
    231
    232
    233
    234
    235
    236
    237
    238
    239
    240
    241
    242
    243
    244
    245
    246
    247
    248
    249
    250
    251
    252
    253
    254
    255
    256
    257
    258
    259
    260
    261
    262
    263
    264
    265
    266
    267
    268
    269
    270
    271
    272
    273
    274
    275
    276
    277
    278
    279
    280
    281
    282
    283
    284
    285
    286
    287
    288
    289
    290
    291
    292
    293
    294
    295
    296
    297
    298
    299
    300
    (gdb) thread apply all bt

    Thread 25 (Thread 0x7fb089023700 (LWP 15960)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0879361ea in Threads::destroy_vm() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087355fa2 in jni_DestroyJavaVM () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb089035942 in JavaMain () from /usr/java/jdk21/bin/../lib/libjli.so
    #6  0x00007fb089039289 in ThreadJavaMain () from /usr/java/jdk21/bin/../lib/libjli.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 24 (Thread 0x7fb08553c700 (LWP 15962)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 23 (Thread 0x7fb08543b700 (LWP 15963)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb08718e80a in G1ConcurrentMarkThread::wait_for_next_cycle() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb08718fa29 in G1ConcurrentMarkThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #9  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 22 (Thread 0x7fb08533a700 (LWP 15964)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    ---Type <return> to continue, or q <return> to quit---
    Thread 21 (Thread 0x7fb084a31700 (LWP 15965)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb087195d7c in G1PrimaryConcurrentRefineThread::wait_for_completed_buffers() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0871960f8 in G1ConcurrentRefineThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #9  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 20 (Thread 0x7fb084930700 (LWP 15966)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08010b1c0, mutex=0x7fb08010b198, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0871eac11 in G1ServiceThread::wait_for_task() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0871eafa0 in G1ServiceThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 19 (Thread 0x7fb084753700 (LWP 15967)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eefcd0 <mutex_init()::PeriodicTask_lock_storage+48>, mutex=0x7fb087eefca8 <mutex_init()::PeriodicTask_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb08767ed4d in WatcherThread::sleep() const () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb08767ee51 in WatcherThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #9  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 18 (Thread 0x7fb084652700 (LWP 15968)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eeead0 <mutex_init()::VMOperation_lock_storage+48>, mutex=0x7fb087eeeaa8 <mutex_init()::VMOperation_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879be7f5 in VMThread::wait_for_operation() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879bf318 in VMThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    ---Type <return> to continue, or q <return> to quit---
    #6  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #9  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 17 (Thread 0x7fb0843c8700 (LWP 15969)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb08739937a in JVM_WaitForReferencePendingList () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb06fdf39c0 in ?? ()
    #5  0x0000000000000004 in ?? ()
    #6  0x00007fb0843c7948 in ?? ()
    #7  0x0000000000000000 in ?? ()

    Thread 16 (Thread 0x7fb0842c7700 (LWP 15970)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be0ab in PlatformEvent::park() () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb08768e3f5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0878da904 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb08739ad7f in JVM_MonitorWait () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb06fdf39c0 in ?? ()
    #6  0x00007fb0842c67a8 in ?? ()
    #7  0x00007fb06fdf3697 in ?? ()
    #8  0x0000000000000000 in ?? ()

    Thread 15 (Thread 0x7fb0841c6700 (LWP 15971)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb08781b93f in os::signal_wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0876a59e5 in signal_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 14 (Thread 0x7fb0648af700 (LWP 15972)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb08776ca82 in ServiceThread::service_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    ---Type <return> to continue, or q <return> to quit---
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 13 (Thread 0x7fb0647ae700 (LWP 15973)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0cd0 <mutex_init()::MonitorDeflation_lock_storage+48>, mutex=0x7fb087ef0ca8 <mutex_init()::MonitorDeflation_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087659353 in MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #9  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 12 (Thread 0x7fb0646ad700 (LWP 15974)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 11 (Thread 0x7fb0645ac700 (LWP 15975)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 10 (Thread 0x7fb0643db700 (LWP 15976)):
    ---Type <return> to continue, or q <return> to quit---
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08017cab8, mutex=0x7fb08017ca90, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be547 in Parker::park(bool, long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0879603c4 in Unsafe_Park () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb06fdf39c0 in ?? ()
    #5  0x00007fb0643da578 in ?? ()
    #6  0x00007fb0643da5e8 in ?? ()
    #7  0x00007fb06fdef180 in ?? ()
    #8  0x00000000868003f8 in ?? ()
    #9  0x00007fb0643da580 in ?? ()
    #10 0x0000000000000000 in ?? ()

    Thread 9 (Thread 0x7fb0642da700 (LWP 15977)):
    #0  0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #2  0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb08767f2f2 in NotificationThread::notification_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 8 (Thread 0x7fb0641d9700 (LWP 15978)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 7 (Thread 0x7fb00efff700 (LWP 15979)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, 
        abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
    ---Type <return> to continue, or q <return> to quit---
    #7  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #8  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #9  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 6 (Thread 0x7fb00eefe700 (LWP 15980)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 5 (Thread 0x7fb00edfd700 (LWP 15981)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 4 (Thread 0x7fb00ecfc700 (LWP 15982)):
    #0  0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
    #1  0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
    #2  0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
    #3  0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
    #6  0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
    #7  0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007fb08852bb0d in clone () from /lib64/libc.so.6

    Thread 3 (Thread 0x7fb00ebfb700 (LWP 15983)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08070b250, mutex=0x7fb08070b228, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb06fdf39c0 in ?? ()
    #6  0x0000000000000000 in ?? ()
    ---Type <return> to continue, or q <return> to quit---

    Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)):
    #0  0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
    #2  0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
    #3  0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
    #4  0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
    #5  0x00007fb06fdf39c0 in ?? ()
    #6  0x0000000000000000 in ?? ()

    Thread 1 (Thread 0x7fb089024740 (LWP 15959)):
    #0  0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0
    #1  0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
    #2  0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
    #3  0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so
    #4  0x000055fa5ded6b0f in main ()

    1. 根因溯源

    其中Thread.Sleep在jdk21+9版本下的调用路径：

    a. Thread.Sleep https://github.com/openjdk/jdk21u/blob/e45287d1ad9dcadf8a23d3271f1b675b8dade0ac/src/java.base/share/classes/java/lang/Thread.java#L498

    b. Thread.Sleep-> sleep0

    c. sleep2 对应JNI实现 JVM_Sleep

    d. JVM_Sleep中使用jvm实现sleep_nanos https://github.com/openjdk/jdk21u/blob/e45287d1ad9dcadf8a23d3271f1b675b8dade0ac/src/hotspot/share/runtime/javaThread.cpp#L1990C18-L1990C29

    e. 其中sleep_nanos内部调用slp->park_nanos(nanos_remaining)

    f. park_nanos调用了glibc的pthread_cond_timedwait,但被faketime劫持

    • faketime中pthread_cond_timedwait,因clock_id不是MONOTONIC，因此计算用了realtime
    • 然后传递至glibc的pthread_cond_timedwait，其调用clock_gettime来做超时判断，这里同样返回的是faketime
    • 导致一直等待，jvm的thread无法unpark

其中sleep_nanos 的实现如下所示：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
bool JavaThread::sleep_nanos(jlong nanos) {
  assert(this == Thread::current(),  "thread consistency check");
  assert(nanos >= 0, "nanos are in range");

  ParkEvent * const slp = this->_SleepEvent;
  // Because there can be races with thread interruption sending an unpark()
  // to the event, we explicitly reset it here to avoid an immediate return.
  // The actual interrupt state will be checked before we park().
  slp->reset();
  // Thread interruption establishes a happens-before ordering in the
  // Java Memory Model, so we need to ensure we synchronize with the
  // interrupt state.
  OrderAccess::fence();

  jlong prevtime = os::javaTimeNanos();

  jlong nanos_remaining = nanos;

  for (;;) {
    // interruption has precedence over timing out
    if (this->is_interrupted(true)) {
      return false;
    }

    if (nanos_remaining <= 0) {
      return true;
    }

    {
      ThreadBlockInVM tbivm(this);
      OSThreadWaitState osts(this->osthread(), false /* not Object.wait() */);
      slp->park_nanos(nanos_remaining);
    }

    // Update elapsed time tracking
    jlong newtime = os::javaTimeNanos();
    if (newtime - prevtime < 0) {
      // time moving backwards, should only happen if no monotonic clock
      // not a guarantee() because JVM should not abort on kernel/glibc bugs
      assert(false,
             "unexpected time moving backwards detected in JavaThread::sleep()");
    } else {
      nanos_remaining -= (newtime - prevtime);
    }
    prevtime = newtime;
  }
}

其中faktime针对pthread_cond_timedwait在0.9.8与0.9.10均实现了劫持，但关于CLOCK_MONOTONIC行为应该是存在差异的，其中0.9.10根据Glibc版本做了自适应调整

参考链接

  1. https://www.ffutop.com/posts/2024-08-06-faketime/

  2. https://cloud.tencent.com/developer/article/2240075

  3. https://github.com/wolfcw/libfaketime/issues/469

  4. https://github.com/wolfcw/libfaketime/commit/0e61d3d1917f530b5e3f66db2ed5dd6acd20e798#diff-9c808c04a6c7f0cf8c7c99d7d03f9e6cfd420f7733dd38d1519d86b4da92a7c3R3571