The Mystery Behind JDK 21 faketime Exercise Overloading CPU

Background

The business team needs to conduct a year-end drill, primarily using faketime to simulate specific future time points. During the drill, we encountered an abnormal CPU spike, which eventually maxed out the CPU quota and triggered a CPU throttle phenomenon.

The above CPU maxing out phenomenon can be consistently reproduced in JDK 21 (with the container image being AlmaLinux 9).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class Main {
public static void main(String[] args) {
java.util.Date now = new java.util.Date();
System.out.println("time:" + now);

for (int k = 0; k < 10; k++) {
Thread t = new Thread(() -> {
try {
Thread.sleep(1000);
} catch (InterruptedException ignore) {
}
});
t.setDaemon(false);
t.start();
}
}
}

Through the custom script /opt/tomcat/bin/extraenv.sh in the Tomcat environment, which is used to set additional environment variables or run custom initialization logic, the faketime (0.9.10) related configuration is as follows:

1
2
3
4
5
6
7
8
cat >>/opt/tomcat/bin/extraenv.sh <<EOF

export LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1
export FAKETIME="${FAKETIME}"
export DONT_FAKE_MONOTONIC=1
EOF
fi
supervisorctl restart tomcat

Technical Background

Introduction to LD_PRELOAD

  • When a dynamically linked program is started, it doesn’t have all the required function code. So, what happens?

    1. The program is loaded into memory.
    2. The dynamic linker identifies any other libraries (e.g., .so files) needed for the program to run.
    3. It loads these libraries into memory as well.
    4. Then, everything is linked together.

    LD_PRELOAD is a system environment variable used to force the loading of custom shared libraries when running a program. It is part of the dynamic linking functionality in the Linux environment, allowing the dynamic manipulation of shared library loading behavior during program execution.

    When a program runs, it loads the required shared libraries (such as the standard C library libc). LD_PRELOAD allows the user to prioritize loading a custom shared library before the program loads its normally required libraries. This enables the user to replace certain functions with custom implementations or modify the program’s behavior without changing the program’s source code. The following are some main use cases:

    a. Function Interception and Overriding

    • By creating a shared library, certain system calls or functions (e.g., malloc(), open()) can be overridden.

    • During the program’s execution, the custom implementation is forced to take priority, modifying the program’s behavior.

      b. Performance Debugging

    • Intercept memory allocation functions (e.g., malloc, free) to detect memory leaks.

    • Record function calls and parameters to help debug program execution.

      c. Compatibility Fixes

    • If the running program relies on outdated libraries or functions, LD_PRELOAD can dynamically provide compatible implementations without modifying the system environment.

      d. Time Faking

    • Tools like faketime use LD_PRELOAD to inject a library that fakes the system time perceived by the program.

      e. Security Testing

    • Intercept and examine sensitive operations like file access or network calls, simulating security attack scenarios or protecting specific resources.

Usage

1
LD_PRELOAD=/PATH/DEST.so <command>

Below are examples of three main scenarios:

a. Intercepting and Modifying System Calls
Suppose we have a custom library myopen.so that overrides the open() function to log file accesses:

1
2
3
4
5
6
7
#include <stdio.h>
#include <fcntl.h>

int open(const char *pathname, int flags, ...) {
printf("File opened: %s\n", pathname);
return open(pathname, flags);
}

Compile into a shared library:

1
gcc -shared -fPIC -o myopen.so myopen.c -ldl

Load it using LD_PRELOAD:

1
LD_PRELOAD=./myopen.so ls

The output will log all file access.

b. Simulating Memory Allocation Issues

Use libmalloc.so to detect memory allocation function calls (e.g., the number of malloc and free calls):

1
LD_PRELOAD=/usr/lib/libmalloc.so my_program

c. Fixing Compatibility Issues

If a program requires an older version of libc.so, you can use:

1
LD_PRELOAD=/path/to/old_libc.so my_program

Key Points to Note When Using LD_PRELOAD:

a. Priority

  • Libraries specified by LD_PRELOAD are loaded first and override the system’s default libraries, but this only applies to dynamically linked programs. Statically linked programs are not affected.

b. Permission Restrictions

  • For security reasons, LD_PRELOAD is typically ignored in setuid or setgid programs to prevent malicious library injection.

c. Scope of Impact

  • It only affects the current terminal session or processes explicitly using LD_PRELOAD and does not alter the global environment.

d. Debugging Complexity

  • Overriding core system functions with LD_PRELOAD may introduce unintended behavior, making debugging more challenging.

Introduction to faketime

libfaketime is a shared library that works alongside the faketime tool. By using the LD_PRELOAD environment variable, it ensures that libfaketime.so.1 is loaded before other libraries when a program starts. It replaces the original time-related logic provided by libc with its custom logic, enabling programs to perceive a specific absolute or relative date and time without affecting the system-wide clock.

1. faketime Configuration Methods

libfaketime supports five ways to provide fake time:

a. Setting the FAKETIME environment variable.
b. Using a file specified by the FAKETIME_TIMESTAMP_FILE environment variable.
c. Using a .faketimerc file in the user’s home directory.
d. Using the system-level /etc/faketimerc file.
e. Setting the FAKETIME_UPDATE_TIMESTAMP_FILE environment variable and executing the date -s "" command.

When using options b, c, or d, you can modify the file’s content to define the time rules.
The behavior is controlled by:

  • The FAKETIME_CACHE_DURATION environment variable, which defines the cache duration to enable hot updates.
  • The FAKETIME_NO_CACHE=1 environment variable to disable caching for real-time fetching.
  • Default cache duration of 10 seconds if no environment variable is set.

2. How It Works

libfaketime intercepts common system calls for time, such as:

  • time()
  • gettimeofday()
  • clock_gettime()

By injecting its implementation, it returns a faked time value instead of the actual system time. This approach only works for dynamically linked programs, as statically linked ones remain unaffected. Key features of faketime include:

a. Fixed Time
Programs perceive a specific fixed time, such as simulating January 1, 2025.

b. Time Offset
Add or subtract a time offset based on the current time, e.g., advancing one day or delaying one hour.

c. Time Acceleration or Deceleration
Simulate faster or slower time progression, such as doubling the speed of time (time acceleration) or slowing it down (time deceleration).

d. Dynamic Time Adjustment
Supports adjusting time rules dynamically via files or environment variables to meet flexible testing needs.


3. Installation and Usage

libfaketime can usually be installed through the system’s package manager:

  • On Debian-based systems:

    1
    sudo apt install libfaketime
  • On RHEL-based systems:

    1
    sudo yum install libfaketime

To inject libfaketime into a target program, use LD_PRELOAD:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="<fake time>" <command>

Common Scenarios:

a. Fixed Time Point
Simulate a fixed time, such as January 1, 2025:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="2025-01-01 00:00:00" date

Output:

1
Wed Jan  1 00:00:00 UTC 2025

b. Time Offset
Shift time forward or backward by one day:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="+1d" date

or:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="-1d" date

c. Time Acceleration
Simulate time passing faster, such as making one second equal to two seconds:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="@2x" sleep 5

A task that normally takes 5 seconds might now complete in 2.5 seconds.

d. Dynamic Time Adjustment
Configure time manipulation rules via a file:

1
LD_PRELOAD=/usr/lib/libfaketime.so.1 FAKETIME="@" ./my_program

The time rules can be dynamically adjusted using configuration files like /etc/faketimerc.

Important Considerations:

a. Affects Only Dynamically Linked Programs
Statically linked programs, such as those written in Golang, use direct system calls and are not intercepted by libfaketime.

b. Impact on Multi-threaded Programs
Multi-threaded programs sensitive to time operations may experience concurrency issues when using libfaketime.

c. Environment Isolation
Using LD_PRELOAD only affects the current terminal session or process and does not impact the global environment.

d. Security Concerns
For security reasons, some sensitive programs (e.g., setuid programs) may ignore LD_PRELOAD to prevent malicious injection.

Tracing the Root Cause

According to issue #425, a specific JDK version requires the removal of the DONT_FAKE_MONOTONIC variable.

1. Identifying the JVM Clock

The java -Xlog:all=info command-line option enables verbose logging in the JVM, which is useful for identifying issues and debugging. In this investigation, it prints critical information:

1
LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1 FAKETIME="@2024-12-22 19:44:32" java -Xlog:all=info Main.java

Sample output:

1
2
3
[0.006s][info][os       ] Use of pthread_condattr_setclock is supported
[0.006s][info][os ] Relative timed-wait using pthread_cond_timedwait is associated with CLOCK_MONOTONIC
[0.006s][info][os,thread] Lookup of __pthread_get_minstack succeeded

It shows that the JVM in JDK 21 determines the availability of CLOCK_MONOTONIC using pthread_condattr_setclock. This ensures that when the JVM uses condition variables (e.g., pthread_cond_timedwait) for timeout operations, it can rely on CLOCK_MONOTONIC to avoid disruptions caused by system time changes.

2. Analyzing the Blocking Point

The following information reveals the call chain and blocking point for Thread.sleep behavior:

1
2
3
4
5
6
7
8
Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb06fdf39c0 in ?? ()
#6 0x0000000000000000 in ?? ()

Debugging Details:

  1. Attach the JVM Process
    Use gdb to attach to the Java process:

    1
    gdb -p 15959
  2. Backtrace the Stack
    Retrieve the call stack for debugging:

    1
    (gdb) bt

    Example output:

    1
    2
    3
    4
    5
    #0  0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0
    #1 0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
    #2 0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
    #3 0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so
    #4 0x000055fa5ded6b0f in main ()
  3. List Threads
    Identify all active threads:

    1
    (gdb) info threads

    Example output:

    1
    2
    3
    4
    5
    6
    7
    8
    Id   Target Id         Frame 
    25 Thread 0x7fb089023700 (LWP 15960) "java" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
    24 Thread 0x7fb08553c700 (LWP 15962) "GC Thread#0" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 ()
    23 Thread 0x7fb08543b700 (LWP 15963) "G1 Main Marker" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
    ...
    3 Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    2 Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    * 1 Thread 0x7fb089024740 (LWP 15959) "java" 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0

    Focus on key threads, such as:

    1
    2
    3    Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    2 Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  4. Backtrace Specific Threads
    For each critical thread, backtrace the stack:

    1
    (gdb) thread apply <thread-id> bt

By analyzing these threads and call stacks, you can pinpoint the blocking points and understand how CLOCK_MONOTONIC interacts with libfaketime.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
(gdb) thread apply all bt

Thread 25 (Thread 0x7fb089023700 (LWP 15960)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0879361ea in Threads::destroy_vm() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087355fa2 in jni_DestroyJavaVM () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb089035942 in JavaMain () from /usr/java/jdk21/bin/../lib/libjli.so
#6 0x00007fb089039289 in ThreadJavaMain () from /usr/java/jdk21/bin/../lib/libjli.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fb08553c700 (LWP 15962)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fb08543b700 (LWP 15963)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb08718e80a in G1ConcurrentMarkThread::wait_for_next_cycle() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb08718fa29 in G1ConcurrentMarkThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fb08533a700 (LWP 15964)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 21 (Thread 0x7fb084a31700 (LWP 15965)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb087195d7c in G1PrimaryConcurrentRefineThread::wait_for_completed_buffers() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0871960f8 in G1ConcurrentRefineThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fb084930700 (LWP 15966)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08010b1c0, mutex=0x7fb08010b198, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0871eac11 in G1ServiceThread::wait_for_task() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0871eafa0 in G1ServiceThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fb084753700 (LWP 15967)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eefcd0 <mutex_init()::PeriodicTask_lock_storage+48>, mutex=0x7fb087eefca8 <mutex_init()::PeriodicTask_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb08767ed4d in WatcherThread::sleep() const () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb08767ee51 in WatcherThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fb084652700 (LWP 15968)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eeead0 <mutex_init()::VMOperation_lock_storage+48>, mutex=0x7fb087eeeaa8 <mutex_init()::VMOperation_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879be7f5 in VMThread::wait_for_operation() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879bf318 in VMThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
---Type <return> to continue, or q <return> to quit---
#6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fb0843c8700 (LWP 15969)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb08739937a in JVM_WaitForReferencePendingList () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb06fdf39c0 in ?? ()
#5 0x0000000000000004 in ?? ()
#6 0x00007fb0843c7948 in ?? ()
#7 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fb0842c7700 (LWP 15970)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be0ab in PlatformEvent::park() () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb08768e3f5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0878da904 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb08739ad7f in JVM_MonitorWait () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb06fdf39c0 in ?? ()
#6 0x00007fb0842c67a8 in ?? ()
#7 0x00007fb06fdf3697 in ?? ()
#8 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fb0841c6700 (LWP 15971)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb08781b93f in os::signal_wait() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0876a59e5 in signal_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fb0648af700 (LWP 15972)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb08776ca82 in ServiceThread::service_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
---Type <return> to continue, or q <return> to quit---
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fb0647ae700 (LWP 15973)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0cd0 <mutex_init()::MonitorDeflation_lock_storage+48>, mutex=0x7fb087ef0ca8 <mutex_init()::MonitorDeflation_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087659353 in MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fb0646ad700 (LWP 15974)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fb0645ac700 (LWP 15975)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fb0643db700 (LWP 15976)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08017cab8, mutex=0x7fb08017ca90, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be547 in Parker::park(bool, long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0879603c4 in Unsafe_Park () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb06fdf39c0 in ?? ()
#5 0x00007fb0643da578 in ?? ()
#6 0x00007fb0643da5e8 in ?? ()
#7 0x00007fb06fdef180 in ?? ()
#8 0x00000000868003f8 in ?? ()
#9 0x00007fb0643da580 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fb0642da700 (LWP 15977)):
#0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb08767f2f2 in NotificationThread::notification_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fb0641d9700 (LWP 15978)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fb00efff700 (LWP 15979)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>,
abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so
---Type <return> to continue, or q <return> to quit---
#7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fb00eefe700 (LWP 15980)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fb00edfd700 (LWP 15981)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fb00ecfc700 (LWP 15982)):
#0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so
#6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so
#7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fb00ebfb700 (LWP 15983)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08070b250, mutex=0x7fb08070b228, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb06fdf39c0 in ?? ()
#6 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---

Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)):
#0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840
#2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so
#4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so
#5 0x00007fb06fdf39c0 in ?? ()
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fb089024740 (LWP 15959)):
#0 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
#2 0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so
#3 0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so
#4 0x000055fa5ded6b0f in main ()

3. Root Cause Analysis

For Thread.Sleep in JDK 21+9, the call path is as follows:

  1. Thread.Sleep
  2. Thread.Sleep → sleep0
  3. sleep0 → JNI Implementation (JVM_Sleep)
  4. JVM_Sleep
    • Utilizes the JVM method sleep_nanos (Source).
  5. sleep_nanos
    • Internally calls park_nanos(nanos_remaining) via the ParkEvent object.
  6. park_nanos
    • Invokes glibc’s pthread_cond_timedwait, which gets intercepted by faketime.

Issue Analysis:

  • In faketime’s pthread_cond_timedwait, if the clock_id is not CLOCK_MONOTONIC, it uses realtime for calculations.
  • The realtime value is then passed to glibc’s pthread_cond_timedwait, which uses faketime’s altered clock_gettime for timeout computation.
  • As a result, the JVM thread remains stuck in waiting, unable to be unparked, because the faketime-altered clock values are inconsistent with the expected monotonic behavior.

Implementation Insight: sleep_nanos

The implementation of sleep_nanos is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
bool JavaThread::sleep_nanos(jlong nanos) {
assert(this == Thread::current(), "thread consistency check");
assert(nanos >= 0, "nanos are in range");

ParkEvent * const slp = this->_SleepEvent;
// Reset the event to avoid immediate return due to races with interrupts
slp->reset();
OrderAccess::fence();

jlong prevtime = os::javaTimeNanos();
jlong nanos_remaining = nanos;

for (;;) {
// Check for interruptions before timing out
if (this->is_interrupted(true)) {
return false;
}

if (nanos_remaining <= 0) {
return true;
}

{
ThreadBlockInVM tbivm(this);
OSThreadWaitState osts(this->osthread(), false /* not Object.wait() */);
slp->park_nanos(nanos_remaining);
}

// Update elapsed time tracking
jlong newtime = os::javaTimeNanos();
if (newtime - prevtime < 0) {
assert(false,
"unexpected time moving backwards detected in JavaThread::sleep()");
} else {
nanos_remaining -= (newtime - prevtime);
}
prevtime = newtime;
}
}
  • The loop ensures that time tracking and remaining nanos are adjusted iteratively.
  • The fallback to realtime (os::javaTimeNanos) when CLOCK_MONOTONIC isn’t available is critical here, but faketime alters the behavior, causing backward time jumps or inconsistencies.

Faketime’s Behavior

  • In versions 0.9.8 and 0.9.10, pthread_cond_timedwait is intercepted.

  • However, the handling of

    1
    CLOCK_MONOTONIC

    differs between these versions:

    • Version 0.9.10 introduces adaptive behavior based on the Glibc version, offering improved compatibility for CLOCK_MONOTONIC.

This discrepancy in handling leads to differences in behavior under faketime when dealing with threads and timers.

Reference

  1. https://www.ffutop.com/posts/2024-08-06-faketime/

  2. https://cloud.tencent.com/developer/article/2240075

  3. https://github.com/wolfcw/libfaketime/issues/469

  4. https://github.com/wolfcw/libfaketime/commit/0e61d3d1917f530b5e3f66db2ed5dd6acd20e798#diff-9c808c04a6c7f0cf8c7c99d7d03f9e6cfd420f7733dd38d1519d86b4da92a7c3R3571