The business team needs to conduct a year-end drill, primarily using faketime to simulate specific future time points. During the drill, we encountered an abnormal CPU spike, which eventually maxed out the CPU quota and triggered a CPU throttle phenomenon.
The above CPU maxing out phenomenon can be consistently reproduced in JDK 21 (with the container image being AlmaLinux 9).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
public class Main { public static void main(String[] args) { java.util.Date now = new java.util.Date(); System.out.println("time:" + now);
for (int k = 0; k < 10; k++) { Thread t = new Thread(() -> { try { Thread.sleep(1000); } catch (InterruptedException ignore) { } }); t.setDaemon(false); t.start(); } } }
Through the custom script /opt/tomcat/bin/extraenv.sh in the Tomcat environment, which is used to set additional environment variables or run custom initialization logic, the faketime (0.9.10) related configuration is as follows:
When a dynamically linked program is started, it doesn’t have all the required function code. So, what happens?
The program is loaded into memory.
The dynamic linker identifies any other libraries (e.g., .so files) needed for the program to run.
It loads these libraries into memory as well.
Then, everything is linked together.
LD_PRELOAD is a system environment variable used to force the loading of custom shared libraries when running a program. It is part of the dynamic linking functionality in the Linux environment, allowing the dynamic manipulation of shared library loading behavior during program execution.
When a program runs, it loads the required shared libraries (such as the standard C library libc). LD_PRELOAD allows the user to prioritize loading a custom shared library before the program loads its normally required libraries. This enables the user to replace certain functions with custom implementations or modify the program’s behavior without changing the program’s source code. The following are some main use cases:
a. Function Interception and Overriding
By creating a shared library, certain system calls or functions (e.g., malloc(), open()) can be overridden.
During the program’s execution, the custom implementation is forced to take priority, modifying the program’s behavior.
Record function calls and parameters to help debug program execution.
c. Compatibility Fixes
If the running program relies on outdated libraries or functions, LD_PRELOAD can dynamically provide compatible implementations without modifying the system environment.
d. Time Faking
Tools like faketime use LD_PRELOAD to inject a library that fakes the system time perceived by the program.
e. Security Testing
Intercept and examine sensitive operations like file access or network calls, simulating security attack scenarios or protecting specific resources.
Usage
1
LD_PRELOAD=/PATH/DEST.so <command>
Below are examples of three main scenarios:
a. Intercepting and Modifying System Calls Suppose we have a custom library myopen.so that overrides the open() function to log file accesses:
Use libmalloc.so to detect memory allocation function calls (e.g., the number of malloc and free calls):
1
LD_PRELOAD=/usr/lib/libmalloc.so my_program
c. Fixing Compatibility Issues
If a program requires an older version of libc.so, you can use:
1
LD_PRELOAD=/path/to/old_libc.so my_program
Key Points to Note When Using LD_PRELOAD:
a. Priority
Libraries specified by LD_PRELOAD are loaded first and override the system’s default libraries, but this only applies to dynamically linked programs. Statically linked programs are not affected.
b. Permission Restrictions
For security reasons, LD_PRELOAD is typically ignored in setuid or setgid programs to prevent malicious library injection.
c. Scope of Impact
It only affects the current terminal session or processes explicitly using LD_PRELOAD and does not alter the global environment.
d. Debugging Complexity
Overriding core system functions with LD_PRELOAD may introduce unintended behavior, making debugging more challenging.
Introduction to faketime
libfaketime is a shared library that works alongside the faketime tool. By using the LD_PRELOAD environment variable, it ensures that libfaketime.so.1 is loaded before other libraries when a program starts. It replaces the original time-related logic provided by libc with its custom logic, enabling programs to perceive a specific absolute or relative date and time without affecting the system-wide clock.
1. faketime Configuration Methods
libfaketime supports five ways to provide fake time:
a. Setting the FAKETIME environment variable. b. Using a file specified by the FAKETIME_TIMESTAMP_FILE environment variable. c. Using a .faketimerc file in the user’s home directory. d. Using the system-level /etc/faketimerc file. e. Setting the FAKETIME_UPDATE_TIMESTAMP_FILE environment variable and executing the date -s "" command.
When using options b, c, or d, you can modify the file’s content to define the time rules. The behavior is controlled by:
The FAKETIME_CACHE_DURATION environment variable, which defines the cache duration to enable hot updates.
The FAKETIME_NO_CACHE=1 environment variable to disable caching for real-time fetching.
Default cache duration of 10 seconds if no environment variable is set.
2. How It Works
libfaketime intercepts common system calls for time, such as:
time()
gettimeofday()
clock_gettime()
By injecting its implementation, it returns a faked time value instead of the actual system time. This approach only works for dynamically linked programs, as statically linked ones remain unaffected. Key features of faketime include:
a. Fixed Time Programs perceive a specific fixed time, such as simulating January 1, 2025.
b. Time Offset Add or subtract a time offset based on the current time, e.g., advancing one day or delaying one hour.
c. Time Acceleration or Deceleration Simulate faster or slower time progression, such as doubling the speed of time (time acceleration) or slowing it down (time deceleration).
d. Dynamic Time Adjustment Supports adjusting time rules dynamically via files or environment variables to meet flexible testing needs.
3. Installation and Usage
libfaketime can usually be installed through the system’s package manager:
On Debian-based systems:
1
sudo apt install libfaketime
On RHEL-based systems:
1
sudo yum install libfaketime
To inject libfaketime into a target program, use LD_PRELOAD:
The time rules can be dynamically adjusted using configuration files like /etc/faketimerc.
Important Considerations:
a. Affects Only Dynamically Linked Programs Statically linked programs, such as those written in Golang, use direct system calls and are not intercepted by libfaketime.
b. Impact on Multi-threaded Programs Multi-threaded programs sensitive to time operations may experience concurrency issues when using libfaketime.
c. Environment Isolation Using LD_PRELOAD only affects the current terminal session or process and does not impact the global environment.
d. Security Concerns For security reasons, some sensitive programs (e.g., setuid programs) may ignore LD_PRELOAD to prevent malicious injection.
Tracing the Root Cause
According to issue #425, a specific JDK version requires the removal of the DONT_FAKE_MONOTONIC variable.
1. Identifying the JVM Clock
The java -Xlog:all=info command-line option enables verbose logging in the JVM, which is useful for identifying issues and debugging. In this investigation, it prints critical information:
[0.006s][info][os ] Use of pthread_condattr_setclock is supported [0.006s][info][os ] Relative timed-wait using pthread_cond_timedwait is associated with CLOCK_MONOTONIC [0.006s][info][os,thread] Lookup of __pthread_get_minstack succeeded
It shows that the JVM in JDK 21 determines the availability of CLOCK_MONOTONIC using pthread_condattr_setclock. This ensures that when the JVM uses condition variables (e.g., pthread_cond_timedwait) for timeout operations, it can rely on CLOCK_MONOTONIC to avoid disruptions caused by system time changes.
2. Analyzing the Blocking Point
The following information reveals the call chain and blocking point for Thread.sleep behavior:
1 2 3 4 5 6 7 8
Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb06fdf39c0 in ?? () #6 0x0000000000000000 in ?? ()
Debugging Details:
Attach the JVM Process Use gdb to attach to the Java process:
1
gdb -p 15959
Backtrace the Stack Retrieve the call stack for debugging:
1
(gdb) bt
Example output:
1 2 3 4 5
#0 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so #2 0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so #3 0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so #4 0x000055fa5ded6b0f in main ()
List Threads Identify all active threads:
1
(gdb) info threads
Example output:
1 2 3 4 5 6 7 8
Id Target Id Frame 25 Thread 0x7fb089023700 (LWP 15960) "java" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () 24 Thread 0x7fb08553c700 (LWP 15962) "GC Thread#0" 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () 23 Thread 0x7fb08543b700 (LWP 15963) "G1 Main Marker" 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () ... 3 Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () 2 Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () * 1 Thread 0x7fb089024740 (LWP 15959) "java" 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0
Focus on key threads, such as:
1 2
3 Thread 0x7fb00ebfb700 (LWP 15983) "Thread-0" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () 2 Thread 0x7fb00eafa700 (LWP 15984) "Thread-1" 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
Backtrace Specific Threads For each critical thread, backtrace the stack:
1
(gdb) thread apply <thread-id> bt
By analyzing these threads and call stacks, you can pinpoint the blocking points and understand how CLOCK_MONOTONIC interacts with libfaketime.
Thread 25 (Thread 0x7fb089023700 (LWP 15960)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0879361ea in Threads::destroy_vm() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087355fa2 in jni_DestroyJavaVM () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb089035942 in JavaMain () from /usr/java/jdk21/bin/../lib/libjli.so #6 0x00007fb089039289 in ThreadJavaMain () from /usr/java/jdk21/bin/../lib/libjli.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb08553c700 (LWP 15962)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb08543b700 (LWP 15963)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb08718e80a in G1ConcurrentMarkThread::wait_for_next_cycle() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb08718fa29 in G1ConcurrentMarkThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb08533a700 (LWP 15964)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit--- Thread 21 (Thread 0x7fb084a31700 (LWP 15965)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb087195d7c in G1PrimaryConcurrentRefineThread::wait_for_completed_buffers() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0871960f8 in G1ConcurrentRefineThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb084930700 (LWP 15966)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08010b1c0, mutex=0x7fb08010b198, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0871eac11 in G1ServiceThread::wait_for_task() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0871eafa0 in G1ServiceThread::run_service() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0870453fb in ConcurrentGCThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb084753700 (LWP 15967)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eefcd0 <mutex_init()::PeriodicTask_lock_storage+48>, mutex=0x7fb087eefca8 <mutex_init()::PeriodicTask_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb08767ed4d in WatcherThread::sleep() const () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb08767ee51 in WatcherThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb084652700 (LWP 15968)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087eeead0 <mutex_init()::VMOperation_lock_storage+48>, mutex=0x7fb087eeeaa8 <mutex_init()::VMOperation_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879be7f5 in VMThread::wait_for_operation() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879bf318 in VMThread::run() () from /usr/java/jdk21/lib/server/libjvm.so ---Type <return> to continue, or q <return> to quit--- #6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb0843c8700 (LWP 15969)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb08739937a in JVM_WaitForReferencePendingList () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb06fdf39c0 in ?? () #5 0x0000000000000004 in ?? () #6 0x00007fb0843c7948 in ?? () #7 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7fb0842c7700 (LWP 15970)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be0ab in PlatformEvent::park() () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb08768e3f5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0878da904 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb08739ad7f in JVM_MonitorWait () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb06fdf39c0 in ?? () #6 0x00007fb0842c67a8 in ?? () #7 0x00007fb06fdf3697 in ?? () #8 0x0000000000000000 in ?? ()
Thread 15 (Thread 0x7fb0841c6700 (LWP 15971)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb08781b93f in os::signal_wait() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0876a59e5 in signal_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb0648af700 (LWP 15972)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb08776ca82 in ServiceThread::service_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so ---Type <return> to continue, or q <return> to quit--- #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb0647ae700 (LWP 15973)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0cd0 <mutex_init()::MonitorDeflation_lock_storage+48>, mutex=0x7fb087ef0ca8 <mutex_init()::MonitorDeflation_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087659353 in MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb0646ad700 (LWP 15974)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb0645ac700 (LWP 15975)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb0643db700 (LWP 15976)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08017cab8, mutex=0x7fb08017ca90, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be547 in Parker::park(bool, long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0879603c4 in Unsafe_Park () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb06fdf39c0 in ?? () #5 0x00007fb0643da578 in ?? () #6 0x00007fb0643da5e8 in ?? () #7 0x00007fb06fdef180 in ?? () #8 0x00000000868003f8 in ?? () #9 0x00007fb0643da580 in ?? () #10 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fb0642da700 (LWP 15977)): #0 0x00007fb088a0aa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb0876be95b in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #2 0x00007fb087666369 in Monitor::wait_without_safepoint_check(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb08767f2f2 in NotificationThread::notification_thread_entry(JavaThread*, JavaThread*) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb0641d9700 (LWP 15978)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb00efff700 (LWP 15979)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb087ef0250 <mutex_init()::MethodCompileQueue_lock_storage+48>, mutex=0x7fb087ef0228 <mutex_init()::MethodCompileQueue_lock_storage+8>, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be937 in PlatformMonitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0876663ec in Monitor::wait(unsigned long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb087026e76 in CompileQueue::get(CompilerThread*) () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb08702ae79 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0872d5cd8 in JavaThread::thread_main_inner() () from /usr/java/jdk21/lib/server/libjvm.so ---Type <return> to continue, or q <return> to quit--- #7 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #8 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #9 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb00eefe700 (LWP 15980)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb00edfd700 (LWP 15981)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb00ecfc700 (LWP 15982)): #0 0x00007fb088a0cb3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 #1 0x00007fb088a0cbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007fb088a0cc6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #3 0x00007fb08776afa2 in PosixSemaphore::wait() () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0879e4d5b in WorkerThread::run() () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb0879283b8 in Thread::call_run() () from /usr/java/jdk21/lib/server/libjvm.so #6 0x00007fb0876b312a in thread_native_entry(Thread*) () from /usr/java/jdk21/lib/server/libjvm.so #7 0x00007fb088a06ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb08852bb0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb00ebfb700 (LWP 15983)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb08070b250, mutex=0x7fb08070b228, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb06fdf39c0 in ?? () #6 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7fb00eafa700 (LWP 15984)): #0 0x00007fb088a0ade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb088c22499 in pthread_cond_timedwait_common (cond=0x7fb0806e7350, mutex=0x7fb0806e7328, abstime=<optimized out>, compat=FT_COMPAT_GLIBC_2_3_2) at libfaketime.c:2840 #2 0x00007fb0876be25f in PlatformEvent::park_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #3 0x00007fb0872d8d59 in JavaThread::sleep_nanos(long) () from /usr/java/jdk21/lib/server/libjvm.so #4 0x00007fb0873989ed in JVM_Sleep () from /usr/java/jdk21/lib/server/libjvm.so #5 0x00007fb06fdf39c0 in ?? () #6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fb089024740 (LWP 15959)): #0 0x00007fb088a08017 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fb089039e6f in CallJavaMainInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so #2 0x00007fb089036ffd in ContinueInNewThread () from /usr/java/jdk21/bin/../lib/libjli.so #3 0x00007fb089037a4d in JLI_Launch () from /usr/java/jdk21/bin/../lib/libjli.so #4 0x000055fa5ded6b0f in main ()
3. Root Cause Analysis
For Thread.Sleep in JDK 21+9, the call path is as follows:
Internally calls park_nanos(nanos_remaining) via the ParkEvent object.
park_nanos
Invokes glibc’s pthread_cond_timedwait, which gets intercepted by faketime.
Issue Analysis:
In faketime’s pthread_cond_timedwait, if the clock_id is not CLOCK_MONOTONIC, it uses realtime for calculations.
The realtime value is then passed to glibc’s pthread_cond_timedwait, which uses faketime’s altered clock_gettime for timeout computation.
As a result, the JVM thread remains stuck in waiting, unable to be unparked, because the faketime-altered clock values are inconsistent with the expected monotonic behavior.
boolJavaThread::sleep_nanos(jlong nanos){ assert(this == Thread::current(), "thread consistency check"); assert(nanos >= 0, "nanos are in range");
ParkEvent * const slp = this->_SleepEvent; // Reset the event to avoid immediate return due to races with interrupts slp->reset(); OrderAccess::fence();
// Update elapsed time tracking jlong newtime = os::javaTimeNanos(); if (newtime - prevtime < 0) { assert(false, "unexpected time moving backwards detected in JavaThread::sleep()"); } else { nanos_remaining -= (newtime - prevtime); } prevtime = newtime; } }
The loop ensures that time tracking and remaining nanos are adjusted iteratively.
The fallback to realtime (os::javaTimeNanos) when CLOCK_MONOTONIC isn’t available is critical here, but faketime alters the behavior, causing backward time jumps or inconsistencies.
Faketime’s Behavior
In versions 0.9.8 and 0.9.10, pthread_cond_timedwait is intercepted.
However, the handling of
1
CLOCK_MONOTONIC
differs between these versions:
Version 0.9.10 introduces adaptive behavior based on the Glibc version, offering improved compatibility for CLOCK_MONOTONIC.
This discrepancy in handling leads to differences in behavior under faketime when dealing with threads and timers.