wrapper.check.deadlock

This property, in combination with the other properties,

are used to configure how the Wrapper monitors a JVM for deadlocked threads. This can be very useful to detect and then work around potentially fatal problems which would otherwise be difficult if not impossible to work around.

Thread deadlock checks require that at least Java version 1.5 is used. Only JVMs will ignore the checks.

Checking for Deadlocks is fairly quick, but it does involve briefly locking and taking a snapshot of all threads so this property is FALSE by default.

Example: (Deadlock Check: Off)
wrapper.check.deadlock=FALSE

Simple Example:

Please read over the property details below, but the following simple example will configure the Wrapper to log the location of a deadlock and then immediately restart the JVM.

Example:
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=60
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL

Deadlock?:

Deadlocks can occur when two or more threads are locking resources in an order which results in all threads waiting indefinitely.

The simplest example is where Thread A locks Object A and then attempts to lock Object B, while another Thread B has Object B locked and is waiting to lock Object A. In this case, Thread A will never release Object A because it is waiting for Object B. This will never happen because Thread B will keep Object B locked indefinitely as it waits for Object A to become available.

wrapper.check.deadlock.interval

The wrapper.check.deadlock.interval property makes it possible to control the interval at which the Wrapper looks for deadlocks in an application. It can be set to any interval starting with once per second, but it defaults to "60" (once per minute). In general, for applications which are known to be stable, it may be fine to greatly decrease the frequency of these deadlock checks.

Example: (every 60 seconds)
wrapper.check.deadlock.interval=60

wrapper.check.deadlock.action

The wrapper.check.deadlock.action property makes it possible to control what the Wrapper does when a deadlock is detected. The default action is to RESTART.

Example:
wrapper.check.deadlock.action=RESTART

Possible actions are:

  • RESTART -

    will stop the current JVM and then restart a new invocation.

  • SHUTDOWN -

    will stop the JVM as well as the Wrapper.

  • DUMP -

    will invoke a thread dump. Note that because deadlocks are by definition permanent, the deadlocked state will persist until the JVM has been restarted, meaning that the thread dump will also be invoked at each interval.

  • USER_<n> -

    will cause a user defined event to be fired in the Professional Edition.

  • NONE -

    is useful because it will prevent any triggers with a higher number from being triggered.

It is possible to specify more than one actions by separating then with a space or comma. When more than one action is specified, they will be executed in rapid succession in the order specified.

The following example will perform a thread dump and then restart the JVM.

Example:
wrapper.check.deadlock.action=DUMP,RESTART

wrapper.check.deadlock.output

The wrapper.check.deadlock.output property makes it possible to control what information the Wrapper causes to be logged when a deadlock is detected. The default output level is FULL.

Example:
wrapper.check.deadlock.output=FULL

Possible output levels are:

  • FULL -

    will cause the WrapperManager class within the JVM to output a report which includes full stack traces for the threads involved in the deadlock.

    Output Example of "FULL":
    INFO   | jvm 1    | WrapperManager Error: Found 2 deadlocked threads!
    INFO   | jvm 1    | WrapperManager Error: =============================
    INFO   | jvm 1    | WrapperManager Error: "Locker-2" tid=18
    INFO   | jvm 1    | WrapperManager Error:   java.lang.Thread.State: BLOCKED
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockSecond(DeadLock.java:64)
    INFO   | jvm 1    | WrapperManager Error:       - waiting on <0x000000002fcac6db> (a java.lang.Object) owned by "Locker-1" tid=17
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockFirst(DeadLock.java:83)
    INFO   | jvm 1    | WrapperManager Error:       - locked <0x0000000029c56c60> (a java.lang.Object)
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.access$100(DeadLock.java:22)
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock$1.run(DeadLock.java:42)
    INFO   | jvm 1    | WrapperManager Error:
    INFO   | jvm 1    | WrapperManager Error: "Locker-1" tid=17
    INFO   | jvm 1    | WrapperManager Error:   java.lang.Thread.State: BLOCKED
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockSecond(DeadLock.java:64)
    INFO   | jvm 1    | WrapperManager Error:       - waiting on <0x0000000029c56c60> (a java.lang.Object) owned by "Locker-2" tid=18
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockFirst(DeadLock.java:83)
    INFO   | jvm 1    | WrapperManager Error:       - locked <0x000000002fcac6db> (a java.lang.Object)
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.access$100(DeadLock.java:22)
    INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock$1.run(DeadLock.java:42)
    INFO   | jvm 1    | WrapperManager Error:
    INFO   | jvm 1    | WrapperManager Error: =============================
    STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.
  • SIMPLE -

    will cause the WrapperManager class within the JVM to output a report which includes only a brief summary naming the threads and objects involved in the deadlock. In many cases, this is enough especially for known problems.

    Output Example of "SIMPLE":
    INFO   | jvm 1    | WrapperManager Error: Found 2 deadlocked threads!
    INFO   | jvm 1    | WrapperManager Error: =============================
    INFO   | jvm 1    | WrapperManager Error: "Locker-2" BLOCKED waiting on a java.lang.Object owned by "Locker-1"
    INFO   | jvm 1    | WrapperManager Error: "Locker-1" BLOCKED waiting on a java.lang.Object owned by "Locker-2"
    INFO   | jvm 1    | WrapperManager Error: =============================
    STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.
  • NONE -

    will cause the WrapperManager class within the JVM to suppress all output. This may be desired for production systems when a problem is known but you don't want or need to maintain too much information in the logs. The Wrapper process, as in the other options, will always log a single entry to provide a reason why the triggered action takes place.

    Output Example of "NONE":
    STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.