How to fix java.lang.OutOfMemoryError: unable to create new native thread [Solution]

There are several types of OutOfMemoryError in Java e.g. OutOfMemoryError related to Java heap space and permgen space, and a new one coming in Java 8, Java.lang.OutOfMemoryError: Metaspace. Each and every OutOfMemoryError has its own unique reason and corresponding unique solution. For example, java.langOutOfMemoryError: Java Heap Space comes when the application has exhausted all heap memory and tries to create an object which requires further memory allocation, At that time JVM throws this error to tell the application that it's not possible to create an object.
Similarly java.lang.OutOfMemoryError: PermGen Space comes when there is no more memory in permgen space and the application tries to load more classes (as class metadata is stored in this area) or tries to create a new String (because prior to Java 7 String pool has also existed on permgen space).

We have already seen how to solve those nightmarish errors and today we will see another nightmare of Java developer, Java.lang.OutOfMemoryError: unable to create a new native thread, this is rather rare for core Java developers but if you have been involved in web and enterprise Java development, you would have seen it on web servers like tomcat and jetty, and enterprise servers like JBoss, WebLogic, and Glassfish.

As the error message "java.lang.OutOfMemoryError: unable to create new native thread" suggests it comes when your application tries to create more and more threads and exhausted the limit imposed by your Server.

The name "native" has a special significance in this error message because when you create a thread, JVM also creates a native thread (native to the operating system like Windows, Linux, Solaris, or Mac OSX), and when JVM fails to create a native thread in response to new Thread(), it throws "java.lang.OutOfMemoryError: unable to create new native thread". In Linux, you can check a number of native threads created by a process by using the ps -eLF | grep java | we -l command.

Btw, If you are serious about mastering JVM and Java performance in-depth then you can also check out Java Application Performance and Memory Management course by Matt Greencroft. It's a great course for experienced Java developers.




When Java.lang.OutOfMemoryError: unable to create new native thread Comes

When I first see this error in one of our Java application I was surprised because it was a simple core Java application which receives the message, process it, and then sends it to another queue. I know for sure that we cannot spawn so many threads to hit this limit because it's usually very large e.g. around 30K in Linux.

We attached our application with JConsole and leave it running for a week, the error was true as our application has a thread leak. It was unnecessary creating a new thread to process a new message and those threads just keep running because of the message processing loop.

In short, there is nothing fancy about this error, if you keep creating threads, eventually you will hit the limit allowed by Operating System to a process and then JVM will not be able to create new native threads and it will throw "java.lang.OutOfMemoryError: unable to create new native thread". Now, the limit is something that depends on OS e.g. it would be different on Windows than Linux and vice-versa.

In UNIX, you can check this limit by using the "ulimit" command as shown below:

$ ulimit -a
max user processes              (-u) 1024

You can see by default number of processes per users is 1024, but you can increase the limit again by using the ulimit command as shown below:

ulimit -u 2048

This is also one of the easiest solutions to this problem. In order to learn about this error and in general about how to troubleshoot performance issues in Java, I would suggest reading a good book on Java profiling and performance like Java Performance The Definitive Guide By Scott Oaks. This will give you enough knowledge about tools and processes to deal with such errors in real-world Java applications.

How to fix java.lang.OutOfMemoryError: unable to create new native thread



java.lang.OutOfMemoryError: unable to create new native thread - Solution

This error is a clear case of resource exhaustion. If you think that 1024 threads are too few for your process you can certainly avoid this error by increasing the limit by using the ulimit command as shown above, at least in Linux. 

There are similar solutions available for other operating systems but just like we can increase heap to avoid the java.lang.OutOfMemoryError: Java Heap Space, you can also avoid "java.lang.OutOfMemoryError: unable to create new native thread" error.

If you think that 1024 is too many threads for your application but you are still getting this error then this is a clear case of thread leak in your Java application. In order to find the thread leak, you need to find the code which is creating threads and carefully review it to avoid the situation I have explained in the last paragraph. 

You can use the jConsole or plain UNIX commands to monitor a number of threads spawned by your process. Alternatively, you can also join these JVM and Java performance Tuning courses to learn more about how to monitor and profile Java applications.

Btw, if you are not sure about the limit in your operating system, you can run the following Java program to find exactly when your JVM will throw "java.lang.OutOfMemoryError: unable to create new native thread" error. 

This program does, nothing but creates threads and put them into sleep for a couple of hours. The program will eventually throw this error and you can check the count just before that to find out how many threads are allowed to your JVM on your machine.

/*
 * Java Program to check when JVM will throw
 * "java.lang.OutOfMemoryError: unable to create new native thread" error.
 */

public class ThreadLimitChecker{

  public static void main(String[] args) {
    int count = 0;
    while (true) {
      count++;     
      new Thread(new Runnable() {
        public void run() {
          try {
            Thread.sleep(10000000);
          } catch (InterruptedException e) {
          }
        }
      }).start();
      System.out.println("Thread #:" + count);
    }
  }
}


That's all about how to solve Java.lang.OutOfMemoryError: unable to create a new native thread in Java. As I said this is a rather uncommon error but it's good to know how to solve it. As a senior Java developer, you are bound to see such kind of performance-related problems in production and you should know how to deal with them. 


Other Java error and exception guides you may like
  • How to solve java.sql.BatchUpdateException: String or binary data would be truncated (guide)
  • How to solve java.lang.ClassNotFoundException: com.mysql.jdbc.Driver error? (hint)
  • How to fix 'javac' is not recognized as an internal or external command (solution)
  • How to avoid ConcurrentModificationException in Java? (tutorial)
  • How to connect to MySQL database in Java? (tutorial)
  • How to solve "could not create the Java virtual machine" error in Java? (solution)
  • java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory error (solution)
  • Common reasons of java.lang.ArrayIndexOutOfBoundsException in Java? (solution)
  • java.lang.ClassNotFoundException : org.Springframework.Web.Context.ContextLoaderListener (solution)
  • java.sql.SQLServerException: The index 58 is out of range - JDBC (solution)
  • How to fix Caused By: java.lang.NoClassDefFoundError: org/apache/log4j/Logger (solution)
  • How to fix "illegal start of expression" compile time error in Java? (tutorial)
  • How to fix "Error: Could not find or load main class" in Eclipse? (guide)
  • 10 common reasons of java.lang.NumberFormatException in Java? (tutorial)
  • How to solve "variable might not have initialized" compile time error in Java? (answer)
  • Fixing java.lang.unsupportedclassversionerror unsupported major.minor version 50.0 (solution)
  • How to solve java.lang.OutOfMemoryError: Java Heap Space in Eclipse, Tomcat? (solution)
  • Cause and solution of "class, interface, or enum expected" compiler error in Java? (fix)
  • java.sql.SQLException: No suitable driver found for 'jdbc:mysql://localhost:3306/mysql [Solution]
  • How to solve java.lang.classnotfoundexception oracle.jdbc.driver.oracledriver? (solution)

As per my experience, I strongly suggest experienced Java developers join advanced courses on Java profiling and performance tuning like these JVM and Performance monitoring courses which cover G1 garbage collectors.

4 comments:

  1. Thank you - your article was really useful in debugging Rundeck crashes.
    One small type "ps -eLF | grep java | we -l"

    should be

    .... | wc -l

    ReplyDelete
    Replies
    1. Thanks Vitaly, yes its word count command to count number of lines , wc -l is correct.

      Delete
  2. Hey, I am getting out of memory error and it got me thinking that ulimit -u is giving user processes while we are getting unable to create native thread error
    How can we relate ulimit -u(user processes limit) with the limit on thread count ?
    Also,ps -eLF | grep java is giving one single process with multiple threads

    ReplyDelete
    Replies
    1. How many threads your program is creating? there is hard limit on number of threads, mostly application doesn't need that many threads but with a bug like creating threads in loop means you can quickly exhaust that limit.

      Delete

Feel free to comment, ask questions if you have any doubt.