How to Debug java.net.SocketException: NET-1.1:E=-19 (not enough free memory)

[alex.colleux]

This guide outlines a specific approach to debugging a NET error encountered during application development on a VEE Port running the LWIP IP stack.
While this is not a universal method, it provides a structured flow that can help identify and resolve the issue. Other debugging techniques may also be applicable.

Overview

The application we are working on is handling multiple network connections:

1. It provides a monitoring web dashboard hosted on the device (Device Monitoring dashboard):
   
   
   
   The monitoring data (CPU, Heap Usage, Threads) is retrieved through multiple REST endpoints exposed by the device.
2. The application also runs a background task to fetch weather data from https://open-meteo.com/. A REST client is used to fetch the weather data.

Environment

• STM32F7508-DK VEE Port 2.2.0
  • NET Pack 9.4.2
  • Network stack: LWIP 2.1.2
• Application networking libraries:
  • ej.library.iot:restserver:4.1.0
  • ej.library.eclasspath:httpclient:1.5.0

Issue Description

A java.net.SocketException: NET-1.1:E=-19 error occurs in the following conditions:

• A user is connected to the Device Monitoring dashboard
• The device is getting the weather data using REST APIs

The error trace shows that the error occurs in the RestClient class when the application tries to get data from https://open-meteo.com/.
It seems like handling too many connections at the same time is causing the error.

Find below the full error trace:

Exception in thread "Thread2" java.lang.RuntimeException: java.net.SocketException: NET-1.1:E=-19
     at java.lang.System.getStackTrace(Unknown Source)
     at java.lang.Throwable.fillInStackTrace(Throwable.java:82)
     at java.lang.Throwable.<init>(Throwable.java:51)
     at java.lang.RuntimeException.<init>(RuntimeException.java:26)
     at com.microej.example.iot.server.RestClient.getData(RestClient.java:93)
     at com.microej.example.iot.server.Agent.run(Agent.java:18)
     at java.lang.Thread.run(Thread.java:311)
     at java.lang.Thread.runWrapper(Thread.java:464)
     at java.lang.Thread.callWrapper(Thread.java:449)
Caused by: java.net.SocketException: NET-1.1:E=-19
     at java.lang.System.getStackTrace(Unknown Source)
     at java.lang.Throwable.fillInStackTrace(Throwable.java:82)
     at java.lang.Throwable.<init>(Throwable.java:37)
     at java.lang.Exception.<init>(Exception.java:18)
     at java.io.IOException.<init>(IOException.java:18)
     at java.net.SocketException.<init>(SocketException.java:36)
     at java.net.Socket.createImpl(Socket.java:153)
     at java.net.Socket.connect(Socket.java:225)
     at java.net.Socket.connect(Socket.java:191)
     at sun.net.NetworkClient.doConnect(NetworkClient.java:153)
     at sun.net.www.http.RestClient.openServer(RestClient.java:194)
     at sun.net.www.http.RestClient.openServer(RestClient.java:216)
     at sun.net.www.http.RestClient.<init>(RestClient.java:126)
     at sun.net.www.http.RestClient.New(RestClient.java:138)
     at sun.net.www.protocol.http.HttpURLConnection.getNewRestClient(HttpURLConnection.java:445)
     at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:427)
     at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:396)
     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:626)
     at ej.rest.web.AbstractResource.fill(AbstractResource.java:53)
     at ej.rest.web.Resty.fillResourceFromURL(Resty.java:390)
     at ej.rest.web.Resty.doGET(Resty.java:338)
     at ej.rest.web.Resty.text(Resty.java:228)
     at ej.rest.web.Resty.text(Resty.java:261)
     at com.microej.example.iot.server.RestClient.getData(RestClient.java:76)
     at com.microej.example.iot.server.Agent.run(Agent.java:18)
     at java.lang.Thread.run(Thread.java:311)
     at java.lang.Thread.runWrapper(Thread.java:464)

Issue Analysis

The NET Error Messages documentation provides information on the nature of the NET-1.1:E=-19 error:

-19 = Not enough free memory.

This error is thrown by the Network Abstraction Layer.

Let’s take a look at the implementation in the stm32f7508_freertos-bsp project:

• The LLNET_ERRORS.h header file provided by the NET pack defines the -19 error code:
  
  
• The J_ENOMEM macro is used in the following function:
  
  
  
  The role of this function is to map the error code of the underlying network stack (LWIP) to the NET pack errors.
• The ENOMEM macro is defined in the errno.h header file of the LWIP library:
  
  
• We can see that the map_to_java_exception() function is used at multiple places in the LLNET Abstraction Layers:
  
  

In conclusion, the NET-1.1:E=-19 error is thrown from one of the LLNET Abstraction Layers and is related to an Out Of Memory error in the LWIP library.

Let’s enable the LWIP debug mode to get more information on what is going on:

• The lwipopts.h configuration file allows to tune LWIP configuration and enable debug. Let’s enable the debug information involving memory management:
  
  
• Run the updated code on the device to get more debug information:

  memp_malloc: out of memory in pool NETCONN
  Exception in thread "Thread2" java.lang.RuntimeException: java.net.SocketException: NET-1.1:E=-19
       at java.lang.System.getStackTrace(Unknown Source)
       at java.lang.Throwable.fillInStackTrace(Throwable.java:82)
  

The error message memp_malloc: out of memory in pool NETCONN is now displayed.

This error means that LWIP has run out of memory in the NETCONN memory pool.
The stack is unable to allocate the necessary memory for new network connections.

Fix Proposal

To address this issue, the following actions can be considered:

1. Optimize network usage: ensure that the application is not creating and maintaining more network connections than necessary. Review the code and identify any areas where network usage can be optimized.

2. Identify and fix memory leaks: look for potential memory leaks in the application or the LLNET Abstraction Layers and address them accordingly.

3. Increase NETCONN memory pool size: try increasing the size of the NETCONN memory pool by adjusting the configuration parameters in theLWIP stack. This can be done by modifying the MEMP_NUM_NETCONN configuration option in the LWIP configuration file (usually lwipopts.h).

In our case, the application is already optimized, and no memory leaks are occurring.

Let’s increase the NETCONN memory pool size:

• In lwipopts.h increase MEMP_NUM_NETCONN:
  
  

An adjustment of the other LWIP configuration options can be necessary for the stack to work properly (MEM_SIZE, MEMP_NUM_PBUF, MEMP_NUM_NETBUF, …).

Run the updated code on the device and monitor the behavior.
If you still encounter the same or similar errors, you can gradually increase the other related configuration options as needed.

In our case, the java.net.SocketException: NET-1.1:E=-19 error is not occurring anymore after this change.

Alex for MicroEJ