Monday, March 29, 2010

Hudson Shows "No buffer space available (maximum connections reached?): JVM_Bind"

These few days we kept bumping into this "No buffer space available" issue with Hudson CI. Initially I thought it was because of we updated Hudson to the latest version and there was a kind of bug in that latest version. I reverted to the latest stable supported version in the Hudson CI website (http://hudson-ci.org). The problem didn't appear for a while, only to reappear again after a while!

Here is the capture in my browser:

I purposely put the same message as in the captured image so that search engine will be able to find when people search using the error message.

java.net.SocketException: No buffer space available (maximum connections reached?): JVM_Bind
at java.net.PlainSocketImpl.socketBind(Native method)
at java.net.PlainSocketImpl.bind(Unknown Source)
at java.net.ServerSocket.(Unknown Source)
at hudson.TcpSlaveAgentListener.(TcpSlaveAgentListener.java:91)
at hudson.model.Hudson.(Hudson.java:598)
at hudson.WebAppMain$2.run(WebAppMain.java:224)

I initially tried to find any solutions to this issue using  Google search without much luck. There are few reference on this kind of error but it happened long time ago and seemed to be unrelated to the problem we encountered.

Initially I was thinking of the user limits on Windows, something like ulimit in UNIX/Linux systems.

After 2 cups of coffee, there come the enlightenment.

It's hard to believe that this problem was caused by this (http://support.microsoft.com/kb/196271). The Hudson CI application is currently running on Windows Server 2003 box. This version of Windows Server has something called  "maximum ephemeral port number".

Quoting from this Wikipedia entry:

"Ephemeral port is a transport protocol port for Internet Protocol (IP) communications allocated automatically from a predefined range by the TCP/IP Stack Software. It is typically used by the TCP, UDP or SCTP as port for the client end of a client-server communication when the application doesn't bind the socket to a specific port number, or by a server application to free up service's well known listening port and establish a service connection to the client host. The allocations are temporary and only valid for the duration of the connection. After completion of the communication session the ports become available for reuse, although most implementations simply increment the last used port number until the ephemeral port range is exhausted."

According to the Microsoft support website:

Windows Vista and Windows Server 2008 use the IANA suggested ephemeral ports range while the Windows Server 2003 is still using port range of 1250 to 5000.

Ephemeral ports are short-lived port, chosen ad-hoc to serve. In most implementations, they usually only add the port numbers by one until get exhausted on port numbers.

May be this is just my silly thought of me, but it seems like they implemented like this pseudo-code:

[sourcecode lang="python"]
next_port_num = 1250

def get_next_ephemeral_port_num():
result = next_port_num
next_port_num = next_port_num + 1
if next_port_num > 5000:
raise ValueError('no buffer space available')
return result
[/sourcecode]

I wasn't really sure that it is the cause of the problem, but I guess no kangaroo being harmed in the making of the game, so I tried it anyway.

The remedy was simply, run the Registry Editor in the Administrator privilege, open  the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and create new DWORD entry, "MaxUserPort", fill with 65534.

The result, after 2 days monitoring, Hudson is still running well.

NOTE:
  • It seems that the problem happens to this specific version of Windows (Windows Server 2003)
  • I haven't been able to reproduce this problem on another platforms

Monday, March 1, 2010

Apache Beehive Moved into Apache Attic

Our long time friend, Apache Beehive framework, which was  from BEA WebLogic Workshop Framework donated to Apache Software Foundation has reached its end of life. Personally I have encountered this framework a few times when dealing with production application developed using BEA WebLogic Workshop Framework 8.1.x (the BEA's proprietary IDE), and also the development that we did using BEA WebLogic Workshop Framework 9.2 (the Eclipse based IDE).

The framework consisted of NetUI, Control and Web Service Metadata.

The Beehive NetUI is a framework built on top of Struts 1.x (I think it was 1.1, but not from 1.3.x for sure). I remembered when using  the BEA WebLogic Workshop IDE, we can configure the page flow using diagrams. It was quite a mess after a while, when you have a lot of page flows though.

The Beehive Controls framework was a superb innovation that time. It simplifies a lot of things (taking into account that at those time it took a lot of extra jobs when you want to configure EJB 2.1 invocation yourself). I believe was based on EJBGen, and predates the Java annotations. The major setbacks with Controls frameworks were that it make it difficult for developers to decouple the application from the BEA WebLogic Workshop Framework, and even more difficult to decouple with the WebLogic Workshop IDE! The IDE did a lot of magic behind the scene (at least for WebLogic 8.1.x), such as generating Ant build files that uses weblogic.jar classes.

Most of the contribution from BEA Systems Inc had been absorbed into the Java Annotations. Well, I think the Beehive Control framework served its duty at its time, yet now it became less and less compelling to use for the reason that it's been obsolete. With almost every comparable framework providing the similar ease of use without having to couple with some jars; who wants to use this "maintenance mode" framework for new developments.

The Beehive WSM (web service metadata), has also served its duty during its time, as new web service standards and frameworks have rolled out one by one. It's been obsolete just as many other web service frameworks. it's cousin, the XMLBeans framework (http://xmlbeans.apache.org) seems to survive much longer. It has just released version 2.5.0 last December 14th, 2009. I personally enjoy using this component framework, generating Java classes and XML mapping. Though we have some other frameworks like Jibx, I personally feel that this XMLBeans is more sophisticated.

Commiters of the project eventually voted to close the Apache Beehive project (http://beehive.apache.org) due to lack of activity. It is moved to Apache Attic (http://attic.apache.org) - such an affectionate name :D

The project was lacking activities, and I agree that it should be shelved into the attic. I believe legacy applications that uses this component should not have any problems because the component is still there, the source code is still there, and more or less it is stable enough.

Recommendations for replacement components are shown in the page, Spring Web Flow/Struts 2 to replace Beehive NetUI, Spring Framework Core to replace Beehive Controls, and Axis2 to replace Beehive WSM.

Good bye, old friend!