Performance Tuning JCAPS - Part 2
It's been close to 2 months since my first Performance Tuning JCAPS post. Since then, we've noticed our servers running out of TCP connections under heavy load. Researching this problem, we learned that the JCAPS' JMS server makes heavy use of TCP in implementing Request/Reply queues. We approached Sun for guidance and they assured us that they've had success implementing high throughput applications using the JMS request reply solution... but how?
It appears a multi-step solution is needed.
So, what's my opinion on all this? Obviously I'm really happy with the results, but the steps leading to the solution is what has me scratching my head a little.
It appears a multi-step solution is needed.
- First, running out of TCP connections in a scenario like ours is a JCAPS bug addressed in an ESR (110348) and rolled into JCAPS 5.1.3 Update Release 3. We installed this update and noticed a marked improvement. Running my simple 75 user simulated test, transactions took 2521 ms to round trip (compared to the previous time of 20128ms). While this is great improvement, an average of 2.5 seconds is still really slow for this simple transaction.
- We reported our findings to Sun and they responded with 3 additional suggestions to try.
- The first suggestion was to turn on the socket pooling in the JMS server. This is accomplished by setting the JMS URL to
stcms://hostname:18007?com.stc.jms.socketpooling=true
in the eDesigner's Environment Explorer setting for this logical host. - The second suggestion was to change the registry setting for
TcpTimedWaitDelayto 30. This setting controls time a TCP resource is made available to the system again after the connection is closed. - the final suggestion was to change the registry setting for
MaxUserPortto 65534, making more TCP port numbers available.
- The first suggestion was to turn on the socket pooling in the JMS server. This is accomplished by setting the JMS URL to
So, what's my opinion on all this? Obviously I'm really happy with the results, but the steps leading to the solution is what has me scratching my head a little.
- ESRs are unnecessarily hard to implement. First you need to determine the prerequisites and corequisites to install with these patches. Sometimes these are listed in the ESR documentation, but often times they aren't and a call to Sun is needed to sort it all out.
Then once you have all the ESR you need, the repository, integration server, and eDesinger all need the patches before you recompile and redeploy the code. Needless to say, this is pretty time consuming. - I did not see any reference to this JMS server URL change in any documentation. It seems that the URL is the only place you can change this setting. There is not administrative console to make this change. Bummer, since it seems pretty important.
- As for the registry settings, environment changes are often the first thing forgotten and the last remembered when upgrading to a new server. I don't like making changes that affect my applications which aren't checked into some sort of source control.
Comments
- is JCAPS be useful for creating user interface for the capturing of data?
- how about using it for implementing workflow?
- is it easy to learn?
Lastly, for someone that like to get some hands on experience with JCAPS, what is the best source of information.
We were using version 5.1.3 and I don't feel it is easy to learn. If you're a java developer, JCAPS makes you do a bunch of things that don't feel natural (see my "2008 Resolutions" post from 12/31/2007). Also look at my post on the repository (More JCAPS Impressions) from November 2007.
In short, I didn't feel JCAPS made me a more productive programmer. I felt I was delivering less code of lower quality than I was without the tool.
Since I've written those posts JCAPS 6 has been released and it is very similar to the latest NetBeans release (which is free). If you can get away without the added JCAPS functionality, I'd try to use that.
Netbeans has built in support for BPEL capabilities if you want to use that to automate your workflow. It also has some tools to aid in creating user interfaces (web or client side).
I agree documentation for STCMS is scarce. At least, JMSJCA documentation comments that socket polling at the JCA adapter level is not neccessary for STCMS as it is implemented in STCMS itself , but it is disabled by default (But, of course, you don't know how to enable it).
I wonder what's the performance loss when you use the JCA option.
We never tried the JCA option.
I just wanted to send you a big thanks for documenting the socketpooling parameter. Finding your web page and reading about your experience gave me the final piece of the puzzle and helped me solve our terrible performance problem!
It seems that the note about changing the URL to activate the feature has been lost in the newer ESR relase notes. We had the right patch installed, but not the required URL modification.
I've sent a request to Oracle/Sun to include the instruction again so others don't have to go through this difficult troubleshooting.
Again, thanks!
/Anders