Loosely Coupled

It's been close to 2 months since my first Performance Tuning JCAPS post. Since then, we've noticed our servers running out of TCP connections under heavy load. Researching this problem, we learned that the JCAPS' JMS server makes heavy use of TCP in implementing Request/Reply queues . We approached Sun for guidance and they assured us that they've had success implementing high throughput applications using the JMS request reply solution... but how? It appears a multi-step solution is needed. First, running out of TCP connections in a scenario like ours is a JCAPS bug addressed in an ESR (110348) and rolled into JCAPS 5.1.3 Update Release 3 . We installed this update and noticed a marked improvement. Running my simple 75 user simulated test, transactions took 2521 ms to round trip (compared to the previous time of 20128ms). While this is great improvement, an average of 2.5 seconds is still really slow for this simple transaction. We reported our findings to Su...

Business transactions can vary widely in their definitions, but they can basically be classified into two types - those where I need an immediate response so I can continue with my work and those where I'm going to tell you what to do and trust that you'll get it done while I move on to something else. These classifications are otherwise known respectively as synchronous and asynchronous transactions. When designing an architecture for solving business problems, it's helpful to figure out which type of problem you're solving and then pick an appropriate technology to support it. For example, HTTP might be a good choice for synchronous applications since it is request/reply by nature. JMS might be a better fit for asynchronous solutions since these structures are typically one way. Blurring the lines is the JCAPS offering of a request/reply mechanism for JMS messaging - called appropriately enough, JMS RequestReply . A colleague called this mechanism "synchronou...

We're getting close to (finally) deploying our JCAPS rewrite to production. Our final task is to load test and "tune" the application - making sure it will handle the expected production transaction load. Prior to executing the load test, we added logging statements to report on the amount of processing time spent in each JCD (or processing unit for those non-JCAPpers out there). Using this info, we can determine the slow running processes and optimize them. For this optimization we could follow a traditional approach and profile the code, or we could take an easier route and configure JCAPS to throw more resources at the bottlenecks. This second option is done in the JCAPS connectivity map by increasing the maximum number of threads allocated to run the process. Clicking the input line to the JCD to open the properties, you can set the "Server session pool size" in the Advanced tab (NOTE: this setting is only for JCDs listening to JMS queues or topics). I...

I wish I was on board at the very beginning of the project I'm working on. Maybe it would shed some light on the direction the client is currently taking. This project is a rewrite of an existing, poorly performing, data processing system. The rewrite is being done in two parts. Phase 1 replaces the front half of the system - responsible for receiving messages, routing the transactions to the appropriate business logic, then finally, transmitting a response. Analysis (performed before I got here) identified this logic as the application bottleneck. This portion of the application was rewritten in JCAPS and it's currently being tested (I haven't seen any performance data for the rewrite yet). The second phase of the project replaces the business logic, currently written in C++, with BPEL . Let me say that again... the second phase of this project replaces compiled C++ with interpreted XML . For speed. Anyone else confused? Now maybe implementing the business...

I started this podcast project so I could learn some things I haven't had time to experiment with in my "real" job. Things like playing more with Ruby, trying out Behavior-Driven Development, and experimenting with Agile practices (plus I really wanted a better podcatcher program and I needed something to do over the winter). While I haven't produced much real code yet, I've created some simple prototypes and wanted to comment on some things I've been looking at. Reading an RSS feed My first prototype was a very simple Ruby program to read and parse an RSS feed. I had found some code here using the standard Ruby RSS Parser so that's where I started. At first I thought it was broken, but after about a minute and a half it returned. This wasn't going to cut it, so I started to look for alternatives and quickly found the feed-normalizer gem. Some quick coding and the total time to access the RSS feed was reduced to about 10 seconds. Not bad...

Migrating my application from Hibernate to Rails has taken longer than I expected. An important client has expressed performance concerns with their release, so I spent the last week looking for ways to tweak the current Hibernate implementation. It has been very frustrating. I don't like to cast blame or plead ignorance, but prior to this exercise, my experience with Hibernate has been limited. The module containing our Hibernate calls was authored by someone else and I've been mostly a client of this module. It wasn't until I looked at our query log in detail did I notice how poorly this module was written and how confusing Hibernate is. When seeing the large amount of queries in the log (often for data not needed in the current view), my first thought was, "we must have a problem in our models... something is causing Hibernate to eagerly fetch related data we have no immediate need for." I went through our models and, indeed, I did find some simple changes ...

The database cleanup is finally complete! Now I need to rewrite our web app's data access code. When we first created the database layer, we used the Hibernate framework. Since then Hibernate3 has been released and an upgrade is scheduled as part of this work. We did have some pain points with Hibernate, however. For instance, when I started evaluating our DB code, I noticed some DB queries that we don't explicitly create. Whether these were the result of a poor object-relational model, lazy instantiation, or some other Hibernate functionality is unknown at this time. I've found Hibernate (or at least how we're using Hibernate) complicated to debug. I'm going to take some time and look at Ruby and Rails as an alternative. I've mentioned before that I've wanted to learn Ruby for a while and this seems like a good excuse. There is a Ruby library for Java called JRuby . The plan is to learn the JRuby Rails extensions and find out the effort to incor...

From my past posts, you may have guessed that I'm in the middle of optimizing a web application that accesses a database. Our studies have shown that a combination of missing indexes, unnormalized data, and inefficient queries are the root cause for this application's poor performance. The optimal solution involves reorganizing the database schema and rewriting our database access layer. Side Note: The books SQL Tuning by Dan Tow , Pro MySQL by Michael Kruckenberg and Jay Pipes , and Hibernate in Action by Christian Bauer and Gavin King have been instrumental in helping to identify solutions to these problems. During a meeting to discuss these changes, another suggestion was made to help performance. There are certain instances where we could save few trips to the database (per request) by reusing session information that is unlikely to change (whether this data should actually be in the session or not is a subject for another post). This enhancement requires ...

I just upgraded my Eclipse IDE and installed the latest Test and Performance Tools Platform (TPTP) release. The only tricky part of the installation was installing the Agent Controller (available from from the TPTP download site ). This is a separate install from the TPTP plugin and must be placed on your PATH. TPTP application profiling is easy. Simply start your application in profiling mode, then start the profiler before you access application functionality you want to benchmark. I'm currently using the tool to identify memory and execution time bottlenecks. The TPTP utility makes it easy to sort on key statistics (like the memory footprint for an object class, or the total time spent executing various methods). I find the method invocation information especially useful. Not only does the tool provide information on which calls are taking the longest, but it also provides details for who called the method and what supporting methods are called (along with the time sp...