Explore IT

Thoughts about a different StackOverflow reputation system

2011-11-19T06:04:00.006-05:00

I love StackOverflow. It's my "go to" place when I run into a tough programming problem.

Reputation on StackOverflow can be a bit intimidating, however. Even when you have a few thousand reputation points (placing you in the top 20% of all StackOverflow users) - the top users have 100x that many points! It can make you feel pretty insignificant. "Catching up" to the leaders doesn't feel like a realistic option.

This week, I listened to a podcast about Cortes' conquest of the Aztec empire. During the conversation, the Aztecs were described as a civilization founded on war. Warriors were celebrated. They earned their own "reputation" - in the form of feathers and other ornaments - by conquering neighboring peoples. What I found most interesting, though, was once a warrior earned a badge for conquests on a particular village, they couldn't earn more rep by sacking the same towns again and again. This forced their warriors to become fighters with diverse skills and experiences as they fought new or stronger opponents.

I thought this example might provide a good alternative to the current StackOverflow reputation model where reputation point gains happen at a constant rate - regardless of the type of answers you provide.

Let's say that a user has gained 5K reputation points for answering Java questions. The top user reputation for questions with the Java tag has upwards of 20K reputation. Is the top user, really 4x more reputable as a Java expert? I wouldn't think so. It could be that the top rep user has just been a StackOverflow user longer. Or maybe they have had the good fortune to provide an answer to a question with a lot of views (and up votes).

I guess my point is, once you've gained a certain amount of reputation answering a particular class of questions, you could be considered an expert in that particular field of study. You've exhibited you know what you're talking about. What, really, does additional reputation provide past this point?

I know earning points is a large part of the appeal for many users so instead of turning off reputation gains outright after reaching a certain plateau, couldn't the rate of growth just slow down a bit? Instead of earning 10 rep points for all up votes, maybe the point rate would decrease by 1 for each 1000 points earned for answers with a particular tag, bottoming out at 1 point per up vote . I don't know - I'm just spit-balling here.

Another nice side effect of this change would be that users with a diverse set of skills, could still earn reputation quickly. Once they've hit the expert experience level for answering Java questions, for instance, they could still earn full reputation points for answering SQL questions, or AJAX. Users with a deep understanding in many, diverse topic areas would rise to the top. Like the Aztec warriors - to gain the most reputation - users would need to expand their skill sets outside of their comfort zone.

A few quick notes

2009-10-09T15:25:00.004-04:00

It's been a quiet few months. I haven't worked on too many things I've felt like writing about.

That's not to say I haven't been busy. One thing I've been reading more about building great teams and delivering more to our customers. I'm just starting to read Creating Magic: 10 Common Sense Leadership Strategies from a Life at Disney. Obvious from the title, this book outlines the approach Disney teaches all of it's Cast Members (employees) on how to be great leaders and provide their visitors with great experiences. I've been wanting to read this for some time and I'll be sure to write more as I work my way through it.

Some related blog posts I've enjoyed recently are:

The Duct Tape Programmer from Joel on Software where he reviews a chapter of the book Coders at Work. The post basically talks about doing the simplest thing possible to get something out the door and into people's hands. Amen to that!
I've also been enjoying the "There is no magic, there is only awesome" series (part 1 & part 2 & part 3) over at the { buckblogs :here }. The series is about being passionate, opinionated, and continuously pushing yourself to be more complete. It's a good read.

Professionalism

2009-07-31T13:21:00.009-04:00

I was inspired by this talk from RailsConf given by Robert "Uncle Bob" Martin. It's a really entertaining talk and you should check it out if you haven't already.

During the talk, the topic of professionalism came up and what it means to be a "professional software developer" (maybe 2/3 of the way through). Professionalism in software development is a topic that's not well defined and certainly has different meanings to different people.

Can you be considered a pro, for instance, if you don't provide a set of automated tests for your code? What if you don't use source control? Do you feel the need to always create a custom build file (make, ant, or maven script) so you can build independent of an IDE? Do your "professional" standards change depending on if you're creating "prototypes" rather than designing code for production?

I know which things I strive to do on every project I work on. But the standards for the teams I work with vary wildly. And this fact alone leads me to question why. Can't we do better?

Doesn't every application need some level of testing? Shouldn't these tests be automated and built into the fabric of every piece of code? Personally, I don't think the benefits of having an automated test suite can be questioned anymore. Same with source control.

Additionally, I feel every project should replace the build file generated by an IDE with one of their own for this simple reason - maintenance. Have you ever looked at the Ant scripts generated by Netbeans? It's enormous - and it has to be to support the wide range of functionality Netbeans allows Java projects to do, like generating code for WS clients for instance. At a minimum, developers should take out all the extraneous (unused) targets and move references to IDE-specific libraries needed to build the project under their own control (treat them just like application dependent libraries). Without this measure, you're locked into a specific IDE and often a specific version of the IDE. By taking this simple step, you can upgrade your development tools without fear that the system won't build.

Excuses not to do the things listed above often fall under the veil of, "We're developing a prototype. We need to do it rapidly and we can't slow down to take on the overhead of automated testing or configuration management". My response is, why not? Isn't this code going to be "delivered" anyway - if not to a client than just internally for evaluation. And really, when is the last time a really cool prototype hasn't been immediately folded into the production code. The overhead to doing these tasks is minimal compared to the time spent repeatedly doing them manually.

All I'm really saying is there are a basic set of things - source control, testing, and repeatable builds - that are common to any software development effort regardless of scope or technology. I'm tired of having the conversation about not only how, but if we're gong to do these most basic things on every new project I join. As a profession, we're better than this (at least we should be). At this point, it's wasteful to have these discussions time and time again. I don't think we can consider ourselves true professionals until we agree to deliver these most basic pieces of functionality for every piece of software we deliver.

More stuff to get excited about

2009-07-08T10:23:00.009-04:00

Chrome OS
I'm excited about Google's announcement to release what sounds like an ultra lightweight OS to basically surf the Internet. I'm hoping this device will allow me (and others) to breathe new life into old, underpowered laptops, allowing them to essentially work as netbooks.

This could also open up new and inexpensive opportunities for schools to recycle or prolong the life of expensive computing equipment and expand horizons of their students. I've been watching the One Laptop per Child (OLPC) movement for some time and I see this as another avenue toward their goal.

I don't share the optimism that this OS will somehow replace Windows. Sometimes you need extra horsepower and functionality. I edit a lot of photos and create some simple video, but the web tools I've tried to do either task can't hold a candle to the four-year-old version of Photoshop Elements I own. I'm not sure CPU intensive apps like this will port very well to the web. It seems the economies of scale afforded by cloud computing may break down when everyone needs 100% of the CPU. (Though I hope I'm wrong)

First person UIs revisited
I just watched this video of something called Sixth Sense. Basically it's like Minority Report meets the Microsoft Surface Computer meets Way Cool and Inspector Gadget. Trust me. Watch the video....

It was cool when the guy was using a wall as the surface computer or got real time product information and reviews at the store. It went to way cool when he used his fingers to frame and take a picture. It went to Inspector Gadget cool when he used his hand as a cell phone and "drew" that watch on his wrist. The tag cloud on his buddy was also pretty neat. I can't wait to see how this technology develops.

Interesting Tidbits

2009-06-26T14:31:00.008-04:00

It's been a while since I had a post to talk about some of the new(er) technology I'm just hearing about. Here are three things that got me excited recently.

First person user interfaces
I came across this article on First person UIs. Basically it works like this: you have a location aware device like a smart phone with GPS and a camera. You're walking around and you see a historical building, business, or other point of interest (POI). You point the device's camera at the object and WHAMO - you instantly get a bunch on info about the object. This can include the name of the POI, how far it is away, a phone number to call, a history, and a bunch of other interesting tidbits. It will be like having a virtual tour guide in your pocket. This is very cool! This concept is also called "augmented reality".

Google App Engine, Big Table, and more on the Cloud
I recently watched this talk on Google App Engine. I've been watching Amazon's EC2 for some time and it was nice to get an overview of Google's cloud service.

Martin Fowler's overview of BigTable was also very helpful. I didn't really understand this data structure beforehand, but his description of it as a nested hash table really gave me a better idea of what it is - and it's usefulness. Lately I've been creating some in memory representations of data structures like this. It's nice to know there's a ready made pattern out there for persisting it should I need to.

Erlang & CouchDB
This week's StackOverflow podcast included an interview with Damien Katz. Damien created CouchDB and wrote it in Erlang. Now, I've heard of both of these items (separately) before, but I knew virtually nothing substantial about these technologies before the podcast.

I'll probably get this all wrong, but Erlang is basically a language for writing distributed, multi-threaded systems. CouchDB uses Erlang to allow a distributed way for users to store "documents" - where a document is some piece of data with a varying set of attributes. From what I can gather, it's sort of organized like BigTable. Two things really got me to thinking during this interview.

Erlang seems like a natural fit for building multi-threaded and distributed systems. It's becoming more important for application developers to learn to code these types of systems as even entry-level computers have multiple cores now. A language like Erlang seems like it may be a good fit to help take advantage of these additional resources.
Toward the end of the podcast, they discussed a StackOverflow question dealing with transactions and locks in CouchDB. Basically, these two mechanisms don't exist in CouchDB - they don't have to. It's automatically handled by the framework. This is so cool! Removing this complexity from my code and allows me (as an application developer) to focus solely on the business logic. That very nice! Thank you very much!

PS
Check out this great presentation from Damien about his passion for what he does. It's pretty powerful and inspiring!

Maven - Why not?

2009-06-16T11:38:00.006-04:00

I guess I still get a little surprised when I see code for a Java project and there's not an automated Maven build. These projects often have an Ant script (many times the default one generated by Netbeans or Eclipse), but seldom a Maven POM file. I wonder why? Especially since Maven is so easy to use and really adds value over a vanilla Ant build. Maven takes an Ant build to a higher level of abstraction to where it's easier to manage and change.

What I think I like most, is the idea of the Maven repository. It's a nice way to store and share application dependencies. In a typical Ant project, dependent JARs will be stored in source control along side the source code (typically in a lib directory). This can make initial checkouts of these projects S-L-O-W. Compounding the problem is when a set of projects use a framework like Spring. Each project will store it's own set of Spring jars with the source, forcing developers to download the same set of jars multiple times for each project.

Maven alleviates this problem somewhat with the notion of a JAR repository (actually a shared directory). For something like Spring, the set of JARs is downloaded once, then shared between all projects needing the jar. The jar repository eliminates the need to store the jars with the source code for the project and makes project downloads quicker while saving disk space.

The list of dependencies each project needs is stored in the projects Maven build file (or POM). The POM lists a projects' JAR dependencies as well as the version of the required JAR. This creates a very detailed snapshot of all the project dependencies for a given release.

Once the project is built for release, Maven allows the opportunity to export the release out to the JAR repository for others to grab. This is a nice way to store and share multiple distributions of your projects - especially internal code libraries which are in turn used by multiple projects.

Maven has it's own plug-in architecture so it's easy to extend. One Maven plug-in I want to mention is Appassember. For projects that include an executable "main" class, Appassembler will copy all the run time libraries to a distribution folder and also create the scripts to start the programs from the command line. This is a great feature and removes some of the tedious tasks in creating releases like these.

I really like Maven and recommend it to everyone.

Spring Revisited

2009-06-15T14:48:00.007-04:00

Recently I retrofitted some code I was working on to utilize Spring. It's been some time since I'd used it and I'd forgotten how much I enjoy coding in this framework. It really helps me write better, cleaner code.

As developers, we always strive to create plug-and-play architectures - you know, code to an interface, achieve easily swappable implementations, etc. That in itself really isn't that hard to do, but Spring really shines in enforcing this idea. My original code (for Jena) needed to grab some Resource references from the model and looked something like this...

{method declaration}
OntologyModel model = {get model reference}
URI resourceUri = new URI("http://www.myCompany.com/MC#myResource1");
Reoruce myResource = model.getResource(resourceUri);
{additional business logic}

The equivalent Spring configuration replacement for this code looks like this...

<bean id="myResource" class="com.myCompany.Resource" factory-bean="model" factory-method="getResource">
<constructor-arg>
    <bean class="java.net.URI">
        <constructor-arg value="http://www.myCompany.com/MC#myResource1"/>
    </bean>
</constructor-arg>
</bean>

The major difference is in the in the {get model reference} statement. In the Java version, I'd normally gain access to the model using a Service Locator or a Singleton. Whatever design choice was made, would find its way into the implementation as "Model.something" or "(OntologyModel)Locator.get("model")".

With Spring, the framework wants you to inject the model dependency into the class. This forces you to pull up the method local OntologyModel reference to a higher level (either as a method parameter or a member variable).

This refactoring has a few pleasant side effects. First of all, it makes the business logic in the method itself a little cleaner - gone is the code clutter to figure out how to get the reference or where it's coming from. The method is reduced to it's essential business logic.

Secondly, the fact that this method or class depends on and utilizes OntologyModel is made explicit through it's interface. This is a good thing (really). In the past, I've argued that one job of the class is to abstract away (or encapsulate) this dependency or complexity, but I've sort of changed my thinking a little on this. Sure, you might not want to expose whether your code is using a JDBC or a JMS connection or gets its value over HTTP, but what you want to change in this case is your level of abstraction. Wrap the HTTP/JMS/JDBC connection in another class interface and expose that instead. It's exactly this functionality which make Spring so appealing to me because now it's trivial to change the Spring configuration between alternative HTTP/JMS/JDBC implementations or stub these out completely for unit testing. With Spring, not only is this functionality "baked in", but the framework encourages this behavior.

Finally, because the behavior of the code can be changed outside the compiler via classpath configuration files, the same jar can used for unit test, QA, and production. Not having to rebuild your code for various environments ensures that your build configuration doesn't introduce any unintentional side effects into your code. I was on a project once where different files were included into a war depending on which region it was destined to run. When incompatibilities were uncovered between the code running in each region, you needed to check the environment of the physical system, any external configuration file overrides, and the code itself. If the same code is deployed to each region, you eliminate one of these things you need to check.

I'm happy to be back using Spring because it helps achieve and enforce the goals I set for myself and my team in developing quality software. My code is cleaner, more modularized, and easier to extend.

Jena Lessons - More On the Reasoner

2009-05-21T08:09:00.008-04:00

Back when I started working with Jena, my first "ontology task" was to iterate through a subset of objects in the model, perform some operations to slightly modify these objects, and then store the objects back into the model. Sounds simple right?

In a traditional database application, this program would consist of a SELECT statement, followed by some business logic, and conclude with a SQL UPDATE. I've probably written code like this hundreds of times. It's simple and runs quickly.

On paper, the ontology task seems even easier. Once opening a Jena ontology model, you can query for objects (Resources) in any number of ways (including a SQL-like SPARQL syntax). From there you simply create or modify properties on the resource. Since the ontology model is in memory, the changes happen real time (no need for an UPDATE). And while you could persist the changes, there really isn't any need to.

While this seems quite easy, I noticed severe performance problems running this program on even moderately sized data sets. Not only was the performance bad, but the application was using HUGE amounts of memory. Much more than I would expect for models of this size.

I immediately started to search for the cause of these problems and all signs pointed to the Jena Reasoner. If you remember from my previous post, the Reasoner is responsible for inferencing - essentially creating relationships between objects not explicitly defined in the data. You see, for each update to the model, the Reasoner ran (traversing much of the model) to discover any new relationships that it could find. As the model became larger and more data changed, the Reasoner process took up more and more of the processing time and system resources.

What I wanted to do was simply shut off the Reasoner temporarily while I did my batch update, then reactivate it once I was done. The Jena API provides no support to "turn off" the Reasoner once the model is created. I knew that there must be some way to do this and I asked some questions.

Eventually someone suggested that I create a new model with no Reasoner and load my existing "Reasoned" model into it. I could then batch all my changes into this simplified model with no Reasoner performance hit. The only trick is to run model.rebind() once the batch changes are complete in order to refresh the main model.

Sidebar: Under the covers, I believe the 2 models are actually sharing the same memory space, but the changes to the "basic model" are happening "'below the radar" of the main model's Reasoner. The rebind() method tells the main model to rescan its cache to get the updates.

All in all, this process works like a champ and I'd recommend it to anyone having performance problems while batch changing a Jena ontology model.

Mentoring - Reading List

2009-04-02T13:35:00.004-04:00

Since outlining a brief list of online technical resources for my rookie coder, I thought it'd be a good idea to also supply a brief reading list of technical books that could have a place on any programmers shelf. For this list, I wanted to focus on books that provide transferable skills - suggestions of different techniques and ways of thinking than someone might otherwise be exposed to. This knowledge can be able leveraged regardless of specific technologies or platforms.

Here's the list:

Core J2EE Patterns – I have an earlier version of this book and it’s a solid J2EE reference. The real benefit in this book is the set of J2EE architectural patterns (which builds on the Gang of Four [GoF] core patterns) and would also apply to server side development in .Net. Some of the info on JSP or EJB may be a little dated with recent additions to the Java language, but still a great reference. Patterns of Enterprise Application Architecture is another great book with a similar theme.
Ruby Pickaxe Book (called so because of the image on the cover). I own the previous version of this other book about Rails and written by the same guys. The first book will teach you Ruby, and the second will teach you the Rails framework. I find Ruby and Rails a joy to work in. My only complaint is that I haven’t gotten to use them more. I was pleasantly surprised how much more I could get done with this simple language and framework. At a minimum it’s a great tool for prototyping and learning language features like dynamic programming, active record, closures, and convention over configuration. Knowing these concepts will help to become a better programmer in any language.
Code Complete. I just started reading this myself and like it so far. It’s highly recommended by the people I respect in the industry, which is why I gave it a shot. Of all of the books here, this may be the best fit for a new programming professional. It provides the foundational knowledge and best practices to create solid software.
RESTful Web Services – this book changed the way I approach web service development. I think it’s a much better way to build services than the traditional WSDL approach. It's simpler and it reduces a lot of the overhead needed in the traditional approach.
SQL Tuning – a fantastic book on SQL tuning. This book was indispensable for me when I needed to refactor a database and some queries for a web application I was working on. I highly recommend although it might not apply directly to anything a new programmer might immediately work on. Still, a great book and I’d definitely put it on my reading list for the future.
Pragmatic Programmer. From the same guys who wrote the Ruby books. This is another book, like Code Complete, teaching principles to write better software. Also like Code Complete, it’s another great reference for someone new to the field.
I also really like Refactoring and Lean Software Development, but I recommend putting these on the “To Read” list until after reading some of the other titles and new programmers gain a little more experience. I actually think the topics in these two books might be some of the most valuable in the long run, but the foundation of the other books (and some experience) is needed to get the full value out of these.

Jena Lessons - The Reasoner

2009-03-30T17:30:00.002-04:00

Earlier this year I started to work with ontology, specifically utilizing Jena to process and manipulate an ontology model. I thought it would be good to share some of what I'm learning as I gain experience programming in this environment.

I thought I'd start with a quick discussion of the Reasoner. First, let me give my 10-cent understanding of ontology and where the Reasoner fits in to all of this.

For me, it's easiest to think of an ontology as an object store. There are classes, and the classes can be related to each other in a number of ways. In an object oriented sense, these relationships can be associations, aggregations, or through inheritance.

Both the class definitions and the instances of these classes are stored in the ontology. Like a regular Java class, an ontology class definition might include a supertype and some distinguishing properties. When an instance of the class is created, the instance gets a unique name (a URI), at least 1 statement to identify it's class, and any number of properties belonging to the unique instance. This "hard data" belonging to the class instance is often referred to as "facts".

Additional knowledge can be "inferred" from the model by combining the instance information (facts) with the data contained in the class relationship. For example, assume we have the following hierarchy

Animal (class)

Mammal (class)

Bear (class)

Yogi (instance)

The model only really "knows" that Yogi is a Bear, but the Reasoner can "infer" with some certainty that Yogi is also a Mammal and an Animal based on the class relationship. Note that there is no "hard data" in the model asserting these facts.

It's the Reasoner that creates this inference information. The example above was intentionally trivial. It's not hard to imagine how a more sophisticated model can result in more complex inferences and a much deeper understanding of the data and the relationships contained within.

Powerful stuff.

Mentoring - Online Technical Resources

2009-03-25T08:56:00.006-04:00

As I stated in my last mentoring post, one thing new professionals need to learn is how to stay technically current and continue to grow their skills in the absence of formal training. Here's a list of some online technical resources and tools I think any new professional would find helpful.

Martin Fowler's Bliki. Martin Fowler always has interesting things to say about object oriented design, agile development, and software development in general. He is a thought leader and always says smart things.
InfoQ This site has a ton of info about a ton of software development topics. What I like most about this site is that there are a lot of videos. I learn better this way. It’s nice to watch someone explain a topic and see the examples. Here’s a Martin Fowler presentation on InfoQ about SOA and ESB.
StackOverflow. This is basically a Q&A site for software developers. It’s fairly new and high quality. If you really get stumped on how to do something or want a wider range of opinions, this is the place to go. People answer your questions then vote up the best answers.
Delicious, not a technical resource as much as a tool to save and organize the things you find online. When you find something helpful on the web, save it here so you can find it again later. It’s also somewhat helpful to search other people’s bookmarks for common tags if you’re not having much luck with a Google keyword search. Here’s a link to my bookmarks.

A suggested "book" reading list is coming soon....

Early Adoption?

2009-03-20T15:45:00.002-04:00

When is the right time to adopt some new technology - the shiny "silver bullet" solution that's going to meet all your current and future business needs, make your team ultra responsive to your users, and totally allow you to kick butt?

Before screaming "get it now! get it now!", consider the following.

This technology is new, cutting edge stuff. Not many people are doing it. Where can you find help and support when you inevitably hit some road blocks? Will you be forced to troll message boards and user groups for solutions? Will you be using the technology in a way that hasn't really been anticipated, pushing you out even further from the mainstream and making it even harder to get good support?
Is this a technology you're buying? If so, how's the company's technical support and reputation? What success stories can they point to?
Can you "make do" with some existing (better supported, more widely implemented) solution until this new tech gains wider adoption? Will the ramp up time to figure out this new technology push out your delivery date? Are your customers OK with that?

For me, I'm taking on this new technology risk only if it's my only option. It would be my last resort. While it's kind of cool to be working on cutting edge stuff, there are just too many risks involved. Too many risks to the business. Risks that, in the current economic climate, companies are less willing to tolerate.

Here are some things I'd like to see before implementing a new technology in my business.

I want to hear success stories. Instances where capabilities of this new technology greatly enhanced some solution.
I want to know that the new tech will make me and my team more productive. For me, productivity is the be all and end all. It allows me to respond to business changes more rapidly, and therefore makes the business more productive and responsive.
I like to see at least one book on Amazon. There's an additional legitimacy and permanence to seeing some topic in print rather than only mentioned in blog posts or online documentation.

Everyone is different and I wouldn't mind hearing other views on this topic.

Mentoring - Intro

2009-03-06T03:27:00.000-05:00

Recently, I was asked to act as a mentor for a junior level developer. It was a little awkward in as much as I hardly know this person - we've only briefly met, we're not working on the same team, and not even for the same company. I'm not even sure what to topics to mentor on.

Remembering back, one of the hardest things for me when I transitioned from a college setting to professional was learning how to stay current and continue to push my skill set forward. Until graduation, you're continuously being fed information and guided on what to learn. After, you need to learn to forage for yourself. Learning how to do this and where to look is where I thought we might begin.

Like any new relationship, I thought the best place to start was with a brief introduction, then see where I might be able to best help this new professional. Here's an excerpt of my introductory email:

(Introductory text..)

Just to give you some insight into my background, I've been professionally developing software for 12 years. Most of that time has been spent with Java/J2EE and web development and I have worked with many Java frameworks like Spring and Hibernate. I've also done some database work (both writing SQL and creating schema). Quality is one of my highest priorities. I encourage things like test-driven development, development wikis, automated builds, and the proper use of source control. As I learn more about what you're doing, I'm sure we'll find more instances where my experience will overlap with your work. If not, I read a lot of technical blogs and books and hopefully I'll be able to point you in the direction of some great resources.

Like I mentioned, the purpose of this first email was to just break the ice and offer a few topic areas where I might be able to offer guidance. We'll see where it goes....

Technical Debt

2009-03-04T13:37:00.009-05:00

This topic seems to be pretty popular lately. First I saw Martin Fowler's comment on technical debt last week. Then Jeff Atwood brought up the topic in this week's StackOverflow podcast (@53:30).

Basically, technical debt is the price you pay down the road for a "less than optimal" design choice today. I particularly liked Atwood's metaphor that you know you've incurred technical debt when you give a long sigh every time you need to modify some part of the code base. You know that the changes are going to be painful (and expensive) to make. It's time to refactor.

A few comments based on my experience.

I've been noticing less technical debt in my own code since starting to follow a test driven development approach more religiously. Maybe there's something about this approach which makes your code more "real than theoretical" from the start. It helps you make better decisions without thinking about it too much. Maybe it's because TDD also keeps you from incurring any unnecessary technical debt from adding unneeded complexity or overbuilding. Or maybe it's because you're constantly refactoring as you add tests for new features and functionality that the technical debt seems less.

Maybe lost in this discussion is opposite problem, the technical debt incurred by overbuilding early. I know I've been involved in a few projects where we knew we would need "X" functionality in a later release. Thinking ahead, we knew the best way to deliver "X" was to use technology "A". "A" would be more time consuming and expensive upfront, but there would be less technical debt to deal with in the future. We were planning ahead and we'd be ready! Problem was, dealing with the extra complexity of "A" put us over budget and behind schedule on the current version. The entire project was canceled before ever getting to the later version and all the technical debt we had hoped to save was for not.

Value Teamwork

2009-02-06T15:50:00.008-05:00

Nova's episode this past week was titled "The Spy Factory". It wasn't quite what I expected, but it was fascinating.

Basically, it was about the NSA and how it was tracking Al Qaeda members and their movements (in the US), long before the September 11th attacks. It also touched on the CIA and how they also tracked Al Qaeda movements into the US. The CIA approached the NSA to share information (since they couldn't legally conduct a domestic investigation on these potential terrorists) but the NSA refused to share any information. Once Al Qaeda activity came stateside, it seems neither agency alerted the FBI. It all seems so foolish. All of these agencies are supposed to work together for our common safety. Was their unwillingness to work together driven by ego, politics, the desire to be national heroes, any or all of the above?

Obviously, this sort of behavior is not limited to government offices. I've seen this same behavior on teams and in organizations I worked in and with. It's a shame. The teams are not as productive as they should be. The organization suffers and - worst of all - the people we are supposed to be serving suffer.

Again and again, I question the motives behind this behavior. What have we been trained or taught that routinely leads to the choice to chase individual recognition rather than group success for the greater good?

Individual honors and being an "all star" routinely take the place of learning to work together. The emphasis to stand out and get recognized is viewed as the way to get ahead in life.

The mirror of this point of view is a communal based system, where pay and status are determined by longevity, rather than performance. I know some people who work in systems like this and it seems at some point, these people lose the desire to excel - the desire to over deliver. I don't think this is the answer either.

Where is the middle ground? How can team successes be rewarded without sacrificing the ingenuity and innovation spurred on and championed by individuals?

I Don't Know

2009-02-03T15:28:00.004-05:00

Why are these three words so hard for some people to mutter? The fear of being caught without an answer is so scary, that some people will say anything to avoid having no answer at all.

It's a big problem.

Consider this: you ask a question and you get an answer. You accept the answer as true because you think, "why would someone mislead me?". And you're right, they're not intentionally misleading you, but they have an intrinsic need to give you an answer - any answer - so that they don't "lose face" or something.

I just can't wrap my head around this behavior because most likely, you'll take some course of action based on the answer to your question. But when you start to notice things - things that don't make any sense and couldn't possibly happen if the information you got was correct -you'll spend a lot of effort reevaluating your beliefs and doing your own research. Eventually (and after some frustration), you'll find something that contradicts the initial information you got (in the form of your original answered question).

Oh boy. Now what do you do? Which answer is right? You dig deeper to find a third source to corroborate either answer.

Now the whole point of asking the question in the first place was to try and avoid all this digging but in fact, you're probably now dong more research than you would've done if you've never gotten that initial answer in the first place.

Eventually, you finally discover that the first person was full of BS. What's worse, when you try and educate them with the right information they'll sometimes start to argue or rationalize their ignorance. They've now lost all credibility in your eyes. You can never again accept what they say at face value and you'll always question the things they say. Everyone loses.

Is this behavior a product of our education and training, where refusing to answer a question you don't know is considered "wrong"? I don't know, but I wish I could stop it.

It's OK to say "I don't know". No lost credibility. No need to raise the BS filter in every future conversation.

New Surroundings

2009-01-24T07:16:00.001-05:00

I would really like to post more often, but frankly, I've been too busy. The New Year brought a change of job scenery and I've been really busy getting up to speed and learning a ton of new things.

That's not to say I haven't been writing at all. As I come up to speed here, I'm taking the opportunity to capture what I learn in a development wiki. My new team has a wiki instance in place, but it's been lightly used to this point. I'm a huge fan of using wikis for development. I like to leverage the wiki as a focal point to capture technical ideas and documentation for the team. Most people seem eager to contribute, so I hope it takes off.

One point here, we're using the wiki included with SharePoint. It's not nearly as hard to edit as MediaWiki (the Wikipedia wiki engine), but I still prefer MoinMoin (used by many Apache projects).

The heart of the software being built here relies on Ontology, and leveraging the data stored in that ontology. Ontology has always been a kind of slippery concept for me to grasp and while I've worked at another place which also did some ontology-related work, this seems "different"somehow. I'm sure it has to do with my limited understanding of this concept. I won't be involved directly in defining any ontologies (nor was I before). Like before, my job will be to create tools to help users sift through the ontology and find meaningful data and relationships so that they can, in turn, make better decisions more efficiently.

The tool used here to create the ontology is Top Braid Composer. We're accessing the data using Sparql, which is sort of like SQL for ontologies. And we can connect to the ontology model directly through Java using the Jena library. At this point, I'm just starting to learn to use these tools.

The UI is being developed in Flex. I've been wanting to do some Flex development for quite some time and am eager to start contributing. The team has chosen the Cairngorm micro architecture to help enforce best practices in the Flex framework.

Other than that, version control is Subversion and we're now starting to use Maven for automated builds.

All in all, it seems like a solid environment to get things done.

Happy 2009!

2009-01-04T18:38:00.005-05:00

As the work calendar turns to 2009, it's natural to reflect on the past year. Professionally, 2008 was a very frustrating year. Part of it had to do with the tools I was using, part of it had to do with the project I was working on, and a lot of my frustration was that I felt like I wasn't being heard.

Instead of rehashing some of the difficulties I've experienced over the past year, I invite you to read some of my previous posts.

So, how do I hope to course correct in 2009?

Do More
Many times in 2008, I found myself waiting for others to complete some part of their work so I could do mine. Other times, we hit limitations of the development platform we were working on and waited for management approval for how we should proceed.

In these instances I often had an alternative in mind for how to proceed, but didn't. I instead hung back and tried to let others do their jobs, make their own decisions, and get consensus on an approach.

I regret this approach.

I wish I had been a little more proactive in these instances and at least started to work on prototypes for my alternative solutions. I have a feeling that these "prototypes" would've eventually become the solutions we needed. Too often these issues got lost in a myriad of meetings or priorities got shuffled and in time, nothing got done.

Communicate Better
I went into 2008 hoping to make a big impact on the organization I was with. The team was new to java and I hoped to start a java users group. I saw where improvements could be made in testing and change control, and made suggestions. Early on, I felt that we might have problems with our application architecture and I voiced my concerns.

My communication style was basically make some suggestions, answer whatever questions I could, then wait for followup. My big mistake was waiting. I needed to be more proactive and follow up on my own. I needed to be more assertive and urgent.

Part of the reason I write this silly blog is so I could learn to communicate my thoughts a little more effectively. I feel it has been a great tool in that respect, but if no one reads it, what good does it really do aside from some academic exercise?

I've heard that How to Win Friends & Influence People and Secrets of Consulting: A Guide to Giving and Getting Advice Successfully are both great books that can help me get my message across better. I'm going to read both of them early this year.

Act Positively
All too often I've heard myself say, "XYZ is not working. Here's what I think we should do..." This is a negative way to approach a problem. People will naturally become defensive when hearing this language and may stop listening before you even get to pose the solution.

I've been doing a lot of reading on Disney the past few months. I wanted to learn more about how they go about executing these enormously large projects, with so many people, so effectively.

One of the tricks they do is always ask "what if" and "why not". They always say "Yes, if..." instead of "No, because...".

This approach enables communication and invites exploration of better ways to do things. It removes any negativity and ego from the conversation.

This is my #1 goal for 2009.

Happy New Year!

Performance Tuning JCAPS - Part 2

2008-12-15T11:56:00.014-05:00

It's been close to 2 months since my first Performance Tuning JCAPS post. Since then, we've noticed our servers running out of TCP connections under heavy load. Researching this problem, we learned that the JCAPS' JMS server makes heavy use of TCP in implementing Request/Reply queues. We approached Sun for guidance and they assured us that they've had success implementing high throughput applications using the JMS request reply solution... but how?

It appears a multi-step solution is needed.

First, running out of TCP connections in a scenario like ours is a JCAPS bug addressed in an ESR (110348) and rolled into JCAPS 5.1.3 Update Release 3. We installed this update and noticed a marked improvement. Running my simple 75 user simulated test, transactions took 2521 ms to round trip (compared to the previous time of 20128ms). While this is great improvement, an average of 2.5 seconds is still really slow for this simple transaction.
We reported our findings to Sun and they responded with 3 additional suggestions to try.
- The first suggestion was to turn on the socket pooling in the JMS server. This is accomplished by setting the JMS URL to
  stcms://hostname:18007?com.stc.jms.socketpooling=true
  in the eDesigner's Environment Explorer setting for this logical host.
- The second suggestion was to change the registry setting for TcpTimedWaitDelay to 30. This setting controls time a TCP resource is made available to the system again after the connection is closed.
- the final suggestion was to change the registry setting for MaxUserPort to 65534, making more TCP port numbers available.
The combination of these changes led to an amazing upgrade in performance to 110 ms for the previously run 75 user test - or as I like to say "pretty darn quick".

So, what's my opinion on all this? Obviously I'm really happy with the results, but the steps leading to the solution is what has me scratching my head a little.

ESRs are unnecessarily hard to implement. First you need to determine the prerequisites and corequisites to install with these patches. Sometimes these are listed in the ESR documentation, but often times they aren't and a call to Sun is needed to sort it all out.
Then once you have all the ESR you need, the repository, integration server, and eDesinger all need the patches before you recompile and redeploy the code. Needless to say, this is pretty time consuming.
I did not see any reference to this JMS server URL change in any documentation. It seems that the URL is the only place you can change this setting. There is not administrative console to make this change. Bummer, since it seems pretty important.
As for the registry settings, environment changes are often the first thing forgotten and the last remembered when upgrading to a new server. I don't like making changes that affect my applications which aren't checked into some sort of source control.

Side Effects May Include...

2008-12-06T18:17:00.001-05:00

Software development is complex. A lot of communication is needed to coordinate with users and between teams to synchronize efforts. On large teams developing distributed systems, effective communication is exponentially harder. In addition to the users and development teams, you also need to deal with system admins, DBAs, network, and change control specialists.

In many environments, this later group of people (the system admins, DBAs, etc.) do not get involved with the project until late stages. The development team might have free reign over development databases for instance, but when the code migrates to a certification environment things are suddenly very different. Teams move from surroundings where they have total control of machine settings, databases, and resources to an environment where they have none.

Often, during the migration to a test or production environment things are often "forgotten". Things that often don't make it into an application's change control system.... database schema updates, new environment settings, property file changes. If these changes are not documented for migration to production, it's a disaster resulting in an embarrassing mad scramble to remember and apply these changes on deployment day. No one feels good after backing out a bad production roll out of a system.

The knee jerk reaction when something like this goes wrong is to add more process - more forms, more user sign offs, an increase in role based access to systems. This seldom however fixes the problem - it just adds more overhead to getting the problem fixed once these problems rear their ugly heads again.

What's really needed is a better process through automation, not more process. Automate testing of the system. Create batch files to apply schema updates, change environment variables, deploy the application, and test in a single step. Use this script in your test environment and run the same script when you migrate to production.

All the time we try to sell the business on automating their processes through the programs we write. It always baffles me that, as software developers, we seldom take our own advice.

Secret Sauce

2008-12-03T15:38:00.002-05:00

Good things always have that "secret sauce" - the element that sets it apart from the pack and makes it better than anything else. The "secret sauce" takes something ordinary and makes it extraordinary. It's a Mac vs. a laptop. Disney World vs. a theme park. A Big Mac vs. a hamburger.

When I hear company execs talk about what sets their companies apart, it seems there's no real "secret" to the sauce after all. It boils down to hard work and a commitment to your customers. A commitment to a better user experience. A commitment to treating every guest interaction "special". A commitment to a good hamburger every time. There's not a magic switch you can flip... it's a commitment to excellence through hard work. Looking for a shortcut to this kind of success is a waste of time.

All too often I've seen software development teams look for magic switches rather than commit to the hard work of fixing the underlying problem. For instance, attempting to fix database problems through caching when the real problem is in the underlying schema. Changing the graphics on a website, while ignoring usability problems. Refusing to correct a broken architecture by continuing to throw more server resources to the application.

I've never seen one of these approaches work long term, and the time spent exploring these workarounds took time and resources away from fixing the actual underlying problem and long term success.

In each instance, the commitment to fixing the problem was abandoned in favor of the quick and inexpensive band aid to hide the flaws.

It's why these products struggle to achieve success while the Mac, Disney World, and the Big Mac continue shine as examples of truly great products.

Right Tool for the Job

2008-11-26T10:43:00.009-05:00

Business transactions can vary widely in their definitions, but they can basically be classified into two types - those where I need an immediate response so I can continue with my work and those where I'm going to tell you what to do and trust that you'll get it done while I move on to something else. These classifications are otherwise known respectively as synchronous and asynchronous transactions.

When designing an architecture for solving business problems, it's helpful to figure out which type of problem you're solving and then pick an appropriate technology to support it. For example, HTTP might be a good choice for synchronous applications since it is request/reply by nature. JMS might be a better fit for asynchronous solutions since these structures are typically one way.

Blurring the lines is the JCAPS offering of a request/reply mechanism for JMS messaging - called appropriately enough, JMS RequestReply. A colleague called this mechanism "synchronous by asynchronous" and I think that's an appropriate description.

In JCAPS, when a RequestReply queue is used, a dynamic connection (actually a TCP socket) is created for each RequestReply request. When the reply is generated, it is placed on the dynamic queue using the JCAPS Message sendto method.

On the surface this seems pretty cool... now I have 2 ways to skin the synchronous cat.

But what's the cost of creating that dynamic JMS connection?

I've created a simple tester that tests some number of simultaneous clients, each submitting 100 transaction requests. Comparing the JCAPS RequestReply mechanism with HTTP response times. Here's the data (all times are milliseconds):

	JMS RequestReply	HTTP
1 User	5 ms	0 ms
25 Users	639 ms	2 ms
50 Users	4791 ms	11 ms
75 Users	20128 ms	654 ms

The RequestReply response time degrades much faster than the HTTP response time. Using JMS in this way is not a good fit for these synchronous transactions. It's a square peg in a round hole.

In this instance JMS (or at least JMS RequestReply) does not seem like the right tool for the job. And further, just because a tool might add some functionality that allows you to do something you couldn't do otherwise, it doesn't mean you should.

Performance Tuning JCAPS

2008-10-28T13:05:00.003-04:00

We're getting close to (finally) deploying our JCAPS rewrite to production. Our final task is to load test and "tune" the application - making sure it will handle the expected production transaction load.

Prior to executing the load test, we added logging statements to report on the amount of processing time spent in each JCD (or processing unit for those non-JCAPpers out there). Using this info, we can determine the slow running processes and optimize them. For this optimization we could follow a traditional approach and profile the code, or we could take an easier route and configure JCAPS to throw more resources at the bottlenecks. This second option is done in the JCAPS connectivity map by increasing the maximum number of threads allocated to run the process. Clicking the input line to the JCD to open the properties, you can set the "Server session pool size" in the Advanced tab (NOTE: this setting is only for JCDs listening to JMS queues or topics).

I'm not sure how I feel about this really. On one hand, it's pretty cool to have this level of control. On the other hand, I'm wondering why I even have to deal with this. I mean, I don't ever need to set the maximum number of requests a web service can handle. The server automatically scales to the number of requests it receives until it runs out of resources.

I'm wondering why JCAPS JMS services cannot follow a similar approach as load becomes heavy? Maybe something similar to a database connection pool that could grow and reuse JMS listeners would provide an extensible solution?

Another question I have is why the architect of this application chose JMS over HTTP for the SOA medium between these services? We have no requirement for guaranteed delivery. This is a real time system, so there's no need to queue up messages if a certain component cannot respond for a while (those transactions will simply timeout). To me, JMS does not seem like the right fit for this application.

I'd be interested to hear other thoughts on this architecture choice. Is it possible I'm missing something? The people who made these decisions are no longer around to ask...

Contractors (ugh)

2008-10-16T10:31:00.008-04:00

Contractors - programming contractors, general home contractors like builders, plumbers or electricians, and even mechanics - have bad reputations. While everyone needs these skilled professionals to complete work where they have little expertise, there an inherent distrust in these relationships. Distrust - where total trust is needed.

Many times it's unwarranted because contractors are as honest and hard working as anyone else. So why is it so? I think a number or relatively minor, easily correctable behaviors lead to this perception.

Personally, I have changed mechanics more than I care to comment on. Not because I felt they were screwing me necessarily, but mostly because they didn't explain the work they were performing to my satisfaction. This led me to doubt their abilities. The doubt and distrust lead me to look for another mechanic who I have more confidence in. It's too bad. I hate shopping around for new mechanics.

At home, we recently added a room to our house. We've all heard horror stories in dealing with these general contractors, so we were pretty picky when it came to selecting one. We checked references, drove by recent jobs, and talked to as many people as we could for recommendations. We selected a good builder and are really happy with the work, but he tried to insist on final payment a few days before the job was complete. This soured the whole experience. Not only that, but he cost himself the opportunity for future work with me or a recommendation to my friends. This was totally unnecessary, stupid, and shortsighted on his part.

At work, one of the other contractors I'm working with went above and beyond the customer specs to create a tool they're finding indispensable for debugging and monitoring the system we built. Now - he's refusing to deliver the source code for this tool. How do you think the customer is reacting to this? (HINT: not good) Even if he had started this tool before he walked in the door at the customer site and didn't charge the customer for any of the enhancements - it's pretty clear that he worked on the changes at the customers site. The customer's perception is that they paid for at least some of the work. They believe they deserve the source so they can maintain and enhance this tool they've become dependent on. If I were the client, I'd feel the same way. I'm not affiliated with this person in any way other than we are both contractors, but this situation is starting to affect my relationship with the client.

It's tough to know how to react in these situations, but as a contractor myself, I feel it's important to at least recognize these trends and not let them interfere with my client relationships. As a contractor, try to:

Explain all work to be done to the customer's satisfaction.
Don't make trivial demands on the customer (like early payment). It sours the customer experience and may cost you future work.
Happily deliver whats expected. I'm sure there are situation where customers expectations are too high, but part of the job is setting expectations at the right level.

Remember, even if you're working for a company directly the business is your client. This advice pertains to you as well. Hopefully it helps replace a lot of the doubt and distrust with confidence and assurance.

Ideal Software Development

2008-10-14T13:22:00.008-04:00

I'm the type of person who constantly looks for ways to improve. No matter how well things go, I'm always striving to "plus" an experience - smooth out the rougher edges to make the next time even better. Sometimes the need to do this drives me a little nuts. It drives my wife really nuts!

When things don't go well, this task can be a little overwhelming (and depressing). There are so many areas for improvement, it's hard to know where to begin.

I've spent the last year or so, reexamining my software development experience - looking for trends in the tools and ideas that helped the teams I was on to be highly productive (and conversely where the lack of some practices led to a lack of productivity and frustration). Combined with many of the things I've read by Spolsky, Fowler, and the Poppendiecks. I've created a list of criteria I'd consider essential in an ideal software development environment.

Talk to the target users of your systems to determine the requirements of the system you are building. Watch them work and try to identify areas to automate - focus on the mundane or frustrating tasks they perform on a daily basis. In a B2B environment, understand the business purpose behind the transactions and the manual interactions they replace. If you can't find or don't have access to your target users, create an idealized version of the people you hope to help and share that vision with your team.

In either case, learn as much about the business and the people you want to help as possible.

Document what you learn and prioritize needs (I prefer a wiki as a low friction method to do this).

Create a schedule based on the requirements gathered while learning about the business. My initial schedule typically looks like a list of the customer needs I've identified sorted by priority.

Try and break these tasks down into subtasks that can be completed in a week or less. This will allow you to work in 1-2 week iterations and you'll instantly know whether you are ahead or behind schedule.

Update the schedule as iterations are completed. I've had good experiences using something as simple as a wiki to track scheduled tasks.

Practice "test first" development. This practice has boosted my productivity more so than any other in this list. The benefits are many.

First, test first development requires you to think about your requirements in detail - by defining the code scope needed to implement those requirements. The tests document the "real" (implemented) requirements for your system. It gives you a place to start.

Secondly, it's the easiest way to know that you're finished (when all the tests pass... you're done). This prevents overbuilding (adding extra source code that may be viewed as robust, but doesn't actually meet any requirement). This "extra" code makes maintenance more complex and expensive.

Finally, the tests give maintainers of the code some assurance that their future changes are not breaking any existing functionality. Again reducing maintenance costs.

Create an automated build. What this means is that you can check out and build any version of the system from source control in a single step (this also implies that you're using source control). Nothing is more frustrating than checking out some source code to work on and not being able to compile it. Sometimes it's because there's no "make file". Sometimes dependent libraries are missing (or you can't identify the correct version). Sometimes it's because someone checked in "broken" code. Either way it's annoying and nothing is worse for productivity than having to track these issues down.

Manage software releases. Create some method to tag releases (don't all source control systems have a way to do this?) and embed this tag somewhere in your delivered source code. This way, when a customer reports a problem (bug) with the software, they can tell you which version of the software they have and you can check out that exact version in an attempt to recreate and fix the problem.

Use a bug database. Track all known issues with your software. Work bug fixes into your schedule based on their priority in relation to new features. Again, this can be as quick and simple as a wiki page to track issues and linked to the schedule. Remember, criteria to recreate bugs become additional tests.

Ironically, I have experience with all of the items on my list but no single team followed all these practices. The most effective teams I've worked on met most of these criteria. The least productive place I worked had no desire to adopt any of these practices (and was a total disaster).

Also of note is that none of this criteria is tied to any specific technology. Java or .Net - SOA or traditional linear development - open or closed source these practices will make your teams more productive.

The bottom line is getting things done, working efficiently, and delivering value to the business - that sounds ideal to me!

Random Thoughts

2008-10-09T14:39:00.008-04:00

I haven't been doing too much development as I wait (and wait and wait) for our acceptance testing to complete. Here's a quick rundown of what I've been thinking about while waiting for that to complete....

I've been reading and participating in some of the discussions on StackOverflow. It's a great way to get a pulse on what others are thinking about and working on. I've also been able to pick up some things I never knew existed before. I've taken part in other question and answer type sites before and I had my doubts how helpful this would really be... I've been pleasantly surprised.
I listened to these great podcasts on alternative energy. Grass as fuel?? Pretty cool. I've been thinking about maybe getting some solar panels for my house. I would never have thought that the Northeast would be a good place for these until I saw a cool episode of Nova sometime back where someone in Mass. had some installed on their house. Now maybe I'll also look into one of these pellet furnaces.
Software Engineering Radio always covers great topics that I wouldn't otherwise have time to look into (AOP, Functional Programming, etc.). One of their more recent episodes covered the topic of Roles in Software Engineering. One of the skills they identified for a good Technical Lead (Architect) is the ability to identify the simplicity in complexity. Amen. For me, there is no greater complement someone can give to my work than to say how simple a solution looks. It takes more effort and skill than many people realize to achieve this.
I've been thinking more and more about what I would view as an ideal development environment. More on this in a future post...

"Nobody knows the trouble I've seen..."

2008-09-16T14:14:00.010-04:00

I haven't had a lot to write about this month. I'm in the middle of this seemingly endless, painful cycle where the more we regression test, the more unimplemented requirements we uncover. That's right, regression testing is defining our requirement set. It's a symptom of this big rewrite I'm working on. The system has been around so long, that no one can remember every intricacy in the existing application. Regression testing is the only way to uncover the missing functionality. It's a nightmare!

Compounding the problem is the difficulty in regression testing. There is a "certification" environment that is supposed to be an exact replica of production. But when we try to run production requests through our new application, the results are different. Our business users review these inconsistencies and often tell us that our responses are actually better than the production version. This is a good news/bad news type of thing. While it's good to be better, in this instance it'd actually be better to be just as bad. Now, instead of an automated test scenario where we could bulk test thousands of requests and responses to programmatically identify inconsistencies, we need to hand check almost every message.

I see long days ahead...

Cheating on Tests

2008-09-03T15:34:00.004-04:00

I just listened to Kent Beck's presentation from this year's RailsConf for the second time (something I almost never do). One of the gems I loved about his presentation was when he described test driven development as "cheating" (at the 27:08 mark).

Exactly! I feel a little foolish for not thinking about it about this way before. It is cheating! How many times in school did I wish I had the answers to tests before I took them? Or wished I even knew the questions? Answer: Every time.

Why? Because it would've been so easy! Imagine all the time I would've saved, with assurance that my work would always be correct and complete.

Extending this idea to software development should be a no brainer - figure out what the software should do ahead of time (and write tests for it). This way you know exactly what to code, when you are done, and that your solution is complete and correct. It's so easy (it feels like cheating).

Schedules

2008-08-28T15:11:00.004-04:00

I'm usually not in charge of scheduling. I've contributed to schedules, but someone else - usually someone from management - drives schedule creation.

I also can't remember a situation where the schedule actually worked. What I mean by this is, a lot of time was spent to create schedules for 2-3 months into the future - then not really revisited (adjusted) for a month or more. By this time something invariably happens (priorities change, initial tasks take longer than expected, vacations/time off weren't accounted for, etc.). This throws the entire thing out of whack and quickly makes it obsolete. A month or more will pass, then the whole process starts again with the creation of another schedule.

For the project I'm currently on we've spent a lot of time creating 2 schedules from scratch and are currently creating a third. The first two have been complete failures and I have low expectations for the third. It's frustrating to spend a lot of time and effort on something you just know has near zero percent chance of success. So, what's gone wrong and what can be done better?

There's poor coordination between departments. Communicating and agreeing on priorities are a problem. We rely on another team to define our business requirements and acceptance test completed functionality. The big mistake is that, to this point, this other team has not been involved in the schedule creation process. Our dates slip when requirement definition and testing are not the highest priority for that other team. It's not their fault that their priorities are different than ours.

Better coordination is needed to get this team on board and committed to working with us to meet our deadlines. We cannot succeed without this. It seems too simple to keep getting wrong, yet it continues to happen.

Our schedule is too vague. Many of the tasks are scheduled with monthly granularity. This is way to long. When a task is running long you don't find out for a month (when it's already past due). I've been doing some reading, and hours rather than weeks or months seem like a better breakdown for schedule tasks. This level of refinement provides a closer to real time status on the overall progress (and potential delays). It allows for a faster reaction to problem areas.

It doesn't seem that hard to me - get everyone on the same page and do a better job defining the steps to complete a task. As I get more involved defining this third schedule, those are the goals I'm setting for myself.

Yes!

2008-08-12T08:43:00.005-04:00

Steve Yegge has this great post today about business requirements and building software that you'd personally use in a domain that you actually know. Great stuff!

I need to figure out a way to print wallet size copies to hand out at requirement meetings.

Second Thoughts on SOA Architecture

2008-07-30T15:21:00.002-04:00

When I first joined this SOA team, I questioned the architecture we were implementing. The application flow is basically linear and uses JMS to pass messages between the various functional components. The components themselves were not designed generic enough to be shared by multiple applications. Like I mentioned earlier, the main driver for rewriting this application was for speed.

The whole thing seemed very unSOA-like to me. Since none of the components were reusable outside this application, why not code the entire project in a "traditional" way? This approach would improve performance, replacing the overhead of multiple marshal/unmarshal operations needed for JMS with direct function calls.

My perspective changed (a little) recently when we needed to add new functionality to the system. We could have modified one of the existing components to implement the new feature. Functionally, it made sense to keep this related logic together in a single module.

The main problem with this approach is that we would need to retest the component to ensure we didn't break anything that was already working (and fully tested). Interestingly, this is where the SOA approach worked to our advantage. An alternative implementation would involve simply creating a new, stand-alone component and inserting it into the proper place in the flow - eliminating the risk in modifying already working code.

The second approach plays to the main strength of SOA - it's inherently extensible. Stand-alone services can be combined, inserted, or deleted without affecting the overall stability/reliability of the individual components - all the while gaining the ability to quickly change the composite application's behavior. While reusability would definitely increase the ROI of developing these components, it shouldn't be looked at as the only benefit of SOA.

*****

A "traditional" application can also be designed in this modular fashion, also allowing new functionality to be plugged in through an additional method call. Here you could realize the speed and the extensible benefits described above. The downside in this approach is the complexity integrating new functionality if it happens to be written in a different language than the main application.

Here again, the loose coupling, inherent in SOA architectures works to your advantage. As long as the new ".Net" service can talk via HTTP, MQ or some other messaging standard - this new functionality can be easily integrated into your Java-based or BPEL composite application (and is more attractive than trying to figure out JNI).

Top Down or Bottom Up

2008-07-25T15:47:00.006-04:00

When developing Java RPC-style web services, I'm torn whether to follow a top-down or bottom-up approach.

Using the bottom-up approach, you create your Java classes first and use the @WebService and @WebMethod annotations to specify that this is your web service interface. The WSDL for the service is generated automatically during the build (pretty nice). This approach has it's advantages in that you never have to leave Java to create the WSDL. Creating a WSDL file is not trivial and using this approach means you never need to think about the WSDL's <service>, <binding>, or <portType> elements and how they're linked together.

Conversely, a top-down approach means you create your WSDL first, then use a utility (either your IDE or a command line program) to generate the Java skeleton. Using this approach you need modify the generated Java stubs to call your business logic. If the interface (the WSDL) changes, you'll need to regenerate the classes, recreating your business calls.

On the surface, it appears the bottom up approach is the way to go. Changes to the web service API are effortless - you only need to update the Java class opposed to updating the WSDL and the Java program. So... what's the downside?

Expressiveness. Your web service is defined by it's API. It let's callers know exactly what's required as input and exactly what will be returned. Take a look as the following snippet from an XML schema I'm using to pass values back from a web service I created.

<xsd:complexType name="my_type">
  <xsd:sequence>
     <xsd:choice>
        <xsd:element name="error_message" type="xsd:string"/>
        <xsd:element name="values" type="xsd:float" minOccurs="1" maxOccurs="unbounded"/>
     </xsd:choice>
  </xsd:sequence>
</xsd:complexType>

This tells the caller, "I'm going to either give you an error message back or an array of floats. If I pass an array of floats, it will have at least 1 entry".

I'm not exactly sure how to best express this using Java classes. First of all, what's the best way to express the <choice> element in a Java class? There is no union data structure like there is in C++. Secondly, is there a Java collection that says "I must have at least 1 element"? Not that I know of.

Now there may be a way to generate the above XSD segment using the WSDL annotation tools, but I'm betting it's not trivial.

Assuming that I'm right, and creating the Java classes to generate the above XSD is not trivial. Let's create a simplified (almost equivalent) Java class and take a look at the WSDL generated from NetBeans for a web method return. Here are the class members for a Java class named MyType (in the class I also created the appropriate getters and setters).

   private String error_message;
  private List<Float> values;

This type is returned by a @WebService @WebMethod defined in a Java class. And here is the corresponding XML schema defs generated by NetBeans.

<xs:complexType name="myType">
  <xs:sequence>
     <xs:element name="errorMessage" type="xs:string" minOccurs="0"></xs:element>
     <xs:element name="values" type="xs:float" nillable="true" minOccurs="0" maxOccurs="unbounded"></xs:element>
  </xs:sequence>
</xs:complexType>

While this schema is similar to the one above, it's less expressive in some important ways. In this case, the caller doesn't really know what to expect the service to return. It looks as if it could return nothing or both values (which isn't true). This ambiguity will cause clients of the service to overbuild - adding unnecessary logic to their systems to handle the empty message or the instance when both values are populated (neither of which will ever happen).

Since the API is the most important part of the web service - after all it's the contact between the provider and the client - I usually lean toward the top-down approach to building services. It's a little more involved for the provider, but by being more verbose and exact in it's usage it's more usable by the client - and that's the important part.

JCAPS 6 Impressions

2008-07-23T15:44:00.004-04:00

I've started to use JCAPS 6 a little of over the past month and thought I'd share some of my initial impressions.

Finally a JCAPS editor I can use! I found the previous version of the JCAPS IDE hard to develop with. It was based on Netbeans 3 and didn't have a lot of the features I had grown accustomed to using (refactoring tools and a local history to name two). It was just plain old, clunky, and slow.

The new version is based on the latest Netbeans and all the goodness is back. And it's fast.

No more repository. Well, almost. JCAPS 6 has a few different modes to work in - repository based or non-repository. A repository based project allows JCAPS developers to work in essentially the same way they did with JCAPS 5 - JCDs, OTDs, connectivity maps, deployment profiles and yes - the repository. It's also the way to migrate legacy JCAPS project to the new platform - through a simple export and import between the repositories.

I haven't used the repository portion of JCAPS 6 much, but some coworkers have and report it works as expected (with an improved interface).

For non-repository JCAPS projects, development is very similar to regular J2EE development - with particular focus on the Java Connector Architecture (JCA) and Java Business Integration (JBI).

I wasn't familiar with either the JCA or JBI before JCAPS 6 so here is my 10-cent definition of each (NOTE: the Wikipedia links above provide more thorough - and probably more correct - definitions):

JCA is an abstraction layer for connecting to third party services like databases and legacy systems. I think of it like generic JDBC that can do more.
JBI provides a standard way for various portions of your SOA to talk to one another (it's the ESB). I see this as an abstraction so you don't really need to know if your composite application is talking to a web service, JMS, an EJB, or possibly JCA components. OpenESB is the JBI implementation used by JCAPS.

The value add for JCAPS is to provide more (and better) JCA and JBI components than are available using the "free" Netbeans. Thus helping to make developers on it's platform more effective.

So, while my dive into JCAPS 6 hasn't been especially deep to this point, so far I like what I'm seeing...

Process Over Technology

2008-07-21T16:23:00.004-04:00

As the struggles continue with the application rewrite I'm currently involved with, I've read and reread these articles by Joel Spolsky, James Shore, and Chad Fowler on why this path generally leads to failure. After I finish reading articles like these I often ask myself: since many of the industry thought leaders think these projects are a bad idea, why are they still so common? I mean, Joel's article was written in 2000. Eight years later (an eternity in technology) that article is still extremely relevant, but it's lesson has failed to penetrate mainstream thought.

There's not enough industry focus on how best to approach legacy system enhancements. Many software development articles that garner the most attention focus on quality improvements delivered through innovations in technology - new programming languages, frameworks, APIs, architectures, and hardware. While that stuff's definitely important, they're relatively little help to application developers looking to extend the life of existing systems.

Martin Fowler's Refactoring book is the best example I've read for extending the life of existing software. This book - and articles like those in the opening paragraph - are timeless because they describe thought patterns for maintaining applications. Their lessons are applicable regardless of any specific technology or language.

In a recent StackOverflow podcast Joel Spolsky and Jeff Atwood discuss a recommended reading list for developers (Jeff had a previous post on the subject). In looking through their lists, I'm not seeing any "Teach Yourself Perl in 8 Hours" or "Ruby for Dummies". Specific technology is not the focus here. These recommendations teach software professionals how to think - improving quality, efficiently, and allowing them to deliver greater value.

I'm starting to believe that there are an increasing number of developers who are starting to think more like this (or at least I'm meeting more of them) - focused on improving processes rather than relying on technology innovations for increased software quality. I hope this mindset spreads to more of the development teams I'm working with.... before the next big rewrite. This thing is painful.

*****

Jeff Atwood also had a recent post on Dealing With Bad Apples. He has a similar take to my Observations in Poor Management.

Head Scratcher

2008-07-15T15:50:00.002-04:00

I wish I was on board at the very beginning of the project I'm working on. Maybe it would shed some light on the direction the client is currently taking.

This project is a rewrite of an existing, poorly performing, data processing system. The rewrite is being done in two parts. Phase 1 replaces the front half of the system - responsible for receiving messages, routing the transactions to the appropriate business logic, then finally, transmitting a response. Analysis (performed before I got here) identified this logic as the application bottleneck. This portion of the application was rewritten in JCAPS and it's currently being tested (I haven't seen any performance data for the rewrite yet).

The second phase of the project replaces the business logic, currently written in C++, with BPEL. Let me say that again... the second phase of this project replaces compiled C++ with interpreted XML. For speed.

Anyone else confused?

Now maybe implementing the business logic in BPEL will make future enhancements to the business logic easier, and that's the reason for choosing BPEL. Maybe. But at what cost to the performance of the system? My gut tells me that BPEL is going to be a dog. I hope not. I haven't been able to locate any BPEL benchmarks and how it compares with Java or C++.

Documenting Requirements

2008-07-11T16:08:00.001-04:00

Almost every place I've worked struggles (in varying degrees) documenting business requirements. Maybe not the initial requirements, but the new requirements/enhancements that emerge as releases are delivered. They usually stem from a conversation that starts out, "I really like this, but it would be really cool if it could also do this other thing I had not thought of until now."

This is a bigger than normal issue at the place I'm currently working. Mostly it's because new requirements are not captured in the same documents as the original specification. Instead, a new document is created and placed into a shared folder on the network (or sometimes a series of different folders). Over time, there are a lot of distributed requirement documents and it's really difficult to figure out what the system is supposed to do (and where to look for this information). Please, place all your application requirements in a single coherent "document"! Make it stupid easy for people to figure out how the application is supposed to work. Right now the ramp up time to learn this application is ridiculously long and the poor state of these requirement documents are a big reason why.

I've used wiki's in the past to capture requirements and I think it worked out pretty well. My practice there was to create a new wiki page for each feature request and then link to that page from the 'overall specification' and 'version release notes' wiki pages. The one thing I've often wished for is a better way to link these pages to the source code changes. I made a practice of creating a bug ticket for each enhancement then cross-linking the ticket to the wiki page, and then using the bug ticket number in the code check-in comment. This process is cumbersome and somewhat difficult to work through in reverse - like finding all the code commits for a given ticket. I'd like it if this functionality were baked into source control systems (maybe it is and I don't know).

The second big problem with documentation on my current team is requirements ownership. The developers are reluctant to update the business requirement documents (I don't know why). The business team is extremely busy and updating these requirement documents is their lowest priority. It doesn't take a genius to see what's going to happen here.... developers start working on features before the requirements are documented and undocumented features are implemented.

Now, while the difficulty in finding the requirements in the first place was a big problem, having undocumented features in your application is a HUGE problem! This practice basically invalidates any documentation you do happen to find. All of a sudden you have no idea if previous requirements have verbally changed or which documented requirements override the verbal requirements that someone vaguely remembers. You have to be disciplined enough to document your requirements before implementing them! There is no excuse for not doing this. None.

Recent SOA questions/thoughts

2008-07-10T16:30:00.002-04:00

We've had some interesting SOA desing questions lately. Here's a quick rundown with my thoughts...

Is it a good idea to wrap DB calls in a web service?

Some argue that this is a waste for internal applications. That these services don't provide any business value - query results are simply regurgitated back to the caller.

While I agree that these calls don't add much business value, the benefit lies in abstracting the DB calls from the caller. In simplest terms, wrapping access means that callers need not concern themselves with managing database connections. Additional changes/enhancements to the schema can now be handled in a single place rather than mandated enhancements to DB clients - even the DB implementation, location, and authentication updates are transparent to users. To me, services like this - services that simplify interactions - are a big part of what SOA is all about.

As SOA systems evolve/extend, is it a better idea to plug in new functionality or extend existing components? For instance, let's say there is an SOA application whose flow consists of calling component A followed by component B (A -> B). New functionality needs to be added to the system. You can either modify one of the existing components to add the new capabilities (A' -> B) or you can create a new component and plug it in (A -> C -> B).

If the new capability is functionally similar to either A or B, this may muddy the waters. Regardless of the situation, I favor the plug in approach. Again, I feel this is one of the big benefits of SOA - the ability to add functionality to a system without changing and retesting existing code. It feels simpler and cleaner. Smaller, highly specialized components are easier to debug and maintain than larger components. They're also more likely to be reused. I feel pretty strongly about this.

One of the things I still struggle with is when is it better to code your SOA interactions through a business process vs "regular" web service code.

My current thinking is that if all you doing is calling services and mapping values, then a business process approach is acceptable. I'd still rather code these interactions in something other than BPEL, but it may only be because I'm more comfortable with that approach. I've just started to code a rather large business process in BPEL and I'm finding it to be OK. The big benefit comes in the ability to visualize the process via BPN. This makes it easy to review the flow with business users.

Something old, something new

2008-07-02T09:37:00.004-04:00

In another discussion recently about SOA, it still surprises me a little that people, technical people, don't get that there's not much new here. For years it's been standard practice to separate functionality into functions, classes, and modules. The idea has always been that these smaller, highly specialized components are easier to share and maintain than monolithic blocks of code.

Functionally, SOA is not much different. The goals are the same - reusability and easy maintenance. The biggest difference - in the case of a web service SOA - is that the shared library included in your application is replaced with an HTTP call. If you're already supporting message-oriented applications using MQ or JMS, these messages are now HTTP requests. The process of decomposing the application into reusable services is essentially the same.

The real point here is, if you already have a modularized architecture and you're having problems with application crashes, a switch to "SOA technology" is not going to help you. My first guess is that (lack of) testing is the problem. These problems will not go away in your big "SOA" rewrite unless you figure out a way to test better. Or you could be having problems scaling your infrastructure. If anything, a switch to a web service solution will increase the number of messages being passed around your network.

My advice, instead of following the knee-jerk reaction that your technology is old and needs a major upgrade, write tests first to determine:

the functionality (behavior) of your application is tested (start by testing the "problem" areas of your app. first)
your infrastructure can handle the current transaction load.

******

If you're reading this and haven't read Chad Fowler's Big Rewrite series, you should check it out.

Targeted Content

2008-06-13T11:05:00.010-04:00

I'm fascinated by all the context sensitive/targeted media that is making it's way into the mainstream.

The other day I was reading an email from my wife having to do with bringing some milk home and something about a recent trip we had taken. The Google ads on this page were some that you'd expect - one on how to save gas, another for a baking product. The one that really surprised me was an ad for silica gel?!?

Silica gel is a moisture absorber and usually found in the little packets that say "Do Not Eat" included when you buy a new pair of sneakers and many other products. This was intriguing to me because there was no mention of anything related to this in the body of the message. However, my wife works for a company that produces these products and sent the email from her work address. Not only was AdSense was smart enough to pick up on the from address and trace it back to an actual company, it knew what the company produces, and suggested an appropriate ad to match their product (it was actually an ad for a competitor). That's pretty cool!

Cognitive Dissonance?

2008-06-06T08:15:00.003-04:00

In this week's StackOverflow podcast, Joel Spolsky describes cognitive dissonance - and it may help explain the the JCAPS love fest at my current client. This conversation happens around the 11-minute mark.

I knew there had to be a name for this...

More Nonsense

2008-06-04T07:55:00.009-04:00

I know I need to let this go, but I can't stop thinking about the nonsense of this BPMN -> JCAPS -> .Net web service architecture proposed by management at the client I'm working with.

The main reason for the base web services to be implemented in .Net rather than directly in JCAPS is that this organization has 10x as many .Net developers as it has Java/JCAPS developers. The thinking is that this ratio will make it easier to find available bodies to maintain and enhance these services. While this makes sense, any guess on how many people know BPMN? A small handful... all of them contractors. There's not even a BPMN modeling tool in place at the company. Yet they're convinced this is the way to go.
Is it reasonable to expect business users to create BPMN models? While I realize the GUI interface makes it less like "coding", I think that it'd be helpful to have a basic understanding of boolean logic, parallelism, and exception processing in creating the business models. While I've seen some amazing Excel applications created by business users, the assumption that these people will be able to create high availability, high throughput applications is a bit of a reach for me.
All the excitement over BPM reminds me of the EJB "revolution" - when the mantra at many companies was "EJB everywhere" and EJBs were the architectural style du jour. EJBs were (poorly) implemented in many places where they weren't a good fit. Many of these applications had scalability and performance problems. I'm wondering if this BPM "revolution" will produce similar results. (Is there a reason it's called Business Model Notation rather than Business Application Notation?)
In a system like this, it's important to think about how service enhancements are rolled out. No one is talking about how these services will be maintained, released, or tested. Will there be a period of backward compatibility for service clients? How will that be handled? This lack of foresight is another reason I feel this company is not ready for SOA.

On numerous occasions I've approached the client to discuss these things with little success. I can't get them to recognize that many of their problems result from their (lack of) process rather than problems with their technology. I wish I knew of a better way to get through to them.

JCAPS = Nonsense (at least here)

2008-06-03T16:10:00.004-04:00

Come hell or high water - this company is hell bent on including JCAPS as part of it's enterprise architecture, even though it doesn't plan to use any of the features that might set it apart from it competitors or open source alternatives.

The long-term plan here is for the business users to use a BPMN-compliant tool (not the one from JCAPS) to create the business processes.

These processes will integrate web service calls, creating a sort of business mashup of these services. The services will be written in .Net, not JCAPS.

The only place JCAPS enters the equation is as the platform to run the BPEL generated from the BPMN. Make any sense? Not to me. Isn't a BPEL engine included in Glassfish? I'm sure this is just one of several free or low cost alternatives to execute BPEL.

What I can't figure out is this company's infatuation with JCAPS. Given the shortcomings they've encountered already, why they are so intent on looking for more places to implement this technology stack?

Observations in Poor Management

2008-05-30T08:51:00.007-04:00

Yesterday two people at the place I'm working gave notice that they'll be leaving the company. There has been a lot of turnover in the six months I've been here. Based on what I've seen, I'm not really surprised. Still, there always seems to be some head scratching by management and some longer tenured employees on why these people are leaving. Here's some of what I've seen.

I've already mentioned the poor tooling and processes in place here - starting with Lotus Notes - so I won't rehash that here other than to mention the tooling shortfall has been brought to management's attention many times with no real action.
Some people I sit near are in a constant state of emergency. Their production systems break daily (even nights and weekends) because the company's trading partners send messages that don't conform to their messaging API. Instead of rejecting these transactions, management's approach is to ignore the problem, asking employees to massage these messages (often manually) into a compliant form. This was going on before I started and I don't understand why the company doesn't choose to reject these messages. Come on! Force the trading partner to update their transmissions instead of putting these poor people through this daily hell!
One person I sit near has openly been searching for a new job the last month. Not that I blame him, but I've never witnessed such an open job search in the workplace before. He routinely leaves during the day for interviews and constantly takes calls. I wouldn't be so offended if his work hasn't suffered, leaving the rest of the team to pick up the slack. Management is aware of both the job search and the lack of productivity which is causing delays, but I've seen no response to this behavior. I can sense the rest of the team starting to wonder why they are working so hard.
Another team has an opening they desperately need to fill. There's a qualified person on different team who wants to apply for the position. Sounds like the perfect fit right? Not so fast. The company wrote the opening as contract-to-hire, and would force the employee to quit and reapply for the contact position! Make any sense? Not to me. When I asked about this, I got some jibber jabber about head counts, and management blah, blah, blah. Give me a break! Do what's right for the organization, the team and the individual here before you wind up losing more members of the overworked team or a qualified individual.

Mismanagement in this company is a problem. Management needs to put decent set of productivity tools in place, it needs to do a better job helping to foster innovation, and it needs to be more responsive and sympathetic to it's employees struggles - otherwise the brain drain will continue - and at greater pace.

*****

Quick addition to my post last month about "Silver Bullet Syndrome". I was listening to a Pragmatic Podcast this morning where Michael Nygard commented that he's sometimes suspicious of vendor solutions. Stating that they are developed by the "same" people as in house programmers with similar time constraints and pressures. The result is often a framework that exhibits similar problems to custom solutions.

He went on to say there are instances where a framework is indeed a better choice (connection pooling, caching, authentication, concurrency, etc). Often these components are battle tested implementations of well known and common technology problems that vary very little between organizations.

Well said.

My point was that... less often do well authored libraries or frameworks exist to address business problems, since these can vary greatly by organization.

Links to Some Useful Resources

2008-05-22T08:00:00.005-04:00

I've been a fan of Joel on Software for some time. Recently Joel (Spolsky) started a new enterprise, stackoverflow.com, with Jeff Atwood. They've been releasing some entertaining and informative podcasts as part of this new endeavor.

Good Stuff. Who knew you could wash your electronics??

*****

Joel has written a few books. One I haven't read yet is titled Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent. While I'm sure this book is helpful, what I really need is a book that can help locate progressive companies which allow employees to work smartly and get things done.

I don't know if it's endemic to the region I live or it's more widespread than I imagine, but none of the companies I encounter here are doing anything close to Agile development, few are working with Ruby, most are afraid of open source, none have heard of REST web services, and most are reactive problem solvers rather than proactive.

They all look first to OTS software solutions, rather than empowering their people to be creative and take ownership of the company's business problems. They're looking for application babysitters rather than application developers.

I've noticed that many of these companies use Lotus Notes for their email. Does anyone outside of western New York use this horrible system? This has become a litmus test for me to gauge the companies I speak with. The way I look at it is: if you can't even get email right, how can you be expected to make good technical decisions in other areas?

*****

It's been a while since I've done some hard core database programming, but I listened to a podcast from WebDevRadio yesterday that blew me away. Brian Moon talked about MySQL scaling and many performance enhancements. This was a great conversation.

Unit Test Saga

2008-05-21T11:32:00.000-04:00

It's been almost a month since I last posted! I think it's partially due to the fact that I haven't had much interesting work to talk about. Mostly I've been frustrated in trying to convince my teammates why automated unit testing is important (and how to better use our horrible source control system). Most of the objections are the familiar "we'll be writing more test code than application code", "how can we test what we haven't written yet", "it takes too long", etc. I think I've finally convinced them of the benefits though, and we're starting to write automated tests! I'm hoping I can now introduce Test Driven Development and convince them to start writing their tests before the code.

I came across a podcast last week from Net Objectives that does a much better job of explaining the benefits of Test Driven Development than I ever could. I thought it was great and I hope it's useful to others as well.

Silver Bullet Syndrome

2008-04-25T13:33:00.003-04:00

I've noticed a trend in my current organization which I'm sure is common to many companies. I'm having a hard time defining it, but let's call it "Silver Bullet Syndrome"... where when faced with some business problem, companies immediately look for an off the self (OTS) technology solution - many times without understanding the full scope of the problem. The OTS solution is viewed as a "Silver Bullet" and will immediately solve all of the organization's woes.

Now I'm not saying all off the shelf software is bad, but often times these business problems are not generic enough to be addressed by a general solution. The lack of flexibility of many OTS solutions compounds the problem, forcing companies to modify business processes or requirements to meet the capabilities of the tool instead of the other way around.

Once the tool is in place however, another problem manifests itself. This often starts with a management mandate like, "OK now that we have our shiny new Silver Bullet 2.0 and have paid $$$ for it, let's get the biggest bang for our buck by using it everywhere!". What I've seen is that the limitations and compromises encountered when solving the original problem are now limitations and compromises of multiple business processes - making the entire organization less effective.

In my current situation JCAPS is playing the Silver Bullet role. JCAPS was initially purchased for it's single customer view capabilities. Even though they've had continuous problems with that implementation, the organization moved forward with a JCAPS-based business automation project. Looking at the suite, someone noticed the JCAPS Business Activity Monitoring (eBAM) functionality and now wants to use this for system level reporting on the system.

Some examples to drive home the point on how the company has modified it's business to meet the capabilities of the tool:

This organization is traditionally .NET and has a large staff of .NET developers and infrastucture. It has little expertise or experience supporting Java enterprise applications.
While some of the EDI messages this organization supports are included in the JCAPS OTDs, some are not. JCAPS does not provide the capability to customize these modules, forcing the company to develop "home grown" solutions anyway (reducing the benefit of the tool).
The system reporting that the company wants is a mixture of tables and charts reporting on the performance of the system. The eBAM module only supports specific chart types (bar, pie, light) and no tables. Furthermore, eBam only works with Oracle or an internal proprietary database. Not a good fit for a company which mainly uses Sybase tables.

Currently the project is behind schedule (and I'm guessing over budget). I believe this company would have a better chance for long term success if the organization focused on using it's tech knowledge to create some custom solutions that meet their exact business needs (and they could support), instead of looking for a "Silver Bullet".

SOA Readyness

2008-03-25T14:31:00.007-04:00

I'm wondering if my company is ready for SOA. Right now the various business units in the company support many pieces of custom built technology which provide essentially the same functionality. Consolidating this functionality into a common, shared set of services is what SOA is all about right? Not only will the footprint of what we need to support be smaller, but we'll be able to meet future business needs faster (getting a jump start by reusing these prebuilt, prepackaged, pretested functionality). Isn't SOA a no brainer in this instance? I'm not so sure.

Building successful SOA components (even ones designed for internal use) need a product development approach. Code before SOA is essentially "set it and forget it". Failures in a particular component have a relatively small impact. In a SOA solution, things like scalability, stability, support, and release management are much more important. The significance of these traits increase proportionately with the number of clients to a particular component. All of a sudden you go from a situation where a software bug might affect a single business unit, to the point where the same bug could affect the entire enterprise.

I've already discussed the company's poor release management procedure. Its system is not conducive to concurrent development. This is needed to support released SOA components while the next version is in development. Strike one.

Currently, there are no test suites or sets of regression tests for this company's systems. Sadly, there is little desire to build any. Regression testing is important because as we enhance the services, we need to ensure we're not breaking any existing functionality. Creating a custom set of tests for each change to a service wastes time and is itself error prone. Without a good set of tests, the stability and support-ability of the service is in serious question. Strike two.

Scalability is always a concern, but since this code will now be responsible for handling more transactions (since it likely supports more than 1 client) any problems with scale will appear sooner and affect more business units. Strikes one and two affect our ability to effectively address these performance problems. Strike three.

It seems like SOA in this environment is a disaster waiting to happen. Until the release and testing issues are addressed, I do not see how it can succeed.

Change Control Gone Wrong

2008-03-20T16:09:00.002-04:00

Change Control & Release Management are two of the most fundamental elements in software engineering. You write code. You release code to the world. All the while, track changes to the code in case something goes horribly wrong and you need to revert to a previous version. It's pretty simple.

Modern change control systems like CVS provide additional functionality. When checking in changes, you can include comments to describe the changes. Tagging the repository ties various versions of source files together (i.e. for a release). Branching involves splitting the repository so parallel development can take place (i.e. to support bug fixes to a release while the next version of development continues on the trunk). Changes on the branch can be merged into the trunk, eliminating the need to apply fixes twice.

These features are fairly standard in most of the change systems software development teams use today. The point of this post is to ask why some organizations are reluctant to use these features?

The process where I currently work is this... Write the code (the method of version control varies by team). While the company has purchased a version control system for production releases, it wasn't available for development teams to use until recently. Even still, the development and production version control systems are completely separate.

What this means is, once the code is ready for promotion, the code is exported from the development version system and imported into the production system via copy/paste to a directory. When an enhancement to the production release needs to be made, the process is for the production code to be imported back into the development system where the changes are made.

I have a few problems with this. First of all, the version history and comments describing the development changes are lost during this import/export process. This information can provide assistance to the development staff in understanding what changes were made, by whom, and why.

Secondly, you lose the ability to merge production fixes automatically to next-release code currently in development.

Understand that the version system this company purchased has the ability to tag, merge, branch and do all the things described above yet they decide not to - instead relying an archaic system of manually copying files around to various directories. This nullifies any value provided by the version system! Why not just copy the release code to tape? All I can guess is, for some reason the company doesn't trust the tool that it bought.

The dirty secret in all of this is, the company's process is a total waste of time and resources. The development team is never going to use the supplied "production code" when a release needs enhancement. The development team is going have a tagged version of the release in the development version system to branch on. It will automatically merge any changes to the trunk when done.

For some reason, the necessity of this charade really bothers me. Getting something so simple wrong, leads me to question how this company can make good decisions when the issues are more complex.

JCAPS Unit Testing - Part 3

2008-03-07T14:42:00.005-05:00

Now that we can test jcdTargets from a jcdUnitTester, the final step is to use a combination of JUnit and HTTPUnit to execute and verify the tests. Basically we turn jcdUnitTester into a simple RESTful Web Service where we post the tests.

The first set of changes are to jcdUnitTester. My first pass at jcdUnitTester hard coded the test into the JCD. Since we'd like to change the message per test, it'd be better to pass this in as parameter on the HTTP Request. Use the getRequest().getParameterInfo().getWebParameterList() method of JCAPS' HttpServer class to get these values.

Other values we'll need, in addition to the test itself, is the queue/topic name where the test will be placed and the queue/topic where jcdUnitTester will listen for the response. Use the same technique described in the previous post to set the replyTo field in the JMS message.

This should be the minimum set of values we'll want to send to jcdUnitTester. Some other optional things I've found useful are parameters for debug level and any needed topic message selectors.

The final thing to create is the JUnit test themselves. Nothing to fancy here. I've used standard JUnit conventions (like the @Test annotation) to identify my test methods. The test methods use HtppUnit's PostMethodWebRequest (set to jcdUnitTester's URL) and setParameter() to set the request parameters.

Once the response returns, WebResponse's getText() method should hopefully provide the string that matches your expected result. Otherwise, you have a bug...

JCAPS Unit Testing - Part 2

2008-03-06T15:52:00.007-05:00

Last time I explained how to trigger a JCAPS JCD from a web browser. This entry will hopefully clarify some of the items in that post, before building on that functionality to execute JUnit tests using HTTPUnit.

First, the JCD created to listen for HTTP requests (let's call it jcdUnitTester) will drop messages on a queue or topic where the JCD we want to test is listening (call it the jcdTarget). The jcdTarget normally processes the incoming message, then passes the result to the next process via a queue, topic, or some other transport mechanism (call it targetResponse). For jcdUnitTester to return the message to browser in the HTTP response, jcdUnitTester must be listening on targetResponse (creating a loop between jcdUnitTester and jcdTarget.)

In my code, most of my jcdTargets involved JCDs sending responses back on a topic or queue. To make the JMS jcdTargets dynamic, I've taken advantage of the JMS eGate class's sendTo method. The targetResponse is obtained from the class's getMessageProperties().getReplyTo() method. To set this value, jcdUnitTester and other classes sending messages to jcdTarget use getMessageProperties().setReplyTo(). This convention is what makes it possible for jcdUnitTester to test individual jcdTargets in an otherwise tightly integrated environment.

*****

One other hint, by default JCAPS will not place messages on the queue for consumption until the JCD exits. This will not work for jcdUnitTester since it needs to send the message to trigger the jcdTarget then wait for the response. To overcome this, I've set the JMS queue/topic properties to "Transacted" and "Nonpersistent" instead of "XA" and "Persistent". I'm sure there are other ways and settings that will achieve the same result, but these have worked for me.

JCAPS Unit Testing - Part 1

2008-03-03T11:21:00.011-05:00

Since starting with JCAPS last November, I've looked for a better, automated way to test my code. During JCAPS training, the test exercises are kicked off by placing files into a "hot folder" monitored by a File eWay. It's very cumbersome to monitor the directory after placing the file - constantly refreshing the window, wondering why the file is not being picked up - or what is taking so long. In my company's environment, the JCAPS development server ran on a shared machine that was cumbersome to access - adding to the difficulty in dropping this file. Since most of our components listened on JMS queues or topics, dropping messages directly to these queues would've been an option, but these were not accessible outside the machine.

I knew there had to be a simpler, more elegant way to trigger our tests. What I needed was a way to drop a JMS message into the system and trigger my JCD. What I wanted was a servlet-like mechanism that I could trigger at will from a web browser to drop the test message into the system, then get the resulting message and return it to the browser.

I started to experiment with the HTTP Server eWay. One of the properties of the HTTP Server External Configuration (from the Connectivity Map) was servlet-url. The plan was connect to this URL to trigger the test. On paper it sounds great, but there are some tricks.

First, you need to know which port on the JCAPS Integration Server accepts HTTP requests. By default, the generic HTTP port is one more than the Integration Server Admin port (or +4 for the HTTPS port). For example, if the Integration Admin port is 18000, the HTTP listener typically runs on port 18001 (18004 for HTTPS). You can verify these values on your systems using the JCAPS Domain Manager.

Second, reaching the servlet is not as simple as typing http://localhost:18001/servlet-url. For some reason JCAPS also requires the deployment profile name, the string "_servlet_", and the servlet-url twice (sperated by a "/"). The URL should look something like this, http://localhost:18001/deploymentName_servlet_servlet-url/servlet-url. It took me a while to discover this, but once I did it works like a charm.

That's it! Now you can execute a JCAPS JCD from the browser. Using input.getResponse().setText() and input.sendResponse() you can send information back to the user.

Next, I'll describe how I integrated this approach with JUnit to remotely unit test my deployed JCAPS modules.

Crazy Lately

2008-02-29T11:03:00.006-05:00

Things have been really hairy at work lately. I haven't had much chance to write. Here's what I've been up to.

JCAPS

One of my goals this year has been to make JCAPS more usable. A tall order indeed. I'm going to hit on the details of some of these items in later posts, but here's a bullet list of some of the issues I've been working through.

My organization uses some non-standard x12 EDI messages. The way to get JCAPS to process these messages is by creating an OTD from an SEF file. I had no luck finding someone to help me create such a file and there are surprisingly few Java libraries that support these x12 messages. I wound up writing my own parser. It works pretty well and can be extended to support other message types.
I've developed an approach that provides an automated way to test my JCAPS collaborations using JUnit. I'll describe this approach in future posts.
The company I'm working for has a change control process that is completely foreign to me. I'm even questioning if it is an actual change control process. I'll describe this approach in a future post, what I perceive to be it's shortcomings, and describe how I think it should work.

Podcatcher

Lately, my free time has been pretty busy as well. I haven't made any progress on the podcatcher lately. I have decided to code the podcatcher in Java, using the prebuilt Netbeans plugins to create the user interface. Once I get something working, I'm going to pick up a copy of Filthy Rich Clients to dress up the interface. From a purely academic standpoint, I wanted to use Ruby and Air to get experience with these technologies. On the other hand, I want the tool to be usable. I think that given the unknowns and "newness" in creating a client-side UI in either Ruby or Air technologies, the Java approach is a safer bet.

Heavy Air

2008-02-16T06:55:00.005-05:00

I downloaded some of the Air samples and the first thing it wants to do is install the application on my machine. I expected an Air application would be fairly lightweight (like a client-side Java application), and as long as I had the Air runtime on my machine the application would run - no administrative rights required. Not so. And too bad.

Installing Eclipse is simply an unzip. There is no need to add entries to the registry or provide administrative rights. It's simple and that's the way it should be.

I wonder why Adobe didn't follow this model?

Podcatcher Prototyping

2008-02-09T12:44:00.000-05:00

I started this podcast project so I could learn some things I haven't had time to experiment with in my "real" job. Things like playing more with Ruby, trying out Behavior-Driven Development, and experimenting with Agile practices (plus I really wanted a better podcatcher program and I needed something to do over the winter). While I haven't produced much real code yet, I've created some simple prototypes and wanted to comment on some things I've been looking at.

Reading an RSS feed

My first prototype was a very simple Ruby program to read and parse an RSS feed. I had found some code here using the standard Ruby RSS Parser so that's where I started. At first I thought it was broken, but after about a minute and a half it returned. This wasn't going to cut it, so I started to look for alternatives and quickly found the feed-normalizer gem. Some quick coding and the total time to access the RSS feed was reduced to about 10 seconds. Not bad, but I still wanted better.

I was using NetBeans and knew that it used the JRuby (1.0.2) interpreter by default. I switched the NetBeans interpreter to the "standard" 1.8.6 release and reran my tests. Both tests came back in about the same time - a little under 6 seconds. Interesting.

While researching RSS, I came across this article describing conditional get. Neither of these Ruby libraries appear to support conditional get, but I found a Java library called Rome which does. Because of my UI concerns (more on that later) I was curious how a Java implementation of this functionality would compare to what I was seeing from Ruby. The Rome prototype accessed the feed in less than 5 seconds, with the conditional get taking a few tenths of a second. Now I had a decision to make, stick with the plan to use Ruby or switch over to Java?

Behavior-Driven Development

I was first turned on to the idea of behavior-driven development (BDD) back in October. My initial plan was to use RSpec with Ruby, but what BDD frameworks are available for Java?? As it turns out, there are several and I decided to try out JBehave and Instinct.

JBehave uses a "Given -> When -> Then" format to specify it's tests which looks very similar to the way RSpec does things. It does have a JUnit like test runner, but I couldn't get it to work from inside NetBeans (I used the Ant target instead). The Ant task didn't produce the green/red visual that I like with JUnit, but the tests did run. It looks as if the most recent release of this library was 12/2006 so I'm not sure how active this project is.

Instinct looks very much like JUnit. It uses annotations to define "specifications" rather than "tests" and "expects" instead of "asserts". Problem was, I couldn't get the example code's maven build to work from inside NetBeans. The most recent least was 12/2007, but it was for version 0.1.6 making stability and maturity a question mark.

User Interface

Because I wanted this podcather to be something I'd use, the user interface is something I've been thinking about as well. I've previously created an Eclipse RCP client application so I knew I could build the UI I wanted there. I wasn't sure I could build a "good enough" interface in Ruby using it's Tk bindings. As luck would have it a recent RailsEnvy podcast reminded me of Shoes, an alternative windowing toolkit for Ruby. Like Instinct, it's still fairly early in development and I'm a little concerned about it's stability.

Still undecided, I decided to take another look at Adobe Air. A while back, I looked into Flex as a framework to create Rich Internet Applications. What drew me to Flex, was how easy it was to create beautiful, interactive, user interfaces. Because Flex applications run in Adobe's Flash player (which is on virtually all browsers), these applications are very portable. Air brings that portability to the desktop, providing a runtime for applications to run in regardless of the operating system (similar to a Java virtual machine).

What's Next

Today I downloaded the Air plug-in for Eclipse and I'm going to spend some time playing with that. If it's as nice as I hope it is, I'll figure out what technologies it integrates with (Java or Ruby) and move on from there.

JCAPS Training Notes

2008-01-29T08:25:00.001-05:00

The JCAPS training I attended (Foundations of Java CAPS II) was excellent. We screamed through the course material in 3 days and had discussions on other JCAPS topics the other 2 days. Here were my impressions.

I enjoyed learning about the eInsight Business Process Manager. The tools and steps involved in creating a business process are very similar to those used to create a JCD. There are a lot of interesting constructs like correlation and the "flow" element, which easily enable parallel processing. One frequent topic of discussion, was when to use an business process (eInsight) rather than a JCD. There's a certain flexibility to creating a business process, gained at the cost of speed. Since you can do virtually everything in a JCD that can be done by a business process, we questioned when each was a better fit. We didn't reach a consensus and I'm sure we'll be talking about this again.
We explored the shortcomings of the repository and the best way to manage releases. In the short term we are going to rely on branching, which is kind of a pain because the entire repository must be branched. Longer term we are waiting for the 5.2 release, which will allow us to move to a different repository version.
Speaking of Java Caps 5.2, it should be released later this year with a beta version coming at the end of this quarter (1Q2008). eDesigner should see the biggest changes. As I understand it, it'll be based on the Netbeans 6.0 release. I'm hoping that the JCAPS functionality will be provided as a plugin to this IDE, keeping intact all other Netbeans functionality. This should allow a more natural way to create subprojects (JARS or 3rd party libraries) which can then be easily integrated into JCDs and other JCAPS components.
We took a quick look at eView which is an interesting utility providing the logic to form a single customer view from information silos. We've been facing some performance and maintenance problems in this area so it was nice to get some insight.
Unit Testing is still a bit of a question mark. The instructor pointed us toward JUnit and JMeter, but I'm having a hard time figuring how to call JCDs from these tools.

Overall, a great week learning!

Podcatcher - Sprint 1

2008-01-20T12:19:00.000-05:00

I've made embarrassingly little progress on my podcatcher so far. My plan was to entirely read Lean Software Development before getting started, but it's taking me a while to get through the book - in a good way. I seldom read more than 5 or 7 pages before I find myself reflecting on current or past projects. This knowledge could've saved me time on previous work by making the most important & well defined features a priority - delaying discussion and decisions on the "nice to have" features until later. Some of these ideas I've stumbled on by accident or heard about from others, but it's great to have them validated and expanded into this set of lean thinking tools. I'm sure this is a book I'll reread many times in the coming years.

Anyway, I want to revisit my requirements list for the podcatcher, prioritize the features, and define the features for my first sprint. Here are the requirements for the podcatcher from a previous post, roughly ordered by importance.

Subscribe to podcasts via RSS.
Download only show descriptions (if this is not the default). Otherwise add functionality to download the entire show. Make this configurable by show.
Delete podcasts (including descriptions).
Build user interface
Import OPML files from iTunes.
Ability to set the genre (or other metadata) for the shows.
Enable playlist creation
Allow the creation of a "favorites list" with user notes or labels and a URL to the show.

The overall goal is to have the podcatcher "live" on the MP3 player or other removable drive so keeping the footprint of the application small will be very important. I'll be monitoring this very closely.

One other note, the user interface is something I've been thinking a lot about lately. I'd like the look and feel to be a native as possible to where the application is running. I know Ruby has a Tk extension bundled with it and I've found some other options, but none look as robust as the Eclipse RCP platform I've worked with previously. I really want to code this in Ruby, so an option may be to use JRuby, with Eclipse for the final solution. I'm going to keep looking and will try out a bunch of options to see which is the best fit for what I want.

Here's what I hope to accomplish in the next week or two, in the first sprint

Download, install, and start to learn RSpec. RSPec is a behavior driven development framework that I'm hoping to use to describe and test the behavior of the podcather.
Define behaviors for downloading podcasts via RSS.
Code these behaviors.

Is Orchestration the new Web 2.0?

2008-01-18T18:35:00.002-05:00

In recent SOA discussions with people at work, the topic of orchestration often comes up. The problem is, like Web 2.0 before it, the meaning of what an orchestration (or orchestration layer) is differs depending on who you're talking with.

In my view, an orchestration simply refers to the process of integrating disparate applications. This idea is obviously not new. It's just a new name for an RPC or ESB-layer but relies on HTTP for communication instead of CORBA or something like JMS. This view is supported on Orchestration's Wikipedia entry which describes an orchestration as

the process of coordinating an exchange of information through web service interactions.

Speaking of the Enterprise Service Bus, it seems there is some confusion whether the "orchestration" lives at the same layer of the ESB or at a layer above the ESB and helps to coordinate ESB communications. I fall into the first camp, believing that "orchestration" is just a new way to do ESB. Wikipedia to the rescue again

the ESB is the piece of software that lies between the business applications and enables communication among them

Because of the confusion in defining these terms, it has been hard for the organization I'm with to gain much traction in implementing their SOA solutions. I'm sure we're not alone...

Web Service Discussion

2008-01-16T20:15:00.000-05:00

Again and again the topic comes up at work about fine vs. coarse grained web services. Is anyone else talking about this? I guess I don't understand what the big debate is about.

I think of a fine grained service as implementing a very specific, atomic operation. Course grained services implement larger, multistep operations... possibly consuming other services.

The way I look at it is this: a business has some piece of functionality it wants to make available for others to use (the service). It shouldn't matter if the steps performed by the service can be broken into smaller components. If no one can use these intermediate (finer-grained) components, the energy spent defining, building, and maintaining this (unused) functionality is wasted. My position is that it's better to wait. Should the need to reuse some of the functionality arise, then refactor the common functionality into a smaller, reusable service.

Am I thinking about this wrong? What am I missing?

JCAPS Training II

2008-01-14T08:18:00.000-05:00

I'm back at JCAPS training this week. This time the focus is on the eInsight Business Process Manager. eInsight is the JCAPS architecture layer used to combine web services. The services are connected using BPEL which is an XML-based language to define business processes. The latest version of NetBeans also includes a BPEL tool, and I'm anxious to see how they compare.

We have some time scheduled at the end of the formal training to explore some of the JCAPS problems we've been experiencing. In addition to the problems I've already outlined with the repository. I'm hoping we can also explore the following:

Lately, we've seen some weird behavior where things that were previously working stopped. The errors in the logs don't seem to make any sense. At first we thought it might be due to some code that was copied to a different location, but I've seen this problem where no code was copied. The only remedy we've found is to recreate both the connectivity map and the deployment profile.
I'm having a hard time following "good" programming practices in out JCAPS architecture. Most of our code resides in JCDs. Some of which do a lot of processing. The JCDs don't readily lend themselves to a very modular architecture. Class hierarchies are not possible, making it hard to implement many design patterns. I'd also like to leverage something like Spring or JUnit, but can't figure out how to integrate these into the tool.
Any debugging or profiling tips we can pick up. Right now, we rely on the logs and it's tedious.

In any event, the BPEL training should be useful. If the instructor can shed any light on the problems we've been having, that will be a bonus.

Podcatcher - Requirements

2008-01-06T19:43:00.000-05:00

One of the goals I set for myself in the New Year was to gain experience with Lean/Agile software development. My test project will be a fairly simple podcatcher to manage the podcasts I listen to. Here is the initial set of requirements for this tool...

Subscribe to podcasts (obviously). All of my current subscriptions are RSS, so I'll focus on this first. Once I have something functioning, I'll add support for Atom feeds. It will also support importing OPML files. This is the format exported by iTunes.
I currently use iTunes to manage my podcasts and the default is to retrieve the latest podcasts on startup. This is OK if you generally listen to all episodes you subscribe to, but for some subscriptions, I like to pick and choose the shows I want to listen to. I know this behavior is generally configurable in iTunes, but I'm not sure this can be individually configured per show. The default behavior for my podcatcher will be to download only show descriptions, with the option to automatically grab the latest shows. This will be configurable by show.
Ability to set the genre (or other metadata) for the shows. This will provide a greater ability to automatically group shows I'd like to listen to.
Playlist creation - a manual way to group shows together for listening.
Allow the creation of a "favorites list" with user notes or labels about the show. This will offer a way to bookmark the URL of favorite episodes along with some notes and tag information. If you want to relisten to an episode at a later time, the episode can be easily downloaded again.
I'd like the podcather to "live" on the player (or any portable storage device). I've switched computers a lot lately and each time I import my OPML file into iTunes, I've needed to manually check for new episodes not downloaded by another iTunes installation. Also, since I've never had an iPod, syncing with the player has always been a manual process. By living directly on the mp3 player, these synchronization issues won't be an issue. Even when switching mp3 players, the podcatcher software can be simply copied between players. This requirement requires that the software footprint be as small as possible.
And finally, delete podcasts (including descriptions).

I guess that's it. On the surface this looks like fairly simple and straightforward project with a useful set of features. I think the next "Lean" thing to do is to prioritize these requirements and set the minimum criteria set needed to meet these requirements, but I'm going to start reading to verify this approach.

2008 Resolutions

2007-12-31T07:38:00.000-05:00

Making resolutions is tradition on New Year's Eve.... evaluate areas you'd like to improve (or new skills you'd like to acquire), and set goals to improve in those areas. Here are two work-related resolutions I'm going to focus on entering 2008.

Make JCAPS Usable

If you've been following my posts, you're aware that I've been frustrated with Sun's JCAPS product. The tool is simply not ready for prime time and it's hurting my productivity. In addition to the problems I've already outlined, developing in JCAPS' eDesigner makes me "feel dirty". You need to go outside to tool to create modules of shared code (jars). This limitation makes it hard to implement classes that follow even the simplest design patterns or integrate with third party frameworks, like Spring. I feel myself falling into bad coding habits.

I'd like to find a compromise. What I'd like to do, is move my code out of the JCAPS JCD and into a separate web service that runs directly on the shared JCAPS application server. The JCD would call this module to do all the heavy lifting, then return to the JCAPS JCD. JCAPS would be used primarily for mapping TCP, MQ, or FTP requests to the web service.

Practice Agile Development

The past year introduced me to many new technologies and ways of thinking. Two I've been most excited about are Ruby and Agile Software Development. Since I'm currently not using either in my "real work", I'm going to use these tools to create a relatively simple and lightweight podcatcher. I'll outline my requirements, approach, and progress in the weeks to follow. I'll be using the Poppendieck's Lean Software Development: An Agile Toolkit as my guide to an Agile approach and the Rspec framework to define and test the behavior of the application. It should be fun and give me some great experience!

Happy New Year!

Vista Woes

2007-12-04T10:40:00.000-05:00

I recently purchased a new laptop preloaded with Vista (Home Premium). Given the choice I'd have preferred to stay with XP, but I had more than enough hardware to support it, so I thought "what the heck". Now I feel like I'm in one of those Mac commercials. Here are some of the issues I'm facing.

When I first booted up Vista, I made the mistake of setting what I intended to be my personal user ("Vinnie") as the Administrator. I was told this is not a good idea for security reasons so I renamed the Administrator account to "Admin" and created a new "regular" user reusing the "Vinnie" account name. On the surface everything appeared OK until I wanted to find the files associated with my account. There are user folders for "Vinnie" and "Vinnie2". The "Vinnie" folder corresponds to my Administrator account. The "Vinnie2" folder corresponds to my "regular" account. When I renamed my Administrator account, why didn't Vista also rename the user files associated with this user?!? Dumb and annoying. I made a mistake, shouldn't the software be able to help me recover?
I have a fairly popular Sansa mp3 player. When I connect it to Vista, I cannot manage all the music on the player. Many of the folders and files are not visible for some reason. I can see all the files if I reconnect the player to an XP machine. I can also access the files in Vista if I manually type in the folder and file names . What gives? In frustration, I accidentally "bricked" my player trying to resolve this issue. I'm hoping I can get the player back in working order with Rockbox.
On XP, I used Photoshop Elements to categorize my photos. Since Vista has a "Compatibility Mode" I was hopeful this wouldn't be a problem. (Insert manual buzzer here) Wrong!

I sill haven't been able to read in my Photoshop catalog on Vista (again it reads in with no problem on a XP machine with a clean Photoshop install)
The application has many odd display artifacts and Adobe Update causes the entire application to crash.
Every time I start the application I get a message telling me that I do not have Administrator privileges and some functionality will not work correctly. There is no way to disable this message, and running as Administrator poses additional annoyances (see below).

You can set up applications to run with Administrator privileges. Unfortunately every time you start the application you need to provide the Administrator credentials. Shouldn't there be a way to say:
"Hey this application will not run how it's supposed to without Administrator privileges. I know this is not the ideal way to run, but there's no other way. I trust this program, so don't ask me for the cotton picking password everytime I need to run this app!"

I've also noticed that when I need to provide Admin access to install an Application, those credentials carry over to when I start the Application, placing any user information into the "Administrator" user folder instead of my "regular" user folder. Am I now running as the Administrator? Shouldn't the Administrator credentials only be needed approve the application install? What a joke! (This behavior goes away if I log out, then log in again.)

Apple is winning over the "average PC user", because their products simply work. I never hear an Apple (or iPod or iPhone) user complain of the problems I'm facing with my Vista PC or mp3 player, or cell phone. And again, it's why they can charge a premium for their products. For Apple it's not about the number of functions, it about the functionality.

More JCAPS Impressions

2007-11-21T07:50:00.000-05:00

I haven't had much chance to post since starting my new position. Getting up to speed in my new role has kept me busy. I have been using JCAPS though and thought I'd post a quick update on my impressions.

Basically the good is still the good, but my team and I are experiencing more pain points, mostly due to the poor repository integrated into the tool. Here's a quick list of things I'd like to see improved.

I've gotten in the habit of diff'ing my changes against the repository version to ensure I'm only checking in changes I've intended. JCAPS does not allow you to do this. There's also no easy way to diff changes between different repository versions. You need to be creative, and go outside to tool for this functionality.
There's no easy way to find all the files you have checked out of the repository. You need to manually keep track of the files you've touched. I've become accustomed to relying on my development tools to keep track of this for me, so this is a pain.
We're having a hard time coming up with a good maintenance plan. The repository does offer tagging and branching, but these terms in a JCAPS repository seem to mean different things than they do in CVS. When you tag a set of files for a JCAPS release, there does not seem to be way to grab all these files as a single unit. In JCAPS, a tag is more of an informational item, like a comment. Branching seems to be more like CVS tagging and branching combined (versioning a set of files together and setting up an alternative development path). Missing is the ability to merge into the trunk.
Aside from the repository, we've been having problems with things as simple as generating javadocs from the the JCAPS Designer. Again, we're forced to go outside the tool to get this functionality.

The more I use JCAPS, the more my wish list for new features grows. I'm hoping the next version of JCAPS will address these problems, but I've been unable to find much information about the upcoming enhancements or a release date. At this point I'm finding JCAPS helpful for quickly laying the interface plumbing (connections to a DB, web service, queue, etc) between applications (or application components), but painful for almost everyhting else.

JCAPS Training

2007-10-31T20:24:00.000-04:00

I've been in JCAPS training all week, and so far my exposure to the product has produced mixed feelings ....

the good....

JCAPS is definitely meant for enterprise-class application development.

The architecture makes connecting to files, messaging queues, and databases trivial. It's scary easy.
Things like monitoring and deploying to multiple environments are baked in. Again, it's nice to not have to worry about developing or searching for tools to manage these tasks.
The JCAPS tools also produce some nice pseudo-documentation (diagrams and high level descriptions depicting what the application is doing and how things are connected). Some may think this is a little cheesy, but I like the idea of getting a high level picture of what is going on without having to surf through multiple source and configuration files. Having some built in transparency is nice (especially in large systems).

the bad....

Is also the UGLY...

The JCAPS is built on NetBeans 3, which is nothing like the version 6 release I'd been working. I feel like I've taken a bad trip back in time. I haven't been able to find a lot of the cool tools I've gotten accustomed to using (i.e. debugging & refactoring tools). The good news is that tighter integration with a more current version of NetBeans is supposed to come with JCAPS 6. I can't wait.
To produce some of the cool diagrams I mentioned earlier, you need to follow some "conventions" in your code. Most of the time this means using fully qualified class names in method declarations (the tool will not recognize import statements). This makes the code overly verbose.
JCAPS provides some web-based management tools. These only work in Internet Explorer :-( The JCAPS Enterprise Designer only runs on Windows.
JCAPS uses an internal (antiquated) version control system. There's no way to plug it into an existing CVS or Subversion repository. Again, I'm told this functionality should be coming soon.

Impressions so far...

JCAPS is a framework that helps make creation and maintenance of enterprise applications very easy. At this point in time the tooling is poor, but upcoming releases should address this.

JCAPS

2007-10-23T09:52:00.000-04:00

In the midst of a job change, I've been spending most of the last few weeks handing my work off to others and writing some final documentation. With that almost complete, I'm turning my attention to my new position where I'll be working to integrate systems together in a Service Oriented Architecture (SOA).

While I have SOA experience (creating & integrating web services as well as CORBA modules) my new company has chosen the Java Composite Application Suite (JCAPS) as it's SOA platform. It's (by far) the most comprehensive SOA solution I've had opportunity to work with. I'm really excited to get started. Here are some things I really like about the architecture and questions I have.

The system is built on an application server and includes things like messaging, as well as consolidated user management and role based access built in. This is a big change from my current situation where I've needed to integrate (and implement) a lot of this functionality myself.
The next layer of the architecture handles integration. Integration with internal services, external services, and legacy systems. This layer also handles the task of single sign-on for users to the various system components.
The next layer is called the eInsight Business Process Manager. It's the part I'm most excited to learn more about. This layer leverages languages like BPEL to allow business leaders to create, model, test, and deploy their own business processes. Delivered via the integrated portal server, I'm envisioning something similar to IBM's QEDWiki. It's been a while since I've worked with portal technology so I'm curious as to whether the JCAPS portlets can deliver a similar experience to QEDWiki's integrated components (wired widgets).

As I learn more about the JCAPS platform, I'll be sure to post more.

Epcot

2007-10-10T15:48:00.000-04:00

NOTE: I realize this post might be a little "out there" for a Tech Blog, but I've been fascinated with Disney since visiting there last Spring. How they provide such a quality experience to so many people is an inspiration. Please indulge me...

On October 1st, Disney's Epcot celebrated it's 25th anniversary. From the beginning, Epcot's goal was to provide people a place to explore the future and the world around them. While learning a little history about Epcot and the various pavilions, I've noticed a few themes common with those I've been exploring.

Mickey Mashup??

In it's original form, the Journey Into Imagination featured Dreamfinder, an interesting character whose goal is

collecting the stuff dreams are made of.... sounds, colors, ideas... anything that sparks the imagination. And I store these sparks... and recombine them into new ideas and inspirations.1

When I heard Dreamfinder describe his ambition, I immediately thought of web services and how these "sparks" of information are also combined together in the form of a mashup to provide new and interesting insights and understanding to the information we have. It's the reason I'm so excited about this technology.

Agile Disney??

In the early days of Disneyland, I've heard that Walt Disney himself would sometimes stroll the park asking guests how they were enjoying the park or ideas of what new attractions they might like to see in the park. Today, Disney makes extensive use of surveys and taps into a very active online community. Keeping in constant touch with it's guests, Disney is not only able to gauge what it's public wants, but what it's willing to pay for. This is similar to the Agile principle of staying in touch with your users to determine and prioritize feature requests.

Some other agile thoughts I've been able to pick up on are:

Not overbuilding. By not building attractions which would generate little public interest, Disney saves a lot of time, money, and valuable real estate.
Continuous improvement. Disney is always looking for ways to improve it's current attractions. Often this means making tough decisions to upgrade to replace popular attractions (Horizons is an example of an attraction that is missed by many Epcot fans, but was replaced by Mission Space which is widely popular)

When it opened in 1982, Epcot introduced the wider public to the power and potential of computers, helped pioneer the "live via satellite" news coverage that is prevalent today, and gave a broader look into the future of transportation and exploration (space and sea). Today Epcot still strives to educate the public on the cutting edge of conservation, technology, and world awareness. Amazing!

Behavior Driven Development

2007-10-09T12:18:00.000-04:00

If you haven't seen some of the Rails vs. _____ (Java, ColdFusion, .Net) links, I made available in the "Related Articles" section, you should check them out. They're really clever. The guys over at RailsEnvy do a great job with these.

They've also released a presentation about testing, introducing me to Behavior Driven Development which is like Test Driven Development, but with purpose. Instead of testing individual functions, Behavior Driven Development tests specific user actions and the result of these actions. The result of creating these behavior driven tests is a specification for how your application will perform... with a final application behaving in this manner after all tests pass. For Ruby, there is a framework called RSpec which can help facilitate this process.

For a while I've wanted to adopt a more test driven approach to my development. There are a lot of benefits: you only build what you need (don't overbuild), you know what you've built works (and can easily prove it), and you can test that future enhancements will not break any existing code. Unfortunately while I have started this process a number of times, I have not been diligent or disciplined enough to make it a habit... believing that the realities of deadlines didn't allow me the "luxury" of creating more automated tests. Hopefully with new inspiration from this presentation (and with the help of RSpec), I can finally make it stick.

Grails, Groovy, & some other stuff

2007-10-08T09:39:00.000-04:00

As the name suggests WebDevRadio is a web development podcast. I'm a big fan of this show. It always delivers an interesting perspective, and the discussions often relate to a technology I'm also exploring.

In a recent episode, the host Michael Kimsal interviewed Jason Rudolph about Jason's new book Getting Started with Grails. I've written before about Ruby and Rails and using something like JRuby to leverage Java's stability and scalability in a production environment. Grails (and Groovy) is an interesting alternative to the Rails/Ruby/JRuby I'd been thinking about. Like Ruby, Groovy is an interpreted language but with some constructs that make it easier for Java developers to transition over. Also, since it's written for the JVM, it gets all the Java maturity "for free" and naturally integrates with existing Java modules (No need for something like JRuby).

Grails brings the Rails goodness to the Groovy platform... including a lot of the tools (scaffolding, included persistence layer, etc.) and concepts (convention over configuration) that make it easy to quickly get something up and running in Rails. I'm going to spend some more time checking it out.

This was a great conversation. They also touched on:

No Fluff, Just Stuff. These sound like great conferences to learn and explore new and cutting edge ideas and technology
Streamlined Framework. This framework adds some productivity enhancements to Rails and ActiveRecord.
Solr. As I understand it, Solr adds a web service front-end to Lucene. I had done some previous work with Lucene and this podcast series had introduced me to Solar. Should I ever need to revisit that work, I'm going to keep this project in mind.

Restlet

2007-10-03T08:44:00.000-04:00

In my last post, I described I was implementing a SOAP web service controller between my Ruby chat bot and a CORBA component my company is developing. The WDSL for this service was straightforward, and with help from the Eclipse Web Service wizard, it was complete in no time.

I had some downtime before the CORBA module I needed to connect with was complete, so I began to look into re-implementing my controller as a REST web service. A little digging led me to Restlet. Restlet is a lightweight framework for developing REST web services in Java. I've seldom had an easier time integrating a new Java package and getting something up and running.

Restlet comes bundled with it's own HTTP server, but I opted to deploy a WAR file to our existing Tomcat installation. There are three basic steps needed to get this to work.

Creating a Resource is the first step. The Resource is the brains of your service. It captures request parameters (from the URL or encoded in the body), then takes the appropriate action dictated by the HTTP method (GET, PUT, POST, or DELETE). The Resource can allow the user to specify a preferred format for for the response (html, text, xml, etc.) and it determines which HTTP response code to return to the caller.
The next step in implementing a Restlet application is to create a Restlet Application class. This class must implement the createRoot method. This method maps incoming requests to your Restlet resources (similar to JSF navigation_rule or Ruby's routes.rb file) .
The final step in creating the REST service is link your Application to Restlet servlet in your web.xml.

Testing this service was kind of fun. At first I used a web browser to submit various requests, but I finally integrated this with my Chat bot using the rest-open-uri Ruby gem described in RESTful Web Services (which is a great book).

Ruby, XMPP, and Soap Web Services

2007-09-14T10:40:00.000-04:00

I've spent a lot of time away from the office lately and haven't had much of a chance to post. Here's a quick update on what I've been thinking about.

First off, the Ruby XMPP-bot I had started is complete. This was insanely easy to implement in Ruby. The only snag I ran into was in creating a new chat room with the bot. The 0.3.1 release of XMPP4R seems to hang when creating a new room. After some research, it was apparent that others experienced this problem and had already submitted a patch. Cool.

My next job is to connect the bot to an application my company developed. Communication to this module has been done previously using CORBA. Unfortunately (or not), I have not been able to find an acceptable Ruby CORBA module (Rinn looks like it's dead and R2CORBA does not appear mature enough). Recently many of our legacy CORBA API's have been replaced with Web Service API's and I'm going to take this approach with this module.

While researching for a Ruby SOAP library, I came across this post describing an integration with SalesForce.com.

I've heard a lot about SalesForce.com and knew it was a CRM solution, but never took an in depth look at all it offered. In addition to all the customer, sales, and marketing tools, SalesForce.com includes integration with many popular third party applications (like Outlook), single sign-on capability for corporate users, and a variety of other utilities. This allows users to seamlessly interact with the SalesForce suite, using their existing work flow rather than being forced to adopt the "SalesForce Way".

What I was most impressed with however, was SalesForce's developer tools (web service API) and it's AppExchange. AppExchange allows third-parties to extend SalesForce functionality though custom application plugins. The developer API allows businesses to roll their own solutions when needed. By acknowledging that they'd never be able fill every business need, and accepting help from it's community to do so, SalesForce has rapidly become the hottest business application around.

QEDWiki

2007-09-04T09:37:00.000-04:00

I've been hearing a lot of buzz about QEDWiki, which is being developed at IBM. QEDWiki is a basically a wiki that can host simple web components. At first glance, QEDWiki looks like a typical portal, displaying aggregated information from around the web in widgets. Think dashboards like PageFlakes or My Yahoo.

I soon learned that QEDWiki is something completely different. My perception changed when I saw these widgets linked together, creating what IBM calls "situational applications". A "situational application" is a simple applications created on the fly using information provided through a web service.

These applications can be created by business users, for business users - without the support of a development team - very much like the advanced Excel applications created and shared today by business teams. I see something like QEDWiki taking the place of these apps with Service Oriented Architectures (SOA) playing the role of the Excel "sheets".

Because QEDWiki is build on a wiki framework, multiple users can collaborate and enhance individual pages - and even individual widgets. To facilitate sharing widgets, IBM has also developed Mashup Hub to categorize and store these reusable components.

I'm interested to see how this technology matures and if it's indeed leveraged by business users.

Ruby Jabber

2007-08-31T08:51:00.001-04:00

The Hibernate tweaks I made a few weeks ago made our application "fast enough" for now. So I'm back off of Rails and I've started work on a XMPP (Jabber) bot to monitor chat channels. For those unfamiliar, XMPP is an open protocol for instant messaging.

I've just gotten started and, again with Ruby, my progress has been quick and easy. After installing the Openfire XMPP server and XMPP4R Ruby Gem, I was off and running. With help from someone who's done this before, I was able to communicate with the server very easily. I'll be sure to post progress and I get deeper into this.

By the way, if anyone's been following my Rails progress, I'm sure I'll be back to finish this at some point. As I've written before, the database design is flawed and will not scale well. We've just bought a little more time before these issues bubble up again.

OLPC and Free Wireless

2007-08-29T08:22:00.000-04:00

I was glad to hear mention of OLPC (One Laptop Per Child) in a podcast I listened to early this week. OLPC (a.k.a. the $100 laptop) is an organization whose goal is to provide inexpensive laptops to children for education.

Integrated collaboration is one of the key features offered in these laptops. The belief that information and ideas are meant to be freely and openly shared is a core principal of open source software... and it's revolutionized the software industry. Allowing children to learn the value of this openness firsthand, will hopefully lead to a more open and collaborative environment in other fields (government for one) when these children enter the workforce as adults.

To help foster these ideals away from school, children will need access to the Internet at home. Fast and cheap, widespread Internet penetration needs to improve. In the neighborhood I live there is one broadband provider and it's rather expensive - making it out of reach for many lower income families. I'm sure my neighborhood is not unique. Another podcast talked about Google and it's success installing a free Wi-Fi network in Mountain View, California.

The barrier to both OLPC and more free Wi-Fi Internet access in the US is, of course, Government. OLPC will only sell laptops to governments (state or federal) and installing free Wi-Fi (even when Google is footing the bill) requires cutting a lot of red tape. In the upcoming Presidential election, I'm looking for a candidate to address these issues and propose a plan to keep our kids on the cutting edge of innovation and education.

Back Riding Rails

2007-08-20T08:48:00.000-04:00

Thankfully, I'm done working with Hibernate for a while and I'm back on Rails.

Since I posted on FRail :include, it seems there has been a lot of plugin development to add this functionality. While all of these plugins help to address the problem, I'm having a hard time finding one to address my specific scenario.

I think I've explained my situation before, but here it is again in a nutshell. The company I work for aggregates a lot of data. The goal of my application is to allow users to sift out the information they don't need, discovering nuggets of truly useful knowledge. The approach we're taking is to have users to start from something they're interested in, and use that object to find other objects with similar attributes and characteristics. What I really need from Rails is a robust way to join a lot of tables together and specify search criteria.

I've decided to roll my own solution for now. What I'm developing in Rails will take in one or more "objects of interest", and use it's attributes to search the DB. Right now, I'm manually creating the join strings in order to get something done quickly. Eventually, I'm hoping to develop a more robust solution (or one of the ":include" solutions evolves to match my needs more closely).

Surprisingly, this solution has not taken as long to implement as I originally thought it would. That's the joy of Ruby on Rails!

TV Innovations

2007-08-16T08:18:00.000-04:00

I love TV. Granted there's a lot of garbage content, but there's also a lot of great content. Since the VCR, it's been possible (although clumsy) to time-shift your TV programs - watching the programs you want, when you want. Things like TiVo and DVR provide a cleaner interface to this functionality while adding useful new features like "pause live TV".

The problem with these technologies is that you need to know the programs you want to watch in advance. If you've just discovered a new series and want to watch previously aired shows, you''ll need to wait for the TV network to rebroadcast (or for the episode to be released on DVD or iTunes).

Enter Move and Joost. These new products offer the promise of time-shifted, on-demand TV, offered over the Internet. Both these products allow the owners of the content (networks) to control distribution, so there's no YouTube-like copyright worries. Joost works more like a file sharing system (think Napster or LimeWire), while Move relies on a smart client to keep the video streaming. Move provides HD streaming right now for ABC and FOX, with Joost's offering currently in beta (I'm eagerly waiting for my invitation).

As this technology evolves, I'd look for local governments and organizations to provide video of their meetings over a similar feed. This could, for instance, help parents "attend" school board meeting they would've otherwise missed because of a conflict with their child's soccer game.

Hibernating

2007-08-13T10:29:00.001-04:00

Migrating my application from Hibernate to Rails has taken longer than I expected. An important client has expressed performance concerns with their release, so I spent the last week looking for ways to tweak the current Hibernate implementation. It has been very frustrating.

I don't like to cast blame or plead ignorance, but prior to this exercise, my experience with Hibernate has been limited. The module containing our Hibernate calls was authored by someone else and I've been mostly a client of this module. It wasn't until I looked at our query log in detail did I notice how poorly this module was written and how confusing Hibernate is.

When seeing the large amount of queries in the log (often for data not needed in the current view), my first thought was, "we must have a problem in our models... something is causing Hibernate to eagerly fetch related data we have no immediate need for."

I went through our models and, indeed, I did find some simple changes where "set" declarations did not have the "lazy" flag set to true.

I had to dig a little deeper to uncover our "many-to-one" relationship problems. Hibernate's "many-to-one" declaration does not allow for a "lazy" parameter... at least not where the relationship is declared. Instead, the "lazy" option needs to be declared in the "class" declaration of the related object?!?! That seems a little backward to me. Confusing things more, declaring "outer-join" in the "many-to-one" relationship does not tell Hibernate to use an "OUTER JOIN" instead of an "INNER JOIN". No, instead it directs Hibernate to ignore the "lazy" class declaration and eagerly fetch the "many-to-one" relationship!?!? For some reason, we actually had this in our model definition.

The initial performance gain I saw was unbelievable! The application had never run so fast. This was despite the fact we have some structural problems with our database (missing indexes, non-normalized data, etc.), causing full table scans for some queries.

The world was right again... until one page access caused the dreaded "org.hibernate.LazyInitializationException: failed to lazily initialize a collection - no session or session was closed". We needed data for one of the relationships that I had set to "lazy".

We had seen this error before, and thought we had licked it. We created a ServletRequestListener that opened and closed the Hibernate session for the request. It seemed to fix the problems we were having at that time.

I started a long session of debugging and it looks as if the shared session is being used for all requests. Also interesting is that, while I changed a number of relationships to lazy, only a single one is failing.

So far, I have not been able figure out what is causing the exception for this particular relationship. The original author of this code doesn't have a clue what the problem can be. My only guess is that there is some code in the Hibernate tree attempting to access this nested data using a different session. I'm going to continue to scour the Internet for any clues.

Times of Change

2007-08-08T08:43:00.000-04:00

Last night, Nova featured the life and contributions of Alberto Santos-Dumont, an early inventor of airplanes and dirigibles. In addition to Santos and the Wrights, the early 1900s brought forth other great inventions from Edison and Tesla, and marked the rise of the automobile... things that are cornerstones of our lives today.

I've always looked back at this time period in wonder and amazement. In a short period of time, the world was turned upside down. The way people travel, communicate, and got their news was forever changed.

I'm happy to be, 100 years later, living in a period similar of similar innovation. The proliferation of Internet has revolutionized the way people meet, shop, and communicate. Innovations like social media and RSS allow individuals to take a more active role in media they consume (not only what, but when). And it seems we're finally making strides identifying clean, renewable energy sources.

It's my hope that the contributions we're making today will still be relevant in 100 years.

Technology Shakedown

2007-08-06T07:54:00.000-04:00

David Berlind recently started a new podcast series entitled "Technology Shakedown". In it, he exposes technology that isn't as good as it could be. He talks about products that work great 80% of the time, but provides major headaches the other 20% - when the average user can't figure out why the product is misbehaving and must spend time concocting convoluted workarounds. A much better description can be found here.

Apple has rebuilt it's business capitalizing on the shortfalls of mediocre technology products. A superior experience is the reason people will shell out $150 for an iPod compared to $30 for a generic MP3 player. And it's the same reason Apple can charge a premium for a Mac compared to the average Windows computer. People love Apple because their products simply work the way they're supposed to - making their users lives more productive and enjoyable. Why more product managers don't comprehend this is beyond me.

*****

The discovery of this series dovetails nicely with my post yesterday about Rails. It not that I don't appreciate all the functionality Rails provides, but I'm finding myself tinkering with aspects of the codebase that should work out of the box - and it's hurting my productivity.

FRail :include

2007-08-05T10:40:00.000-04:00

I really like developing with Rails, but I've been frustrated the last few days by some things Rails should do quite easily, yet doesn't. I've already written here and here about some of my struggles with Rails :include clause. Here's one more...

Let's say you have a has-and-belongs-to-many relationship. If you :include this relationship in a find query, Rails will automatically create the join clause including the lookup table! That's cool.

Unfortunately, it looks as if only the :include clause provides this piece of functionality. Why isn't the same functionality provided via the :join clause or through a general purpose module? This doesn't make any sense to me!

I'm working on a database application where I don't need to display a lot of data, but the data displayed is heavily filtered by it's connections to other information in the database. I need to join a lot of tables to perform the filtering operation and using :include would be the most elegant way to express my queries. But to :include all the data from the join tables in the response kills the performance of my application (because :select is ignored).

Oh well. It's back to hand coding queries for me. At least until the powers that be agree this is much needed functionality (or someone creates a cool plugin I can use).

Freebase

2007-08-02T10:05:00.000-04:00

I wanted post a quick note about new technology I heard about today via podcast. The technology is freebase created by Metaweb.

As you might guess from the name, freebase is a free database for the web containing all sorts of information contributed by users. In other words, it's a huge Wikipedia-style database that anyone can contribute or query to find interesting (and possibly little known) information and relationships between seemingly unrelated objects.

For instance, assuming the information is in the database, I could query to find "all wooden roller coasters in the northeast longer than 5200 feet." That's pretty cool. Assuming I found some results, I could whittle down the set further, looking for which roller coasters have been ridden by Rock Stars or with lower height restrictions (so I could ride with my daughter).

Because the data can be contributed by the average person, the breadth and depth of the information to search for will grow quickly (and hopefully be of high quality).

The concept is great! Freebase is in pre-Alpha right now and I've signed up for an account to try it out. Once I get in, I'll let you know what I think.

wRESTling with REST

2007-07-30T15:36:00.000-04:00

I'm still struggling with the decision to create a REST or traditional web service. Over the weekend I started reading RESTful Web Services and I'm now thinking that a RESTful approach is very doable. Here are my thoughts:

REST resources are inherently stateless and shareable. All the information needed to reconstruct the view of a resource exists in the URI. Although I listed this a a "pro" before reading the book, I'm more convinced of the benefits now.
I had a concern about long query criteria in the URI, but I'm thinking of adopting the Ruby-like approach of referring to ids. The RESTful authors prefer a more human readable URI (as it is self documenting), but many of my filter criteria cannot be accessed through unique strings.
I believe I was trying to do too much at the beginning (not keeping it simple). My object relationships are complex, with most objects linked to several related (non-subordinate) lists of objects. I wanted to have a single interface with options to include specific subsets of data as well as page through the results.

As I thought more about this, I began to think that these subsets should actually be separate URI requests. Clients will probably make more requests to the service for data, but underneath, my Ruby controllers were making multiple requests to the database anyway. Here I'm hoping that the additional overhead of multiple GET requests will be balanced by the simplicity and usability of the interface (and no SOAP overhead).

As I become more familiar with the concepts of REST, I'm getting more comfortable thinking of my objects in terms of URIs and resources opposed to RPC calls and parameters. It's kind of similar to the shift in thought from procedural programming to object oriented design.

Should I REST?

2007-07-27T10:22:00.000-04:00

Now that the models are complete, I want to expose access to them as a web service. I'm thinking REST is the way to go. A RESTful approach seems simple. It appears to solve pesky "back button" problems and session timeout issues, but I have one big question: How to fit all the information I need into the URL?

The application I'm working on has A LOT of data. The main objective of the application is to allow filtering of this data down to a useful, manageable set. It wouldn't be uncommon for a user to ask for something like:

   Give me everything connected to Item1, Item2 and Item3,
   Hide anything that has to do with Item4 or Item5
   In addition, give me anything that mentions Items 6, 7, or 8
   and is new to the system in the past month.

To put all this information in the URL is a scary proposition.

In it's previous life as a Java web application, all this search criteria was built incrementally and stored in a user's session. Since the client of this service could be servicing many clients of it's own, this isn't a valid approach.

I don't want to return all the information for each object either. Allowing clients to specify the information they need, as well as pagination, is required.

Writing an "Old School" SOAP/XML web service may be the best answer. I'm already familiar with creating this type of service and all the criteria/request information can easily be captured in a WSDL definition.

I'm thinking, however, that other people must've encountered similar challenges in designing their RESTful web services. I'm picking up a copy of RESTful Web Services to try and find the answers and best practices to these questions.

Rails Progress

2007-07-20T14:28:00.000-04:00

All my models are implemented in Rails and I wanted to give a quick update on how things are going.

For now, I'm been using the rails views and controllers to test my code (automated tests coming soon). This has worked out great for me. Since my application only allows users to view and filter database records, I don't have a lot of validation checks, but I wanted to get a feel for the performance of pages need to display a lot of data from various tables. One page I tested performs 35 queries (including all the counts and show field commands) and takes about 5 seconds using the Webrick server (from NetBeans so I believe JRuby is being used). The development log lists the time spent in the database as less than 1 second. I'm sure performance could be better, but this is a big improvement over what we had before and the database I'm using contains more records.
I'm still noticing that when you ":include" more than one model in a "find" command, more than 1 query is performed. The authors don't mention this in Agile Web Development with Rails so I'm wondering if I have something wrong or if this is a performance enhancement??? Here is a snippet...
```
     @mp_events = Model.find :all,
    :include => [{:model_data1 => 
                [:nested_data1, :nested_data2]}, 
                :model_data2],
    :conditions => query
```
Something cool about this statement, "nested_data1" and "nested_data2" eager load data connected to Model through "model_data1". This works great! I found the information about nested includes here. This author claims a single query as well. I'm wondering if my extra "conditions" clause triggers Ruby to produce the extra query.
One bummer about ":include" is that it does not seem to work with the nested ":select" option. I wanted to total a few of the columns where data was grouped and couldn't get it to work. I resorted to "find_by_sql".

We're now talking about exposing access to these models through a web service. I also need to test more queries with more complex criteria. I'll continue to post interesting tidbits as I encounter them.

Good Migrations

2007-07-10T13:21:00.000-04:00

I was out last week, but finally got my schema in a Rails friendly format and the scaffolding utility is working great! I thought I'd hit on some of the pain points I experienced during this process and how I worked through them.

MySql stinks when you need to change the schema of a database containing data. To make things easy, I've been working with an empty database and using the MySQL administrator to capture the SQL. The plan was to run this SQL against existing databases to upgrade them. This process doesn't work like I'd hoped.

For some columns, the SQL query fails when I try to change a column name (like changing the primary key name to the Rails default 'id'). There are no foreign key constraints on this field and it does not fail on every table where the column name must change. I can't figure out why MySQL bombs on these specific statements. Maybe it's because these table have large row counts (~1M rows)??? Luckily, I won't be creating models for these tables (they are not used in my application), so I'll ignore upgrading these for now.

I've run into similar problems with other update scripts (but these don't fail consistently). I had thought these other statements encountered some sort of race condition, but I'm not sure.

After I updated the schema I went back to DBDesigner to update the schema diagram. This tool does not look at foreign key constraints to visualize relationships (it builds relationships based on column names. Since I now have 33 'id' columns, it gets very confused). I grabbed the alpha release of MySQL Workbench, but found it painful to use.

Eventually, I found SchemaSpy which uses Graphviz to create some great looking diagrams. What I like most is that it allows you to drill down to specific tables for a localized view of how things are connected - leaving the rest of the clutter behind. Also, it was very easy to setup and use!! Awesome tool!!

Finally, I used the Rake db:schema:dump utility to create a schema.rb file. I'll be able to use this as version 1 of my migration file. Like everything else I'm finding in Rails, this utility saved me a lot of time!

More Rails Progress

2007-06-29T08:21:00.000-04:00

I've been playing with Rails over the past few days and want to quickly express my experience so far.

Creating joins, lookup tables, tuning, grouping, sorting, and setting limits are all easy. You can do something like:

Actor.sum(:appearances,
:joins => "INNER JOIN appearances_show_lookup asl
           ON asl.appearances_id
             = appearances.appearances_id",
:group=>'asl.appearances_id',
:order=>'sum(appearances) DESC', :limit=>10)

which translates into

SELECT m.appearances_id, sum(appearances)
FROM appearances m
INNER JOIN appearances_show_lookup md
  ON md.appearances_id=m.appearances_id
GROUP BY m.appearances_id
ORDER BY sum(appearances) DESC
LIMIT 10;

or...

Actor.find(:all,include=>:appearances_type,
:conditions=>"appearances_type.name='guest'",
:limit=>10)

SELECT *
FROM appearances
LEFT OUTER JOIN appearances_type
  ON appearances_type.appearances_type_id =
     appearances.appearances_type_id
WHERE (appearances_type.name='guest')
LIMIT 10

or expand it to 3 tables...

Actor.find(:all,
:include=>[:appearances_type, :show_scores],
:conditions=>"appearances_type.name='guest'",
:limit=>10,
:group=>'appearances.appearances_id' )

SELECT DISTINCT appearances.appearances_id
FROM appearances
LEFT OUTER JOIN appearances_type
  ON appearances_type.appearances_type_id
    = appearances.appearances_type_id
LEFT OUTER JOIN appearances_show_lookup
  ON appearances_show_lookup.appearances_id
    = appearances.appearances_id
WHERE (appearances_type.name='guest')  LIMIT 10

SELECT *
FROM appearances
LEFT OUTER JOIN appearances_type
  ON appearances_type.appearances_type_id
    = appearances.appearances_type_id
LEFT OUTER JOIN appearances_show_lookup
  ON appearances_show_lookup.appearances_id
    = appearances.appearances_id
WHERE (appearances_type.name='guest')
AND appearances.appearances_id
    IN ('5', '6', '7', '8', '9', '10',
        '11', '12', '13', '14')
GROUP BY appearances.appearances_id

Since this final Ruby/Rails code actually produces 2 queries (and I'd prefer an inner join to an outer join as the default), I'm going to opt to create my own queries using find_by_sql like this...

Actor.find_by_sql "
  SELECT m.*,
    SUM(md.appearances) appearances,
    mt.name type_name
  FROM appearances m
    INNER JOIN appearances_type mt
    ON m.appearances_type_id
      = mt.appearances_type_id
    INNER JOIN appearances_show_lookup md
    ON md.appearances_id
      = m.appearances_id
  WHERE mt.nominal = false
  GROUP by m.appearances_id
  ORDER BY appearances desc
  LIMIT 10"

and produces identical SQL.

Even though you can override defaults for table names, primary keys and the like, I can't get the scaffolding scripts to work unless I set them to the Rails defaults (opinionated software!). I'm going to try and use the Rails Migrations tool for these updates. I'll be sure to provide updates on my progress.

Like many others, I love what I've experienced in Rails so far. It's a breath of fresh air from my previous Hibernate and Java experience. I love the ability to make a change then instantly test it using the Ruby console. It's really great! Now my concerns are shifting to performance and scalability.

Rails Progress

2007-06-26T13:36:00.001-04:00

I picked up a copy of Rails for Java Developers by Stuart Halloway and Justin Gehtland and am playing around more with Rails. The book is really good, but I really need a migration guide from Hibernate to Rails (or an existing web application to Rails). I'm looking for more information on creating Rails models that can be accessed through a complex set of filters and associations. The things I need to know are:

How to create something like a 8 table Rails join?
How best to represent lookup tables?
What options are available to tune the queries? Can I capture the Rails query before it's sent to the DB for debugging (outside the MySQL log)?
How do I specify things like limit, paging, group by, sum, distinct, etc?

I'm going to reread the ActiveRecord chapter of this book again. I also have a copy of Agile Web Development with Rails on the way.

Many articles I'm finding on the Internet draw the same conclusions - if your schema and object dependencies are not too complex Rails is the way to go... otherwise use Hibernate. Some of these articles are a little dated though (2005).

I did learn about some cool features built into Rails that I'd like to use, so I'm not giving up hope yet. These include

schema versioning (and rollback)
migration support between various database implementations (is this available in Hibernate?)

I hope to have an answer to some of these questions soon.

More Info

Hibernate vs. Rails: The Persistence Showdown
Ruby on Rails and J2EE: Is there room for both?
Rolling with Ruby on Rails Revisited (Part 2)

JRuby

2007-06-22T10:19:00.000-04:00

The database cleanup is finally complete! Now I need to rewrite our web app's data access code. When we first created the database layer, we used the Hibernate framework. Since then Hibernate3 has been released and an upgrade is scheduled as part of this work. We did have some pain points with Hibernate, however. For instance, when I started evaluating our DB code, I noticed some DB queries that we don't explicitly create. Whether these were the result of a poor object-relational model, lazy instantiation, or some other Hibernate functionality is unknown at this time. I've found Hibernate (or at least how we're using Hibernate) complicated to debug. I'm going to take some time and look at Ruby and Rails as an alternative.

I've mentioned before that I've wanted to learn Ruby for a while and this seems like a good excuse. There is a Ruby library for Java called JRuby. The plan is to learn the JRuby Rails extensions and find out the effort to incorporate this framework into our existing Java APIs. I'll continue to post progress...

Discussion

Has anyone else explored JRuby as a replacement for Hibernate? How does the performance compare? What about ease of use?

More Info

JRuby on Wikipedia

Standards

2007-06-19T07:54:00.001-04:00

I just listened to an interview with Tim Berners-Lee, inventor of the World Wide Web. While most of the interview focused the Semantec Web, what I found most interesting, were his thoughts on Rich Internet Applications. Of course he thinks this is an exciting development, but with various technologies (Flash, JavaFX, Silverlight) competing in this space, he feels that standardization would help speed its adoption. He points to how standardizing HTML in an open and royalty-free way helped to speed the adoption of the Web.

Standardizing this technology would be a great step. Right now, these technologies run as a plugin on the browser. Once standardized, browsers could build this functionality right in, making adoption seamless. Open-sourcing the technology is not enough. The biggest problem with standardization is that it takes so long.

It's a good thought, hopefully it will gain some traction.

******

More news on the DB front. The more I delve into the database schema and the data stored in the tables, the deeper the problems run. Examining the data more closely, I'm uncovering more normalization problems that I hadn't noticed in my initial review of the schema. This is a mess!

We're using DBDesigner to diagram our schema. While the tool provides great looking diagrams, the pain is organizing the tables manually to see all the relationships. Does anyone know of other tools to diagram a schema where this is not an issue?

It's All About Software (and Passion)

2007-06-15T09:17:00.000-04:00

Just to touch on Apple and Microsoft again... I've watched the conversation between Steve Jobs and Bill Gates from this year's D5 conference a number of times. The thing that keeps amazing me is that these guys, earlier than almost anyone else, got that it's all about the software. They realized that we had this great, new technology... but it was way too hard for the average person to use.

Gates referred to a speech that Jobs gave years ago where he said, "We build the products that we want to use ourselves." I always hear successful people say that the key to success is being passionate about what you're doing and what you're producing. Microsoft and Apple are two examples where this approach has obviously worked. I'm convinced these giants are more similar than different.

It would be foolish not to apply these ideals into our own software projects. To have a successful project we need to become our target users. Think like they think. Learn their passions and pain points, then build solutions that alleviate the pain and fuel their passion.

I've found that when I'm passionate about something, my desire is to make it perfect. This is where an Agile approach pays off. From the beginning, you take an approach that you'll endlessly be tweaking and improving (evolving) what you have, always making it a better extension of yourself and your goals.

As we've learned from Microsoft and Apple, the result is a group of users who are passionate about our software.

Masking the Problem

2007-06-13T09:58:00.000-04:00

From my past posts, you may have guessed that I'm in the middle of optimizing a web application that accesses a database. Our studies have shown that a combination of missing indexes, unnormalized data, and inefficient queries are the root cause for this application's poor performance. The optimal solution involves reorganizing the database schema and rewriting our database access layer.

Side Note: The books SQL Tuning by Dan Tow, Pro MySQL by Michael Kruckenberg and Jay Pipes, and Hibernate in Action by Christian Bauer and Gavin King have been instrumental in helping to identify solutions to these problems.

During a meeting to discuss these changes, another suggestion was made to help performance. There are certain instances where we could save few trips to the database (per request) by reusing session information that is unlikely to change (whether this data should actually be in the session or not is a subject for another post). This enhancement requires that we check the database for updates (using a new query) , and reuse the session information if the database has not changed since the previous request.

On the surface this looks like a reasonable approach, but I don't like making these types of changes as a first step to improve performance. First of all, it violates the 80/20 rule (or 90/10 or whatever) because the time saved by this enhancement is small in relation to all the processing we're doing per request. Secondly, enhancements like these mask the real problem - that we have a bad database design and we a doing a poor job accessing it. Finally, I'm a big believer in the KISS principle and when something goes wrong with application caching, my experience has been that this added complexity is difficult to debug.

Don't get me wrong, I feel that optimizations like caching are necessary in web applications, I just feel they shouldn't be explored until other, more obvious, issues have been addressed.

Discussion

What is your approach to identifying and tackling performance problems?

OLPC

2007-06-11T08:33:00.000-04:00

One Laptop Per Child (OLPC) is a movement to create inexpensive, full featured laptops to be used by children for education. The design of this machine is geared at promoting creativity and collaboration. Each system features wireless capabilities for students to connect and communicate with teachers and other students. The user interface centers on activities that children participate in to learn. This differs from a traditional UI which focuses on tasks and applications. The laptop is enclosed in a durable case with a kid friendly handle. It also includes many power saving features, including the ability to charge the battery using human power (like turning a crank or pulling a string). This is an important consideration for areas with little or no access to power.

The price of the hardware is very low (around $100). In addition, much of the software for this system is being developed in an cost friendly, open source model. I'm excited about the possibilities this accessibility will bring to an educational system strapped for financial resources. Costs associated with keeping textbooks up to date should dramatically decrease once these are distributed in electronic form. Some science and math experiments can be simulated, saving costs associated with these activities.

Since I started playing around on my parent's Apple IIe, one of the things I loved most about programming that it allows me to be creative, at a low cost and space requirement. Programs like OLPC, will help provide these opportunities to the next generation of children. With tools like Ruby and Hackety Hack at their disposal these kids will be able to create greater things than I ever thought possible.

Discussion

What other programs, like OLPC are being developed? Has your community, either in the US or worldwide, started to work with these organizations? (I have written to my congressman - Charles Schumer - but have not gotten a response yet.)

Gearing Up

2007-06-05T09:20:00.000-04:00

I've previously written about Rich Internet Applications and how they bring desktop-like functionality to web applications. The knock on this technology has always been "how do I work if I don't have an Internet connection?" Last week, Google announced Google Gears, a framework designed to address this problem.

Google Gears provides 3 modules developers can used to make their web applications available offline.

LocalServer allows web applications the ability to store, then serve, web content locally when a user does not have an Internet connection. These resources can be updated when the user is reconnected to the Net.
Database provides persistent storage to the application when working offline.
WorkerPool allows long running operations to be "chunked", and run in the background without blocking the browser.

It's likely Google developed this technology to widen the appeal and availability of it's Google Docs & Spreadsheats offering. Making this utility open source and free to use will hopefully lead to a quick and widespread adoption.

I'm excited to see how the first applications make use of this technology and if the distinction between online and offline is truly seamless.

More Info

Wikipedia entries for Google Gears and Google Docs & Spreadsheets

Performance Testing

2007-06-04T08:24:00.000-04:00

I just upgraded my Eclipse IDE and installed the latest Test and Performance Tools Platform (TPTP) release. The only tricky part of the installation was installing the Agent Controller (available from from the TPTP download site). This is a separate install from the TPTP plugin and must be placed on your PATH.

TPTP application profiling is easy. Simply start your application in profiling mode, then start the profiler before you access application functionality you want to benchmark. I'm currently using the tool to identify memory and execution time bottlenecks. The TPTP utility makes it easy to sort on key statistics (like the memory footprint for an object class, or the total time spent executing various methods).

I find the method invocation information especially useful. Not only does the tool provide information on which calls are taking the longest, but it also provides details for who called the method and what supporting methods are called (along with the time spent in each). This makes it extremely easy to drill down in the application to find to true cause of the slowdown.

Discussion

I've read that the profiling tool included with NetBeans is very good. Has anyone used this? How does it compare?

New Age Appliances

2007-06-01T08:37:00.000-04:00

I'm pretty excited about the latest consumer tech products I've seen from Apple and Microsoft. Although they seem to target slightly different segments, both have new offerings which push user interface and interaction to a new level.

iPhone

Early this year, I watched the Macworld video where Steve Jobs introduced the iPhone. I was blown away. It was easy to see the new possibilities offered by this device. I mean, the phone I have now offers many advanced features that are clumsy and frustrating to use (web access, email, text messaging, etc) . The iPhone makes these tasks intuitive (and it looks fun). As the average user gets a hold of this, I'm guessing we'll see a spike in web traffic as mobile computing goes mainstream (sort of like what the iPod did for digital music). It will forever change the way people interact with the Internet and our products.

Microsoft Surface Computing

Yesterday I watched a video demoing Microsoft Surface Computing. I don't know if I'd want one of these in my home, but the retail possibilities were unbelievable. Having a system that can recognize objects from the physical world (credit card, phone, camera, *wine glass*) and appropriately interact with these objects is astonishing. I can't wait until one of these devices make it to a retail outlet near me... or I could plan a trip to Vegas.

Common Ground

Technology-wise, both products leverage similar technology - multi-touch displays and stunning user interfaces. Where they both excel, though, is in delivering an incredible user experience - and that's what will make them successful.

Discussion

What new technology or user experiences have you excited?

Not a DBA (Part 2) and Some Quick Hits

2007-05-30T08:46:00.000-04:00

Since my last post, I picked up a copy of SQL Tuning by Dan Tow. This is exactly the resource I've been looking for. Dan Tow has developed a process of diagramming queries, and then using these diagrams to determine an optimized query plan. I'm only half finished with the book, and I'm amazed by the results I'm already seeing. This book has given me a new perspective (and confidence) to solve the problems I described.

I'll post more on my progress in a future post.

Ruby

For a while now, I've wanted to learn Ruby (and Rails). I'm impressed with how much you can do with this language in a few lines of code. If nothing else, it will provide a quick and easy way to mock up prototypes.

Last week, I downloaded Hackety Hack, which is a lightweight Ruby IDE. I've completed the Hackety Hack tutorial and can't wait to create more complex examples.

del.icio.us

I've started to (finally) place my bookmarks into del.icio.us. I thought I'd link to it from here to make it easy to see the things I've been looking at.

Not a DBA

2007-05-25T08:35:00.000-04:00

I don't consider myself an expert in database design or administration, but like many web programmers, my applications often rely on fast and efficient interaction with a database. I am fairly comfortable normalizing a schema and identifying fields to index, but I sometimes struggle creating efficient queries... especially when multiple table joins are needed.

I'm looking for suggestions on how I can improve in this area.

Dynamically Joining Tables

Currently, I'm working with a database containing multiple sets of interconnected objects. Let's assume they are television shows, actors in the shows, and times the shows are on. The goal is to allow users to apply a set of filters to find the shows that interest them. Some simple use scenarios include:

Find all shows where ActorA appears.
Find all shows on between 10AM and 12 PM.

Some harder scenarios include:

Which show or actor appears most frequently.
Which shows are on between 10AM and 12 PM, staring ActorA, but not ActorB.
Which shows are on between 10AM and 12 PM, or star ActorC.
Which shows star ActorA and ActorC, but not ActorB or ActorD.

As you can see from these examples, users can combine filter criteria (actors, times, etc) in a variety of different ways to find the information they need. My team has created a query system that relies on inner queries to search for this information. Performance is not that good. I know that sometimes the order nested queries execute can yield better performance. We're struggling to find the algorithm that will provide the right mixture.

We're also exploring other ideas like

utilizing views to try and simplify things. I'm concerned we're just pushing down the complexity to another level.
creating stored procedures to perform the filtering logic. Again, will we find performance gains by pushing the complexity down a level?

Discussion

I'm sure this problem is not unique. What techniques have others employed to find performance gains (query optimization, views, stored procedures, something else).

How do you verify that the gains are real? Do you solely look at query plans? If not, how do you keep the query cache from skewing the results?

Eclipse RCP

2007-05-23T11:06:00.000-04:00

My work had focused on web application development for so long, I had to pause when I was asked to develop a desktop application. I had little idea where to begin. I started searching for application frameworks that met the following criteria:

I could write the application in Java
The resulting application would be cross platform with a native look and feel.
The framework was mature, easy to use, and widely adopted.

I quickly narrowed the field down to 2 choices Eclipse RCP and NetBeans. Both met the criteria I was looking for. Either would have been a reasonable choice, and I chose Eclipse. First of all, I was comfortable developing in the Eclipse IDE. Second, Eclipse had a larger installed base and an avid community following, with over 1200 commercial plugins available.

I was happy with my choice. Since I had been coding with JSF for some time, the leap to an event based system was not that large. The Forms plugin allowed me to easily create an attractive interface.

More importantly, my clients loved the product. The system was designed to dynamically configure distributed application components. Allowing users to drag and drop these components to create the architecture, made the process easy and intuitive. Real time statistics delivered via web services added to the usefulness of this simple tool.

Discussion

Does the combination of

Rich Internet Application technology
widespread availability of high-speed wireless Internet connections (and mobile devices)
Cheap mult-core processors

spell the end of desktop applications?

I believe so. It's hard to imagine a desktop application anymore that does not need to interface with information on the network. This technology combination will allow new web apps to service more users faster and cheaper - without giving up usability features that, until recently, were only available on the desktop.

JavaServer Faces

2007-05-21T08:54:00.000-04:00

I made the switch to JSF in late Spring, 2005. My team had just adopted the use of Spring and Hibernate - and loved using them. We had just started a new web development effort and knew we'd also need a flexible and easy to use UI framework.

I was a big Struts fan, but often got frustrated with JSP and how much code seemed to be needed in my pages (foreach, if, etc). In addition to providing very modular web UI components, JSF provided many other features I liked and had gotten used to: a scoped managed bean facility and configurable page navigation were two biggies. I also liked how JSF abstracted the idea of a request into an Action, with a corresponding series of listeners that could be called to "set the stage".

The one thing I really missed was the ability to dynamically arrange page fragments, like I had done using Tiles. There were some Tiles/JSF jars that others had written, but I could not get them to work correctly. Eventually I just wrote my own. Amazingly, it was not that hard to do.

JSF was great for me. It is one of those technology innovations that sometimes seem to rise to the surface right when you need something new. Unfortunately (or not), with new web UI developments like Ruby and RIAs, it's lifespan will probably be rather short.

Discussion

Have you used JSF? What was you experience like?

More Info

MyFaces

Props

2007-05-18T11:50:00.000-04:00

It's Friday and I don't feel like writing anything I have to think about too much. I thought I'd start a list of some of the online resources I regularly use for news and inspiration. I hope to add to this list as I go. Props to these great innovators!

Kayak

Kayak is a travel search engine. Kayak makes excellent use of AJAX, adding results as they are discovered, and allowing users to tweak results (times, airlines, stops, etc) without a page refresh.

I love how natural this application is to use. I also appreciate how willing they are to share the technology they're using and other cool stuff.

CSS Zen Garden

This site really cemented, for me, just how powerful and flexible CSS can be. I can spend hours browsing designs and learning new techniques. I do not have an extensive background in design and this site has really helped me refine my user interfaces.

Amazon

I'm always impressed with the great innovations Amazon has developed for users browsing books... recommendations, reviews, ratings, wishlists, etc. I don't know if they were the first website to make extensive use of user contributed content, but they've done the best job of it in my opinion. I'm a big Amazon fan.

ZdNet

I turn to ZDNet for most of my news about the technology industry. I especially enjoy the "Between the Lines" features and podcasts.

Discussion

Thanks to all these great sources for the knowledge and help they provide. What sources to you regularly turn to as a reference or for ideas?

Web Services

2007-05-17T10:16:00.000-04:00

I'm a big fan of Web Services and Service Oriented Architectures (SOA). I've developed a handful of web services for my company to use internally and there are a few reasons I enjoy working with this technology.

Ease

What I think I like best, is how easy it is to create a web service. I use Eclipse and this IDE provides a web service wizard (I'm sure other tools have something similar). All you need to do is create the WSDL and most of the heavy lifting is done for you (opening and closing connections, serializing the object to the wire, etc). Hook the generated code into your business objects and it's ready to go. Since Web Services use HTTP, installing your WAR file on the web server is the final step. No other setup is needed.

I did not find CORBA as easy to use. In CORBA, you need to communicate what ports you want to use, optionally set up a CORBA name service, etc. If there was a port conflict on one of the machines you were using, you needed to reconfigure a bunch of nodes.

Lego-like

To me, Service Oriented Architectures epitomize the type of modules we're all taught to build in school. We all strive to build reusable, scalable code, with open interfaces, that can be used in several types of applications. The goal was always to have a set of these modules which can be combined in different ways to create cool things (like Lego blocks).

Prior to SOA, sharing a module's functionality meant sharing jar files. Updates were hard to deploy and track. Since SOA lives on the web, updates are instant and painless (as long as the API does not change). It seems like the ideal distribution method.

Unlimited Possibilities

Because of this Lego-like architecture and because the functionality is available over the web, people are combining these services in many interesting ways. These combinations are commonly referred to as mashups. The first mashup I saw plotted data obtained from a local law enforcement web site onto a Google map - providing a visual representation of high crime areas. These unexpected and useful combinations are what help make SOAs exciting.

Discussion

Which mashups have helped get you excited about this technology?

More Info

WSDL Tutorial (W3C Schools)
Web Services Programming (Yahoo Group)