Loosely Coupled

I've been a fan of Joel on Software for some time. Recently Joel (Spolsky) started a new enterprise, stackoverflow.com , with Jeff Atwood. They've been releasing some entertaining and informative podcasts as part of this new endeavor. Good Stuff. Who knew you could wash your electronics?? ***** Joel has written a few books. One I haven't read yet is titled Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent . While I'm sure this book is helpful, what I really need is a book that can help locate progressive companies which allow employees to work smartly and get things done. I don't know if it's endemic to the region I live or it's more widespread than I imagine, but none of the companies I encounter here are doing anything close to Agile development, few are working with Ruby, most are afraid of open source, none have heard of REST web services, and most are reactive problem solvers rather than proactive....

As the name suggests WebDevRadio is a web development podcast. I'm a big fan of this show. It always delivers an interesting perspective, and the discussions often relate to a technology I'm also exploring. In a recent episode, the host Michael Kimsal interviewed Jason Rudolph about Jason's new book Getting Started with Grails . I've written before about Ruby and Rails and using something like JRuby to leverage Java's stability and scalability in a production environment. Grails (and Groovy ) is an interesting alternative to the Rails/Ruby/JRuby I'd been thinking about. Like Ruby, Groovy is an interpreted language but with some constructs that make it easier for Java developers to transition over. Also, since it's written for the JVM, it gets all the Java maturity "for free" and naturally integrates with existing Java modules (No need for something like JRuby). Grails brings the Rails goodness to the Groovy platform... including a lot of ...

Migrating my application from Hibernate to Rails has taken longer than I expected. An important client has expressed performance concerns with their release, so I spent the last week looking for ways to tweak the current Hibernate implementation. It has been very frustrating. I don't like to cast blame or plead ignorance, but prior to this exercise, my experience with Hibernate has been limited. The module containing our Hibernate calls was authored by someone else and I've been mostly a client of this module. It wasn't until I looked at our query log in detail did I notice how poorly this module was written and how confusing Hibernate is. When seeing the large amount of queries in the log (often for data not needed in the current view), my first thought was, "we must have a problem in our models... something is causing Hibernate to eagerly fetch related data we have no immediate need for." I went through our models and, indeed, I did find some simple changes ...

I wanted post a quick note about new technology I heard about today via podcast . The technology is freebase created by Metaweb . As you might guess from the name, freebase is a free database for the web containing all sorts of information contributed by users. In other words, it's a huge Wikipedia-style database that anyone can contribute or query to find interesting (and possibly little known) information and relationships between seemingly unrelated objects. For instance, assuming the information is in the database, I could query to find "all wooden roller coasters in the northeast longer than 5200 feet." That's pretty cool. Assuming I found some results, I could whittle down the set further, looking for which roller coasters have been ridden by Rock Stars or with lower height restrictions (so I could ride with my daughter). Because the data can be contributed by the average person, the breadth and depth of the information to search for will grow quickly (an...

All my models are implemented in Rails and I wanted to give a quick update on how things are going. For now, I'm been using the rails views and controllers to test my code (automated tests coming soon). This has worked out great for me. Since my application only allows users to view and filter database records, I don't have a lot of validation checks, but I wanted to get a feel for the performance of pages need to display a lot of data from various tables. One page I tested performs 35 queries (including all the counts and show field commands) and takes about 5 seconds using the Webrick server (from NetBeans so I believe JRuby is being used). The development log lists the time spent in the database as less than 1 second. I'm sure performance could be better, but this is a big improvement over what we had before and the database I'm using contains more records. I'm still noticing that when you ":include" more than one model in a "find" comman...

I was out last week, but finally got my schema in a Rails friendly format and the scaffolding utility is working great! I thought I'd hit on some of the pain points I experienced during this process and how I worked through them. MySql stinks when you need to change the schema of a database containing data. To make things easy, I've been working with an empty database and using the MySQL administrator to capture the SQL. The plan was to run this SQL against existing databases to upgrade them. This process doesn't work like I'd hoped. For some columns, the SQL query fails when I try to change a column name (like changing the primary key name to the Rails default 'id'). There are no foreign key constraints on this field and it does not fail on every table where the column name must change. I can't figure out why MySQL bombs on these specific statements. Maybe it's because these table have large row counts (~1M rows)??? Luckily, I won't be creat...

I've been playing with Rails over the past few days and want to quickly express my experience so far. Creating joins, lookup tables, tuning, grouping, sorting, and setting limits are all easy. You can do something like: Actor.sum(:appearances, :joins => "INNER JOIN appearances_show_lookup asl ON asl.appearances_id = appearances.appearances_id", :group=>'asl.appearances_id', :order=>'sum(appearances) DESC', :limit=>10) which translates into SELECT m.appearances_id, sum(appearances) FROM appearances m INNER JOIN appearances_show_lookup md ON md.appearances_id=m.appearances_id GROUP BY m.appearances_id ORDER BY sum(appearances) DESC LIMIT 10; or... Actor.find(:all,include=>:appearances_type, :conditions=>"appearances_type.name='guest'", :limit=>10) to SELECT * FROM appearances LEFT OUTER JOIN appearances_type ON appearances_type.appearances_type_id = appearances.appearances_type_id WHERE...

I picked up a copy of Rails for Java Developers by Stuart Halloway and Justin Gehtland and am playing around more with Rails. The book is really good, but I really need a migration guide from Hibernate to Rails (or an existing web application to Rails). I'm looking for more information on creating Rails models that can be accessed through a complex set of filters and associations. The things I need to know are: How to create something like a 8 table Rails join? How best to represent lookup tables? What options are available to tune the queries? Can I capture the Rails query before it's sent to the DB for debugging (outside the MySQL log)? How do I specify things like limit, paging, group by, sum, distinct, etc? I'm going to reread the ActiveRecord chapter of this book again. I also have a copy of Agile Web Development with Rails on the way. Many articles I'm finding on the Internet draw the same conclusions - if your schema and object dependencies ar...

The database cleanup is finally complete! Now I need to rewrite our web app's data access code. When we first created the database layer, we used the Hibernate framework. Since then Hibernate3 has been released and an upgrade is scheduled as part of this work. We did have some pain points with Hibernate, however. For instance, when I started evaluating our DB code, I noticed some DB queries that we don't explicitly create. Whether these were the result of a poor object-relational model, lazy instantiation, or some other Hibernate functionality is unknown at this time. I've found Hibernate (or at least how we're using Hibernate) complicated to debug. I'm going to take some time and look at Ruby and Rails as an alternative. I've mentioned before that I've wanted to learn Ruby for a while and this seems like a good excuse. There is a Ruby library for Java called JRuby . The plan is to learn the JRuby Rails extensions and find out the effort to incor...

I just listened to an interview with Tim Berners-Lee , inventor of the World Wide Web. While most of the interview focused the Semantec Web , what I found most interesting, were his thoughts on Rich Internet Applications. Of course he thinks this is an exciting development, but with various technologies (Flash, JavaFX, Silverlight) competing in this space, he feels that standardization would help speed its adoption. He points to how standardizing HTML in an open and royalty-free way helped to speed the adoption of the Web. Standardizing this technology would be a great step. Right now, these technologies run as a plugin on the browser. Once standardized, browsers could build this functionality right in, making adoption seamless. Open-sourcing the technology is not enough. The biggest problem with standardization is that it takes so long. It's a good thought, hopefully it will gain some traction. ****** More news on the DB front. The more I delve into the database schema and...

From my past posts, you may have guessed that I'm in the middle of optimizing a web application that accesses a database. Our studies have shown that a combination of missing indexes, unnormalized data, and inefficient queries are the root cause for this application's poor performance. The optimal solution involves reorganizing the database schema and rewriting our database access layer. Side Note: The books SQL Tuning by Dan Tow , Pro MySQL by Michael Kruckenberg and Jay Pipes , and Hibernate in Action by Christian Bauer and Gavin King have been instrumental in helping to identify solutions to these problems. During a meeting to discuss these changes, another suggestion was made to help performance. There are certain instances where we could save few trips to the database (per request) by reusing session information that is unlikely to change (whether this data should actually be in the session or not is a subject for another post). This enhancement requires ...

Since my last post, I picked up a copy of SQL Tuning by Dan Tow . This is exactly the resource I've been looking for. Dan Tow has developed a process of diagramming queries, and then using these diagrams to determine an optimized query plan. I'm only half finished with the book, and I'm amazed by the results I'm already seeing. This book has given me a new perspective (and confidence) to solve the problems I described. I'll post more on my progress in a future post. Ruby For a while now, I've wanted to learn Ruby (and Rails ). I'm impressed with how much you can do with this language in a few lines of code . If nothing else, it will provide a quick and easy way to mock up prototypes. Last week, I downloaded Hackety Hack , which is a lightweight Ruby IDE. I've completed the Hackety Hack tutorial and can't wait to create more complex examples. del.icio.us I've started to (finally) place my bookmarks into del.icio.us . I thought ...

I don't consider myself an expert in database design or administration, but like many web programmers, my applications often rely on fast and efficient interaction with a database. I am fairly comfortable normalizing a schema and identifying fields to index, but I sometimes struggle creating efficient queries... especially when multiple table joins are needed. I'm looking for suggestions on how I can improve in this area. Dynamically Joining Tables Currently, I'm working with a database containing multiple sets of interconnected objects. Let's assume they are television shows, actors in the shows, and times the shows are on. The goal is to allow users to apply a set of filters to find the shows that interest them. Some simple use scenarios include: Find all shows where ActorA appears. Find all shows on between 10AM and 12 PM. Some harder scenarios include: Which show or actor appears most frequently. Which shows are on between 10AM and 12 PM, staring Ac...