Performance Whack-a-Mole by Josh Berkus

July 25th, 2007

This turned out to be a good talk, chock-full of concrete practical advice.

Here’s our audience poll:
95% use DB apps
15% are consultants (like the presenter)

Clients just say: “the database is slow,” but anything in the big application stack (application, middleware, database, operating system, hardware_ can make things slow. Every layer in the stack needs do its job to scale the application. Otherwise, we are leaving significant performance gains on the table. In fact, “database” problems are usually somewhere else — like in the middleware.

You have many performance problems in your application, but most aren’t significant. Performance is usually degraded by a few problems — 10% of problems cause 90% of slowdown. (And big problems mask smaller problems.)

Apps tune differently. The presenter uses three rough categories: web, online transaction processing, data warehouse.

  • Entire database can fit in RAM
  • 90% or more of database queries are simple reads
  • CPU-bound
  • Performance moles: caching, pooling, connection time


  • Database slightly larger than RAM to 1TB
  • Many small data write queries
  • CPU or I/O bound
  • Moles: locks, cache, transactions, write speed, log

Data Warehouse

  • 100GB to 100TB database
  • Large complex reporting queris
  • Large bulk loads of data
  • Also called “business intelligence”
  • Moles: Sequential scans, resources, bad queries

Your first step should be to measure a baseline. Check basic setup (versions, configurations) for sanity. Then measure each layer.


  • OS tools are simple, easy to use, and non-invasive
    • ps, mpstat
  • Benchmark. Invasive
    • micorbenchmarkL bonnie++ for files system perf
  • DB
    • DBTrace
    • DB query log
    • Query analysis (troublehoot “bad” queries)
    • ogbench, Wisconsin, TPCB, OSDB, PolePosition
  • Application
    • app server tools
    • workload simulation and screen scraping
  • lwp and log replay tools
  • bug detectors
  • valgrind, MDB, GDB, DTrace

OK, we’ve got the baseline, now what? What are the symptoms; when do they occur?

The Performance Moles


  • Symptoms: one device saturating I/O — other resources are OK
  • heavy writes or very large DB
  • causes
    • bad I/O hardware, software or configuration
    • not enough RAM
    • too much data requested
    • bad schema


  • Symptoms: maxed-out CPU, but RAM available
  • mostly-read loads or complex calculations
  • causes
    • too many queries
    • insufficient caching or pooling
    • too much data requested
    • bad queries or schema
  • Note: most DB servers should be CPU-bound at max load

Locking Mole

  • Symptoms: Resource aren’t maxed-out
  • lots of pessimistic locking
  • causes
    • long-running transaction or stored procedures
    • longly held open cursors
    • unneeded pessimistic locking
    • poor transaction management

App Mole

  • Symptoms: DB server maxed, app server is OK
  • common in J2EE
  • causes
    • not enough app servers
    • too much data/ too many queries
    • bad caching.pooling
    • driver issues
    • ORM frameworks

Open Source Geospatial Boot Camp by Schuyler Erle

July 25th, 2007

Holy fast talking presenter, Batman!

Today’s audience poll:
Used open source software: 95%
Comfortable with the command line: 90%
Used open source GIS software: 20%

We ran though some GIS/cartography terms that are handy to know when doing GIS.

Did you know? The Earth is not a sphere. In fact, It’s an irregular ellipsoid. There are many different standards for modelling the Earth. You should use WGS 1984 as your ellipsoid model.

Projection: trade-off between conformal (preserve angle) and equidistant (distance) and equivalent (area). Got it? We have to make trade-offs because we are translating a sphere, er, ellipsoid onto a flat surface.

    There are lots of GIS data formats

  • OS tools help hide this
  • GEOTiff (raster)
  • ESRI Shapefile (vector)
  • GML (XML)

Open source tool: Quantum GIS — useful for browsing GIS data. Looks like PostgreSQL, at least, has good GIS support.

You’d think I’d have gained more information from this talk. Unfortunately, the presenter spent lots of time fiddling with stuff to make it work. This “show you how it is with GIS!” “It’s fun!” Well … I don’t know. I do think that this is how it with GIS. It’s certainly been my experience with my GPS and mapping software. Almost the antithesis of what I like: polished, clean interfaces, adherence to conventions. Makes me think that GIS is not for me.

A Taste of Haskell

July 24th, 2007

Back in the Oregon Convention Center. Pretty handy that two of the biggest open source get-togethers are a short ride up the Esplanade. Con-way even ponied-up for the OSCON tutorials.I went to the Haskell talk because all the cool kids are were talking about it. (Gerry tells me that they are on to Erlang now.) Anyway, Simon Peyton-Jones from Microsoft Research (yes, that’s right) gave an energized talk to all of us later-to-the-party programmers.Things didn’t look too promising at the start, though. We had some presenter equipment set-up hijinks: “Any Ubuntu experts here? How do you increase the font size in Emacs?” Once that was all squared-away, I noticed that the presenter was barefoot.Audience poll! 30% have written a Haskell program.OK, what is Haskell? A functional, lazy, (really) strongly-typed, compiled, general purpose language. It’s about value, not state … what? Yeah, I know, that doesn’t make much sense. It did make sense to me about three hours later, but it’s beyond my ability to enlighten you via this blog. Sorry! Try or’s a research language, and the presenter is one of the inventors. He defines a successful research language as one that dies a slow death rather than an immediate death, so Haskell is successful.We’re using xmonad as an example of a Haskell application. It’s an X11 window manager. It’s not exactly Enlightenment. And the first big diagram is a state machine! Uh, isn’t Haskell about values, not state? We find out later that since we’re dealing with I/O — the outside world — we can’t keep everything pure. Turns out that, for quite a while, Haskell was a nifty language that couldn’t do any I/O, which rather limited its practical value.OK, here’s a code sample. The syntax seems reasonable, but everything looks verbose to me after Ruby. Functions are defined by pattern matching. In some sense, they are a “collection of equations.” The code below basically says: for an empty list, return an empty list; for a list of one, return a list of one followed by an empty list; for more than one, return the same list with the first and second elements swapped.swap :: w -> w -> wswap [] = []swap (w : []) = w : []swap (w1 : w2 : ws) = w2 : w1 : wsThis is pretty cool. Pattern matching forces us to think of all cases. The presenter didn’t mention this, but I wonder if they are driving at the goal of mathematically-provable programs. It reminds me of Eiffel and its design by contract.Haskell has a lot of the things you’d exepect. There’s a library called Test.QuickCheck that passes in 100 (random?) values into each of your functions to ensure that you’ve covered all the cases. (As you might imagine, functions are easy to test.) There’s a documentation generator, a package repository and general-purpose libraries.We dived a whole level deeper into Haskell, but I’m not sure I can do it justice. So, the big picture?

  • No side effects at all
  • Purity makes the interface explicit
  • Types are everywhere

Haskell is trying to be useful and safe (safe == no side effects). It is is very safe, but needs to become more useful. From Haskell’s perspective, Java, C#, and Ruby are very useful languages that are becoming safer. Apparently Haskell’s not useless, as there’s a company in Portland that employs twenty Haskell coders and would like to hire some more.Did the presenter just say, “The nettle must be grasped?”

Final Keynote by Dave Thomas

May 30th, 2007

I have to say that all the RailsConf keynotes I went to were good, and I heard that the others were good, too. Is that level of quality sustainable? I hope so.

Dave noted that EuroRailsConf was the most profane conference ever. He thought this was a bad thing.

We all were getting a bit loopy at this point, so my notes are short and fragmented. Here’s what I can piece back together.

Ze Frank talked about what to do when you have many amateur authors. This was an interesting problem that we wrestled with back in the day at Gap Inc.’s intranet. You can ignore them (obvious bad consequences). You can try and control them, but this doesn’t scale. You can shut them down (draconian). He suggests that you instead facilitate conversation. Which sounds fine, but fuzzy now that I write it down here. Should have gone to his talk, I guess.

Dave warned about Cargo Cult programming — definitely a danger in the Rails community. He then listed some assumptions we should question:
– The web browser: it’s a freakin’ full-duplex mainframe terminal!
– Object-oriented programming. The world is not hierarchial, it’s chaotic. Why are we so class-centric? Try to write a program without classes and you might be surprised by how it works.
– Relational databases

Laying Tracks: How to Contribute to the Ruby on Rails Open Source by Josh Susser

May 30th, 2007

This talk was practical advice on how to do just what the title says: contribute to Rails.

Josh comes from a Smalltalk background. He also worked on OpenDoc, “A technology that no one has heard of it.” You are wrong, my friend — I remember when it was all the buzz at MacWorld Boston ‘96, and my boss was wondering if we should become OpenDoc developers. Ah, CyberDog, we shed a tear for you.

Anyway, Josh now works at Java Card.

The session’s audience:
15% use the Rails Trac
10-15% opened a ticket
10% contributed a patch
5% had patch accepted

Prerequisites for contributing: Rails Trac account, Subversion, subscribe to the Rails Core discussion list.

First steps:
1. Create a local, vanilla Rails projects for this work
2. svn co to vendor
3. Set up tests
4. Ensure tests run

You need to use TDD, follow code style, write docs, write tests, and ensue that the rdoc builds. Use SVN diff to generate patch. Create a Trac ticket with [PATCH] in the subject.

How to get some patch accepted
– Make tiny changes
– Discuss big patches first on Rails Core
– Need good code
– Need tests
– Get buy in for Rails developers

“Get used to disappointment. “There’s a backlog.

JRuby on Rails: A Taste of Honey (for the Enterprise) by Charles Nutter, Thomas Enebo

May 30th, 2007

Here’s a real advantage of a conference over email: I learned that Charles Nutter has an awesome Midwestern accent! I’ve got a whole new voice in my head while I read his posts, and there are a lot of them. Does Charles Nutter sleep? He seems to churn out emails and blog comments about JRuby 24/7.

On to the talk. For the record, I am not sure how I feel about the whole Rails running on JRuby inside a WAR on Glassfish stack at the enterprise data center. Sounds like a tall, tipsy stack of dinnerware to me. But I do like JRuby. It’s been really handy at C o n – w a y for calling EJBs, using mainframe proxies, and running JUnit test suites. And there’s a real possibility that the Java VM could become a great VM for Ruby.

This session’s poll:
25% of the audience are Java developers
25% ex-Java developers
15% ex-Perl users

Who uses which database?
40% MySQL
5% SQLite
10% MS SQL Server
20% Oracle
1% DB2

Why use JRuby instead of C Ruby?
– Unicode support
– Native JVM threads
– The big pain point …

What are the biggest differences?

1) Database support. Ruby MySQL works about the same. JRuby can use JDBC, of course, and JNDI to look up connection pools. DB2 support is not 100%? (Works pretty good for me.) Oracle ActiveRecord tests are not 100%

2) Native Ruby extensions don’t work out of the box. Need porters.

3) Command-line performance because Java start-up time is slow.

Charles and Thomas demoed Goldspike, a way to package a Rails app into a J2EE WAR:
rake war:standalone:create

And they demoed a Glassfish gem! Hm, well, theoretically, if all this just works, it could be pretty cool. Though I dread digging through the stack trace.

RESTful Modeling and Active by Record F. Morgan Whitney

May 30th, 2007

Vonage seems to have done a really nice job of integrating Rails with their legacy J2EE apps (cracks me up to write “J2EE apps”). This talk was somewhat of a rehash of Mapping Rails to Legacy Systems, with more details about the REST bits.

Turns out that Vonage essentially implemented Rails 2.0’s ActiveRecord six months before the Rails team.

Angel on the Shoulder by Jarkko Laine

May 30th, 2007

Another good guy and kindred spirit, Jarkko’s central point was how to build an effective framework for developers. Or, precisely, what are the principles that make Rails’ framework effective? Good stuff, but not enough for a full-length RailsConf session. Don’t mean to sell you short, Jarkko — I’ve spent a bit of time recently building an internal testing framework while using Rails, so maybe the principles feel obvious.

It really boils down to the carrot-and-stick approach: make it easy for developers to do the right thing. And the right thing should be sensible, terse, and as intuitive as possible. The “wrong” thing should be hard to find, documented with warnings and deprecations, and lead to many more lines of code. (This is certainly characteristic of Rails. Whenever it takes me more than 3-4 lines of code for something simple, I immediately suspect that David’s already thought of a better way to do it.)

Carrot: How dead simple it is to write an automated test in Rails
Carrot: All the magic REST methods
Stick: CruiseControlRB

Jaarko says that you should ask: “How does this help the user kick ass?” He’s the lead a developer at, and he wrote, “Beginning Ruby on Rails E-Commerce.”

Test Dictionary Open Sourced

May 26th, 2007

I finally rallied some focus and motivation, and released Test Dictionary on SourceForge. This is a very small Java lib jar that is handy for generating realistic test data:
String name = Dictionary.getRandomProperName();
String word = Dictionary.getRandomWord();
String word = Dictionary.getRandomWordTermCommonNameOrConnector();

The Business of Rails

May 19th, 2007

This was a panel discussion: Joe O’Brien, Nathaniel Talbott, Justin Gehtland, Geoffrey Grosenbach, Robby Russell, Andre Lewis.

When did you decide to work for yourself?
– When CEO decided it was more important to have his boat instead HR staff
– Job ended
– Wanted to pursue music career (Robbie)
– Wanted to write code, and work with Rails
– Wanted to junk the traveling consultant lifestyle
– Wanted to build things for yourself

“Never do fixed bid.”

Companies come to Planet Argon asking for Rails — they don’t need convincing. Other people bid projects in RoR and Java, and let the customer decide.

How to compete with cheap people? How to talk people out of fixed bid? Tailor response to customer:
– Hourly rate or 2-3 month projects ballpark estimate
– Maybe for a big client, you could double or triple your price and do a fixed bid
Talk to client, and if you don’t convince them, bail on the work. Maybe do first iteration at time and materials, and do a bang-up job. If you measure and refine your process, you can do fixed bid.

Words of wisdom: “Can’t rewrite GMail for $20K.”

How many are thinking of going out and doing consulting o your own? 20% of the audience.


Do marketing via user groups, etc. Getting leads sucks, but sales is actually fun. Don’t be superficial about it. Give back to community.

You can email for a contract template

Employees cost about the same as contractors, but you get more value because of collaboration
Incorporate, fool! So that your personal bank account is not at risk
– LLC or S Corp
– Talk to a CPA
– Need to build business credit history; fix separate taxes.
– Get a separate bank account
– AMEX card and buy something on it every month
Go to a local bank, where they will take your business seriously
– When that $100K check comes in, the banks will put a 15-day hold on it if they don’t know you

How do you manage to only work 40 hours a week?
– You don’t
– Hard to separate work and home when you work at home
– Have to be firm with yourself
– Find an office space