The pg_query_analyzer gem: PostgreSQL query analysis in your development logs

I just wrapped up a small gem called pg_query_analyzer that’s based on some old code that can be found on the interwebs in various states of disrepair. This latest incarnation is a simple gem that provides a PostgreSQL query analysis in the development log of a Rails app.

This is useful for finding indexes that your database may benefit from. There are alternatives available (e.g. rails_indexes) that evaluate your application and guess at which indexes might help you, and provide example migrations to add them. That kind of thing is definitely cool, but I’ve found that performing a manual analysis using tools like this gem I’m releasing will typically lead to the best results. Perhaps you would consider using both techniques? It’s smart, at least, to verify that the database indexes you add to your application are having the intended effect of speeding things up!

I hope you’ll find the README I’ve provided with this gem to be helpful in your quest for a fast and scalable application using PostgreSQL.

Good luck, and good indexing 😉

Weekly Digest, 3-24-10

Seriously. Not weekly at all. Still, it’s a digest of links, I’d say.

Sinatra 1.0 Released: Major Milestone for Ruby’s Best Webapp DSL

In November 2007, we casually mentioned a new Ruby webapp library called Sinatra. It took a year to capture the imagination of the Ruby community as a whole and we eventually covered it in more depth but today we’re proud to (exclusively) announce that Sinatra has today reached its landmark 1.0 release!

Introducing Cramp

Cramp is an asynchronous framework, always running inside EventMachine reactor loop. Cramp isn’t a good fit for most of the web applications out there. However, Cramp is good at holding and working with a large number of open connections. Hence it’ll work great for things like comet, long polling, streaming API or even when your application needs to handle thousands of concurrent connections.

Untangling Evented Code with Ruby Fibers

Ruby 1.9 Fibers are a means of creating code blocks which can be paused and resumed by our application (think lightweight threads, minus the thread scheduler and less overhead). Each fiber comes with a small 4KB stack, which makes them cheap to spin up, pause and resume.

Official Google Blog: A new approach to China: an update

Finally, we would like to make clear that all these decisions have been driven and implemented by our executives in the United States, and that none of our employees in China can, or should, be held responsible for them.

datagraph’s rack-throttle

Rack middleware for rate-limiting incoming HTTP requests.

Raiding Eternity

The Cloud is just the internet. And the internet is just a bunch of hard drives. The internet is really good at replicating discrete bits of self-contained data. There are probably a few million copies of any given Loretta Lynn song out on all the hard drives of the world, because lots of people care about Loretta Lynn. But my photos on Flickr only live on a few hard drives in the world. The hard drives in the database servers. The hard drives in the networked-attached storage devices that are used to backup the database servers.

Consuming the Twitter Streaming API

How do you integrate this into a Ruby app? Standard HTTP clients such as RestClient and HTTParty aren’t appropriate, since they’re designed for atomic HTTP requests, not streaming. With this API, you want to keep the socket open indefinitely, decoding JSON one line at a time.

Notational Velocity

NOTATIONAL VELOCITY is an application that stores and retrieves notes.

Graph Databases

Graph databases are a type of datastore which treat the relationship between things as equally important to the things themselves.

simplabs’s highlight at master – GitHub

Highlight is a simple syntax highlighting plugin for Ruby on Rails. It’s basically a wrapper around the popular pygments highlighter that’s written in Python. If pygments is installed on the machine and in the PATH, that binary is used, otherwise the plugin falls back to the web API created by Trevor Turk. [Awesome!]

Saying Yes to NoSQL; Going Steady with Cassandra

Digg is committed to the use and development of open source software and we’re keen to avoid the cost of proprietary large-scale storage solutions. We were inspired by Google and Amazon’s broad use of their non-relational BigTable and Dynamo systems. We evaluated all the usual open source NoSQL suspects. After considerable debate, we decided to go with Cassandra.

News flash

We don’t deserve anything. Publishers can do whatever they want. If you don’t like it, don’t send them nasty emails or browse their sites with ad-blockers: just don’t support them. Don’t read their content, don’t link to them, and don’t talk about them. Since money’s not usually involved, vote with your attention and read elsewhere.

Curing RSS addiction and continuous partial attention

The fundamental problem is the disconnect between when we find good content and when we actually want to read it: these occur at completely different times in our day and in completely different contexts.

How Google’s Algorithm Rules the Web

Throughout its history, Google has devised ways of adding more signals, all without disrupting its users’ core experience. Every couple of years there’s a major change in the system — sort of equivalent to a new version of Windows — that’s a big deal in Mountain View but not discussed publicly.

Daring Fireball: Tits and Apps

These apps were allowed for about a year and a half. Some developers were prospering by them. And then, boom, they were gone. The reason Apple ought to be concerned about this is that it unsettles all developers — even those whose apps and ideas for future apps were nowhere along the lines of girls-in-bikinis. What developers see here isn’t Apple managing its own brand. What developers see is that the App Store is a shaky foundation upon which to build a business.

Customized Google Forms

Google Forms are a great, free way to collect information from anyone, stored directly into a Google Spreadsheet, and then have Google notify you each time a form is submitted (optionally). The downside is that you can only use one of their pre-packaged themes. You can’t have your company logo and corporate “look”. Pooey to Google, we say. So we wrote custom_google_forms.

The Skinny on Scopes (Formerly named_scope)

So here we are with Rails 3 completely refactoring the internals of ActiveRecord – what’s up with our beloved named_scope? Well, the simple answer is that it’s been renamed to scope and you can use it just as you’re used to … but that’s taking the easy way out. Let’s see what else we can do with scope in Rails 3.

Be lucky – it’s an easy skill to learn

My research revealed that lucky people generate good fortune via four basic principles. They are skilled at creating and noticing chance opportunities, make lucky decisions by listening to their intuition, create self-fulfilling prophesies via positive expectations, and adopt a resilient attitude that transforms bad luck into good.

Notes on MongoDB

For an article in a German magazine I’ve been researching MongoDB over the last week or so. While I didn’t need a lot of the information I came across I collected some nicely distilled notes on some of its inner workings.

Cassandra @ Twitter: An Interview with Ryan King

I received a short email from Ryan King, the lead of Cassandra efforts at Twitter simply saying that he would be glad to talk about these efforts.

mzsanford’s twitter-text-rb

Twitter text processing library (auto linking and extraction of usernames, lists and hashtags)

Why the GPL does not apply to premium WordPress themes

At its core, the GPL is simply a fancy way of controlling other people’s work through the imposition of copyright restrictions. Those who seek to extend the GPL beyond the bounds allowed by copyright law, do not promote freedom but instead take freedom away.

defunkt’s resque

Resque is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.

The 2-Billion-Eyed Intermedia

We are developing an intense, sustained conversation with this large thing. The fact that it is made up of a million loosely connected pieces is distracting us. The producers of Websites, and the hordes of commenters online, and the movie moguls reluctantly letting us stream their movies, don’t believe they are mere pixels in a big global show, but they are. It is one thing now, an intermedia with 2 billion screens peering into it.

EngineYard to exit Herakles and the infrastructure business in 6 months or less?

I think EngineYard, in spite of the fact they made their salt on being a hosting and infrastructure company, is sensibly getting out of the plumbing business. They are going much more to the software-as-a-service (expertise in virtual / cloud Rails hosting, general and specific Rails expertise, etc.) vs infrastructure-as-a-service side and their propensity for Xen and virtualization makes the decision much more logical and transparent.

Happy Birthday Devise

Let’s celebrate and talk a bit about history.

How to upgrade plugins to Rails 3.0

Rails 3.0 beta is out and it’s now time to upgrade all the plugins available. To show you how to do it I’ve decided to create a small plugin compatible with Rails 2.x and Rails 3.0. It’s a wrapper around Rack::Cache to insert it automatically in a Rails application.

Cultivate Teams, Not Ideas

Success is rarely determined by the quality of your ideas. But it is frequently determined by the quality of your execution. So instead of worrying about whether the Next Big Idea you’re all working on is sufficiently brilliant, worry about how well you’re executing.

YouTube Disco

YouTube Music Discovery Project and Playlist Creation Tool

Run Node.js as a Service on Ubuntu Karmic

The core of our new project runs on Node.js. With Node you can write very fast JavaScript programs serverside. It’s pretty easy to install Node, code your program, and run it. But how do you make it run nicely in the background like a true server?