By 1830 this evening, Tesco on Eastcheap had pushed over 3,000 people through their self service checkouts.

Fact.

A departure board that sees 10 minutes into the future - at last!

The classic light boxes still function, though. Nostalgia there.

Having recently posted a question asking Why are there so many NoSQL databases? over at Hacker News, I thought it would be useful to summarise the responses, and to draw any common thoughts out.

Background

There is a "standard set" of traditional databases if you are developing a web (or non-web, of course) application; MySQL, PostgreSQL, SQLite and perhaps Oracle and SQL Server if you're in an enterprise environment.

However, with the NoSQL anti-database movement gaining in momentum and becoming more widespread, people are starting to look towards the new schema-free, key-value and document store databases that are hitting the market.

The problem is the proliferation of NoSQL options, and trying to boil everything down to understand which options suits your needs most closely.

Why are there so many NoSQL options?

Without delving into too much detail, there have been many recent innovations ^1 in the field (Facebook created Cassandra, Google created BigTable and MapReduce, Amazon created SimpleTable and LinkedIn created Project Voldemort). Innvoations that came about to solve the relatively new challenge of scaling web applications ^2 for millions of users.

The general view is that things will settle down in the future, with a few clear front-runners emerging next year ^3. In the mean time, a useful - and related - analogy is that of SQL. There were many different ways of communicating with relational databases, and a common syntax was needed ^4; SQL was the result of compromise and common ground between all those different query languages.

Whilst I don't think that NoSQL projects will merge to create a common system, it does seem likely that some will lag behind in their development, and be superceeded by the better-engineered solutions.

How to decide on a NoSQL option

In the mean time, all you can do is read read read. No one is going to tell you which path you should take; you need to research it yourself and fit it with your requirements before commiting.

Remember that there are three general camps for NoSQL systems:

• Key-value stores
• Column stores
• Document stores

And there is now redis, which straddles these camps.

My advice...

A good jumping off point is NOSQL: scaling to size and scaling to complexity (which gives a good high level overview of the concepts), then browse some of the posts over on MyNoSQL to see which projects are active and what new technologies are being added.

Good luck, and thanks to all the respondees on Hacker News.

• Milkshake Shop show how changing a user input error message for processing credit cards to be less...accusatory...has dramatically increased sales.

• I spent some time trying to setup GT.M and MD/B on Ubuntu to immerse myself in key/value databases. It has been around since 1986, and shows some impressive benchmarks (compared to the CouchDB, MongoDB, Tokyo crowd), but I didn't quite get mine off the ground. More research needed. One downside, and comeback on the criticism it presents against modern day key/value databases is that whilst it is open-source, the support is commercialised.

• kuro5hin proposes the greatest program ever written - a chess game for the Sinclair ZX-81, with AI, and with only 672 bytes of memory; astonishing.

• The Google phone (Nexus One) is out. I'm more inclined to agree with the "two years too late" and "iPhone competitor" camps, instead of the "wow fantastic" camp. Certainly wouldn't for out £330 for one offline whilst already owning a (first generation) iPhone.

• ...However, I would fork out £130 for the Boxee Box; it would replace an archaic laptop plugged into the TV via VGA.

I have been doing some reading into key-value databases (schema-free/NOSQL), and am particularly taken with redis, which has had some limelight of late.

Many people have been advocating it as an alternative to memcached as a web-app caching system, so I thought I'd dip my toe in the ocean of caching with redis.

Note that I don't actually need to cache anything, it's just the performance angle appeals to me.

After moving from the standard apt-get install nginx to a from-source version of Nginx (compiled with the Nginx redis module, of course), I was able to do a few basic tests with ab (ApacheBench).

Using the following command:

ab -n 1000 -c 50 localhost/

I get some results:

• Without redis caching: 2,864 requests per second (mean)
• With redis caching: 3,354 requests per second (mean)

Conclusive?

Not really, just indicative. We all know Nginx is very accomplished at serving static files, and this was a very simple "Hello" index page, 15 bytes long.

However, this would certainly make me think hard about deploying redis alongside Nginx for any low-write/high-read web applications or sites.

The results aren't particularly astounding, but the simplicity of integrating redis with Nginx and the fact that it is so transparent should make any Nginx user think about going down this route.

From a Mac OS X terminal window, you can use:

mate myfile.rb

to open a given file in TextMate.

You can also ask TextMate to open all the files and folders in the current directory as a project, so all files are accessible from the project drawer:

mate *

Useful functionality from MacroMates (developer of TextMate); one of those small features that makes you go "ahhh...that's nice".

The term category and tag are used interchangeably throughout this posting; they are assumed to be the same thing.

Having recently adopted Jekyll to power this website, I have been doing a bit of hacking/extending to get some added features in. A few days ago it was integrating Twitter with Jekyll, and now it's generating a tag cloud.

Back in July, Alex Young blogged about his Jekyll migration, and thoughtfully included a link to some code he wrote to list all posts broken out by category/tag.

I wanted to take this a bit further, and generate a per-category page which listed all the postings for that category, but also to generate a tag cloud.

Generating tag pages

After making a few changes to Alex's code, I ended up with a tag stub in a Rakefile which loops through all the categories used on the site and generates a static HTML page with a list of all the postings in that category.

Remember that this code snippet requires you to define your per-post categories in the YAML header of each post, e.g.

    categories:
  - jekyll
  - blog
  - ruby
  

(You need to mkdir tags in your Jekyll directory before executing the code below)

Now, the Rakefile segment:

    desc 'Generate tags page'
task :tags do
  puts "Generating tags..."
  require 'rubygems'
  require 'jekyll'
  include Jekyll::Filters
  
  options = Jekyll.configuration({})
  site = Jekyll::Site.new(options)
  site.read_posts('')
  site.categories.sort.each do |category, posts|
    html = ''
    html << <<-HTML
---
layout: default
title: Postings tagged "#{category}"
---
    <h1 id="#{category}">Postings tagged "#{category}"</h1>

    html << '<ul class="posts">'
    posts.each do |post|
      post_data = post.to_liquid
      html << <<-HTML
        <li>#{post_data['title']}</li>
      HTML
    end
    html << '</ul>'
    
    File.open("tags/#{category}.html", 'w+') do |file|
      file.puts html
    end
  end
  puts 'Done.'
end
  

There is also a gist here

Now you can run rake tags and it will generate a number of HTML files in the tags/ subdirectory; regenerating through Jekyll will then copy these files over to your site. Navigating to /tags/jekyll.html should list all your Jekyll related posts.

Generating your tag cloud

The below snippet does something similar, but just loops through each category and counts the number of tagged postings. It then does some very rudimentary font-size scaling to make the more popular tags bigger.

    puts 'Generating tag cloud...'
require 'rubygems'
require 'jekyll'
include Jekyll::Filters

options = Jekyll.configuration({})
site = Jekyll::Site.new(options)
site.read_posts('')

html =<<-HTML
---
layout: default
title: Tag cloud
---

<h1>Tag cloud</h1>

    HTML

    site.categories.sort.each do |category, posts|
      html << <<-HTML
      HTML
      
      s = posts.count
      font_size = 12 + (s*1.5);
      html << "twitter?) with any queries or improvements, or post a comment below.

• A link and thread on the word for the "smell of rain" - most notably when rain falls on dry earth. That word is Petrichor.

• An interesting posting on The Physics of Space Battles - not tremendously scientific, but certainly an enjoyable diversion for the Sci-Fi inclined.

• Theodore Dalrymple's (pen name for Anthony Daniels) Not With a Bang But a Whimper: The Politics & Culture of Decline looks like a good last-minute stocking filler. An old boy at the pub claimed that Theodore "is a sage" - high praise.

• The Guardian's crowdsourcing MP expenses system is a fantastic demonstration of social technology. Simon Willison talks about the lessons learnt from building such a system (twice).

• Also from Simon Willison is a superb introduction to Node.js - it does look tempting.

Having migrated Justkez.com to be based on Jekyll, I was pondering how I might include my recent twitterings on the front page of the site. In the Wordpress world, this would have been done via a plugin which may or may not have hung the loading of the page, might have employed caching, but would certainly have had some overheads.

Not in Jekyll.

Integrating Jekyll and Twitter

It is rather simple to create a Ruby script to pull down your most recent Twitter updates and dump them into a file.

It is also simple to wrap each update in some rudimentary HTML.

We then use the Liquid include tag to insert the updates where desired.

The Ruby script

I have the following sat in my ~/bin/ directory, which can be executed by a cron job at whatever interval you see fit:

[code] require 'twitter' twitter_user = 'JohnDoe' # TODO: Change to your Twitter username puts '<ul id="twitter_list">' Twitter::Search.new.from(twitter_user).each do |r| d = DateTime.parse(r.created_at).strftime('%d %b') puts "<li><span class=\"gentle\">#{d}</span> #{r.text}