Ruby

Sinatra, Sequel, HAML, PostgreSQL, UTF-8 and Ruby 1.9.1

Lately, with my friend and colleague Joseph, we made some experiments with the Sinatra Ruby Framework.

As part of the experiment we’ve chosen to stick with our DB of choice PostgreSQL and our preferred template engine HAML, but we decided to give the Sequel ORM a try as well as using Ruby 1.9.1. We also wanted to store our datas as UTF-8 (this part is the most painful of all).

Our first goal was to have a simple application tying everything together and testable with RSpec and Cucumber.

Using Sinatra and HAML is a snap, as it’s a core feature of Sinatra, just be sure to use HAML version >= 2.2.0 as it includes some work to support new Ruby 1.9 String Encoding.

Then comes Sequel, using it is as simple as requiring it and feeding it with database connection information, just be sure to set the :encoding => 'UTF-8' (cf : Sequel::Database.connect method).
Sequel is great in that it has adapters for most commons connectors, first we tried DataObjects’s do_postgres as it should support asynchronous query (and it does !), but we had to fall back to the PG one and even to a patched version.

Let me explain the problem here, and be warned it’s not limited to Sequel, but to any ORM using currently available db connectors, when using a charset different from ASCII-8BIT under Ruby 1.9.
What happens is that ORMs do not force any encoding on String returned by the database connectors even when you specified an encoding (commonly used to set the connection’s “client_encoding”). Current ruby connectors (under Ruby 1.9) do not use the database/client’s connection encoding as a “hint” to determine and set the encoding of returned values.
This is not a problem on Ruby 1.8, but on Ruby 1.9 you get some weird results, the string returned from db have a default encoding “ASCII-8BIT” (in fact default for BINARY), as ORMs do not force encodings this result in a String with the bad Encoding.
Try to display it on a page and you’re welcomed with friendly “incompatible character encodings: ASCII-8BIT and UTF-8.” messages or try to use Webrat with RSpec matchers and you get “incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)”.

So here are the libraries versions to use to have a working Sinatra, Sequel with Postgres, HAML, RSpec, Cucumber stack :

• Sinatra >= 0.9.4
• Rack-Test >= 0.4.1
• Webrat >= 0.5.0
• kamk-pg >= 0.8.0.3 (http://github.com/kamk/pg/tree/master)
• Sequel >= 3.0.0
• RSpec >= 1.2.8
• Cucumber >= 0.3.94

Then you need to monkey-patch Rack (this is highly untested, it worked for my current app but it should not be used in production environment) :

module Rack
  module Utils
    def escape(s)
      regexp = case
        when RUBY_VERSION >= "1.9" && s.encoding === Encoding.find('UTF-8')
          /([^ a-zA-Z0-9_.-]+)/u
        else
          /([^ a-zA-Z0-9_.-]+)/n
        end
      s.to_s.gsub(regexp) {
        '%'+$1.unpack('H2'*bytesize($1)).join('%').upcase
      }.tr(' ', '+')
    end
  end
end

This does 2 things :

• it’s using bytesize($1) to correctly handle multibyte chars (taken from http://github.com/rack/rack/commit/ce26ede59d9e833ee233edff8d9ec73c6ecb0998)


• it’s using a regexp with “u” (unicode) modifier when dealing with UTF-8 Strings under Ruby 1.9.

There you go, you can finally test your Sinatra apps outside-in even when using non-us languages under Ruby 1.9. Isn’t that great ?[…]

RubyGems mirror update

Due to the increasing time/resources needed to generate the indices for my rubygems mirror, I decided to stop generating the ones for old RubyGems versions (<= 1.2.0) and switch to the —update option.

This greatly help with the load generated on the server by the mirroring/indexing process. From now on I will only generate the legacy indices (for RubyGems <= 1.2.0) twice a day : 12AM UTC+2/12PM UTC+2.

[…]

Creating your own RubyGem mirror

In case you want to create your own RubyGem mirror here’s how I did mine.

From now on, let’s pretend we will store the mirrored gems in : /data/rubygems.mirror

First you need to create a config file (YAML format) specifying the source and destination for your mirror (put it in /data/gemmirror.config for example) :

---
- from: http://gems.rubyforge.org
  to: /data/rubygems.mirror

- from is the master mirror you will pull from
- to is the directory where your mirror files will be stored

Then all you have to do is to add this commands as a cron task :

/usr/bin/gem mirror --config-file=/data/gemmirror.config && /usr/bin/gem generate_index -d /data/rubygems.mirror

/usr/bin/gem mirror —config-file=/data/rubygemmirror : will mirror the gems using informations in /data/rubygemmirror file
/usr/bin/gem generate_index -d /data/rubygems.mirror : will generate an optimized index to reduce the datas transfered on index update

Here’s what you asked for Nick ![…]

RSpec and HAML Helpers

Today I faced a problem with specing a helper where I use the haml_tag/puts methods (haml_tag was previously open).

Here’s a solution working with Rails 2.0.2, RSpec 1.0.3 and HAML 1.8.2 :

in spec_helper.rb add :

Spec::Runner.configure do |config|
  ...
  
  # Activate haml to spec helpers
  config.with_options(:behaviour_type => :helpers) do |config|  
    config.include Haml::Helpers  
    config.include ActionView::Helpers  
    config.prepend_before :all do  
       # Update from Evgeny comment, with HAML >= 1.8.2, you can call
       init_haml_helpers

       # Old way for HAML <= 1.8.2
       # @haml_is_haml = true  
       # @haml_stack = [Haml::Buffer.new]  
    end  
  end

  ...
end

then you can write your helper spec like this :

describe ApplicationHelper do
  describe "#top_navigation_menu when logged in" do
    it "returns the menu" do
      @user = mock_model(User)
      capture_haml {
        top_navigation_menu(@user)
      }.should_not be_empty
    end
end

The interesting method is capture_haml, which does the same as Rails builtin capture but for HAML.
haml_tag/puts methods write output directly to the buffer and does not return the generated content as a String, thus we cannot just test on the method output.[…]

RSpec 1.1

David Chelimsky anounced this morning the release of RSpec 1.1 (as of now the website is not up to date) RSpec 1.1 and just now the brand new website hosting.

What’s in this release ?

• StoryRunner : merge from Dan North ’s RBehave library. It aimed at providing a clean way to express and automate Acceptance Tests.

• Nested Example Groups : allows you to nest your describe blocks resulting in group sharing common specifications.

• Support for Rails 2.0.1

• Test::Unit interoperability : switch smoothly from Test::Unit to RSpec by allowing you to run your Test::Unit tests with RSpec. The goal here is to provide a way to progressively transition your tests to RSpec syntax.

More infos on David blog post about the RSpec 1.1 release

Congratulation and many thanks to everyone involved in this release ![…]

RubyConf 2007 videos online

I was really looking for that and it finally happened : Confreaks released RubyConf 2007 videos.[…]

RubyGems mirror update

My RubyGems mirror just got an update :

• switch to rubygems 1.9.5 0.9.5
• index generation

If all goes well, it should be compatible with older rubygems versions and give a significantly boost to those using rubygems >= 1.9.5. 0.9.5[…]

Date, Time and old days

Not so long ago a bug emerged on my current application, a vicious one.
Let’s say you have a user and store its bithday date in database (say MySQL) as a DateTime (yep I know, it’s a little to precise for this kind of data).
Rails conveniently retrieve this field as a Time, but as you probably do not be aware of is the range limitation of Time’s dates. In fact the Time class is often used because it’s faster than Date or DateTime, but the tradeoff is it handle date based on epoch time (01-01-1970), where Date/DateTime use Complex class and as such can handle a broader range of date.

Well, my user’s birthday date was somewhere in 60s and so can’t be handle by a the Time class. But it’s not as simple, Rails fallback to DateTime if parsing MySQL datetime field does not work with Time.parse. This results in an object perfectly valid when using it, the problem arise when you try to update the corresponding record in database.

Remember my initial datetime field was fetched as a DateTime ruby object and as Rails include all fields from the record on update it tries to translate the birthday to something MySQL recognize. ActiveSupport include some convenients methods to do that : Time#to_s and Date#to_s takes a parameter allowing us to do something like Time.today.to_s(:db) as well as some convenient Date to Time conversion. Unfortunately, no out of range detection is done in Date#to_time which leads me to the first headache. Then ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS hash does not contains a :db key so calling Date#to_s(:db) results in an erroneous string for the db. (this is corrected in rails edge but not in 1.2-stable branch).

All in all :

1. be aware of Time class limitation
2. ruby open classes are wonderful : just drop this code in your lib directory and that should make it until rails is patched.

module ActiveSupport
  module CoreExtensions
    module Date
      module Conversions
        # Add db and rfc822 format
        DATE_FORMATS.merge!({
          :db       => "%Y-%m-%d %H:%M:%S",
          :rfc822   => "%a, %d %b %Y %H:%M:%S %z"
        })
      end
    end
  end
end 

[…]

Ruby 1.8.6 and Date

If you have a localized application and use Gettext, you probably use some method to handle localized Date.
Mine was a widely used approach consisting of redefining various Date.rb constants (Date::ABBR_DAYNAMES, Date::MONTHNAMES, etc.) to use the Gettext Object#_() method.

Recently I was forced to use Windows™® as my primary work OS, so I installed latest Ruby One-Click Installer which comes with Ruby 1.8.6. (I’m running 1.8.5 on all other machines).

This is where I was bitten by the Ruby 1.8.6 Date change, all my lovely constants are now frozen and I can’t localized them anymore (at least so simply).

So stick with 1.8.5 as long as you don’t really need to upgrade, there are also some threads issues with this release… so you also have weird issues if you use backgroundrb.[…]