The Agile Developer

Localizing Gems (vendor everything) with Native Extensions

Matthew Deiters — Wed, 13 Feb 2008 16:24:00 +0000

Localizing all libraries used by a rails app is a good strategy to building robust systems. Localizing or 'vendor everything' mitigates version conflicts between applications and simplifies deployments as we move from one environment to another. It drastically increases the portability of our application and transparency of what we have dependencies on. Additionally it enables something I consider very important,checkout development, which essentially means a developer can check out the code from scm for the first time and be running the development environment...no lengthy "how to setup development environment" documents.

We localize all our dependencies (i.e. gems) in the enterprise environment since by nature we internally run our rails apps in a "shared hosting" environment and would frequently run into the above issues if we didn't localize. It is trival to localize pure ruby gems and there are even other gems to help us make it stupid easy to 'vendor everything', see http://gemsonrails.rubyforge.org/. Personally I don't find it that hard to just use the "gem unpack command" instead. But what about localizing gems that have native extensions...

Localizing Hpricot

If your rails app is doing any extensive XML processing you should avoid REXML as its terribly slow. Hpricot proves to be a nice XML parser for ruby but it depends on some c libraries. Since we localize everything we are going to need to figure out how to freeze the native extensions to our rails app. The tricky part of native extensions is they are unique to each OS/architecture. To set the stage we have to run our apps on several architectures:

Windows

Mac OSX Intel

32 Bit RedHat

64 Bit RedHat

1. Vendoring the gem

This is the easiest part, assuming you have the hpricot gem installed on you development box, run from the command line:
cd vendor gem unpack hpricot

You should now have the hpricot gem source in your vendor folder.

2. Building the native extension by hand

We need to understand the anatomy of a gem to do this next bit. Generally gems with native extensions will have a ext folder where the c source code and such live. When you typically install a gem the install process will compile these c source code and generate our ruby extension targeted for our particular OS Architecture and put the compiled library in under the lib folder of our gem...for example on mac the installed hpricot gem has a i686-linux folder with the compiled library under the hprict lib folder. But up until now we have only unpacked our gem to the vendor folder the native extension has not been built so our hprioct's lib folder will only have ruby source.

To build the native extension by hand we need to navigate the the C source code in the ext folder, create the make file, and then run the makefile to compile.
cd vendor/hpricot-0.6/ext/hpricot_scan ruby extconf.rb make

3. Moving the compiled library to the lib folder

If everything went smoothly above, then you will see a new file under the ext/hpricot_scan folder. On macs this is hpricot_scan.bundle. When you install a gem with "gem install hpriot" this file is moved for us to the lib folder, but because we are building by hand we need to move it. We want to move it to a folder under the vendor/hpricot/lib folder target for the architecture we just built. I do this by getting the OS platform as ruby sees it.

ruby -e "puts RUBY_PLATFORM"

On my mac, the output is "i686-darwin8.9.1" so I create a i686-darwin8.9.1 folder and put the hpricot_scan.bundle in this folder. I'll repeat step 2 (compiling the source files) above and create folders and copy the compiled library for each OS architecture (mac, various linux flavors, etc) I plan to run my rails app on. This might be a little time consuming if your app needs to run on 3 or 4 environments but once the libraries are built we can reuse this localized gem for all other rails projects. For our current project I have serveral folders with their own compiled library under the froozen hpricot's lib folder: i486-linux, i686-darwin8.9.1, i686-linux, x86_64-linux.

4. Configuring Rails

At this point we have our localized gem ready but we need to tell rails about it. In our envionrment.rb file we want to add:

config.load_paths << "#{RAILS_ROOT}/vendor/hpricot-0.6/lib" config.load_paths << "#{RAILS_ROOT}/vendor/hpricot-0.6/lib/#{RUBY_PLATFORM}"

This first puts the froozen hpricot in our load paths so we load up the localized gem in vendor. The second line is so we load up the right library for the right OS platform we are running on.

Let me know if you have any issues/questions: mdeiters@gmail.com

Enterprise Capistrano: Deploying to Servers With Unique Credentials

Matthew Deiters — Mon, 28 Jan 2008 19:40:00 +0000

Capistrano makes an assumption that all servers in your application stack share the same credentials. As much logical sense as this makes, in large enterprise landscapes it is often not a reality. The nature of this world (Enterprise) is that trying to shift the mindset of all shareholders (infrastructure, support teams, security, etc) and then changing long running practices is not always feasible. This is especially true when doing so on numerous severs servicing numerous other systems that depend on the already in place mismatched credentials. For us...we have our normal application servers and database servers that DO share matching credentials. But along with them we use a separate server env to store and host our assets (images, css, js) with its own credentials.

Still desiring to leverage capistrano for its core automation benefits we had to monkey-patch capistrano so we could supply server specific username and passwords. Drop this in your lib folder or somewhere else more relevant. WARNING: This is basically cut-paste from the capistrano internals with some added sugar - I take no responsibility for the class design and multiple responsibilities in this rather big method.

#This is a core class in the capistrano library. it is overridden so that we can support servers that have different user names and passwords module Capistrano class SSH class << self alias :original_connect :connect def connect(server, options={}, &block) special_server_settings = options[:server_authentication] ? options[:server_authentication][server.to_s] : nil if special_server_settings.nil? return original_connect(server, options, &block) else methods = [ %w(publickey hostbased), %w(password keyboard-interactive) ] password_value = nil ssh_options = (options[:ssh_options] || {}).dup ssh_options[:username] = special_server_settings[:user] || server.user || options[:user] || ssh_options[:username] ssh_options[:port] = server.port || options[:port] || ssh_options[:port] || DEFAULT_PORT begin connection_options = ssh_options.merge( :password => password_value, :auth_methods => ssh_options[:auth_methods] || methods.shift ) connection = Net::SSH.start(server.host, connection_options, &block) Server.apply_to(connection, server) rescue Net::SSH::AuthenticationFailed raise if methods.empty? || ssh_options[:auth_methods] password_value = special_server_settings[:password] password_value = password_value.call if password_value.is_a? Proc retry end end end end end end

Your capfile will look something similar to this:

role :app, 'thisapp.rocks.com' role :web, 'thisapp.rocks.com' role :db, 'thisdb.rocks.com' role :assets, 'my.asset.server01.com' set :user, 'the-primary-user' set :server_authentication, { 'my.asset.server01.com' => { :user => 'assetusr' , :password => 'password'}} } ....

2007 Retrospective

Matthew Deiters — Thu, 24 Jan 2008 04:58:00 +0000
I left ThoughtWorks a couple months ago and rejoined the ranks with independent contractors...since leaving TW I've had a chance to work in unique environments with Ruby and Rails. Where it seems the majority of Rails projects are product companies (i.e. startups, ecommerce and general web sites) I have for the most part worked with large organizations and now spend most of my time with a large global management firm (a.k.a "An Enterprise"). The obstacles and challenges are very different then the Rails sweet spot but they provide the opportunity to close the gaps ruby has from other platforms and fill a niche (and no I do not mean JRuby). I hope to roll out most of lessons learned into the community over this next year. The goal is to write useful content more frequently as I blogged very little in 2007 and wrote very little useful content for Ruby prior. The paucity of postings is mostly due to day to day work with my new clients and traveling:

NY 6 times

India

France

Thailand

Australia

Hawaii

Named Routes: When _url is better then _path

Matthew Deiters — Wed, 11 Jul 2007 17:04:00 +0000

After you configure a resource in your rails routes configuration, you have access to generate named routes with _path and _url. The main difference between _url and _path can be demonstrated in the following example.

map.resource :people

we can now do the following:

person_path(@person) #=> /people/1 person_url(@person) #=> http://localhost:3000/people/1

_path versus _url

So when do we use one over the other?

The current school of thought is that in the view we use _path (i.e. person_path) as everything is relative to the current url. When we need to do any redirection in the controllers we would use _url (i.e. person_url) because the browser needs a absolute url for redirection.

The perceived advantage of relative paths (_path) in our view is that the less characters you have like href="/people/1" over href="http://localhost:3000/people/1" then therefore less the bytes that have to be passed across the wire. That is good to note but by using _path in views you are potentially setting yourself up for some problems. Primarily reverse proxy issues that still continue to plague rails. If our rails application is sitting on a server, lets say http://app1.server.com, but we have it behind a load balancer and other various reverse proxies (i.e. single sign on) so that it is publicly accessible with url of "http://my.doman.com/rails-app/" then we will have some troubles. There are several issues to note with the reverse proxy problems and there are various solutions but for the sake of staying on topic lets only look at the _path _url problem. Standard views generally are fine with relative paths as they will generate '/person/2' so the browser typically will look at the url in our example as "http:/my.coman.com/rails-app/people/1". But any view rendered from an ajax request is going to look to the browser like "http:/my.coman.com/people/1" omitting the sub directory "rails-app".

My suggestion is that if you are not 100% sure of your deployment environment when initially developing a rails application, program a little more defensively for the sake of a couple bytes and use _url named routes instead of _path for all urls. By doing this you can be a little more prepared to adapt to your production environments as all the reverse proxy fixes will only work with absolute urls and not relative ones.

Updated Blog

Matthew Deiters — Wed, 11 Jul 2007 03:15:00 +0000
I finally ditched Typo and rolled my own blog site. Typo seemed to have a nasty habit of consuming all my VPS memory. Good bye Typo

RSPEC View tests that use helpers

Matthew Deiters — Wed, 02 May 2007 19:39:14 +0000

RSpec 0.9.1 is out and is a great library for BDD. But I have some problems when testing views. The syntax is a bit clunky when a view calls a helper method. Lets say I have a view:

<%= if current_user.is?(:admin) %> render some html for admins <% end %>

When testing my view specs 9.1 enables helpers to be loaded using the :helpers option for the render method

Example (lifted straight from rspec_rails file view_spec_spec.rb):

describe "Given a view requiring an explicit helper", :behaviour_type => :view do before(:each) do render "view_spec/explicit_helper", :helper => 'explicit' end it "the helper should be included if specified" do response.should have_tag('div', :content => "This is text from a method in the ExplicitHelper") end end

What this does is when the view is rendered it will mixin the helper ExplicitHelper dynamically. There are a couple problems with this but the main one being my view is now dependent on the helper's implementation. And since helpers are modules, gook luck trying to mock module methods. The second problem is in my example the current_user method is actually on the controller and not in a helper at all. So to resolve both issues I wanted to specify helper methods that I would mock with out depending on a helper module.

To do do this, the following code to the end of your spec_helper.rb.

module Spec module Rails module DSL class ViewEvalContext < Spec::Rails::DSL::FunctionalEvalContext def mock_helper_method method @controller.template.expects(method) end end end end end

This will allow you to right views that do not depend on any helper's implementation.

describe "/content/index", :behaviour_type => :view do it "should allow editing content when user is admin" mock_helper_method(:current_user).returns(mock) render '/content/index' response.should have_tag('div.edit-content') end end

Note: I am using the mocha library for mocking which uses the method 'expects'. If you are using rspec's built in mocking then replace:

@controller.template.expects(method)
with
@controller.template.should_recieve(method)

Build Action Plugin for CruiseControl.rb

Matthew Deiters — Tue, 01 May 2007 14:58:14 +0000
Download at: http://rubyforge.org/projects/cc-post-build/
I created this plugin to deploy to our integration environment every time the build is successful.

Installation:

Download .zip at ruby forge

Unzip files to cruisecontrol.rb /builder_plugins/installed directory

Configure different post build actions for each project in the cruise_config.rb file

Configuration should look like:

Project.configure do |project| # other configurations # enable action on successful build project.post_build_action.on_successful_build = "cap deploy_to_integration -p somepassword" end

Encapsulation 101

Matthew Deiters — Wed, 15 Nov 2006 19:24:27 +0000

Just cause you have state class doesn't mean you achieved object orientation

public class State { readonly int Id; }

The following breaks encapsulation

public void DoStuffWithState(State targetState) { if(state.id == targetState.id) { .... } }

This is much better

public void DoStuffWithState(State targetState) { if(state == targetState) { .... } }

Using Operator Overloading is key to achieve proper encapsulation

public class State { private readonly int Id; public static bool operator ==(State x, State y) { return x.Id = y.Id; } }

YAGNI and Designing For the Future

Matthew Deiters — Wed, 04 Oct 2006 15:28:22 +0000

YAGNI (You aren’t going to need it)

Simply stated "Always implement things when you actually need them, never when you just foresee that you need them." There are several reasons why this is good practice.

Designing for the Future
Once an application is created, there is one thing that is for sure…the system will eventually be asked to change to meet certain new requirements by its users.

That means the future of every application is change and therefore it would make sense to write software in a way for it to evolve gracefully. But when pairing with some one new to "real" object thinking...designing for the future seemed to conflict with YAGNI.

These two aspects of programming in fact are not contradictions at all but applicable in different contexts.

The ultimate goal of OOP is to provide a system with limited "Implementation" dependencies. Meaning Class A does not depend on how class b is implemented and so on. In a system, YAGNI would manifest itself when a developer would want provide some type of feature that they feel the business will eventually need OR introduce a measure of complexity just for "when" things change drastically.

I think the simplest example would be factories for the creation of every different type in a system (I am not sure why many have such a desire to abstract a constructor) At this point YAGNI (if it were a person) would say stick to the simplest code to implement the task and do only what the users/business ask for.

But we can still designing a system for the future without requiring additional complexity by:

narrowing down interfaces and responsibilities of class

Creating Behavior in a way so that it can be leveraged in other contexts

So what does that mean, lets take an example of Filtering an collection of object based on some criteria Lets say we have a filter class that should filter items based on its location. It can filter anything that implements IsupportLocationFiltering
interface ISupportLocationFiltering { State GetState(); } class FilterRule { private readonly State TargetState; public IList Filter(IList itemsToFilter) { IList filteredItems = new ArrayList(); foreach(IsupportLocatinFiltering supporter in itemsToFilter) if(items.GetState() == TagetState) filteredItems.Add(supporter) return filteredItems; } }
The above FilterRule breaks encapsulation and therefore is not designed for the future. The reason being is the filter asks for the State and proceeds to compare it against the state that we want to find items from. First we are assuming that all types that implement ISupportLocationFiltering can be filtered by matching states. Maybe we match states for all objects today but tomorrow what if we introduce a new type that can support "Location Filtering" but is done by checking to see if a zip code is from the target state. We can easily accommodate this by using encapsulation without adding complexity and achieving flexibility.
interface ISupportLocationFiltering { bool IsIn(State state); } class FilterRule { private readonly State TargetState; public IList Filter(IList itemsToFilter) { IList filteredItems = new ArrayList(); foreach(IsupportLocatinFiltering supporter in itemsToFilter) if(items.IsIn(TagetState)) filteredItems.Add(supporter) return filteredItems; } }
Now its up to each implementation of ISupportLocationFiltering to know if it can be filtered by a state. Our Interface is much more useful then just for filtering so we can accommodate future changes by renaming it to something more useable
interface ISupportALocation { bool IsIn(State state); }

Biztalk Orchestration to BPEL

Matthew Deiters — Tue, 08 Aug 2006 20:00:23 +0000

Another “awsome” feature in Biztalk is the power to export an orchestration to BPEL (Buisness Processing Execution Language).

Unfortunatly pretty much 99% of the things most BizTalk devs do in orchestrations exclude you from using this feature. Here is the list of some 20 “do not do“ if you want to leverage BPEL. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/ebiz_prog_orch_zopr.asp

I am not sure who is to blame...BPEL or BizTalk...anyone?

The Agile Developer

Localizing Gems (vendor everything) with Native Extensions

Localizing Hpricot

1. Vendoring the gem

This is the easiest part, assuming you have the hpricot gem installed on you development box, run from the command line: cd vendor gem unpack hpricot You should now have the hpricot gem source in your vendor folder.

2. Building the native extension by hand

3. Moving the compiled library to the lib folder

4. Configuring Rails

Enterprise Capistrano: Deploying to Servers With Unique Credentials

2007 Retrospective

Named Routes: When _url is better then _path

_path versus _url

Updated Blog

RSPEC View tests that use helpers

Note: I am using the mocha library for mocking which uses the method 'expects'. If you are using rspec's built in mocking then replace:

Build Action Plugin for CruiseControl.rb

Encapsulation 101

YAGNI and Designing For the Future

YAGNI (You aren’t going to need it)

Designing for the Future

Biztalk Orchestration to BPEL

This is the easiest part, assuming you have the hpricot gem installed on you development box, run from the command line:
cd vendor gem unpack hpricot

You should now have the hpricot gem source in your vendor folder.