Cloning Permissions And ACLs On OS X

Sometimes permissions get messed up, it’s normally easy to fix, but if the problem also breaks your access control lists (ACLs) then the fix can be much more time consuming (especially when external drives are involved).

Most of the time, for a quick fix, I use the commands below to clone the permissions and ACLs from a known good folder to my broken folder. Usually, this is all I need.

chown $(stat -f%u:%g "$SRCDIR") "$DSTDIR"
chmod $(stat -f%Mp%Lp "$SRCDIR") "$DSTDIR"
(ls -lde "$SRCDIR" | tail +2 | sed 's/^ [0-9]*: //'; echo) | chmod -E  "$DSTDIR"

Either set the environmental variables as you need to or replace them directly (liberal use of sudo may also be required depending on the folder being updated).

I can’t take credit for the above commands, I discovered them on StackOverflow a good while ago and added them to my useful snippets - if I come across the original I will add the necessary credit.

Push Email With FastMail (Sieve) & IFTTT

As I was writing this post FastMail released a new app with push notification support, it wasn’t really for me as I prefer a unified inbox but if you think it might suit you - check it out.

I recently went through the process of prising my email from the jaws of Gmail and getting it into FastMail, the switch went pretty smoothly but I missed Mailbox’s push notifications (even if it gets IMAP support I probably won’t use it any more - having my email flowing through another cloud server never felt quite right).

The simplest thing to do would be to set up the email channel in IFTTT, forward all incoming mail to trigger@recipe.ifttt.com then use the iOS notifications channel to push the alerts. The main problem with this is related to privacy; the email body will also, at some point, end up on IFTTT’s servers.

To solve this problem I created a custom sieve rule inside FastMail’s advanced rules section that strips everything other than the subject. It’s pretty simple and looks like this:

if true {
  notify :method "mailto" :options ["[email protected]"] :message "/ $subject$ /";
}

This rule should probably go after your junk mail filters (which probably look similar to the below), unless you want to get notified about your junk mail too of course.

if not header :contains ["X-Spam-known-sender"] "yes" {
  if allof(
    header :contains ["X-Backscatter"] "yes",
    not header :matches ["X-LinkName"] "*" 
  ) {
    setflag "\\Seen";
    fileinto "INBOX.Junk Mail";
    stop;
  }
  if  header :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"] ["5"]  {
    setflag "\\Seen";
    fileinto "INBOX.Junk Mail";
    stop;
  }
}

I also have emails coming through to a work Google account that I like to get push notifications for (without using the Gmail app). IFTTT is a little limited as it only supports one email address in one email channel so I set up Gmail to forward all my emails to FastMail. I didn’t really want all this unrelated email cluttering up my inbox so I added another rule, just after my notification rule, to discard them:

if header :matches ["x-forwarded-for"] "[email protected]" {
  discard;
}

And that’s all there is to it really - I get a (nearly instant) push notification that I have a new email and I can open the mail app of my choice when it’s convenient. One added benefit seems to be a little extra battery life after turning off fetch & push in the native iOS Mail app.

Digitising My Negatives

As I mentioned before, I recently began shooting on film again (a little bit anyway). Of course, this meant I wanted someway of getting my images from the negatives and into my digital library for a bit of light editing and sharing.

I found myself on Amazon looking at the plethora of cheap negative scanners. Most of these consist of a 5mp CCD and a backlight; photos are scanned quickly, to JPEG, and mostly without the need to involve a computer. From what I could find though, this type of scanner has three problems: highly compressed JPEG output, relatively low resolution ‘scans’ and extremely mixed quality output.

Maybe I was worrying too much about image quality - but they weren’t good enough for me. I wanted RAW images and higher-resolution output.

The most obvious choice would’ve been something like the Plustek OpticFilm 8100 or a negative-compatible flatbed scanner. I could’ve scanned as TIFF at high resolution and have been done. The main problem with this solution was the price, I couldn’t justify the high cost for something I probably wouldn’t be doing often.

To this end, I decided to make the use of what I already owned (or could make pretty easily).

The Setup

The camera setup itself wasn’t too complicated, I used my Nikon D700 with and old 105mm Micro-Nikkor. The equipment isn’t massively important though, as long as you have a decent DSLR, mirrorless or high-end compact / bridge with a lens that can get close enough to fill the frame with a negative then you’re probably going to get higher quality shots than a cheap dedicated negative scanner. RAW shooting is a massive plus though; the results will need some white-balance correction.

All of this needs to be mounted on a fairly sturdy tripod that can take the weight of your setup pointing straight down.

A couple of things you will need to be able to do though: manual focus, or at least have the ability fix the focus, and a self-timer / cable release function. Shooting so close, things can get blurry quickly.

One useful little accessory is a macro focusing rail, it allows you to finely tune the focus without having to mess around with the camera’s settings too much. It can be especially helpful with older, heavier lenses that tend to fall out of focus when the camera is pointing towards the ground and nudged slightly.

Probably the most difficult bit of the whole setup was coming up with some way to backlight the negative a suitable amount and evenly. Fortunately an Amazon shipping box, printer paper, a torch and a lot of packing tape came to the rescue.

As I didn’t have anything suitable to diffuse the light directly under the negative I made a relatively long tube (appox. 30cm) and lined it with printer paper that curved up towards the negative-sized aperture I cut at one end of the box. This produced diffuse enough light that evenly lit the negative.

A special shout out should probably go to the torch I used, it was the extremely bright LED Lenser P7. This is probably the best torch I’ve ever bought, super-bright for normal torching with a neutral enough light temperature for small photography-related projects like this.

Now for the stuff that really matters…

The Settings

For my negatives I shot in manual mode: 1/50s, f/7.1, ISO 200. I left automatic white-balance enabled as I was shooting in RAW and the white balance would definitely have to be corrected in post-processing anyway.

I chose not to quite fill the frame with the negative to ensure I made the most of the lens’ sharpness in the centre. After cropping, most of my shots worked out at around 8mp, which was pretty good going and definitely better than the cheap negative scanners.

The Results

Straight out of the camera this is how the negatives looked:

Inverting the photo quickly got me to something that looked more sensible. The blue cast to the image is the nature of the colour film and this is what needs to be white-balanced away. This can take a lot of playing with to get right but once you’ve done it for a single image, it should be the same for the whole roll.

After a little pushing & prodding with your image editor of choice (mine is Aperture but I guess that won’t be true for much longer). You can get something that looks perfectly acceptable.

To be honest, this photo probably isn’t the best example, but you can find some of the better ones (B&W and colour) in my Flickr album.

Something that did surprise me during this process was the amount of dynamic range I got from the negatives by digitising them in this way, I could see details from the negatives that the original prints didn’t even give the smallest clue to. The large RAW photos also gave me a lot of latitude when I was editing, it was nice to maintain the atmosphere of film with the advantages of a digital editing workflow.

Did it take a while to do all this: yes, would I have been better off getting a scanner: possibly, would it have been anywhere near as satisfying or fun: definitely not!

Using GitLab Omnibus With Passenger

GitLab is a great self-hosted alternative to GitHub. I’d set it up for other people before but it always seemed to be more hassle than should be to update and maintain (especially with it’s monthly update cycle) so I’d never set it up for myself.

Thankfully GitLab now has omnibus packages available to make installation and maintenance much easier. Unfortunately these packages contain all of GitLab’s dependencies including PostgreSQL, Nginx, Unicorn etc.. This is great for running on a server dedicated to GitLab but not terribly useful for my setup.

I already had a Postgres database I wanted to make use of along with an Nginx + Passenger setup for running Ruby application. The following describes the configuration changes I needed to make to fit GitLab omnibus into my setup.

The first steps are to create a PostgreSQL user and database for your GitLab instance and install your chosen omnibus package from installation instructions, we need to add some config first.

The first bit of config goes in /etc/gitlab/gitlab.rb, this sets up the external URL for your GitLab instance, configures the database and disables the built-in Postgres, Nginx and Unicorn servers:

external_url "http://git.yourdomain.com"

# Disable the built-in Postgres
postgresql['enable'] = false

# Fill in the values for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_host'] = '127.0.0.1'
gitlab_rails['db_port'] = '5432'
gitlab_rails['db_username'] = 'username'
gitlab_rails['db_password'] = 'password'

# Configure other GitLab settings
gitlab_rails['internal_api_url'] = 'http://git.yourdomain.com'

# Disable the built-in nginx
nginx['enable'] = false

# Disable the built-in unicorn
unicorn['enable'] = false

Now you can run sudo gitlab-ctl reconfigure, this should set up all of GitLab’s configuration files correctly, with your settings, and migrate the database. You’ll also need to run sudo gitlab-rake gitlab:setup to seed the database (this is a destructive task, do not run it on an existing database).

The final bit of configuration goes in /etc/nginx/sites-enabled/gitlab.conf (this assumes you have Nginx + Passenger installed from their instructions):

server {
  listen *:80;
  server_name git.yourdomain.com;
  server_tokens off;
  root /opt/gitlab/embedded/service/gitlab-rails/public;

  client_max_body_size 250m;

  access_log  /var/log/gitlab/nginx/gitlab_access.log;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;

  passenger_ruby /opt/gitlab/embedded/bin/ruby;
  passenger_set_cgi_param PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";
  passenger_user git;
  passenger_group git;

  passenger_enabled on;
  passenger_min_instances 1;

  error_page 502 /502.html;
}

Most of the above configuration comes directly from GitLab’s omnibus configuration with a few customisations for Passenger. The main configuration options are correctly setting the user and group so there are no permission issues and ensuring that the correct directories exist in the $PATH variable to prevent errors in GitLab’s admin interface.

Currently, files uploaded into GitLab may not appear correctly using these instructions due to a permission issue, this should be corrected with a future omnibus release, more discussion can be found in this merge request.

And we’re done.

That’s about everything, hope it works for you too.

Monitoring Internet Connection Speed With Munin

Considering that an internet connection is now deemed to be a human right you’d think that ISPs, that generally seem to rip us off anyway, would’ve managed to make their connections nice and reliable, especially when you take into account that the internet has been round for a good few years now and they’ve had plenty of time to get it right. Unfortunately this isn’t the case & I decided that I wanted a way to make sure that I wasn’t getting (too) ripped off by my ISP.

To this end I decided to make the use of the tools I already had available, a Speedtest to test the connection.

(The following is entirely Debian & Munin 2 biased, you may need to tweak it for your particular setup.)

The first job was to find some way to run Speedtest from the command line, fortunately while I was stumbling around the internet I came across speedtest-cli which makes life much easier. So first we need to get a copy of the script and put it somewhere:

git clone git://github.com/sivel/speedtest-cli.git speedtest-cli

(You’ll probably be needing to make sure you have a copy of Python installed too, for more info check out speedtest-cli’s GitHub page.)

Then we need to get some useful numbers from the script, we do this as a cron job because the script can take a while to run, use a lot of bandwidth & tends to timeout when run in Munin directly.

Create the file /etc/cron.d/speedtest containing the following (modifying the speedtest-cli path to suit your needs of course):

0,20,40 * * * * root /usr/local/speedtest-cli/speedtest-cli --simple > /tmp/speedtest.out

Finally we need a Munin script to create the graphs, the script below should go in /etc/munin/plugins/speedtest, don’t forget it make it executable too, or it might not run (chmod +x /etc/munin/plugins/speedtest):

#!/bin/bash

echo "graph_category network"
echo "graph_title Speedtest"
echo "graph_args --base 1000 -l 0"
echo "graph_vlabel DL / UL"
echo "graph_scale no"
echo "down.label DL"
echo "down.type GAUGE"
echo "down.draw LINE1"
echo "up.label UL"
echo "up.type GAUGE"
echo "up.draw LINE1"
echo "graph_info Graph of Internet Connection Speed"

OUTPUT=`cat /tmp/speedtest.out`
DOWNLOAD=`echo "$OUTPUT" | grep Download | sed 's/[a-zA-Z:]* \([0-9]*\.[0-9]*\) [a-zA-Z/]*/\1/'`
UPLOAD=`echo "$OUTPUT" | grep Upload | sed 's/[a-zA-Z:]* \([0-9]*\.[0-9]*\) [a-zA-Z/]*/\1/'`

echo "down.value $DOWNLOAD"
echo "up.value $UPLOAD"

Finally, restart munin-node (probably something along the lines of /etc/init.d/munin-node restart), give it an hour or so and enjoy your new statistics.

Check back soon.

Automatic Build Numbers In Xcode

As part of the project I’m working on at the moment I wanted a way to automatically update a targets build number in Xcode. My preferred setting for the build number is the current SCM revision, the script below automatically updates the build number in the Info.plist file each time the application is built.

Just add the script as a ‘Run Script’ build phase for your chosen target in Xcode. The script is only designed for Git, SVN & Git-SVN as they’re all I use at the moment.

#!/bin/bash

#
# Automatically sets the build number for a target in Xcode based
# on either the Subversion revision or Git hash depending on the
# version control system being used
#
# Simply add this script as a 'Run Script' build phase for any
# target in your project
#

git status
if [ "$?" -eq "0" ]
then

    git svn info
    if [ "$?" -eq "0" ]
    then
        buildNumber=$(git svn info | grep Revision | awk '{print $2}')
    else
        buildNumber=$(git rev-parse HEAD | cut -c -10)
    fi
elif [ -d ".svn" ]
then
    buildNumber=$(svn info | grep Revision | awk '{print $2}')
else
    buildNumber=$(/usr/libexec/PlistBuddy -c "Print :CFBundleVersion" Source/Application-Info.plist)
fi

/usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber" Source/Application-Info.plist

Check back soon.

'Merge All Windows' AppleScript

A friend reminded me of how nice it would be to merge all the windows in Safari back together using LaunchBar (or QuickSilver for that matter). So I wrote a quick AppleScript to accomplish the task:

on gui_scripting_status()
  tell application "System Events"
    set ui_enabled to UI elements enabled
  end tell
  if ui_enabled is false then
    tell application "System Preferences"
      activate
      set current pane to pane id "com.apple.preference.universalaccess"
      display dialog "The GUI scripting architecture of Mac OS X is currently disabled." & return & return & "To activate GUI Scripting select the checkbox \"Enable access for assistive devices\" in the Universal Access preference pane." with icon 1 buttons {"Okay"} default button 1
    end tell
  end if
  return ui_enabled
end gui_scripting_status

on click_menu(app_name, menu_name, menu_item)
  try
    tell application app_name
      activate
    end tell
    tell application "System Events"
      click menu item menu_item of menu menu_name of menu bar 1 of process app_name
    end tell
    return true
  on error error_message
    return false
  end try
end click_menu

if gui_scripting_status() then
  click_menu("Safari", "Window", "Merge All Windows")
end if

The ‘gui_scripting_status()’ routine is taken and modified from code that can be found here: http://www.macosxautomation.com/applescript/uiscripting/index.html.

Check back soon.

Clearing Out Old Sessions

A while ago I started setting up my websites to use ActiveRecord as a session store, this means that the session information for all the visitors to my website is placed in a table in my database. It may or may not be the best way to store sessions but it’s certainly faster than the filesystem and my VPS doesn’t really have the memory capacity for an ‘in memory’ store.

Anyway, one day I decided to perform some DB maintenance, check tables where okay etc, upon logging into the DB I noticed that my sessions table had grown quite large, almost 125,000 records, little did I realize that the sessions are persisted forever in the DB.

I didn’t think it was the best idea to keep all the session data so wrote the following script and put it in lib/tasks/session.rake:

namespace :session do
  desc "Prune old session data"
  task :prune => :environment do
    sql = ActiveRecord::Base.connection()
    sql.execute "SET autocommit=0"
    sql.begin_db_transaction
      response = sql.execute("DELETE FROM sessions WHERE `updated_at` < DATE_SUB(CURDATE(), INTERVAL 1 DAY)");
    sql.commit_db_transaction
  end
end

This gave me a ‘session:prune’ rake task. The task removes all sessions older than 1 day from the sessions table. I then added a CRON job for in the following format:

0 0 * * * cd /home/user/railsapp && rake RAILS_ENV=production session:prune > /dev/null 2>&1

The job above basically calls the session:prune rake task at midnight every night.

The code in the task in MySQL specific but without a model representing the session table I couldn’t (or at least couldn’t think of a way) to make the code any more ruby-fied. In the event that you do have, or decide to create a session model the following code may work in your task (warning: untested):

Session.destroy_all("created_at" < (Time.now - 1.day))

Hope the above snippet solves at least one of your ActiveRecord session woes.

Check back soon.

Clearing Out The rFlickr Cache

Assuming you’ve followed the Caching Your Photographs tutorial at some point, you’ll probably have had a lot of fun either deleting the cache every time you upload a new photo or you’ve written your own automated method by now. For those of you that haven’t written your own method of dumping the cache yet, here’s how I do it.

First of all, I created a lib/actions folder in the root of my rails project. Inside this folder I created the file ‘photography_action.rb’ with the following contents:

class PhotographyAction
  def self.clear_cache
    ActionController::Base.new.expire_fragment(%r{photography.cache})
  end
end

The above fragment naming assumes that your photos are on a page called ‘photography’ if they are elsewhere, change the fragment to expire that page instead.

Fairly simple I think you’ll agree, you may also be asking yourself ‘why the extra file?’, the main reason for the new file is so that the cache clearing can be executed from a new rake task that doesn’t remove all your cached pages or from an admin page on the website.

You’ll also need to update your config.load_paths in environment.rb. After updating, mine looks like this:

config.load_paths += %W( #{RAILS_ROOT}/app/sweepers #{RAILS_ROOT}/lib/actions )

Inside some action in some, preferably protected, controller somewhere, add the following (redirecting to anywhere you fancy):

PhotographyAction.clear_cache
redirect_to :action => 'index'

Now for the rake task. Inside the directory lib/tasks (create it if it doesn’t exists) create the file photography.rake then put the following code inside the file:

namespace :photo do
  namespace :cache do
    desc "Clear out photography cache"
    task :clear => :environment do
      PhotographyAction.clear_cache
    end
  end
end

You should then be able to run:

rake photo:cache:clear

From the base of your project in order to clear the cache.

Bear in mind, the code above is literally just a convenient way of clearing out the fragment cache so new photos show up on your photo page, it does not delete photos, nor does it perform a refresh automatically, although, you could add it to a CRON job.

When I get chance, I intend to automate this process and build it into rFlickr along with a new, improved, caching mechanism. I’m sure the above will tide you over for now though.

Check back soon.

New rFlickr Ruby Gem

After I started to use the rFlickr gem it didn’t take me long to realize that development of the gem had all but halted, yes it worked, which was more than the original Flickr gem did, but it was still a little bit out of date and in the end, a little bit broken.

In one of my older posts I documented a fix for the gem and provided a download to unzip into your plugins folder, however, with the advent of the wonderful GitHub and its marvelous gem support I’ve decided to move the project onto GitHub.

I have preserved the original gem’s GPL license and copied the source code from its original repository on RubyForge to a new, public, GitHub repository. In the process of the move I have dropped old code from the project, updated the readme & license information and generally performed a little house-keeping.

You can find the project at: http://github.com/digitalpardoe/rflickr/. You can install the gem using one of the following methods. First involves adding GitHub as a gem source (always a good idea) and installing the gem:

gem sources -a http://gems.github.com
sudo gem install digitalpardoe-rflickr

The second method it to add the gem as a gem dependency to the environment.rb of your Rails project:

config.gem 'digitalpardoe-rflickr', :lib => 'flickr', :source => 'http://gems.github.com'

And run a rake task to install the gem:

sudo rake gems:install

Whilst performing the code migration I also added the fix that was documented in my original post and implemented support for the (not so) new ‘farm’ based Flickr URLs for images (which should make things easier to implement).

The future plans for rFlickr include new tests, improved usage examples, updated readme / documentation and implementation of missing API methods, time permitting of course.

Until the readme is updated please refer to the original post for information on how to use rFlickr.

That’s all for now, enjoy the new gem and as they say, if you don’t like it, fork it.

Calculating Work Days In Java

In work this week I came across a couple of problems in which I needed to performs some calculations involving dates and a number of “work days” rather than just normal numbers of days. My first few attempts filed miserably, so I did some Googling to see if I could find anyone else that had come across the same problem and of course, there were plenty of people.

The code I eventually used I found shoehorned into the middle of a coding help forum and, with a little bit of tweaking, it calculated the number of whole work days in between two dates. The code is as follows:

public static int calculateDuration(Date startDate, Date endDate)
{
  Calendar startCal = Calendar.getInstance();
  startCal.setTime(startDate);

  Calendar endCal = Calendar.getInstance();
  endCal.setTime(endDate);

  int workDays = 0;

  if (startCal.getTimeInMillis() > endCal.getTimeInMillis())
  {
    startCal.setTime(endDate);
    endCal.setTime(startDate);
  }

  do
  {
    startCal.add(Calendar.DAY_OF_MONTH, 1);
    if (startCal.get(Calendar.DAY_OF_WEEK) != Calendar.SATURDAY && startCal.get(Calendar.DAY_OF_WEEK) != Calendar.SUNDAY)
    {
      workDays++;
    }
  }
  while (startCal.getTimeInMillis() <= endCal.getTimeInMillis());

  return workDays;
}

This code solved my first problem, however I couldn’t find any code to solve my second problem, how to calculate a final date from a start date and a specified number of work days, to do the calculation I came up with the following code:

public static Date calculateEndDate(Date startDate, int duration)
{       
  Calendar startCal = Calendar.getInstance();

  startCal.setTime(startDate);

  for (int i = 1; i < duration; i++)
  {
    startCal.add(Calendar.DAY_OF_MONTH, 1);
    while (startCal.get(Calendar.DAY_OF_WEEK) == Calendar.SATURDAY || startCal.get(Calendar.DAY_OF_WEEK) == Calendar.SUNDAY)
    {
      startCal.add(Calendar.DAY_OF_MONTH, 1);
    }
  }

  return startCal.getTime();
}

Hope this code works as well for you as it did for me.

Check back soon.

Update: Improved the calculating code with the suggestion from Anass (below) and made some more accuracy & business logic improvements to the calculateDuration method. Enjoy.

Flickr BBCode

phpBB is one of the most widely used forum systems on the internet and Flickr is one of the most popular photo sharing websites, unfortunately there is still no concrete way to integrate the two of them. As a temporary solution I came up with the following piece of BBCode to allow for the easy display of Flickr images in phpBB forums.

First, you will need to navigate to the “Admin Control Panel” of phpBB then to the “Posting” section. From here navigate to the screen that allows you to add custom BBCode.

In the “Usage” section paste the following code:

[flickr=<a href="{URL1}" title="{TEXT1}"><img src="{URL2}" width="{NUMBER1}" height="{NUMBER2}" alt="{TEXT2}" /></a>][/flickr]

In the “HTML” section paste:

<a href="{URL1}" title="{TEXT1}"><img src="{URL2}" width="{NUMBER1}" height="{NUMBER2}" alt="{TEXT2}" /></a><br /><a href="{URL1}">{TEXT2}</a>

Then in the “Help” section paste:

Flickr: [flickr=<Flickr, Medium Size, Copy & Paste HTML>][/flickr]

And that’s it, you can use the tag by navigating to one of your photo’s Flickr “All Sizes → Medium” page (e.g. http://www.flickr.com/photos/digitalpardoe/2900618617/sizes/m/), copying the code in the “1. Copy and paste this HTML into your webpage:” box. And inserting it after the “=” in the Flickr BBCode tag, e.g.:

[flickr=<a href="http://www.flickr.com/photos/digitalpardoe/2900618617/" title="Isolation by digital:pardoe, on Flickr"> <img src="http://farm4.static.flickr.com/3095/2900618617_26bc8abc12.jpg" width="500" height="334" alt="Isolation" /></a>][/flickr]

Check back soon.

Caching Your Photographs

If you have at some point followed my now fairly ancient rFlickr tutorial you may have noticed that your photo page loads quite slowly, and that my photo page loads fairly quickly. To get my page to load as quickly as it does required a small custom caching method and a willingness on my part to sacrifice some bandwidth. Here’s how I did it.

This tutorial assumes that you have already worked through the previously mentioned rFlickr tutorial and have something similar to it set up. It also assumes that you have some knowledge of Rails, not that my knowledge was particularly wide ranging at the point I wrote this caching method.

First of all, you will need a table in your database to store the information about your photographs, I suggest the structure illustrated in the migration below:

class CreatePhotos < ActiveRecord::Migration
  def self.up
    create_table :photos, :id => false do |t|
      t.column "flickr_id", :string, :limit => 25, :null => false
      t.column "title", :string, :limit => 250
      t.column "description", :text
      t.column "url", :string, :limit => 250
    end

    add_index :photos, :flickr_id
  end

  def self.down
    drop_table :photos
  end
end

Once you have created this table you will need to create some folders to store the cached images, I created the following folders and will be using them throughout this tutorial:

"#{RAILS_ROOT}/public/images/flickr_cache/small/"
"#{RAILS_ROOT}/public/images/flickr_cache/large/"

Then generate the model for this photos table:

cd /your/rails/application
./script/generate model Photo

Your view from the first tutorial can remain almost the same (details at the end of the post), however, to see the greatest speed improvement I suggest caching it, i.e:

<% cache do %>
... Your view code here. ...
<% end %>

Then modify your view method in your photography controller to read something like:

def view
  unless read_fragment({})
    check_cache
    @photos = Photo.find(:all)
  end
end

The above code will make sure that a cached photography page doesn’t already exist, if it doesn’t, then and only then will it check that the photograph cache is up to date and query the database.

We have not yet created a check_cache method, this method is the core method to make the photography page load much, much faster, even when the photography page’s cache does not exist. The method should be placed as the last method in your photography controller, the code is as follows:

private
def check_cache
  if ENV['RAILS_ENV'] == 'production'
    flickr = Flickr.new(RAILS_ROOT + "/config/flickr.cache", FLICKR_API_KEY, FLICKR_SHARED_SECRET)
    @photos = flickr.people.getPublicPhotos(flickr.people.findByUsername(FLICKR_USERNAME))

    @db_photos = Array.new
    Photo.find(:all).each { |p| @db_photos.push(p.flickr_id) }

    for photo in @photos.reverse
      if [email protected]_photos.include?(photo.id)

        db_photo = Photo.new
        db_photo.flickr_id = photo.id.to_i
        db_photo.title = photo.flickr.photos.getInfo(photo.id).title
        db_photo.description = photo.flickr.photos.getInfo(photo.id).description
        db_photo.url = photo.flickr.photos.getInfo(photo.id).urls.values[0]

        db_photo.save

        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/small/" + photo.id + ".jpg"),"w").write(open(photo.url('s')).read)
        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/large/" + photo.id + ".jpg"),"w").write(open(photo.url).read)
      end
    end
  end
end

The following code will only run if you are in production mode (and probably test mode too). It will then load the necessary information from Flickr using the methods outlined in the rFlickr tutorial post. The method then iterates through the collection of photos from Flickr in reverse, so they appear in the same order in the database as the order they appear on Flickr.

It will then check if the photo already exists in the database, if it does not it will store a copy of the photograph’s information in the database and download previews of the images from Flickr to your server, previews of images can then be loaded from your server rather than Flickr’s slow servers.

To take advantage of the cache you will also need to modify your view to access the thumbnails from your newly created local repository rather than from Flickr’s servers, i.e:

<% for photo in @photos.reverse %>
  <%= image_tag("/images/flickr_cache/small/" + photo.flickr_id + ".jpg", :alt => photo.title) %>
<% end %>

Hope this goes some way to helping you improve the speed of your website.

Check back soon.

Update: As I have just been reminded in the comments, I forgot a piece of code to make this tutorial work correctly. You should put the line:

Either in the bottom of you environment.rb file or just under the ‘class’ line in the controller that is responsible for your photography page.

Update: The code in the check_cache method has been updated slightly, it now makes less round trips to the database, should speed things up if your DB server is in a different location to your application server. Also, any redirection problems you may have faced before should be solved by the new rFlickr ruby gem that has support for farm URLs built in.

rFlickr And Rails 2.0

Any of you that followed my tutorial on setting up rFlickr and have subsequently upgraded to Ruby on Rails 2.0 may have noticed some server errors on your photo pages, this is due to the fact that a single line of code in rFlickr does not work correctly with Rails 2.0.

Line 644 of lib/base.rb had to be modified from:

def from_xml(xml,photo=nil)

to:

def self.from_xml(xml,photo=nil)

Hope this gets you back on your feet, so to speak. Check back soon.

Update: You may also want to install the updated version of actionwebservice because it is no longer bundled with the Rails distribution. Other people have reported rFlickr not working because of a lack of this gem, however, the lack of this gem didn’t affect my setup at all.

Setting Hard Drive Icons In OS X

Another day and I’m finally getting around to writing another tutorial. This time I thought I concentrate on something I’ve had quite a lot of emails about, ’you’ve given me the icons, how do I go about using them’. I suppose it’s quite an important thing, if your supplying people with something, to tell them how to use it. (Warning: this is a Mac only tutorial).

In previous version of Mac OS X, before Leopard you could get away with applying HD icons though copy and paste alone, and from that point on everything would work fine. Unfortunately, one of the first things I noticed about applying HD icons in Leopard is that copy and paste gives bad results, especially for the scaled down versions of the icon in the new Finder sidebar.

I eventually found the correct way to apply the icons, it involves a little bit of Terminal usage but don’t be afraid. (I have a sneaking suspicion you may also need the Apple Developer Tools installed, these should have been included on your Leopard DVD). I believe this tutorial also applies to setting HD icons on previous versions of Mac OS X.

The first command we will execute in the Terminal copies the icon resource (.icns file) to the correct place on the target HD. You will need to fill in the paths in a way suitable for your system but you can do this by dragging and dropping onto the terminal window.

sudo cp -f /Path_to_Icon/icon_file.icns /Path_to_Drive_Root/.VolumeIcon.icns

Not too difficult was it. The next line will tell the HD to accept a thrid-party icon.

sudo SetFile -a C /Path_to_Drive_Root

The final line will restart the Finder so you can see the changes.

killall Finder

And that’s it. You should see your new icon in the Finder and the correct scaled down version in the Finder sidebar.

Hope you get it working. Check back soon.

Update: After many emails I have written a small utility called Set Icon that automates this process, if you are not feeling confident with the Terminal commands you can download it here.

Update: I’ve just got my hands on a copy of Snow Leopard and the above instructions still seem to work correctly, happy icon changing.

Using rFlickr

I said it was coming and here it is, my little tutorial on how to use the rFlickr Ruby on Rails gem to create a photography section like the one on my own website. The first thing to note is that pretty much all of the options available in the Flickr API (here) are available for use in rFlickr due to the fact it is all based around XML. There is a laborious process of configuration to go through, however, to make everything work, but once this is done you should have no problems.

Firstly install the rFlickr gem, I should at this juncture note the fact I am primarily a UNIX user so will aim these instructions at other UNIX users, mainly because I don’t know the specifics for Rails installations on Windows. So lets dive in ($ denotes the terminal prompt and \ denotes line continues below):

sudo gem install rflickr --include-dependencies

The second thing you will need to do is make sure you have a Flickr account with some photos on it then pay a visit to http://www.flickr.com/services/api/keys/ and sign yourself up for an API key, once you have generated the key make a note of the key itself and the ‘secret’ that you are given, you will be needing these quite a bit.

The next thing to do is to basically follow the tutorial here, albeit with a few modifications, I have re-written the tutorial in full below.

cd /your/rails/application
./script/console

To make differences clear the Rails console prompt will be shown as >>, don’t forget to replace the x’s with your information.

>> require 'flickr'
>> API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> SHARED_SECRET = "xxxxxxxxxxxxxxxx"
>> flickr = Flickr.new("/tmp/flickr.cache", API_KEY, SHARED_SECRET)
>> flickr.auth.token
=> nil

The above just sets up your rFlickr object and makes sure that you don’t already have a key.

>> flickr.auth.getFrob

This returns a value that you will need to save somewhere.

>> flickr.auth.login_link
=> "http://some.link.flickr.com"

Click or copy the link you are given into a browser and authorize the API for usage, don’t worry, we’re almost there.

>> flickr.auth.getToken('that_frob_number_we_saved' )
>> flickr.auth.cache_token
>> exit

Right, this is as far as the tutorial online goes, but there are some other useful steps we need to take to make everything more useable, mainly the moving of the token as the /tmp directory may get cleared by our host.

cp /tmp/flickr.cache /your/rails/application/config/flickr.cache
rm /tmp/flickr.cache

Now we can get onto the actual programming and leave the authentication business behind.

We’re going to need a controller to use, for the purposes of this tutorial I will use a controller named Photography, it should save me some time as that’s what mine is called, the page to be rendered will be called view.

In the file photography_controller.rb we will need the following information, rename as necessary to your application.

class PhotographyController < ApplicationController
  API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  SHARED_SECRET = "xxxxxxxxxxxxxxxx"

  def index
    view
    render :action => 'view'
  end

  def view
    flickr = Flickr.new(RAILS_ROOT + "/config/flickr.cache", API_KEY, SHARED_SECRET)
    @photos = flickr.people.getPublicPhotos(flickr.people.findByUsername("YOUR_FLICKR_NAME"))
  end
end

Then in the view.rhtml that you will have created in your views folder (or at least, are creating now) paste the following code.


<% for photo in @photos %>
  <a href="<%= photo.flickr.photos.getInfo(photo.id).urls.values %>"><img src="<%= photo.url('s') %>" /></a>
<% end %>

And that’s it, your basically done, all the thumbnails will link directly to your Flickr page, easy wasn’t it? The main problem occurs when you load your newly created page, it’s very, very slow, due to the speed of the Flickr API (I think). In order to improve the situation I would recommend using either page or fragment caching, but that will be covered in a future tutorial.

Hopefully this will have given you a few pointers in using the rFlickr gem, read through the Flickr API for more inspiration if you are feeling adventurous. Check back soon.

Update: If you are having problems with Rails 2.0, take a look at this fix.

Update: I have written a new tutorial on caching your photos page, that should speed it up a lot, assuming you are having problems.

TextMate, Java & Compiling [Updated]

For those of you that do not already know TextMate is a brilliant text editor for Mac OS X, including syntax highlighting and the most useful snippets system I have ever come across. If your a budding coder and want to try it out visit: macromates.com, and download a copy.

This post illustrates some of the bundle items I have either modified or added to TextMate to make it easier to use for Java development. If you use TextMate already then hopefully you will know how to use the bundle editor if you don’t then there is some good documentation available here.

In the code snippets below a ‘\’ at the end of a line indicates the code continues but is illustrated on the line below.

  • Compile, but no run.
echo "<h2>Compiling $TM_FILENAME</h2>"
cd "$TM_DIRECTORY"
javac -encoding UTF8 "$TM_FILENAME" &> >("${TM_RUBY:-ruby}" -rtm_parser -eTextMate.parse_errors)
  • Run in terminal.
osascript <<EOF
  tell application "Terminal"
    activate
    try
      close front window
    end try
    do script "cd '$TM_DIRECTORY'; java '${TM_FILENAME%.java}'"
  end tell
EOF
  • Compile and run in TextMate.
echo "<h2>Compiling $TM_FILENAME</h2>"
cd "$TM_DIRECTORY"
javac -encoding UTF8 "$TM_FILENAME" &> >("${TM_RUBY:-ruby}" -rtm_parser -eTextMate.parse_errors)
if (($? >= 1)); then exit; fi

{ java -Dfile.encoding=utf-8 "${TM_FILENAME%.java}" echo -e "\nProgram exited with status $?."; }|pre
  • Compile and run in terminal.
echo "<h2>Compiling $TM_FILENAME</h2>"
cd "$TM_DIRECTORY"
javac -encoding UTF8 "$TM_FILENAME" &> >("${TM_RUBY:-ruby}" -rtm_parser -eTextMate.parse_errors)
if (($? >= 1)); then exit; fi

osascript <<EOF
  tell application "Terminal"
    activate
    try
      close front window
    end try
    do script "cd '$TM_DIRECTORY'; java '${TM_FILENAME%.java}'"
  end tell
EOF
  • Compile and run package item. This also relies on you setting a CLASSPATH variable in you .bashrc, and will only work for packages one deep, still useful if you are just starting out with Java.
echo "<h2>Compiling $TM_FILENAME</h2>"
cd "$TM_DIRECTORY"
javac -encoding UTF8 "$TM_FILENAME" &> >("${TM_RUBY:-ruby}" -rtm_parser -eTextMate.parse_errors)
if (($? >= 1)); then exit; fi

export CURRENT_DIR=`pwd | sed -n 's/.*\///p'`

{ java -Dfile.encoding=utf-8 $CURRENT_DIR."${TM_FILENAME%.java}" echo -e "\nProgram exited with status $?."; }|pre

This post was just an illustration of how to make compilation in TextMate for Java more useable, in the next post I will be adding some new snippets, the ones that are missing from the original TextMate bundle.

Update: I have now (finally) expanded on this post, you can read about more changes here.

Rich Text File (RTF) Into NSTextView, Cocoa & Obj-C

I was searching for an example of how to do this for ages and came up with nothing. All the pros out there will probably say it is easy and give shorter ways of doing this but for noobs and people not in the know I wrote this little snippet. This is especially useful for creating easy to maintain change logs or for adding simple read-me and help files to your application.

First of all you will need to create a NIB file with some sort of controller class in it. In the controller class create an outlet of type ‘NSTextView’, for this example I will call it ‘controller_Out_TextView’. Then generate the files for this controller so they appear in your Xcode project.

Secondly, create a window and add an NSTextView to it from the palette, connect the NSTextView to the outlet in the controller.

You will then need to create your rich text file with the necessary text in it, make sure it is added to your project and will be copied to the application bundle. For this example the text file will be called (rather originally) file.rhtml.

Finally add the following code to the ControllerName.m file between the @implementation and @end, or in your awakeFromNib method, if you already have one.

- (void)awakeFromNib
{
  NSBundle * myMainBundle = [NSBundle mainBundle];
  NSString * rtfFilePath = [myMainBundle pathForResource:@"file" ofType:@"rtf"];
  [controller_Out_TextView readRTFDFromFile:rtfFilePath];
}

Compile and run, hoepfully all of your text will be displayed.

Textilize Alternative

The textilize methods in Rails has never really been what I wanted. When it wasn’t adding paragraphs it was messing with my line brakes but it made such a good job of making HTML and XHTML that validates. In order to make it work better I re-wrote the method to be more suitable.

Put the following in the application_helper.rb file and call it using custom_text(“String”).

def custom_text(text)
  if text.blank?
    ""
  else
    text = RedCloth.new(text).to_html
    if text[0..2] == "<p>" then text = text[3..-1] end
    if text[-4..-1] == "</p>" then text = text[0..-5] end
    text = text.gsub("<p>", "")
    text = text.gsub("</p>", "")
    return text
  end
end

Job done, you should get correct HTML and XHTML but no paragraphs and no messing with line breaks.

An Email Form with Ruby on Rails

The one thing that I’ve been struggling with the most during my Ruby on Rails learning curve is how to create a simple email form for any web applications I may want to create. There was nothing to be found on Google so in my initial attempt I worked through the ActionMailer documentation on the RoR wiki website. I put it all together and, feeling really pleased with myself, clicked the send button. Nothing at all, just lists of errors, so I modified it and still more errors. After posting on Ruby Forum and with the help of a talented member I finally got it working.

This post is for all the people like me who don’t know what to do.

First of all, this tutorial assumes that you already have a basic RoR web application set up with a ‘Contact’ controller and a main view for the index in the controller.

Now the real work starts. Open up a terminal session (or SSH) and navigate to your rails application’s directory (for this tutorial the rail application will be ‘rails_app’).

cd /rails_app

Then create the mailer, ‘Emailer’ is the name of the model that will be produced.

./script/generate mailer Emailer

Now, that’s all the terminal work done. Lets start the setup.

In your chosen editor open the file ‘/rails_app/config/environment.rb’ and place the following inside the ‘Rails::Initializer’ block customizing all the necessaries to your SMTP server’s configuration.

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  :address => 'localhost',
  :domain => 'website.co.uk',
  :port => 25,
  :authentication => :login,
  :user_name => "smtp_username",
  :password => "smtp_password"
}

config.action_mailer.perform_deliveries = true
config.action_mailer.raise_delivery_errors = false
config.action_mailer.default_charset = "utf-8"

Now open the file ‘/rails_app/app/models/emailer.rb’ and make it look like the below, again customizing the necessaries (but do not touch the variables).

class Emailer < ActionMailer::Base
  def contact_email(email_params)
    # You only need to customize @recipients.
    @recipients = "[email protected]"
    @from = email_params[:name] + " <" + email_params[:address] + ">"
    @subject = email_params[:subject]
    @sent_on = Time.now
    @body["email_body"] = email_params[:body]
    @body["email_name"] = email_params[:name]
    content_type "text/html"
  end
end

In the controller for your contact page you will need to make the following changes, basically, adding a definition for ‘send_mail’.

def send_mail
  Emailer::deliver_contact_email(params[:email])
end

Go to the ‘/rails_app/views/emailer’ folder and create a file called ‘contact_email.html.erb’. In this file place the following code.

<p>Name:</p>

<p><%= @email_name %></p>

<p>Message:</p>

<p><%= @email_body %></p>

The final step is to create a form on the ‘Contact’ page that will allow the website to user to input and submit the email. An example for is illustrated below.

<% form_for :email, :url => { :controller => 'contact', :action => 'send_mail' } do |f| %>
  <%= f.label :name %><br />
  <%= f.text_field :name, :size => 60 %><br />
  <%= f.label :address %><br />
  <%= f.text_field :address, :size => 60 %><br />
  <%= f.label :subject %><br />
  <%= f.text_field :subject, :size => 60 %><br />
  <%= f.label :body %><br />
  <%= f.text_area :body, :rols => 10, :cols => 60 %><br />
  <%= submit_tag "Send Email "%>
<% end %>

That’s about it really, you may want to add your own error handling as this example only covers the basics. If you go to the contact page you can see a working example of the above tutorial in action. There are probably easier ways of doing this than the method I have described, but I like my method and hopefully it will work for you.

Update: The code in this post was updated for Rails 2.0.x on 17th March 2008.

Update: Modified the settings code to be ‘smtp_settings’ rather than ‘server_settings’, thanks goes to Lyle for discovering the problem (below).

Update: Modified the code to work for Rails 2.3.x onwards (but not Rails 3).