Clearing Out The rFlickr Cache

Assuming you’ve followed the Caching Your Photographs tutorial at some point, you’ll probably have had a lot of fun either deleting the cache every time you upload a new photo or you’ve written your own automated method by now. For those of you that haven’t written your own method of dumping the cache yet, here’s how I do it.

First of all, I created a lib/actions folder in the root of my rails project. Inside this folder I created the file ‘photography_action.rb’ with the following contents:

class PhotographyAction
  def self.clear_cache
    ActionController::Base.new.expire_fragment(%r{photography.cache})
  end
end

The above fragment naming assumes that your photos are on a page called ‘photography’ if they are elsewhere, change the fragment to expire that page instead.

Fairly simple I think you’ll agree, you may also be asking yourself ‘why the extra file?’, the main reason for the new file is so that the cache clearing can be executed from a new rake task that doesn’t remove all your cached pages or from an admin page on the website.

You’ll also need to update your config.load_paths in environment.rb. After updating, mine looks like this:

config.load_paths += %W( #{RAILS_ROOT}/app/sweepers #{RAILS_ROOT}/lib/actions )

Inside some action in some, preferably protected, controller somewhere, add the following (redirecting to anywhere you fancy):

PhotographyAction.clear_cache
redirect_to :action => 'index'

Now for the rake task. Inside the directory lib/tasks (create it if it doesn’t exists) create the file photography.rake then put the following code inside the file:

namespace :photo do
  namespace :cache do
    desc "Clear out photography cache"
    task :clear => :environment do
      PhotographyAction.clear_cache
    end
  end
end

You should then be able to run:

rake photo:cache:clear

From the base of your project in order to clear the cache.

Bear in mind, the code above is literally just a convenient way of clearing out the fragment cache so new photos show up on your photo page, it does not delete photos, nor does it perform a refresh automatically, although, you could add it to a CRON job.

When I get chance, I intend to automate this process and build it into rFlickr along with a new, improved, caching mechanism. I’m sure the above will tide you over for now though.

Check back soon.

Clearing Out The rFlickr Cache

New rFlickr Ruby Gem

After I started to use the rFlickr gem it didn’t take me long to realize that development of the gem had all but halted, yes it worked, which was more than the original Flickr gem did, but it was still a little bit out of date and in the end, a little bit broken.

In one of my older posts I documented a fix for the gem and provided a download to unzip into your plugins folder, however, with the advent of the wonderful GitHub and its marvelous gem support I’ve decided to move the project onto GitHub.

I have preserved the original gem’s GPL license and copied the source code from its original repository on RubyForge to a new, public, GitHub repository. In the process of the move I have dropped old code from the project, updated the readme & license information and generally performed a little house-keeping.

You can find the project at: http://github.com/digitalpardoe/rflickr/. You can install the gem using one of the following methods. First involves adding GitHub as a gem source (always a good idea) and installing the gem:

gem sources -a http://gems.github.com
sudo gem install digitalpardoe-rflickr

The second method it to add the gem as a gem dependency to the environment.rb of your Rails project:

config.gem 'digitalpardoe-rflickr', :lib => 'flickr', :source => 'http://gems.github.com'

And run a rake task to install the gem:

sudo rake gems:install

Whilst performing the code migration I also added the fix that was documented in my original post and implemented support for the (not so) new ‘farm’ based Flickr URLs for images (which should make things easier to implement).

The future plans for rFlickr include new tests, improved usage examples, updated readme / documentation and implementation of missing API methods, time permitting of course.

Until the readme is updated please refer to the original post for information on how to use rFlickr.

That’s all for now, enjoy the new gem and as they say, if you don’t like it, fork it.

New rFlickr Ruby Gem

Guess Who’s Back

As you may have noticed, it’s been a long time since my last post. There isn’t really any good reason for this. Plenty has happened, I just haven’t got round to writing any of it down.

First off I’d like to mention the website, it went through a fairly radical redesign a few months ago and I mentioned nothing about it. For some reason it’s not in my nature to be happy with what I make hence the many faces and iterations of the website. This website, whilst being my home on the internet, is also the test bed for my RoR programming, you may get tired of hearing about its re-designs and re-codes but that’s part of the reason I created it. Anyway, another re-design is coming, this time it’s not visual but all back end, the main difference you will notice is that I am doing away with user accounts and having a more open comment system (I could be shooting myself in the foot with this decision, we’ll have to see how the spam bots take it). To the people that have commented on the blog already, your comments will be preserved and, when I roll out the changes, I intend to reply to all the comments I haven’t yet replied to.

The second thing I wanted to mention, again website related, is my hosting. A good proportion of my posts seem to be apologizing for the downtime of the website. I was actually getting pretty bored of this so decided to, quite literally, take matters into my own hands. The website is now hosted on a virtual private server set up and maintained by me. This again, may be a case where I’ve shot myself in the foot. For those of you interested, the VPS is provided by the wonderful folks at Bytemark Hosting.

Number three. Many of the posts of my website relate to the use of the ‘rflickr’ RubyGem. Development of this gem seems to have been at a stand still for a good while now, I’ve therefore taken the decision to clone it and try to continue development in my spare time. More on this in a later post.

Four. Any of you interested in my photography will have noticed a lack of it over the past few months, it’s not that I haven’t been taking any photographs, it’s just that I’ve not published any. To try and remedy this I uploaded a batch of photos today that have been sitting on my computer for a while. You can take a look at them on the photo page of the website or on my Flickr page.

Guess Who’s Back

Caching Your Photographs

If you have at some point followed my now fairly ancient rFlickr tutorial you may have noticed that your photo page loads quite slowly, and that my photo page loads fairly quickly. To get my page to load as quickly as it does required a small custom caching method and a willingness on my part to sacrifice some bandwidth. Here’s how I did it.

This tutorial assumes that you have already worked through the previously mentioned rFlickr tutorial and have something similar to it set up. It also assumes that you have some knowledge of Rails, not that my knowledge was particularly wide ranging at the point I wrote this caching method.

First of all, you will need a table in your database to store the information about your photographs, I suggest the structure illustrated in the migration below:

class CreatePhotos < ActiveRecord::Migration
  def self.up
    create_table :photos, :id => false do |t|
      t.column "flickr_id", :string, :limit => 25, :null => false
      t.column "title", :string, :limit => 250
      t.column "description", :text
      t.column "url", :string, :limit => 250
    end

    add_index :photos, :flickr_id
  end

  def self.down
    drop_table :photos
  end
end

Once you have created this table you will need to create some folders to store the cached images, I created the following folders and will be using them throughout this tutorial:

"#{RAILS_ROOT}/public/images/flickr_cache/small/"
"#{RAILS_ROOT}/public/images/flickr_cache/large/"

Then generate the model for this photos table:

cd /your/rails/application
./script/generate model Photo

Your view from the first tutorial can remain almost the same (details at the end of the post), however, to see the greatest speed improvement I suggest caching it, i.e:

<% cache do %>
... Your view code here. ...
<% end %>

Then modify your view method in your photography controller to read something like:

def view
  unless read_fragment({})
    check_cache
    @photos = Photo.find(:all)
  end
end

The above code will make sure that a cached photography page doesn’t already exist, if it doesn’t, then and only then will it check that the photograph cache is up to date and query the database.

We have not yet created a check_cache method, this method is the core method to make the photography page load much, much faster, even when the photography page’s cache does not exist. The method should be placed as the last method in your photography controller, the code is as follows:

private
def check_cache
  if ENV['RAILS_ENV'] == 'production'
    flickr = Flickr.new(RAILS_ROOT + "/config/flickr.cache", FLICKR_API_KEY, FLICKR_SHARED_SECRET)
    @photos = flickr.people.getPublicPhotos(flickr.people.findByUsername(FLICKR_USERNAME))

    @db_photos = Array.new
    Photo.find(:all).each { |p| @db_photos.push(p.flickr_id) }

    for photo in @photos.reverse
      if !@db_photos.include?(photo.id)

        db_photo = Photo.new
        db_photo.flickr_id = photo.id.to_i
        db_photo.title = photo.flickr.photos.getInfo(photo.id).title
        db_photo.description = photo.flickr.photos.getInfo(photo.id).description
        db_photo.url = photo.flickr.photos.getInfo(photo.id).urls.values[0]

        db_photo.save

        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/small/" + photo.id + ".jpg"),"w").write(open(photo.url('s')).read)
        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/large/" + photo.id + ".jpg"),"w").write(open(photo.url).read)
      end
    end
  end
end

The following code will only run if you are in production mode (and probably test mode too). It will then load the necessary information from Flickr using the methods outlined in the rFlickr tutorial post. The method then iterates through the collection of photos from Flickr in reverse, so they appear in the same order in the database as the order they appear on Flickr.

It will then check if the photo already exists in the database, if it does not it will store a copy of the photograph’s information in the database and download previews of the images from Flickr to your server, previews of images can then be loaded from your server rather than Flickr’s slow servers.

To take advantage of the cache you will also need to modify your view to access the thumbnails from your newly created local repository rather than from Flickr’s servers, i.e:

<% for photo in @photos.reverse %>
  <%= image_tag("/images/flickr_cache/small/" + photo.flickr_id + ".jpg", :alt => photo.title) %>
<% end %>

Hope this goes some way to helping you improve the speed of your website.

Check back soon.

Update: As I have just been reminded in the comments, I forgot a piece of code to make this tutorial work correctly. You should put the line:

Either in the bottom of you environment.rb file or just under the ‘class’ line in the controller that is responsible for your photography page.

Update: The code in the check_cache method has been updated slightly, it now makes less round trips to the database, should speed things up if your DB server is in a different location to your application server. Also, any redirection problems you may have faced before should be solved by the new rFlickr ruby gem that has support for farm URLs built in.

Caching Your Photographs

rFlickr And Rails 2.0

Any of you that followed my tutorial on setting up rFlickr and have subsequently upgraded to Ruby on Rails 2.0 may have noticed some server errors on your photo pages, this is due to the fact that a single line of code in rFlickr does not work correctly with Rails 2.0.

Line 644 of lib/base.rb had to be modified from:

def from_xml(xml,photo=nil)

to:

def self.from_xml(xml,photo=nil)

Hope this gets you back on your feet, so to speak. Check back soon.

Update: You may also want to install the updated version of actionwebservice because it is no longer bundled with the Rails distribution. Other people have reported rFlickr not working because of a lack of this gem, however, the lack of this gem didn’t affect my setup at all.

rFlickr And Rails 2.0

Using rFlickr

I said it was coming and here it is, my little tutorial on how to use the rFlickr Ruby on Rails gem to create a photography section like the one on my own website. The first thing to note is that pretty much all of the options available in the Flickr API (here) are available for use in rFlickr due to the fact it is all based around XML. There is a laborious process of configuration to go through, however, to make everything work, but once this is done you should have no problems.

Firstly install the rFlickr gem, I should at this juncture note the fact I am primarily a UNIX user so will aim these instructions at other UNIX users, mainly because I don’t know the specifics for Rails installations on Windows. So lets dive in ($ denotes the terminal prompt and \ denotes line continues below):

sudo gem install rflickr --include-dependencies

The second thing you will need to do is make sure you have a Flickr account with some photos on it then pay a visit to http://www.flickr.com/services/api/keys/ and sign yourself up for an API key, once you have generated the key make a note of the key itself and the ‘secret’ that you are given, you will be needing these quite a bit.

The next thing to do is to basically follow the tutorial here, albeit with a few modifications, I have re-written the tutorial in full below.

cd /your/rails/application
./script/console

To make differences clear the Rails console prompt will be shown as >>, don’t forget to replace the x’s with your information.

>> require 'flickr'
>> API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> SHARED_SECRET = "xxxxxxxxxxxxxxxx"
>> flickr = Flickr.new("/tmp/flickr.cache", API_KEY, SHARED_SECRET)
>> flickr.auth.token
=> nil

The above just sets up your rFlickr object and makes sure that you don’t already have a key.

>> flickr.auth.getFrob

This returns a value that you will need to save somewhere.

>> flickr.auth.login_link
=> "http://some.link.flickr.com"

Click or copy the link you are given into a browser and authorize the API for usage, don’t worry, we’re almost there.

>> flickr.auth.getToken('that_frob_number_we_saved')
>> flickr.auth.cache_token
>> exit

Right, this is as far as the tutorial online goes, but there are some other useful steps we need to take to make everything more useable, mainly the moving of the token as the /tmp directory may get cleared by our host.

cp /tmp/flickr.cache /your/rails/application/config/flickr.cache
rm /tmp/flickr.cache

Now we can get onto the actual programming and leave the authentication business behind.

We’re going to need a controller to use, for the purposes of this tutorial I will use a controller named Photography, it should save me some time as that’s what mine is called, the page to be rendered will be called view.

In the file photography_controller.rb we will need the following information, rename as necessary to your application.

class PhotographyController < ApplicationController
  API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  SHARED_SECRET = "xxxxxxxxxxxxxxxx"

  def index
    view
    render :action => 'view'
  end

  def view
    flickr = Flickr.new(RAILS_ROOT + "/config/flickr.cache", API_KEY, SHARED_SECRET)
    @photos = flickr.people.getPublicPhotos(flickr.people.findByUsername("YOUR_FLICKR_NAME"))
  end
end

Then in the view.rhtml that you will have created in your views folder (or at least, are creating now) paste the following code.

<% for photo in @photos %>
  <a href="<%= photo.flickr.photos.getInfo(photo.id).urls.values %>"><img src="<%= photo.url('s') %>" /></a>
<% end %>

And that’s it, your basically done, all the thumbnails will link directly to your Flickr page, easy wasn’t it? The main problem occurs when you load your newly created page, it’s very, very slow, due to the speed of the Flickr API (I think). In order to improve the situation I would recommend using either page or fragment caching, but that will be covered in a future tutorial.

Hopefully this will have given you a few pointers in using the rFlickr gem, read through the Flickr API for more inspiration if you are feeling adventurous. Check back soon.

Update: If you are having problems with Rails 2.0, take a look at this fix.

Update: I have written a new tutorial on caching your photos page, that should speed it up a lot, assuming you are having problems.

Using rFlickr

Scheduling & Photos

Things appear to be moving along more quickly than expected this week, I posted my main application iSyncIt and the My Book Icons on a few forums and the Apple software website and suddenly the visits to my website have rocketed from 200 a day up to 700 hits or more a day. Not as massive as some websites but it’s certainly an incentive for me to do some more work.

After many requests I have started to implement a more suitable scheduling system in iSyncIt, I added the original system for people that change their calendar and contacts very often but it seems people want more flexibility so I am currently coding something more suitable and user programmable.

Also, partially as an experiment and partially due to me wanting to unify my websites into a single website I have today programmed and uploaded a photography section to the website. This portion of the website showcases some of the better photographs that I take, for other people to see. It has been created using the ‘rflickr’ gem and my Flickr Photostream. I had several problems creating the section due to the lack of documentation for the ‘rflickr’ gem, so I will be writing a tutorial in the coming days to help other people who want to create something similar.

Well, thats all for now, check back soon.

Scheduling & Photos