Digitising My Negatives

As I mentioned before, I recently began shooting on film again (a little bit anyway). Of course, this meant I wanted someway of getting my images from the negatives and into my digital library for a bit of light editing and sharing.

I found myself on Amazon looking at the plethora of cheap negative scanners. Most of these consist of a 5mp CCD and a backlight; photos are scanned quickly, to JPEG, and mostly without the need to involve a computer. From what I could find though, this type of scanner has three problems: highly compressed JPEG output, relatively low resolution ‘scans’ and extremely mixed quality output.

Maybe I was worrying too much about image quality – but they weren’t good enough for me. I wanted RAW images and higher-resolution output.

The most obvious choice would’ve been something like the Plustek OpticFilm 8100 or a negative-compatible flatbed scanner. I could’ve scanned as TIFF at high resolution and have been done. The main problem with this solution was the price, I couldn’t justify the high cost for something I probably wouldn’t be doing often.

To this end, I decided to make the use of what I already owned (or could make pretty easily).

The Setup

The camera setup itself wasn’t too complicated, I used my Nikon D700 with and old 105mm Micro-Nikkor. The equipment isn’t massively important though, as long as you have a decent DSLR, mirrorless or high-end compact / bridge with a lens that can get close enough to fill the frame with a negative then you’re probably going to get higher quality shots than a cheap dedicated negative scanner. RAW shooting is a massive plus though; the results will need some white-balance correction.

All of this needs to be mounted on a fairly sturdy tripod that can take the weight of your setup pointing straight down.

A couple of things you will need to be able to do though: manual focus, or at least have the ability fix the focus, and a self-timer / cable release function. Shooting so close, things can get blurry quickly.

One useful little accessory is a macro focusing rail, it allows you to finely tune the focus without having to mess around with the camera’s settings too much. It can be especially helpful with older, heavier lenses that tend to fall out of focus when the camera is pointing towards the ground and nudged slightly.

Probably the most difficult bit of the whole setup was coming up with some way to backlight the negative a suitable amount and evenly. Fortunately an Amazon shipping box, printer paper, a torch and a lot of packing tape came to the rescue.

As I didn’t have anything suitable to diffuse the light directly under the negative I made a relatively long tube (appox. 30cm) and lined it with printer paper that curved up towards the negative-sized aperture I cut at one end of the box. This produced diffuse enough light that evenly lit the negative.

A special shout out should probably go to the torch I used, it was the extremely bright LED Lenser P7. This is probably the best torch I’ve ever bought, super-bright for normal torching with a neutral enough light temperature for small photography-related projects like this.

Now for the stuff that really matters…

The Settings

For my negatives I shot in manual mode: 1/50s, f/7.1, ISO 200. I left automatic white-balance enabled as I was shooting in RAW and the white balance would definitely have to be corrected in post-processing anyway.

I chose not to quite fill the frame with the negative to ensure I made the most of the lens’ sharpness in the centre. After cropping, most of my shots worked out at around 8mp, which was pretty good going and definitely better than the cheap negative scanners.

The Results

Straight out of the camera this is how the negatives looked:

Inverting the photo quickly got me to something that looked more sensible. The blue cast to the image is the nature of the colour film and this is what needs to be white-balanced away. This can take a lot of playing with to get right but once you’ve done it for a single image, it should be the same for the whole roll.

After a little pushing & prodding with your image editor of choice (mine is Aperture but I guess that won’t be true for much longer). You can get something that looks perfectly acceptable.

To be honest, this photo probably isn’t the best example, but you can find some of the better ones (B&W and colour) in my Flickr album.

Something that did surprise me during this process was the amount of dynamic range I got from the negatives by digitising them in this way, I could see details from the negatives that the original prints didn’t even give the smallest clue to. The large RAW photos also gave me a lot of latitude when I was editing, it was nice to maintain the atmosphere of film with the advantages of a digital editing workflow.

Did it take a while to do all this: yes, would I have been better off getting a scanner: possibly, would it have been anywhere near as satisfying or fun: definitely not!

Digitising My Negatives

Me On The Internet

As I never seem to have anything interesting enough to write a blog post of any length about it seems like a good time to mention all the other (far more frequently updated) places that you can find me on the internet.

The popular places (where I’m known as digitalpardoe):

If you must:

  • Facebook: https://www.facebook.com/digitalpardoe

Really?!:

There’s probably a lot more places you can find me that I’ve forgotten about, but if you haven’t already realised, my username tends to be digitalpardoe so just try a search on your ‘social’ site of choice.

Strangely, in putting this together, I seem to have thought of something longer to write about.

Check back soon.

Me On The Internet

Moving To Typo

As you may or may not be aware digital:pardoe has, for the past 4 years, been running atop a custom blogging engine that I developed as way to learn Ruby on Rails. Whilst the system has (nearly) always been stable and (nearly) always fast I felt it was time to retire it, from everyday use at least.

When using the digital:pardoe blogging engine the ‘blogging’ always felt secondary to the actual development of the blog, I was always found myself doing far more of the ‘adding new features’ than the ‘adding new posts’ which, at least in recent months, is not what I intended to happen.

Unfortunately, the loss of my bespoke blogging engine also means a loss of some of the bespoke features I added to the website. The downloads (previously ‘software’) area is now very cut down – everything is still available as before though. The ‘photo’ section has now disappeared completely, if you want to see my photographs you’ll have to visit my Flickr account instead. I’ve made every effort to redirect old pages to their new location but if you find a page that is missing please contact me so I can fix the problem.

In the near future I intend to release the digital:pardoe blogging engine source code (once I’ve cleaned it up of course) as it may be a useful reference to other new RoR developers. Don’t expect the default Typo theme to stick around for long either, I’m currently in the process of porting a digital:pardoe theme to Typo.

And, if you hadn’t guessed already, digital:pardoe is now Typo powered.

Check back soon.

Moving To Typo

Clearing Out The rFlickr Cache

Assuming you’ve followed the Caching Your Photographs tutorial at some point, you’ll probably have had a lot of fun either deleting the cache every time you upload a new photo or you’ve written your own automated method by now. For those of you that haven’t written your own method of dumping the cache yet, here’s how I do it.

First of all, I created a lib/actions folder in the root of my rails project. Inside this folder I created the file ‘photography_action.rb’ with the following contents:

class PhotographyAction
  def self.clear_cache
    ActionController::Base.new.expire_fragment(%r{photography.cache})
  end
end

The above fragment naming assumes that your photos are on a page called ‘photography’ if they are elsewhere, change the fragment to expire that page instead.

Fairly simple I think you’ll agree, you may also be asking yourself ‘why the extra file?’, the main reason for the new file is so that the cache clearing can be executed from a new rake task that doesn’t remove all your cached pages or from an admin page on the website.

You’ll also need to update your config.load_paths in environment.rb. After updating, mine looks like this:

config.load_paths += %W( #{RAILS_ROOT}/app/sweepers #{RAILS_ROOT}/lib/actions )

Inside some action in some, preferably protected, controller somewhere, add the following (redirecting to anywhere you fancy):

PhotographyAction.clear_cache
redirect_to :action => 'index'

Now for the rake task. Inside the directory lib/tasks (create it if it doesn’t exists) create the file photography.rake then put the following code inside the file:

namespace :photo do
  namespace :cache do
    desc "Clear out photography cache"
    task :clear => :environment do
      PhotographyAction.clear_cache
    end
  end
end

You should then be able to run:

rake photo:cache:clear

From the base of your project in order to clear the cache.

Bear in mind, the code above is literally just a convenient way of clearing out the fragment cache so new photos show up on your photo page, it does not delete photos, nor does it perform a refresh automatically, although, you could add it to a CRON job.

When I get chance, I intend to automate this process and build it into rFlickr along with a new, improved, caching mechanism. I’m sure the above will tide you over for now though.

Check back soon.

Clearing Out The rFlickr Cache

New rFlickr Ruby Gem

After I started to use the rFlickr gem it didn’t take me long to realize that development of the gem had all but halted, yes it worked, which was more than the original Flickr gem did, but it was still a little bit out of date and in the end, a little bit broken.

In one of my older posts I documented a fix for the gem and provided a download to unzip into your plugins folder, however, with the advent of the wonderful GitHub and its marvelous gem support I’ve decided to move the project onto GitHub.

I have preserved the original gem’s GPL license and copied the source code from its original repository on RubyForge to a new, public, GitHub repository. In the process of the move I have dropped old code from the project, updated the readme & license information and generally performed a little house-keeping.

You can find the project at: http://github.com/digitalpardoe/rflickr/. You can install the gem using one of the following methods. First involves adding GitHub as a gem source (always a good idea) and installing the gem:

gem sources -a http://gems.github.com
sudo gem install digitalpardoe-rflickr

The second method it to add the gem as a gem dependency to the environment.rb of your Rails project:

config.gem 'digitalpardoe-rflickr', :lib => 'flickr', :source => 'http://gems.github.com'

And run a rake task to install the gem:

sudo rake gems:install

Whilst performing the code migration I also added the fix that was documented in my original post and implemented support for the (not so) new ‘farm’ based Flickr URLs for images (which should make things easier to implement).

The future plans for rFlickr include new tests, improved usage examples, updated readme / documentation and implementation of missing API methods, time permitting of course.

Until the readme is updated please refer to the original post for information on how to use rFlickr.

That’s all for now, enjoy the new gem and as they say, if you don’t like it, fork it.

New rFlickr Ruby Gem

Guess Who’s Back

As you may have noticed, it’s been a long time since my last post. There isn’t really any good reason for this. Plenty has happened, I just haven’t got round to writing any of it down.

First off I’d like to mention the website, it went through a fairly radical redesign a few months ago and I mentioned nothing about it. For some reason it’s not in my nature to be happy with what I make hence the many faces and iterations of the website. This website, whilst being my home on the internet, is also the test bed for my RoR programming, you may get tired of hearing about its re-designs and re-codes but that’s part of the reason I created it. Anyway, another re-design is coming, this time it’s not visual but all back end, the main difference you will notice is that I am doing away with user accounts and having a more open comment system (I could be shooting myself in the foot with this decision, we’ll have to see how the spam bots take it). To the people that have commented on the blog already, your comments will be preserved and, when I roll out the changes, I intend to reply to all the comments I haven’t yet replied to.

The second thing I wanted to mention, again website related, is my hosting. A good proportion of my posts seem to be apologizing for the downtime of the website. I was actually getting pretty bored of this so decided to, quite literally, take matters into my own hands. The website is now hosted on a virtual private server set up and maintained by me. This again, may be a case where I’ve shot myself in the foot. For those of you interested, the VPS is provided by the wonderful folks at Bytemark Hosting.

Number three. Many of the posts of my website relate to the use of the ‘rflickr’ RubyGem. Development of this gem seems to have been at a stand still for a good while now, I’ve therefore taken the decision to clone it and try to continue development in my spare time. More on this in a later post.

Four. Any of you interested in my photography will have noticed a lack of it over the past few months, it’s not that I haven’t been taking any photographs, it’s just that I’ve not published any. To try and remedy this I uploaded a batch of photos today that have been sitting on my computer for a while. You can take a look at them on the photo page of the website or on my Flickr page.

Guess Who’s Back

Just An Introduction

I’ve been promising to write this post for a while now, it is exactly what is says in the title, an introduction. A good friend of mine, after much persuasion has finally ‘got his blog on’. Fortunately he has a far better blogging style than me and blogs far more often so it shouldn’t be too much of a strain on the senses to bookmark him and/or add him to your feed reader. Funnily enough he is also a keen photographer so don’t forget to check out his Flickr page, you’ll find a link from his blog.

So here he is, (drum roll please), Mr Rob Young: robyoung.me.uk.

Be nice, I don’t want him scared back off the internet, it took long enough to get him on here.

Don’t forget to check back here sometimes too.

Just An Introduction

Flickr BBCode

phpBB is one of the most widely used forum systems on the internet and Flickr is one of the most popular photo sharing websites, unfortunately there is still no concrete way to integrate the two of them. As a temporary solution I came up with the following piece of BBCode to allow for the easy display of Flickr images in phpBB forums.

First, you will need to navigate to the “Admin Control Panel” of phpBB then to the “Posting” section. From here navigate to the screen that allows you to add custom BBCode.

In the “Usage” section paste the following code:

[flickr=<a href="{URL1}" title="{TEXT1}"><img src="{URL2}" width="{NUMBER1}" height="{NUMBER2}" alt="{TEXT2}" /></a>][/flickr]

In the “HTML” section paste:

<a href="{URL1}" title="{TEXT1}"><img src="{URL2}" width="{NUMBER1}" height="{NUMBER2}" alt="{TEXT2}" /></a><br /><a href="{URL1}">{TEXT2}</a>

Then in the “Help” section paste:

Flickr: [flickr=<Flickr, Medium Size, Copy & Paste HTML>][/flickr]

And that’s it, you can use the tag by navigating to one of your photo’s Flickr “All Sizes → Medium” page (e.g. http://www.flickr.com/photos/digitalpardoe/2900618617/sizes/m/), copying the code in the “1. Copy and paste this HTML into your webpage:” box. And inserting it after the “=” in the Flickr BBCode tag, e.g.:

[flickr=<a href="http://www.flickr.com/photos/digitalpardoe/2900618617/" title="Isolation by digital:pardoe, on Flickr"> <img src="http://farm4.static.flickr.com/3095/2900618617_26bc8abc12.jpg" width="500" height="334" alt="Isolation" /></a>][/flickr]

Check back soon.

Flickr BBCode

Caching Your Photographs

If you have at some point followed my now fairly ancient rFlickr tutorial you may have noticed that your photo page loads quite slowly, and that my photo page loads fairly quickly. To get my page to load as quickly as it does required a small custom caching method and a willingness on my part to sacrifice some bandwidth. Here’s how I did it.

This tutorial assumes that you have already worked through the previously mentioned rFlickr tutorial and have something similar to it set up. It also assumes that you have some knowledge of Rails, not that my knowledge was particularly wide ranging at the point I wrote this caching method.

First of all, you will need a table in your database to store the information about your photographs, I suggest the structure illustrated in the migration below:

class CreatePhotos < ActiveRecord::Migration
  def self.up
    create_table :photos, :id => false do |t|
      t.column "flickr_id", :string, :limit => 25, :null => false
      t.column "title", :string, :limit => 250
      t.column "description", :text
      t.column "url", :string, :limit => 250
    end

    add_index :photos, :flickr_id
  end

  def self.down
    drop_table :photos
  end
end

Once you have created this table you will need to create some folders to store the cached images, I created the following folders and will be using them throughout this tutorial:

"#{RAILS_ROOT}/public/images/flickr_cache/small/"
"#{RAILS_ROOT}/public/images/flickr_cache/large/"

Then generate the model for this photos table:

cd /your/rails/application
./script/generate model Photo

Your view from the first tutorial can remain almost the same (details at the end of the post), however, to see the greatest speed improvement I suggest caching it, i.e:

<% cache do %>
... Your view code here. ...
<% end %>

Then modify your view method in your photography controller to read something like:

def view
  unless read_fragment({})
    check_cache
    @photos = Photo.find(:all)
  end
end

The above code will make sure that a cached photography page doesn’t already exist, if it doesn’t, then and only then will it check that the photograph cache is up to date and query the database.

We have not yet created a check_cache method, this method is the core method to make the photography page load much, much faster, even when the photography page’s cache does not exist. The method should be placed as the last method in your photography controller, the code is as follows:

private
def check_cache
  if ENV['RAILS_ENV'] == 'production'
    flickr = Flickr.new(RAILS_ROOT + "/config/flickr.cache", FLICKR_API_KEY, FLICKR_SHARED_SECRET)
    @photos = flickr.people.getPublicPhotos(flickr.people.findByUsername(FLICKR_USERNAME))

    @db_photos = Array.new
    Photo.find(:all).each { |p| @db_photos.push(p.flickr_id) }

    for photo in @photos.reverse
      if !@db_photos.include?(photo.id)

        db_photo = Photo.new
        db_photo.flickr_id = photo.id.to_i
        db_photo.title = photo.flickr.photos.getInfo(photo.id).title
        db_photo.description = photo.flickr.photos.getInfo(photo.id).description
        db_photo.url = photo.flickr.photos.getInfo(photo.id).urls.values[0]

        db_photo.save

        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/small/" + photo.id + ".jpg"),"w").write(open(photo.url('s')).read)
        open(File.expand_path("#{RAILS_ROOT}/public/images/flickr_cache/large/" + photo.id + ".jpg"),"w").write(open(photo.url).read)
      end
    end
  end
end

The following code will only run if you are in production mode (and probably test mode too). It will then load the necessary information from Flickr using the methods outlined in the rFlickr tutorial post. The method then iterates through the collection of photos from Flickr in reverse, so they appear in the same order in the database as the order they appear on Flickr.

It will then check if the photo already exists in the database, if it does not it will store a copy of the photograph’s information in the database and download previews of the images from Flickr to your server, previews of images can then be loaded from your server rather than Flickr’s slow servers.

To take advantage of the cache you will also need to modify your view to access the thumbnails from your newly created local repository rather than from Flickr’s servers, i.e:

<% for photo in @photos.reverse %>
  <%= image_tag("/images/flickr_cache/small/" + photo.flickr_id + ".jpg", :alt => photo.title) %>
<% end %>

Hope this goes some way to helping you improve the speed of your website.

Check back soon.

Update: As I have just been reminded in the comments, I forgot a piece of code to make this tutorial work correctly. You should put the line:

Either in the bottom of you environment.rb file or just under the ‘class’ line in the controller that is responsible for your photography page.

Update: The code in the check_cache method has been updated slightly, it now makes less round trips to the database, should speed things up if your DB server is in a different location to your application server. Also, any redirection problems you may have faced before should be solved by the new rFlickr ruby gem that has support for farm URLs built in.

Caching Your Photographs