I’ve been a Flickr user since the service was first launched. Since then, I’ve had many different cameras, connected to multiple machines, and used multiple methods to upload content.
In many cases, I’d export photos from my photo-management software (I’ve used everything from Image Capture and iPhoto to Lightroom and more recently Aperture) to a folder on the desktop, deleting it when I was done.
So although I still have all of the original photographs imported into and stored in my Aperture library, because of the less-than ideal workflow I’d used, I don’t necessarily know which photos — or which versions (cropped, WB-adjusted, etc.) of the photos — I’ve exported and uploaded.
These issues, combined with the goal of loading my better photographs (which all make it to Flickr) onto my AppleTV for easier family viewing, has lead me to try and create a complete, easily updatable local backup of all my Flickr-photos in their originally-uploaded sizes.
After trying many, many apps to accomplish this task, I’ve only been able to find one that actually works — and with only a single modification to the source code.
FlickrTouchr
After asking the Twitterverse for suggestions and trying many different applications, scripts, and techniques unsuccessfully, my friend Willie recommended I take a look at FlickrTouchr. FlickrTouchr is a Python script that you run via the command-line, which works fine on Mac OS X and Linux, and would probably work just fine on a Windows machine too (info here).
The author, Colm MacCárthaigh, created the script to synchronize his iPod touch with his Flickr account. MacCárthaigh writes:
The whole thing is a single python file, as implementing the flickr API calls myself rather than use any of various python flickr libraries (as they each seem poorly maintained) seemed easier. [Once it] authenticates against your flickr account, it will progress through syncing your entire flickr account into a directory hierarchy. It keeps a small cache of authentication data in the directory too. […] If you have the same photo in multiple sets it will use hard-links to preserve local disk-space.
My modifications
MacCárthaigh wanted to download photos optimized for his iPod touch, but I need to download the original, full-size photos. So I modified the script (which was released under the Apache 2.0 license) to grab the original-size photos when possible.
You can download my version of the script from GitHub. Please feel free to help me improve it in other ways if you’d like.
Running FlickrTouchr
Download the script, open a Terminal window, and from the command-prompt, just run the script and specify the name of the folder you’ll be using to store your backup, like this:
mkdir BackupFolder python flickrtouchr.py BackupFolder
The script will open your default web browser, allowing you to authorize FlickrTouchr to access your library, and will commence downloading every photograph you’ve uploaded, organizing them into folders by set, and using hard-links to save space.
Even better, because FlickrTouchr doesn’t rely on external libraries, it should just work on any system that can run Python without additional software or prerequisites.
Each time you run the script, it will find anything new that you’ve added, and update your backup as needed.
Subscribe via RSS
Jason G
05 June 2009 at 2:25 pm
Dan,
Thanks for writing this up. One question, do you have to have a pro account to download the fill sized originals?
Jason
Wolf
05 June 2009 at 2:49 pm
Also wanted to thank you for the writeup, backing up Flickr has been in my to do list like… forever.
Grant
05 June 2009 at 3:52 pm
That last part is killer!
Dan Benjamin
05 June 2009 at 7:05 pm
A few people have pointed out that there was an indenting issue, which a friend fixed. It’s updated now in GitHub, with credit.
Thanks.
Mario Aeby
05 June 2009 at 11:30 pm
Great tool - Thanks for sharing! It’ll spend the whole night backing up my Flickr feed ;-)
Just one small suggestion: What about using jhead to write a photo’s tags into the JPEG itself? I’ve spent hours on Flickr assigning tags to the photos. It’d be a pity to leave them on the servers.
(Yes, I know, I could add this feature by myself, but unfortunately I’m quite busy currently. If someone finds this idea exciting: Give it a try! After looking at the Python code it doesn’t look like rocket science to implement this feature)
Robert Zolkos
06 June 2009 at 7:53 am
Nice write up. Just checking - but you did know you can look at your Flickr photos using your AppleTV? Has had flickr in the Photos menu since v2 I think.
All the best.
jeremy tarling
06 June 2009 at 7:59 am
Just what I was looking for, thanks!
Dan Benjamin
06 June 2009 at 12:06 pm
@Robert - You’re right, but the solution isn’t perfect. Although the AppleTV allows you to browse Flickr, it does not permit you to browse private photos (your own or those of others), nor does Flickr browsing work while offline, nor does it provide a local backup, hence the solution I’ve posted here.
JAA
06 June 2009 at 1:57 pm
Thanks! Excellent idea and execution. I’m backing up a rather large collection - does it also back up our short video clips?
Thanks!
fozbaca
06 June 2009 at 7:44 pm
All my photos are downloading corrupted. I’m using Python 2.6.2 on Windows7RC. Any suggestions?
Will Wilkins
06 June 2009 at 9:55 pm
Awesome script! I have been looking for something like this for a while. However, I am receiving the following error where I am assuming the script is trying to create the hard link to an already downloaded picture.
Traceback (most recent call last):
File “/Users/username/Pictures/Flickr/flickrtouchr.py”, line 304, in <module>
if inodes.has_key(photoid) and os.access(inodes[photoid], os.R_OK):
TypeError: coercing to Unicode: need string or buffer, NoneType found
Is this a permissions issue or do you know what would be causing it?
Thanks!
Chris
08 June 2009 at 2:23 am
I think this script may be having trouble retrieving videos. The original script that doesn’t backup full-sized images was able to complete successfully. Unfortunately, this script fails the same way Will Wilkins experienced. It appears it is failing to download a video when this error occurs.
Rod Begbie
08 June 2009 at 5:14 am
The bug Will & Chris are experiencing is fixed on my fork of the project: http://github.com/rodbegbie/hivelogic-flickrtouchr/tree/master I’ve pinged Dan about merging the fixes back into his repo.
Dan Benjamin
08 June 2009 at 9:44 am
@Will, @Chris, @Rod - I’ve merged in the fixes, thanks!
tayker
08 June 2009 at 9:51 am
I’ve always been fond of SourceForge app flickrBackup. Plus, I like using a GUI. I haven’t tried it with videos though.
Travis Isaacs
08 June 2009 at 11:57 am
This is *exactly* what I have been looking for! Syncing my entire Aperture library to the Apple TV just wont work.
When script comes to a set with a slash in the name, such as Canon EF 16-35mm f/2.8 L, is creates a folder named “Canon EF 16-35mm f” and a sub-folder called “2.8 L”. It’s not a big deal, but iPhoto picks up only the sub-folder name as the album title.
I keep my photos in sets by lens, in case you are wondering: http://www.flickr.com/photos/tbisaacs/collections/72157607477011585
Shay
08 June 2009 at 7:04 pm
Thank you for this—I’m sure it’s something many long time Flickr users have wanted.
I am curious to know if there’s a way to download specific sets?
Will Wilkins
12 June 2009 at 2:36 am
@Will, @Chris, @Rod and @Dan - Thanks for the update. Works awesome now.
Mark
20 June 2009 at 7:54 am
One Word ... INCREDULOUS!
Could life be any better today? This script is absolutely what I was looking for in developing a half descent photo workflow on my Macs of Goodness. It works perfectly with Aperture!
I am in much the same situation as you were with photos all over the place ... what a MESS! Some only on Flickr some in Aperture and some just dormant in folders. Not an ideal situation.
I was looking at FlickrEdit ... which does the awesome of downloading things like descriptions and thingies from Flickr ... but it just not have that type one line and let it do the business feel to it.
The beautiful thing about using this with Aperture ... is that the information that you put into your photos using Aperture ... is still there after downloading the original photo. BRILLIANT! A few adjustments to my WorkFlow ... and this script will be catalyzing component that screams ... it works!
Thank you ... THANK YOU!
Adrian Cooke
26 July 2009 at 3:28 pm
This is for Jason G: I have a free account and flickrtouchr.py fails to backup my photos (but does backup my favourites). I saw your question here, then found this on Flickr → Your Account → Privacy and Permissions → Global settings: “Who can download your stuff? Anyone (but no-one can download the original files, because you have a free account).” Charming behaviour, that. Love the script though, thanks Dan.
Matt Haughey
05 August 2009 at 8:30 pm
huh, it seems to have choked downloading my entire favorites. Two things about that: 1. Should it even download favorites? They’re not “mine” really so much that I need a copy of them forever (plus the photo sizes are small). 2. Maybe it is dying on private or no-download-offered photos?
It worked on all my sets but died on favorites which frankly, I’d rather it not even archive since it’s not my own material.
Graham
06 August 2009 at 12:28 pm
@Dan, is there anyway to run this without a GUI environment? I want to run this from my Linux Server which doesn’t have a GUI (desktop) to allowing you to authorize FlickrTouchr to access your library.
DG
10 August 2009 at 2:46 am
There’s a resolution reported for an error when using out-of-the-box python on Mac OS X 10.4 at this link: http://www.stdlib.net/~colmmacc/2008/01/21/flickrtouchrpy-keep-your-flickr-album-and-favourites-on-an-ipod-touch-or-iphone/
In particular, it seems to correct: AttributeError: ‘tuple’ object has no attribute ‘query’. At least, it appears to correct it—haven’t completed the entire first back-up yet on my 10.4 system and have no clue what the resolution actual does since I’m not familiar with python.
Nick
12 August 2009 at 5:25 pm
I’m using Python 2.6.2 on Windows Vista (64bit) and the latest version fot he .py script but I’m still getting the following error after its downloaded a load of files.
Traceback (most recent call last):
File “flickrtouchr.py”, line 309, in <module>
os.link(inodes[photoid], target)
AttributeError: ‘module’ object has no attribute ‘link’
I’ve also just checked all the files it did download and they all appear to be corrupt and unviewable.
Any ideas?
gopal
18 August 2009 at 2:46 am
Your article was really very useful… thanks a lot for the information!
Automatt
18 August 2009 at 4:54 pm
Thanks for this! I’ve been running the backup script for a little over a day, and it’s astounding to me how much photo data I have stored on Flickr.
Jaume Orejuela
21 August 2009 at 8:07 am
I love this game!!! ho yeah!
tom
21 August 2009 at 10:34 pm
is there a way to determine how big your flickr “account” is prior to downloading? i’ve got something in the range of 10,000 images up there, obviously i could do some rough math based on the max image size i’ve uploaded and multiply that by 10,000 which would give be a very approx worst case scenario, but is there a way to come up with a more precise idea first. i’m short enough on HD space as it is… :(
Rom Feria
16 September 2009 at 10:11 am
Traceback (most recent call last):
File “flickrtouchr.py”, line 309, in <module>
os.link(inodes[photoid], target)
OSError: [Errno 1] Operation not permitted
HELP.
BTW, running on Linux.
rince
05 November 2009 at 10:34 am
Hi,
I’ve been using this great tool for some time without issues, but last night as i tried the script i got the following error…
i must admit it had been a while since i used it, so i’m not sure what happened…
File “flickrtouchr.py”, line 7
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”
^
SyntaxError: invalid syntax
running ARch linux, python 2.6.4-1 [installed)
thx
rince
05 November 2009 at 6:19 pm
nevermind my last comment….
instead of using the actual code, i just did a right click on the script and ‘saved link as’.... which didn’t actually work :)
so now it’s running again as it should :S
sry
Jason Ryan
06 November 2009 at 6:44 am
Great concept, but -and correct me if I am wrong- it only works if you upgrade to a paid account?
Even though all of my shots are CC licensed, the script still fails to download any of them; from Adrian Cooke’s comment above, I assume this is the reason…
numa
08 November 2009 at 4:51 am
Great! This is what I’ve been looking for!
Myke
13 November 2009 at 12:37 am
This is awesome! Thank you!
Dara
08 December 2009 at 11:33 am
hi, this is exactly what i need as i dont want to fork out to flickr again for another year of pro and account stops been pro in 6 days.
have the same problem as nick (12 aug)above. anyone come up with a solution to this? using 2.6.2 on windows xp.
do all photos have to be in at least one set?
i had never even heard of python before i saw this article by the way, so keep it simple!
any suggestions appreciated…
C:\Python26>python flickrtouchr.py flickrbackup
flickrtouchr.py:25: DeprecationWarning: the md5 module is deprecated; use hashli
b instead
import md5
In order to allow FlickrTouchr to read your photos and favourites
you need to allow the application. Please press return when you’ve
granted access at the following url (which should have opened
automatically).
http://api.flickr.com/services/auth/?api_key=e224418b91b4af4e8cdb0564716fa9bd&pe;
rms=read&frob=72157622835338977-b3528609c819ff44-642995&api_sig=1aa48d6c1fcb348c
d6632dd53a000407
Waiting for you to press return
the hive ... in set ... work rest play
living rooms ... in set ... work rest play
liberty lights ... in set ... work rest play
work ... in set ... work rest play
grand canal basin ... in set ... work rest play
dusk drive ... in set ... work rest play
... in set ... work rest play
liberty hall playhouse ... in set ... work rest play
... in set ... work rest play
big tv ... in set ... work rest play
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
... in set ... mosaic
Traceback (most recent call last):
File “flickrtouchr.py”, line 309, in <module>
os.link(inodes[photoid], target)
AttributeError: ‘module’ object has no attribute ‘link’
Dave
15 December 2009 at 10:30 pm
flickrtouchr.py:25: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
usage: flickrtouchr.py directory
I’m not too sharp when it comes to these things. Would anyone help me? OS X 10.6.2