I take a ridiculous number of photos and videos on my phone. Every couple of months I’d get one of the most annoying notifications you can get on your phone. It’s the kind of notification that’s often followed by an expletive:

The problem with this message is that you can’t solve it quickly. You either need to carefully delete photos that are poorly composed or out of focus, or move them off the device. For the latter, Google suggests two methods:

1. Move files with your Google Account; or,
2. Move files with a USB cable.

In an increasingly connected world with age-old software like ssh and rsync, the only options on Android are sending your files to Google’s cloud, which has massive privacy implications, or using a USB cable. In other words, Android sucks at transferring files.

There are many other cloud storage services besides Google, the most popular being Dropbox; however, they all come with the same privacy implications.

It gets worse - even if it were possible to use ssh and rsync, there’s no way to run it automatically.

A few years ago I built a personal mini-itx based server running freenas for storing files and sharing them on the home network.

In the beginning I used the second option: I moved photos and videos manually from my phone to the server using a USB cable. Not only was it tedious, it would only happen when my phone storage was full, which is too late.

Then I discovered syncthing!

Syncthing is a peer-to-peer file synchronization application. Similar to Dropbox, it synchronizes folders you choose between different devices. Unlike Dropbox, you install it on your own devices, retaining ownership and control of your files. freenas also happens to have a syncthing plugin, so it can be installed very easily.

The rest of this article assumes you have FreeNAS running on a server with a dataset for storing photos.

Warning: You are responsible for your data. Make a back up before proceeding.

How does it work?

Syncthing is installed on both the freenas server and your Android smart phone. The folder where photos are saved by the Android camera app is added, with read/write permissions, to syncthing and shared with the freenas instance.

It’s tempting to add the photos as read-only; however, we want the server to be able to remove the photos from the phone, so read/write permission is essential.

Whenever photos are taken they will be synchronised with the server (assuming that both the server and phone are able to talk to one another. i.e. they are on the same network and both turned on).

To free up space on the phone and simultaneously organise the photos, a bash script is run periodically, using cron, to move the photos to the server’s photo storage location, using the photo’s exif data to organise the files. When syncthing syncs, the photos are removed from the smart phone.

Install syncthing on freenas

1. Install the syncthing plugin;
2. Change the owner of your photos dataset to the syncthing user;
3. Create a new dataset called syncthing and set both the owner and group to syncthing;
4. Add the new dataset to the syncthing jail’s storage;

Install syncthing on Android device

1. Install the syncthing app from the Google Play store;
2. Add the freenas syncthing instance as device;
3. Add the folder to syncthing where the camera app saves photos, with read/write permissions. Select to share the folder with freenas syncthing;
4. Confirm the share on the freenas syncthing instance.

Install script on freenas

• Create a new jail called photo_processing;
• Add the photos dataset to the new jail’s storage;
• Add the syncthing dataset to the new jail’s storage;
• SSH into the jail and install exiftool and git with: pkg install p5-Image-ExifTool-10.40 git
• Clone the script with: git clone https://gist.github.com/jonblack/9e907739527a56877212362e2844e5db photo_org and give the script execute permission: chmod a+x sort_photos.sh (Study the script first. Trust no-one!).
• Add a cron job on freenas that runs the following command as root: jexec -U syncthing photo_processing /root/photo_org/sort_photos.sh /mnt/syncthing/photos /mnt/photos/phone

The script

The following crude bash script moves files from the syncthing folder to a more permanent location on the server. During copying exiftool is used to extract date information to organize the files into a year/month/file_timestamp folders.

The script is evidence I have no idea what I’m doing in bash.

 1#!/usr/bin/env sh
 2
 3phone_photos="$1"
 4photos_root="$2"
 5
 6for filename in `find "$phone_photos" -type f`; do
 7    # Some paths can be ignored. For now lets hard code them.
 8    case "$filename" in
 9      *.thumbnail*) continue;; # thumbnails
10      *.stfolder*) continue;;  # syncthing folder marker
11    esac
12
13    # Get the creation date from the media format meta data. If not found,
14    # use the file modified time.
15    c_date=`exiftool "$filename" -CreateDate -d "%Y-%m" | cut -d : -f 2 | tr -d '[:space:]'`
16    c_fdt=`exiftool "$filename" -CreateDate -d "%Y%m%d%H%M%S" | cut -d : -f 2 | tr -d '[:space:]'`
17    if [ "$c_date" == "0000" -o "$c_date" == "" ]; then
18        # The arguments for stat are specific to freenas and are not very
19        # portable. This could break at any time. exiftool has
20        # FileModifyDate, try that some time.
21        c_date=`stat -f %Sm -t %Y-%m "$filename"`;
22        c_fdt=`stat -f %Sm -t %Y%m%d%H%M%S "$filename"`;
23    fi
24    if [ $? -eq 0 ]; then
25        c_date_y=`echo "$c_date" | cut -d - -f 1`;
26        c_date_m=`echo "$c_date" | cut -d - -f 2`;
27        if [ ! -d "$photos_root/$c_date_y/$c_date_m" ]; then
28            mkdir -p "$photos_root/$c_date_y/$c_date_m";
29        fi
30
31        # Add date/time to new filename to make files as unique as
32        # possible. Cameras are rubbish at file naming, and duplicates
33        # are possible when multiple cameras are used.
34        filename_base=$(basename "$filename");
35        filename_raw="${filename_base%.*}";
36        filename_ext="${filename_base##*.}";
37        new_filename="$filename_raw"_"$c_fdt"."$filename_ext"
38        echo mv "$filename" "$photos_root/$c_date_y/$c_date_m/$new_filename";
39        mv "$filename" "$photos_root/$c_date_y/$c_date_m/$new_filename";
40    fi
41done

Improvements

This is far from a perfect solution, but it-works-for-me™. Below are some improvements that would be nice to add at some point. Consider them an exercise for the reader 😏.

• Skipping files should be loaded from an ignore text file, not hardcoded into the script
• Automatic flagging of photos that are poorly composed, out-of-focused, and over/under-exposed. A nice machine learning problem, perhaps?
• Different organisation methods, such as geographical or autobiographical.