Cambak: a simple tool for derushing digital cameras

In December 2019, a few days before the big public transport strike that impacted all French citizens (and unconsciously gave us a first taste of how to work remotely at home and prepare us for what we will experience a few months later with the first lockdown), I got my first own interchangeable lens hybrid camera, a Sony A6400.

To take control of my new toy, during the first months, I took a lot of pictures. At home, in the street, at work during a Christmas event, in Portugal when I was there for the holiday season and once again, etc. In just few months, thousands of pictures have been taken.

Almost every time, when I got home, I exported my shots of the day to my home server. But sometimes, several days went without exporting them and when it was the moment to do it, it was a pain. Moving all files in dated folders, by type of files, was time consuming. Because I’m a software engineer and also because we are lazy persons, we want to automate everything (even if it’s useless). Here’s how a small piece of software called Cambak born.

2020: The first release of Cambak

#

On february 2020, I decided to create a tool in Python to automate this fastidious task for me. His name, Cambak, is the contraction of camera and backup. His first release on GitHub was on March 1st.

The goal of cambak was simple: take all pictures, RAW files and movies from the SD Card and copy them to my home server structured by date, camera and per type of file. Here’s an example of what it looks:

λ kalimdor ~ → tree -d cambak-old-dest
cambak-old-dest
└── 2021-03-20
    └── a6400
        ├── Pictures
        ├── RAW
        └── Videos

In terms of options, three were available: two were mandatory (-t, -n), the last one (-f) was optional but it never worked. It was made around the amazing stdlib of Python and used click to generate the CLI. To give you an idea, here’s what the help text looked like:

λ kalimdor cambak → λ git old* → poetry run cambak --help
usage: cambak [-h] -t TYPE -n NAME [-f] src dest

positional arguments:
  src                   Source folder (mounted card/usb camera volume)
  dest                  Destination folder (local, network volume)

options:
  -h, --help            show this help message and exit
  -t TYPE, --type TYPE  Type of camera
  -n NAME, --name NAME  Name of the camera
  -f, --force           Override if file already exists in the dest folder

In a daily usage, the command to export my files from my SD Card was (almost) this one:

λ kalimdor cambak → λ git old* → poetry run cambak -t SonyNex -n A6400 cambak cambak-old-dest
Total of 192 files to copy.
Copying cambak/DCIM/100MSDCF/DSC08395.JPG - Remaining: 191/192
Copying cambak/DCIM/100MSDCF/DSC08396.JPG - Remaining: 190/192
...

One of the big problem with this version was a stupid overengineered design concept of Camera classes. The goal was to create a class that contains all the technical information of a camera or a series/type of cameras. This decision was so stupid that made the tool very niche and unusable by other persons without developping specifically something for its camera. I call this: intellectual masturbation.

For example, for Sony Alpha/NEX cameras, the class that contains all information the tool needs to collect files looked like this:

class SonyNex(Camera):
    """General support for Sony NEX cameras (Alpha 5, 6, 7 and 9)"""

    img_folders = ["DCIM/*MSDCF"]
    raw_folders = img_folders
    vid_folders = ["PRIVATE/M4ROOT/CLIP"]

    img_extensions = [".JPG"]
    raw_extensions = [".ARW"]
    vid_extensions = [".MP4"]

overengineered right?

An another problem I encounter with the time was the command itself. It was soooo long and boring to use, and it results of always using my command history to avoid typing the whole command every single time. Can I just specify the path where my SD card is mounted and let the tool make the rest? It almost what I made… two years later.

2022: The full rewrite

#

Two years later after the first release, and two weeks before my departure for a trip to Stockholm with a friend, I decided to rewrite completely the script to become more powerful and more fun to use. This time, because I’m learning new things, I decided to use Go instead of Python.

To avoid making the same errors, I decided to make the tool more generic and open to all cameras. I also was focused to make the tool for flexible and customizable without sacrificing the simplicity to use that I had in mind.

To simplify the development of the CLI, I used Cobra, an easy and powerful library to create CLIs.

λ kalimdor cambak → λ git main → cambak help
Cambak is a simple but powerful too for derushing cameras.

The program use a configuration file located in '$HOME/.config/cambak.yaml'.
During the first execution, a default configuration file will be created. You
can override it by an another configuration file by using the --config flag.

For more information, please consult: https://github.com/themimitoof/cambak.

Usage:
  cambak [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  extract     Copy files from a source media to a local/remote destination
  help        Help about any command
  version     Return the version of Cambak

Flags:
      --config string   Path of the configuration file (default "/home/themimitoof/.config/cambak.yaml")
  -h, --help            help for cambak

Use "cambak [command] --help" for more information about a command.

Because I have some ideas for the future, I decided to split the CLI in commands. For the moment, I have only one useful command called extract contains all the logic of the tool.

λ kalimdor cambak → λ git main → cambak extract --help
The cambak extrator will copy/extract files from a source media (eg:
SD card, MTP drive, local/remote folder) to a local or remote destination folder.

By default, the folder destination structure is the following:

<destination folder>
└── <YEAR>
    └── <MONTH>-<DAY>
        └── <CAMERA_NAME>
            ├── Pictures
            ├── RAW
            └── Movies

You can change the destination format by using the '--format' flag or change the value
in the configuration file.

For more information, please consult: https://github.com/themimitoof/cambak.

Usage:
  cambak extract [flags]

Aliases:
  extract, copy, cp

Flags:
  -A, --all             Import all medias files type
  -c, --clean           Delete source file after been copied
      --dry-run         Only log what the extractor will do if this flag was not set
  -f, --format string   Structure format in the destination folder.
  -h, --help            help for extract
  -m, --merge           Merge the source file if it already exists in the destination folder
  -M, --movies          Import movies files
  -n, --name string     Name of the camera
  -P, --pictures        Import pictures files
  -R, --raws            Import RAWs files
  -s, --skip            Skip the source file if it already exists in the destination folder

Global Flags:
      --config string   Path of the configuration file (default "/home/themimitoof/.config/cambak.yaml")

As you can see, compared to the previous version, a lot of options are now available and make the tool more flexible. In addition, I introduced a configuration file to specify a specific workflow if you use the same every time. This avoid typing every single type all parameters and options to run the tool. In my case, here’s the one I use every time:

extract:
    pictures: true
    raws: true
    movies: true
    destination: "/mnt/okinawa-smb/Medias/backup-brut"
    format: '%y/%m-%d/%n/%t'
    conflict: skip
    camera_name: A6400
    clean_after_copy: false

Basically:

• it copies all pictures, RAW files and movies from my source media and copy them to my home server (called, yes, Okinawa).
• The structure of the destination folder looks like <YEAR>/<MONTH>-<DAY>/<CAMERA_NAME>/{Pictures,RAW,Movies}.
• In case I re-run the command and files already exists, I simply ignore them to save few (precious) seconds.
• The name of my camera is… A6400.
• It donesn’t clean the source media after having copied all files to its destination folder. This is just for safety, I do a complete format of the SD Card via the camera a few days/weeks later in case of an accident with a source file. This gives me the time to recover an image if it as not been yet archived in the automated weekly incremental backup of my home server.

And now, each time I plug my SD card, I run Cambak with a simple stupid command that looks like this one:

λ kalimdor ~ → cambak cp /mnt/my-sdcard
✅ 30 files collected (20 pictures, 8 RAWs, 2 movies). 0 files skipped.
Copying files...
 100% |██████████████████████████████████████████████| (30/30, 71 it/s)
✨ All files have been copied!

I just give the path where the SD Card is mounted, press enter and voilà. Dead simple right?

If I take a look at the destination folder, it looks like this:

λ kalimdor cam-export-demo → tree .
.
└── 2022
    ├── 03-16
    │   └── A6400
    │       ├── Pictures
    │       │   ├── DSC00251.JPG
    │       │   └── DSC00252.JPG
    │       └── RAW
    │           ├── DSC00251.ARW
    │           └── DSC00252.ARW
    └── 03-19
        └── A6400
            ├── Pictures
            │   ├── DSC00253.JPG
            │   ├── DSC00254.JPG
            ...

Let imagine one day, I made a session with three cameras and I don’t want to use the same destination folder that I normally use. I can simply use the below commands to override the configuration from the configuration file by the one specified in argument and voilà!

cambak cp /mnt/my-sdcard ~/Documents/FoodP0rn-Shoot  # With this command, I will use the default camera name set in the configuration file
cambak cp -n iphone13 /mnt/my-sdcard ~/Documents/FoodP0rn-Shoot  # This one and the next one, override the camera name with another one
cambak cp -n xt4 /mnt/my-sdcard ~/Documents/FoodP0rn-Shoot

Customize the destination structure

#

Let’s talk about an option called --format. This thing is simple but amazing and can be improved in the future to be even more powerful.

It is a some sort of templating mechanism that let you configure how you want to structure your files. By default, Cambak will structure them like this: <YEAR>/<MONTH>-<DAY>/<CAMERA_NAME>/{Pictures,RAW,Movies} (%y/%m-%d/%n/%t). If you don’t like this structure, you can easily change it for something more adapted to your needs.

Today, only few verbs/variables are available for the moment but already gives a lot of flexibility:

Let’s take a simple example. Imagine I want to create a folder that contains the name of a session or a trip + the date of the session. A lot of possibilities are available. It could be something like Stockholm Trip - %y-%m/%t that gives Stockholm Trip - 2022-03/{Pictures,RAW,Movies} or include even more information with for example Stockholm Trip - %y-%m/%m-%d/%n/%t which gives Stockholm Trip - 2022-03/03-19/A6400/{Pictures,RAW,Movies}.

This feature will make you fall in love with Cambak, trust me! I love this feature!

Ideas for the future

#

I have some ideas for the future of Cambak. For the moment there are only ideas or I need more investigation.

I would like to implement a date range and a rating options to get specific range of files. This feature requires reading EXIF metadatas but the main problem I got on testing multiple libraries was performance issues. Not because the libraries were slow, but because you need to open, read and parse each file individually. When you have few files on a Class 10 or a UHS-2 SD card, it’s OK but when you have hundreds of files, the process became very slow.

Another idea was to create a some sort of daemon that listen for new mounted SD cards, MTP devices, etc. and extract the files automatically. For the moment, I don’t have the need for this kind of feature but who knows? Probably one day I will need it, or probably you on reading this post wants this kind of feature.

Finally, this is one is more for fun and to learn new things. I would like to create a GTK4 app to visualize, sort and prepare an extraction job. Basically everything that an editing software or something like Shotwell already do. I don’t think I will work on it anytime soon but who knows, probably one day, or never 🤷‍♂️.

Should you use it?

#

If like me, you store all your pictures in a “RAW” folder before using your favorite photo editing software to create a session or fill your catalogue, yes I recommend you to use or at least give a try to Cambak. You probably will create or improve your workflow whether you are a professional or an amateur just like me.

Certainly, opening a terminal to type a command is not the most user-friendly thing for photographers and other persons who exclusively use GUIs and other clickodromes that never open a terminal but the tool is simple and have some interesting features that you can probably need.

This project is not an extraordinary piece of software with AI, machine learning, blockchain and other bullsh*t hype things but it answers to a problematic and gave me the opportunity to learn new things and discover things from the stdlib of Go. And that’s probably the most important thing about this project: learning and experimenting new things 😇.

Cambak is an open source project under MIT license. If you are interested, take a look to it’s GitHub page and take a look to the release page to get the latest binaries or package for your operating software.