Calibre Library Management Scripts

A collection of bash scripts for managing, backing up, and restoring a large Calibre ebook library on macOS, with automated nightly backups and a self-hosted CalibreWeb instance.

These scripts were written under the FourM identity namespace and are designed to run on a headless Mac Mini serving as a home server.

---

Background

This toolset was born out of a library corruption event that threatened 900+ hours of curation work across 8,500+ books. The recovery process exposed several weaknesses in Calibre's built-in backup/restore mechanism and led to building a more robust, transparent backup architecture based on rsync with hard-link deduplication for local snapshots and rclone with versioning for iCloud.

See SCRIPTS.md for detailed documentation of each script including design decisions and known issues encountered during development.

---

Architecture

Live library: ~/Calibre Library (internal disk)
     │
     ├── Nightly rsync snapshots ──► /Volumes/Extreme/CalibreBackups/
     │   7 dailies, 4 weeklies, 2 monthlies, 1 yearly
     │   Hard-link deduplication via --link-dest
     │   ~1GB per day incremental on a 190GB library
     │
     ├── Nightly rclone sync ──────► ~/Documents/Backups/Calibre/ (iCloud)
     │   current/   — rolling mirror, always reflects last backup
     │   versions/  — dated delta folders of changed/deleted files (7 kept)
     │   Syncs to iCloud automatically
     │   rclone used instead of rsync because iCloud does not support hard links
     │
     └── Syncthing live mirror ────► local backup machine internal disk
         Continuous, best-effort, same-day recovery
              │
              └── Sync.com client ──► Cloud storage

Recovery options in order of speed:

1. Local backup machine (Syncthing) — same-day work preserved
2. Last nightly on external drive — known good, previous night, full point-in-time snapshot
3. iCloud current — offsite full mirror, always reflects last backup
4. iCloud versions — offsite per-file recovery for recently changed/deleted files
5. Sync.com cloud — ultimate offsite fallback

---

Scripts Overview

Script	Purpose
config.sh.example	Template for local machine configuration (copy to config.sh)
calibre_nightly_backup.sh	Automated nightly backup with rotation
calibre_check_integrity.sh	Scan library for corrupt PDF/EPUB files
calibre_sync.sh	Copy missing books between two library folders
calibre_update_metadata.sh	Update book metadata from OPF files
calibre_restore_preview.sh	Preview a backup snapshot before restoring
calibre_restore_finalize.sh	Finalize a restore after preview approval
setup_calibreweb.sh	Install or reinstall CalibreWeb
start_calibreweb.sh	Start CalibreWeb in a tmux session
stop_calibreweb.sh	Stop CalibreWeb, Calibre app and worker processes
install_calibre_backup_launchd.sh	Install nightly backup as a launchd agent
info.fourm.calibre-backup.plist	launchd plist for 2am nightly schedule
calibre_fts_search.sh	Full-text search from the CLI, saved as ids / snippets / a search expression
calibre_ids_to_search.sh	Turn a list of book ids (or a catalog CSV, or calibredb JSON) into a search expression
calibre_tag_ids.sh	Add or remove one tag across a list of ids without destroying the other tags

The three full-text-search scripts are documented in depth in FULL_TEXT_SEARCH.md, which also covers why a search result in the Calibre GUI does not survive quitting the app, and every route for making one that does.

---

Requirements

• macOS (tested on Intel Mac Mini running macOS 15)
• bash 3.2+ (macOS default — all scripts are bash 3.2 compatible)
• Python 3.12 (for CalibreWeb)
• poppler for PDF integrity checking: brew install poppler
• tmux for CalibreWeb session management: brew install tmux
• rclone for iCloud versioned backup: brew install rclone
• rsync (included with macOS)
• Calibre desktop app installed at /Applications/calibre.app

---

Quick Start

First time setup

cp config.sh.example config.sh
# Edit config.sh with your paths and settings
nano config.sh

Setting up CalibreWeb

chmod +x setup_calibreweb.sh
./setup_calibreweb.sh

Then start it:

./start_calibreweb.sh
# Attach to session: tmux attach -t calibreweb

Setting up nightly backups

Install the launchd agent:

chmod +x install_calibre_backup_launchd.sh
./install_calibre_backup_launchd.sh

Verify it loaded:

launchctl list | grep calibre
# Should show: -  0  info.fourm.calibre-backup

Running a manual backup

./calibre_nightly_backup.sh

Logs are written to $LOG_DIR/calibre_backup_YYYYMMDD_HHMMSS.log.

---

macOS Permissions

The backup script requires Full Disk Access for /path/to/bin/rclone, /usr/bin/rsync and /bin/bash to write to external drives when run via launchd:

System Settings → Privacy & Security → Full Disk Access → Add /path/to/bin/rclone, /usr/bin/rsync and /bin/bash

Without this, launchd jobs will fail with Operation not permitted on external volumes.

---

Key Design Decisions

Hard-link deduplication (external drive) — rsync --link-dest creates snapshots where unchanged files are hard links to the previous snapshot rather than copies. A 190GB library with daily changes costs ~1GB per additional snapshot rather than 190GB. Deleting an old snapshot only frees space for files that exist exclusively in that snapshot.

rclone with --backup-dir (iCloud) — iCloud Drive does not support hard links. Using rsync --link-dest on iCloud produces broken snapshots where all versions share inodes and mutate together — every snapshot ends up reflecting the latest state rather than the state at creation time. rclone's --backup-dir solves this: before each sync it moves changed and deleted files into a dated versions folder, giving genuine point-in-time recovery for individual files without relying on hard links.

No compression — snapshots are raw files in Calibre's native folder structure. Any file can be dragged out of a snapshot in Finder or restored with a simple cp -R or rclone sync. No special tools needed.

Bash 3.2 compatibility — macOS ships with bash 3.2 (due to GPL licensing). All scripts avoid bash 4+ features: no declare -A, no ${var,,} lowercase, no mapfile. Associative arrays are replaced with sorted temp files and grep.

Always exit 0 — the integrity check script always exits 0 so corrupt files are logged but never abort the backup. The backup script itself handles external drive absence gracefully rather than dying under set -euo pipefail.

---

Configuration

All machine-specific values live in config.sh which is gitignored and never committed. Copy config.sh.example to config.sh and fill in your values:

cp config.sh.example config.sh

Key values to set:

LIBRARY="$HOME/Calibre Library"
LOG_DIR="$HOME/path/to/Logs"
VENV_DIR="$HOME/path/to/venv/calibre-web-env"
REVIEW_DIR="$HOME/Code/FourM/Reviews"
HOST_BACKUP="/Volumes/YOUR_EXTERNAL_DRIVE/CalibreBackups"
ICLOUD_BACKUP="$HOME/Documents/Backups/Calibre"
CALIBRE_HOST="https://your-calibre-domain:YOUR_PORT"
PORT=YOUR_PORT
CERT_FILE="$HOME/path/to/your_cert.pem"
KEY_FILE="$HOME/path/to/your_key.key"

REVIEW_DIR is where calibre_fts_search.sh writes its results. It was added after the first config.sh files were already in circulation, and config.sh is gitignored and hand-carried between machines, so an older copy will not have it — the script falls back to $LOG_DIR/reviews and tells you where the files went rather than refusing to run.

Keep it outside the repo. .snippets.txt holds passages copied out of your books and this repo is public. Because you choose the path, no committed .gitignore can know what to exclude — put REVIEW_DIR inside the repo and you are editing .gitignore by hand and relying on remembering to. The default sits next to $LOG_DIR, which is outside the repo for the same reason.

See config.sh.example for the full list, including CALIBREDB, CALIBRE_WEB_CONFIG, TMUX_SESSION and the retention counts.

Every path a script needs lives here, including the ones under $HOME. They used to be derived inside each script, on the grounds that a $HOME path gives nothing away — which is true, but it meant the same location was spelled out in four or five places and changing one of them meant hunting down the rest. One definition per path is one place to change it and one place for it to be wrong. $HOME is still expanded at runtime, so no username is ever written down.

The one exception is SCRIPT_DIR. Each script derives its own directory, because that is what it uses to find config.sh — it cannot come from the file it is used to locate. Set SCRIPT_DIR in config.sh anyway if you want to override it; scripts honour it when present and fall back to their own directory when it isn't.

---

License

MIT