pack-aggregate: cheaply consolidate loose objects and small packs

.gitignore

@@ -119,6 +119,7 @@
 /git-name-rev
 /git-notes
 /git-p4
+/git-pack-aggregate
 /git-pack-redundant
 /git-pack-objects
 /git-pack-refs

Documentation/config/pack.adoc

@@ -39,6 +39,22 @@ is set to "multi", reuse parts of just the bitmapped packfile. This
 can reduce memory and CPU usage to serve fetches, but might result in
 sending a slightly larger pack. Defaults to true.
 
+pack.aggregateMaxObjects::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-objects`
+	option. Defaults to `100000`; see that option for details.
+
+pack.aggregateMaxInputPackSize::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-input-pack-size`
+	option. Defaults to `0` (automatic); see that option for details.
+
+pack.aggregateMaxLooseObjects::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-loose-objects`
+	option. Defaults to `100000`; see that option for details.
+
+pack.aggregateMaxPacks::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-packs`
+	option. Defaults to `10000`; see that option for details.
+
 pack.island::
 	An extended regular expression configuring a set of delta
 	islands. See "DELTA ISLANDS" in linkgit:git-pack-objects[1]

Documentation/config/repack.adoc

@@ -64,3 +64,21 @@ repack.midxNewLayerThreshold::
 When the tip layer has fewer packs than this threshold, those packs are
 excluded from the geometric repack entirely, and are thus left
 unmodified. Must be at least 1. Defaults to 8.
+
+repack.aggregateOnce::
+	If set to true, linkgit:git-repack[1] will run
+	linkgit:git-pack-aggregate[1] once before inspecting the packs
+	and loose objects to repack. This can quickly reduce a large
+	number of files before the more thorough repack begins.
+	Defaults to false. Can be overridden on the command line with
+	`--aggregate-once` or `--no-aggregate-once`.
+
+repack.aggregateLoop::
+	If set to true, linkgit:git-repack[1] will spawn a background
+	linkgit:git-pack-aggregate[1] while repacking, until redundant-pack
+	cleanup begins. The aggregator rolls up small packs and loose
+	objects that arrive after the main pack-objects process records
+	its input exclusions, preventing them from accumulating in
+	`objects/pack/` and slowing other Git operations on busy servers.
+	Defaults to false. Can be overridden on the command line with
+	`--aggregate-loop` or `--no-aggregate-loop`.

Documentation/git-pack-aggregate.adoc

@@ -0,0 +1,207 @@
+git-pack-aggregate(1)
+=====================
+
+NAME
+----
+git-pack-aggregate - Quickly roll up loose objects and small packs
+without doing any delta compression or search
+
+SYNOPSIS
+--------
+[verse]
+'git pack-aggregate' (--once | --loop) [--interval=<seconds>]
+                   [--min-loose=<n>] [--min-packs=<n>]
+                   [--max-loose-objects=<n>]
+                   [--max-objects=<n>] [--max-packs=<n>]
+                   [--max-input-pack-size=<bytes>]
+                   [--keep-pack=<pack-name>]
+                   [--exclude-pack-file=<path>]
+                   [--exclude-loose-file=<path>]
+                   [--[no-]progress]
+                   [--parent-pipe-fd=<n>]
+
+DESCRIPTION
+-----------
+
+`git pack-aggregate` rolls accumulated loose objects and small
+packfiles into new packs.  It can run one cycle as a quick recovery
+step, or run cycles periodically to keep new material under control
+while longer-running maintenance proceeds.  Each cycle has two steps:
+
+1. Bundle local loose objects (minus any listed in
+   `--exclude-loose-file`) into new packs and unlink the loose copies.
+2. Aggregate small local packs (minus any excluded by `--keep-pack` or
+   `--exclude-pack-file`, or those referenced by the multi-pack-index,
+   or those carrying a `.keep`, `.promisor`, `.mtimes`, or `.bitmap`
+   sidecar) into new packs and unlink the source packs.  If step 1
+   writes a single pack, it is naturally a candidate here and will
+   normally be folded into the step-2 output.  Multiple loose-rollup
+   packs are left for a later cycle.
+
+No delta search is performed and no bitmaps or commit-graphs are updated.
+Existing packed representations are reused where possible.  When an
+object has both delta and full-object representations in the input
+packs, aggregation prefers the existing delta.  All output packs are
+written with `pack-objects --window=0 --mark-unoptimized`, so each one
+ships with an `.unoptimized` sidecar (see linkgit:gitformat-pack[5]).  A
+subsequent thorough repack (linkgit:git-repack[1]) honors that marker
+and reconsiders intra-pack deltas at that time.
+
+The `pack.packSizeLimit` setting can split either step's output into
+multiple packs.  All outputs from a batch are installed before its
+input packs or loose objects are removed.  Size-based splitting may
+require expanding deltas whose bases are in another output pack.
+By default, step 2 skips packs larger than half that limit to avoid
+repeatedly copying nearly full packs without reducing the pack count.
+See `--max-input-pack-size` for a smaller input-size threshold.
+
+This command serves two related purposes.  With `--once`, it provides a
+quick recovery step for a repository that has accumulated many loose
+objects or small packs.  With `--loop`, it keeps new material under
+control while a long-running repack is in flight.  Both reduce the
+number of files that unrelated Git operations must scan, while leaving
+a later thorough linkgit:git-repack[1] to optimize deltas and pack
+layout.  The `repack.aggregateOnce` and `repack.aggregateLoop`
+configuration variables (see linkgit:git-repack[1]) let `git repack`
+request either behavior independently.
+
+Like linkgit:git-repack[1], `git pack-aggregate` takes no locks of its
+own.  Callers that need serialization must arrange it themselves (for
+example by running under `git gc`).
+Callers can declare "do not touch these inputs" with `--keep-pack`,
+`--exclude-pack-file`, and `--exclude-loose-file`.
+Rereading exclusion files each cycle is not a synchronization protocol:
+publish exclusions before starting concurrent aggregation, and keep the
+protected inputs excluded until aggregation stops.
+When coordinating with a concurrent repack, protect every pack it may
+read or include in its replacement MIDX.  Exclude existing packs with
+`--exclude-pack-file` and protect new outputs with `.keep` files before
+their indexes become visible.  Keep these protections through the MIDX
+write and until aggregation stops.  `git repack` arranges this itself
+when spawning the aggregator and removes only its own `.keep` files
+afterward.
+
+OPTIONS
+-------
+
+--once::
+	Run a single cycle and exit.  Exactly one of `--once` or
+	`--loop` is required.
+
+--loop::
+	Run cycles forever, sleeping `--interval` seconds between
+	cycles.  Stops on `SIGTERM`, `SIGHUP`, or `SIGINT`.
+
+--interval=<seconds>::
+	Number of seconds to sleep between the end of one cycle and the
+	start of the next.  Defaults to 60.  Only meaningful with
+	`--loop`.
+
+--min-loose=<n>::
+	Skip aggregation of loose objects if fewer than `<n>` remain
+	after applying `--exclude-loose-file`.  Defaults to 5.
+
+--min-packs=<n>::
+	Skip aggregation of small packfiles if fewer than `<n>`
+	aggregatable packs remain after applying all exclusions.  Defaults
+	to 5.
+
+--keep-pack=<pack-name>::
+	Exclude the given pack from aggregation.  `<pack-name>` is the pack
+	file name without a leading directory (e.g. `pack-123.pack`).
+	This option can be repeated to keep multiple packs.
+
+--exclude-pack-file=<path>::
+	Read a list of pack basenames (one per line) from `<path>` and
+	never touch any of those packs.  Lines may name a basename with
+	or without a `.pack` or `.idx` suffix; the suffix is stripped.
+	Blank lines and lines beginning with `#` are ignored.  When
+	`git pack-aggregate` is spawned by a long-running `git repack`,
+	this file is populated by that `pack-objects` itself via
+	`--emit-input-packs`, and may contain a conservative superset
+	of the packs it will actually consume (such as in `--geometric`
+	mode).
+
+--exclude-loose-file=<path>::
+	Read a list of loose object IDs (one per line) from `<path>`
+	and never pack or unlink any of those loose objects.  Blank
+	lines and lines beginning with `#` are ignored.  When `git
+	pack-aggregate` is spawned by a long-running `git repack`, this
+	file is populated by that `pack-objects` itself via
+	`--emit-input-loose`.
+
+--parent-pipe-fd=<n>::
+	An inherited pipe file descriptor whose write end the parent
+	process holds open.  When the parent exits the pipe is closed
+	and `git pack-aggregate` exits at the next cycle boundary,
+	without waiting out the remainder of `--interval`.  This is an
+	internal plumbing option used by `git repack` to keep its
+	companion aggregator from outliving it.
+
+--progress::
+--no-progress::
+	Show progress while creating packs.  The default is to show progress
+	for `--once` when standard error is connected to a terminal, and to
+	suppress progress for `--loop`.
+
+--max-loose-objects=<n>::
+	Process at most `<n>` loose objects per batch.  All output packs
+	are installed and the loose copies are removed before the next
+	batch begins, so repository performance can improve incrementally
+	during recovery from an extreme backlog.  Loose objects created
+	after the cycle starts are left for the next cycle.
+	If multiple packs are produced, they are left for a later aggregation
+	cycle instead of being copied again immediately.
+	Defaults to the `pack.aggregateMaxLooseObjects` configuration value,
+	or 100000 if that is unset.  A value of `0` disables the limit.
+
+--max-objects=<n>::
+	Skip packs estimated to contain more than `<n>` objects, leaving
+	them untouched by step 2.  The object count is estimated cheaply
+	from the size of the pack's `.idx` file rather than by opening it.
+	Legacy v1 indexes may admit slightly more objects than the limit.
+	This keeps `git pack-aggregate` focused on rolling up the small
+	packs it is meant for, instead of repacking a large base pack
+	(which would amount to a near-full repack).  Defaults to the
+	`pack.aggregateMaxObjects` configuration value, or 100000 if that
+	is unset.  A value of `0` disables the limit.  Note that packs
+	already referenced by the multi-pack-index are excluded
+	regardless of this setting, so on a normally-maintained
+	repository the large base pack is skipped anyway.
+
+--max-input-pack-size=<bytes>::
+	Skip input packs whose `.pack` file is larger than `<bytes>`.
+	This byte-size gate applies in addition to `--max-objects`;
+	it does not limit loose-object rollup or individual output packs.
+	The suffixes `k`, `m`, and `g` are supported.
++
+Defaults to `pack.aggregateMaxInputPackSize`, or `0` (automatic).
+Automatic selection uses half of `pack.packSizeLimit`, after applying
+the same 1-MiB minimum for nonzero output limits as `pack-objects`.
+An explicit positive input limit above that ceiling is an error;
+a lower limit is allowed.  If `pack.packSizeLimit` is unset or zero,
+automatic selection imposes no byte-size gate, and any explicit
+positive input limit is allowed.  An explicit `0` overrides the
+configuration and restores automatic selection.
+
+--max-packs=<n>::
+	Process at most `<n>` input packs per `pack-objects` invocation.
+	When more than `<n>` aggregatable packs are present, split them
+	into approximately balanced batches by pack count.  The final batch
+	may be smaller, and each batch can produce multiple output packs.
+	Limiting input packs helps control memory use for pack data and
+	indexes on repositories with very many packs.  Batching happens on
+	whole-pack boundaries, keeping each delta and its base in the same
+	input batch.
+	Defaults to the `pack.aggregateMaxPacks` configuration value, or
+	`10000` if unset.  A value of `0` disables the input-pack limit and
+	processes all input packs in one batch.
+
+SEE ALSO
+--------
+linkgit:git-repack[1], linkgit:git-pack-objects[1],
+linkgit:gitformat-pack[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite

Documentation/git-pack-objects.adoc

@@ -143,6 +143,14 @@ options which imply `--revs`.
 	have an mtime older than `<approxidate>`. If unspecified (and
 	given `--cruft`), then no objects are eliminated.
 
+--mark-unoptimized::
+	Write an `.unoptimized` marker file alongside each output pack. The
+	marker signals that objects within the pack have not been fully
+	delta-searched against other objects within the same pack and
+	that future repacking should consider them. Any deltas that do
+	exist within this pack can still be reused, however.
+	Incompatible with `--stdout`.
+
 --window=<n>::
 --depth=<n>::
 	These two options affect how the objects contained in
@@ -246,6 +254,15 @@ depth is 4095.
 	wholesale enforcement of a different compression level on the
 	packed data is desired.
 
+--prefer-reused-deltas::
+--no-prefer-reused-deltas::
+	Only meaningful with `--stdin-packs`. When the same object is
+	present as a base in one included pack and a delta in another,
+	record the delta copy rather than the base. This costs a
+	cheap object-header read per duplicate, and has no effect
+	under `--no-reuse-delta`. Disabled by default; use
+	`--no-prefer-reused-deltas` to keep the first copy encountered.
+
 --compression=<n>::
 	Specifies compression level for newly-compressed data in the
 	generated pack.  If not specified,  pack compression level is

Documentation/git-repack.adoc

@@ -13,6 +13,7 @@ SYNOPSIS
 	[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]
 	[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]
 	[--filter=<filter-spec>] [--drop-filtered [--dry-run]]
+	[--[no-]aggregate-once] [--[no-]aggregate-loop]
 
 DESCRIPTION
 -----------
@@ -274,6 +275,9 @@ picks the smallest set of packfiles such that as many of the larger
 packfiles (by count of objects contained in that pack) may be left
 intact.
 +
+Selection also accounts for packs forced into the roll-up by `.unoptimized`
+markers, including another pack if needed to preserve the progression.
++
 Unlike other repack modes, the set of objects to pack is determined
 uniquely by the set of packs being "rolled-up"; in other words, the
 packs determined to need to be combined in order to restore a geometric
@@ -337,6 +341,25 @@ created for any new pack(s) without disturbing the existing chain.
 	Pass the `--path-walk` option to the underlying `git pack-objects`
 	process. See linkgit:git-pack-objects[1] for full details.
 
+--aggregate-once::
+--no-aggregate-once::
+	Run linkgit:git-pack-aggregate[1] once before inspecting the
+	packs and loose objects to repack. This can quickly reduce a
+	large number of files before the more thorough repack begins.
+	Packs named by `--keep-pack` are excluded from this preliminary pass.
+	Overrides the `repack.aggregateOnce` configuration variable.
+	Off by default.
+
+--aggregate-loop::
+--no-aggregate-loop::
+	Spawn a background linkgit:git-pack-aggregate[1] while
+	repacking, until redundant-pack cleanup begins. The aggregator
+	rolls up small packs and loose objects that arrive after the main
+	pack-objects process records its input exclusions, preventing them
+	from accumulating in `objects/pack/` and slowing other Git
+	operations on busy servers. Overrides the `repack.aggregateLoop`
+	configuration variable. Off by default.
+
 CONFIGURATION
 -------------
 
@@ -361,6 +384,7 @@ SEE ALSO
 --------
 linkgit:git-pack-objects[1]
 linkgit:git-prune-packed[1]
+linkgit:git-pack-aggregate[1]
 
 GIT
 ---

Documentation/gitformat-pack.adoc

@@ -12,6 +12,7 @@ SYNOPSIS
 $GIT_DIR/objects/pack/pack-*.{pack,idx}
 $GIT_DIR/objects/pack/pack-*.rev
 $GIT_DIR/objects/pack/pack-*.mtimes
+$GIT_DIR/objects/pack/pack-*.unoptimized
 $GIT_DIR/objects/pack/multi-pack-index
 
 DESCRIPTION
@@ -357,6 +358,31 @@ All 4-byte numbers are in network byte order.
     and a checksum of all of the above (each having length according
     to the specified hash function).
 
+== pack-*.unoptimized files
+
+The optional `.unoptimized` file is an empty marker sitting alongside a
+`pack-*.pack` (and its `.idx`). It signals to `git pack-objects` that
+the delta layout of the pack should not be trusted: even when two
+objects appear together in the same pack and neither is stored as a
+delta, the next packing run should still call out to its delta search
+routine for the pair instead of assuming a prior pack-objects already
+considered (and rejected) the pair.
+
+This is intended for producers that intentionally skip delta search
+when writing a pack (for example, processes that bulk-import objects
+or aggregate multiple existing packs without recomputing deltas).
+Without this marker, the same-pack delta skip in `git pack-objects`
+would silently inherit those producers' lack of delta search into
+future repacks.
+
+The contents of the file are currently ignored. Producers should
+write an empty file; consumers must tolerate (and ignore) any
+content.
+
+The marker only affects whether `git pack-objects` will attempt to
+compute new deltas for object pairs that share the marked pack. It
+does not disable reuse of existing on-disk deltas.
+
 == multi-pack-index (MIDX) files have the following format:
 
 The multi-pack-index files refer to multiple pack-files and loose objects.

Documentation/meson.build

@@ -99,6 +99,7 @@ manpages = {
   'git-name-rev.adoc' : 1,
   'git-notes.adoc' : 1,
   'git-p4.adoc' : 1,
+  'git-pack-aggregate.adoc' : 1,
   'git-pack-objects.adoc' : 1,
   'git-pack-refs.adoc' : 1,
   'git-patch-id.adoc' : 1,

Makefile

@@ -1465,6 +1465,7 @@ BUILTIN_OBJS += builtin/multi-pack-index.o
 BUILTIN_OBJS += builtin/mv.o
 BUILTIN_OBJS += builtin/name-rev.o
 BUILTIN_OBJS += builtin/notes.o
+BUILTIN_OBJS += builtin/pack-aggregate.o
 BUILTIN_OBJS += builtin/pack-objects.o
 ifndef WITH_BREAKING_CHANGES
 BUILTIN_OBJS += builtin/pack-redundant.o