git-chunks

Split large pushes into chunked commits to beat SCM push size limits and per-push scan/hook timeouts (secret scanning, pre-receive hooks, CI triggers)

Liveno hosted version

Sourcegithub.com/jishnuteegala/git-chunks ↗

Readme

Split large pending changes into chunked commits to reduce push size and per-push scan time.

The problem

If you've ever tried to push a large import — vendored dependencies, design assets, a repo migration — you've probably hit one of these:

remote: fatal: pack exceeds maximum allowed size (2.00 GiB)        # GitHub
remote: GitLab: Push size limit exceeded                           # GitLab
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
error: remote unpack failed: error VS403500: size of your push exceeds the limit  # Azure DevOps
remote: pre-receive hook declined                                  # timed-out server-side scan
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500  # server timed out processing the push

A single oversized push fails for two distinct reasons:

Hard size limits — most hosts cap the size of one push (GitHub: 2 GB/pack; GitLab/Bitbucket/Azure DevOps: configurable, often much less).
Per-push scan and hook timeouts — secret scanning, push protection, malware/DLP scanning, and custom pre-receive hooks all run against each push, and are killed after a time budget. A push with thousands of files or gigabytes of content can blow that budget, and the whole push is rejected even though it's under the size limit. Enterprise-managed GitLab/GitHub instances are especially prone to this.

The workaround is always the same tedious loop: stage some files, commit, push, repeat.

git-chunks automates that loop. It splits your pending changes into multiple commits based on criteria you set (max files and/or working-tree bytes per commit), optionally pushing after each one. Smaller commits usually reduce each push's pack and server-side scan workload, but the configured size is a planning heuristic, not a hard wire-size guarantee. Server policy findings must be fixed; this tool does not bypass them.

Because the binary is named git-chunks, git picks it up automatically as a subcommand: git chunks.

Install

# npm
npm install -g git-chunks

# Bun
bun add -g git-chunks

# pnpm
pnpm add -g git-chunks

# Homebrew (macOS / Linux)
brew install jishnuteegala/tap/git-chunks

# winget (Windows)
winget install jishnuteegala.git-chunks

# Scoop (Windows)
scoop bucket add jishnuteegala https://github.com/jishnuteegala/scoop-bucket
scoop install git-chunks

# Chocolatey (Windows; available after community moderation)
choco install git-chunks

# AUR (Arch Linux)
paru -S git-chunks-bin

# Go
go install github.com/jishnuteegala/git-chunks/cmd/git-chunks@latest

# Install script (Linux / macOS; installs to ~/.local/bin by default)
curl -fsSL https://raw.githubusercontent.com/jishnuteegala/git-chunks/main/scripts/install.sh | bash

The install script downloads the appropriate release archive and verifies it against the release's checksums.txt before installing. Review the script before piping it to a shell, or download it and run sh install.sh. Set INSTALL_DIR to choose another destination or pass a version such as curl -fsSL https://raw.githubusercontent.com/jishnuteegala/git-chunks/main/scripts/install.sh | bash -s v0.1.0. latest and stable both select the latest published release. To remove that installation:

curl -fsSL https://raw.githubusercontent.com/jishnuteegala/git-chunks/main/scripts/uninstall.sh | bash

The Winget and initial Chocolatey submissions may require community moderation before their install commands become available.

Linux packages

git-chunks is not in the official Debian/Fedora/etc. archives, so plain apt install git-chunks won't work. Instead, every release attaches native packages you download and install manually. For example (amd64):

VERSION=$(curl -s https://api.github.com/repos/jishnuteegala/git-chunks/releases/latest | grep -Po '"tag_name": "v\K[^"]*')

# Debian / Ubuntu
curl -LO "https://github.com/jishnuteegala/git-chunks/releases/download/v${VERSION}/git-chunks_${VERSION}_linux_amd64.deb"
sudo dpkg -i "git-chunks_${VERSION}_linux_amd64.deb"

# Fedora / RHEL
sudo dnf install "https://github.com/jishnuteegala/git-chunks/releases/download/v${VERSION}/git-chunks_${VERSION}_linux_amd64.rpm"

.apk (Alpine) and .pkg.tar.zst (Arch) packages are also attached to each release; Arch users can use the AUR package for managed updates. These manual installs don't auto-update.

Prebuilt binary archives are also on the Releases page - the build matrix covers Linux, macOS (darwin), and Windows on amd64 + arm64. Each release contains these checksummed payloads (replace ${VERSION} with the release version):

git-chunks_${VERSION}_linux_amd64.tar.gz
git-chunks_${VERSION}_linux_arm64.tar.gz
git-chunks_${VERSION}_darwin_amd64.tar.gz
git-chunks_${VERSION}_darwin_arm64.tar.gz
git-chunks_${VERSION}_windows_amd64.zip
git-chunks_${VERSION}_windows_arm64.zip
git-chunks_${VERSION}_linux_amd64.deb
git-chunks_${VERSION}_linux_arm64.deb
git-chunks_${VERSION}_linux_amd64.rpm
git-chunks_${VERSION}_linux_arm64.rpm
git-chunks_${VERSION}_linux_amd64.apk
git-chunks_${VERSION}_linux_arm64.apk
git-chunks_${VERSION}_linux_amd64.pkg.tar.zst
git-chunks_${VERSION}_linux_arm64.pkg.tar.zst
checksums.txt

Usage

# 20 files per commit
git chunks -n 20

# max 50 MB per commit, push after each commit
git chunks -s 50M -p

# combine criteria, custom message, preview first
git chunks -n 100 -s 100M -m "import legacy assets" --dry-run

# machine-readable plan, persistent log
git chunks -s 50M --dry-run --json
git chunks -s 50M -p --log push.log

Flags

Flag	Description
`-n, --max-files`	Max files per commit
`-s, --max-size`	Max total on-disk size of regular working-tree files per commit (`500K`, `50M`, `1G`)
`-m, --message`	Commit message prefix (default: `chunk`), suffixed with `(i/total)`
`-p, --push`	Push after each commit
`--remote`	Remote to push to (default: `origin`)
`--branch`	Branch to push (default: current)
`--retries`	Push retry attempts with exponential backoff (default: 3, maximum: 6)
`--dry-run`	Show the chunk plan without committing
`--json`	Output the `--dry-run` plan as JSON
`--log`	Append timestamped progress to a log file
`-q, --quiet`	Suppress progress output (errors still shown)
`-C, --repo`	Path to git repo (default: current dir)
`--version`	Print version

At least one of --max-files / --max-size is required.

Usage for AI agents and scripts

git-chunks is non-interactive and can resume after a push failure. The recipe:

# 1. Preview the plan as JSON (stdout; progress goes to stderr)
git chunks --max-size 50M --dry-run --json

# 2. Execute: commit in chunks and push each one, with retries and a log
git chunks --max-size 50M --push --retries 3 --log git-chunks.log

Exit codes: 0 success, 1 runtime error, 2 usage error.
If a push fails after retries, committed work is preserved. Rerunning first pushes those existing commits, then processes changes still pending.
Only run it in a trusted repository: Git hooks and repository configuration execute with your privileges. Remotes and credential helpers must also be trusted.
A machine-readable summary of this tool lives in llms.txt.

Resumability

Completed chunks are recoverable after a push failure. Chunks are committed one at a time, so:

If a push fails (even after retries), committed work is preserved. Rerun the command; it pushes existing unpushed commits before creating another chunk.
If that resume push fails, the command stops without creating an additional commit.

Notes

A single file larger than --max-size still gets its own commit — a file can't be split. If it exceeds your platform's hard limit you'll need Git LFS for that file.
Untracked files are included; deleted files count as 0 bytes. Start with an empty Git index: staged changes are rejected before any commit is made.
--max-size sums current on-disk sizes of regular working-tree files. Deletions, symlinks, submodules, Git history, compression, and protocol overhead are not represented, so actual push size can be higher or lower.
Chunking may reduce hook and scan time, but findings from server security or policy checks must be remediated rather than bypassed.
Progress goes to stderr; --dry-run output goes to stdout (pipe-friendly with --json).

Development

See CONTRIBUTING.md for contribution guidelines and the full validation checklist.

go test ./...
go build ./cmd/git-chunks

Releasing

Releases are fully automated with Conventional Commits, release-please, and GoReleaser:

Commits to main use conventional commit messages (feat:, fix:, perf:, ...)
release-please maintains a release PR with the next semver bump and CHANGELOG
Merging that PR creates an immutable tag and draft GitHub release, which triggers the publisher to:
- Build the OS/arch matrix (linux/darwin/windows x amd64/arm64)
- Attach archives + checksums to the release, with a changelog grouped by type
- Publish the Homebrew cask to jishnuteegala/homebrew-tap
- Publish the Scoop manifest to jishnuteegala/scoop-bucket
- Publish git-chunks + per-platform binary packages to npm
- Publish the GitHub release after its directly controlled channels verify
- Publish and verify the AUR package
- Submit the Chocolatey package and verify it once moderation makes it public
- Open and verify a PR to microsoft/winget-pkgs after release assets are public

No manual steps between merging and published packages.

Required repo secrets:

Secret	Purpose
`PACKAGES_GITHUB_TOKEN`	PAT with write access to the tap + scoop bucket repos
`WINGET_GITHUB_TOKEN`	PAT for the `winget-pkgs` fork used to open PRs to microsoft/winget-pkgs
`NPM_TOKEN`	Temporary npm fallback during migration to trusted publishing
`AUR_KEY`	Unencrypted SSH private key authorized by the AUR account
`CHOCOLATEY_API_KEY`	Chocolatey Community Repository API key

One-time setup: create homebrew-tap and scoop-bucket repos, fork microsoft/winget-pkgs, authorize the AUR SSH key, and create a Chocolatey API key. First-time Winget and Chocolatey submissions are externally moderated.

License

MIT — see LICENSE.

Loading readme…