Chapter 10-3 — Monorepo Performance | DevInHyderabad☕ ☕ 18 min read

Monorepo Performance

Monorepo slow hai toh Git kaa dosh nahi, strategy ka dosh hai.

01The Monorepo Challenge: Size and Speed

Monorepo means ALL your projects live in ONE repository — frontend, backend, mobile, shared libraries, infrastructure code, everything together.

The benefits are powerful: atomic commits across projects (change API and frontend in one commit), shared versioning (all packages always in sync), easy refactoring (search and replace across the entire codebase), and a single source of truth.

The challenge: the repository becomes HUGE. Real monorepos at Google, Meta, and Microsoft range from 10GB to over 100GB. Even smaller companies hit 5-10GB easily.

Performance problems multiply: slow clone (downloading gigabytes of data), slow git status (scanning millions of files), slow git log (processing millions of commits), slow CI (checking out the entire repo every build).

Git was designed for smaller repositories. Monorepos push Git to its limits and require special strategies to stay fast.

# The monorepo problem
git clone https://github.com/huge-company/monorepo.git
# Cloning into 'monorepo'...
# Receiving objects:  12% (52384/423892), 1.23 GiB | 2.50 MiB/s
# (45 minutes later...)
# Receiving objects: 100% (423892/423892), 9.87 GiB | 3.10 MiB/s

# Even after cloning, daily operations are slow
time git status
# real    0m8.234s  ← 8 SECONDS just for status!

time git log --oneline -1
# real    0m2.145s  ← 2 seconds for a single log entry

time git diff
# real    0m5.678s  ← 5 seconds for a diff

# The root causes:
# 1. Millions of files in working directory
# 2. Millions of commits in history
# 3. Large binary assets (images, videos, ML models)
# 4. Bloated .git directory with full history
Real monorepo sizes: Google's Piper has 1 billion+ files and 35 million+ commits. Meta's monorepo has 4 million+ commits. Microsoft's Windows repo is 300GB+. These companies use custom VCS or heavily modified Git. For most organizations, standard Git with optimization strategies (shallow clone, sparse checkout, partial clone) handles repos up to several hundred thousand files comfortably.
02Shallow Clone: Just the Latest

git clone --depth 1 creates a shallow clone — only the latest commit, no history.

It is dramatically faster for large repos: you download only the files at the latest state, not the entire history. A 10GB repo might only download 500MB with a shallow clone.

Use cases: CI/CD builds, deployment scripts, automated testing — anywhere you need the code but not the history.

Limitations: you cannot view history beyond the depth, you usually cannot push from a shallow clone, and git blame and git log do not work for older commits.

--depth N gets the last N commits. You can deepen later with git fetch --unshallow.

# Shallow clone — only the latest commit
git clone --depth 1 https://github.com/huge-company/monorepo.git
# Cloning into 'monorepo'...
# Receiving objects: 100% (1243/1243), 45.23 MiB | 12.50 MiB/s
# Done in 30 seconds instead of 45 minutes!

# Limitations
git log --oneline
# abc1234 (HEAD -> main) latest commit
# (only 1 commit visible — no history!)

git blame src/app.js
# fatal: no such path in the commit history  ← doesn't work for deep history

git push origin main
# ERROR: cannot push from a shallow clone

# Deepen if needed
git fetch --unshallow   # get full history
git fetch --depth=100   # get last 100 commits

# Shallow clone with single branch (even faster)
git clone --depth 1 --single-branch --branch main https://github.com/huge/repo.git
# Only downloads main branch, not all branches
💡 Pro Tip: Use --depth 1 for CI where you only need to build and test. Use --depth 2 if you need to diff against the previous commit. Never use shallow clone for daily development — you lose git blame, git log, git bisect, and other debugging tools that depend on history.
03Sparse Checkout: Only What You Need

Sparse checkout lets you clone a repository but only check out SPECIFIC directories.

You still get the full commit history, but your working directory only contains the files you care about. This is the perfect balance for monorepo developers.

Perfect for monorepos where you work on one package but need to see full history and make cross-project commits.

Git 2.25+ introduced cone mode (--cone) which is much faster for large monorepos because it uses directory-level patterns instead of complex glob patterns.

The workflow: git sparse-checkout init --cone then git sparse-checkout set packages/frontend

# Step 1: Create a sparse checkout clone
git clone --filter=blob:none --sparse https://github.com/huge/monorepo.git
cd monorepo

# Initially, almost no files are checked out
ls
# (only root-level files like README.md)

# Step 2: Set the directories you want
git sparse-checkout init --cone
git sparse-checkout set packages/frontend packages/shared

# Now only those directories exist
ls packages/
# frontend/  shared/
# (backend/, mobile/, infra/ are NOT checked out!)

# Step 3: Add more directories later
git sparse-checkout add packages/backend

# Step 4: Remove directories you no longer need
git sparse-checkout set packages/frontend
# (removes packages/shared and packages/backend from working dir)

# Step 5: See what is checked out
git sparse-checkout list
# packages/frontend

# Full history is available even for unchecked-out dirs
git log -- packages/backend/api.js  # still works!
04Partial Clone: Blobs on Demand

git clone --filter=blob:none downloads the commit and tree objects but NOT the file content (blobs).

Git downloads blobs on demand — when you check out a file or run git diff, it fetches only the needed blobs from the server.

This combines the best of both worlds: full history is available, but you do not download gigabytes of data you do not need.

--filter=blob:limit=1m — only download blobs smaller than 1MB; large files are fetched on demand.

--filter=tree:0 — do not even download tree objects; fetch everything on demand (ultra-thin clone).

Requires server support — GitHub, GitLab, and Azure DevOps all support it.

# Blobless clone — full history, no file content initially
git clone --filter=blob:none https://github.com/huge/monorepo.git
# Cloning into 'monorepo'...
# Receiving objects: 100% (423892/423892), 234.56 MiB
# (Only commits and trees — much smaller than 9.87 GiB!)

# File contents are fetched on demand
cat src/app.js
# (Git automatically fetches the blob for this file)

git diff HEAD~1
# (Git fetches the needed blobs to compute the diff)

# Full history works!
git log --oneline | head -20  # all commits visible
git blame src/app.js           # works perfectly

# Treeless clone — even thinner
git clone --filter=tree:0 https://github.com/huge/monorepo.git
# Doesn't even download tree objects
# Fetches everything on demand — smallest initial download

# Size-limited clone
git clone --filter=blob:limit=1m https://github.com/huge/monorepo.git
# Downloads all blobs < 1MB
# Large files (images, videos) are fetched on demand

# Combine with sparse checkout for maximum efficiency
git clone --filter=blob:none --sparse https://github.com/huge/monorepo.git
# Blobless + sparse = minimal download
💡 Pro Tip: Blobless clones are the best default for developers working with large repositories. You get full history access (git log, git blame work), but only download file contents as needed. The initial clone is 5-20x smaller than a full clone, and subsequent blob fetches are tiny because they only download what you actually access.
05CI/CD Optimization for Monorepos

CI pipelines for monorepos need special optimization to avoid checking out the entire repo every time.

Strategy 1: Shallow clone — use fetch-depth: 1 in GitHub Actions. Fastest checkout, but no history.

Strategy 2: Path-based triggers — only run CI for directories that changed. Saves massive CI time in monorepos.

Strategy 3: Caching — persist the .git directory between CI runs. Incremental fetches instead of full clones.

Strategy 4: Sparse checkout in CI — only check out the directories being tested. Smaller working directory, faster operations.

Strategy 5: Monorepo tools (Nx, Turborepo, Lerna) — intelligent task scheduling based on dependency graphs. Only build and test affected packages.

# GitHub Actions optimized for monorepo
name: CI
on: [push, pull_request]

jobs:
  # Strategy 1: Shallow clone (fastest)
  fast-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1  # shallow clone

  # Strategy 2: Path-based triggers
  frontend-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2  # need 2 for diff
      - name: Check if frontend changed
        run: |
          if git diff --name-only HEAD~1 HEAD | grep -q "^packages/frontend/"; then
            echo "Frontend changed, running tests..."
            cd packages/frontend && npm test
          else
            echo "Frontend not changed, skipping."
          fi

  # Strategy 3: Sparse checkout in CI
  backend-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          sparse-checkout: |
            packages/backend
            packages/shared
          sparse-checkout-cone-mode: true

  # Strategy 4: Cache .git between runs
  cached-checkout:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/cache@v3
        with:
          path: .git
          key: git-cache
      - run: git fetch  # incremental update instead of full clone
💡 Pro Tip: For GitHub Actions, use fetch-depth: 0 only when you need full history (e.g., for git blame or semantic-release). For everything else, fetch-depth: 1 is 10x faster. If you need to detect which files changed, use fetch-depth: 2 and run git diff HEAD~1 --name-only.
Google's monorepo has over 1 billion files and 35 million commits. They use a custom version control system (Piper) because standard Git cannot handle that scale. But for most organizations, Git's built-in optimization features (shallow clone, sparse checkout, partial clone) combined with monorepo tools like Nx or Turborepo provide excellent performance up to several hundred thousand files.

Lo kar liya — Key Points:

  • ✅ Monorepos store all projects in one repository but face performance issues with size: slow clone, slow status, slow CI
  • ✅ Shallow clone (--depth 1) downloads only the latest commit — fastest but no history, cannot push, cannot blame
  • ✅ Sparse checkout lets you check out only specific directories while retaining full commit history
  • ✅ Partial clone (--filter=blob:none) downloads commit and tree objects but fetches file content on demand
  • ✅ Combine sparse checkout with partial clone for maximum efficiency: minimal download, full history on demand
  • ✅ CI/CD optimization includes shallow clones, path-based triggers, caching, and sparse checkout
  • ✅ Choose clone strategy based on needs: shallow for CI, sparse for monorepo dev, blobless for large repo dev
Course Search
Search across all chapters & stages
📖

Search the course

Type any topic — branching, stash, rebase, hooks — and jump straight to that chapter.

merge branchesgit stashundo commitrebase