Chapter 10-5 — CI Clone Optimization | DevInHyderabad☕ ☕ 15 min read

CI Clone Optimization

Fast CI means fast feedback. Slow CI means developers wander off to make chai.

01The CI Clone Bottleneck

CI pipelines start with git clone, which downloads the entire repository history. For large repos, cloning can take 5-15 minutes — that is wasted CI time and compute cost.

Every minute of CI time is multiplied by the number of pipeline runs per day. 50 runs/day × 10 min clone = 8+ hours of wasted compute daily.

The clone step is often the slowest part of CI, slower than building or testing. And most CI jobs do not need the full repository history — they just need the code to build and test.

Optimizing the clone step is the highest-impact CI improvement for large repositories.

# The CI clone problem
# Typical CI pipeline times for a large monorepo:

# Step 1: git clone              → 8 minutes  ← BOTTLENECK!
# Step 2: npm install             → 3 minutes
# Step 3: build                   → 4 minutes
# Step 4: test                    → 5 minutes
# Total:                          → 20 minutes
# Clone is 40% of total time!

# Why is clone so slow?
# 1. Downloads ALL history (10+ years of commits)
# 2. Downloads ALL branches and tags
# 3. Downloads ALL blobs (every version of every file)
# 4. Network latency to remote server

# Most CI jobs only need:
# - The latest version of the files
# - On the current branch
# - With no history required for build/test

02Shallow Clone for CI: fetch-depth

GitHub Actions uses actions/checkout which defaults to a full clone. Set fetch-depth: 1 for a shallow clone — only the latest commit.

This reduces clone time from minutes to seconds. Shallow clone = only the latest commit, no history.

Limitations: git log will not show history, git blame will not work, some Git operations fail. But for most CI jobs (build, test, lint, deploy), shallow clone is perfect.

Use fetch-depth: 0 only when you specifically need full history (e.g., semantic-release).

# GitHub Actions: Shallow clone (FASTEST)
steps:
  - uses: actions/checkout@v4
    with:
      fetch-depth: 1  # Only latest commit
      # Clone time: 30 seconds instead of 8 minutes!

# GitLab CI: Shallow clone
variables:
  GIT_DEPTH: 1  # Only latest commit

# Bitbucket Pipelines
clone:
  depth: 1  # Shallow clone

# CircleCI
checkout:
  post:
    - git fetch --depth=1  # Ensure shallow

# When to use fetch-depth: 0 (full history)
# - semantic-release (needs to analyze commit history)
# - git blame in CI (needs full history)
# - changelog generation from commits
# - git bisect in CI

03Sparse Checkout in CI: Only What You Build

For monorepos, CI often only needs to build one package, not the entire repo. Sparse checkout lets you check out only specific directories.

Combine sparse checkout with partial clone for maximum efficiency. actions/checkout supports sparse checkout natively.

Only check out the directories needed for the current CI job. This reduces both clone time and disk usage.

# GitHub Actions: Sparse checkout for monorepo
jobs:
  frontend-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          sparse-checkout: |
            packages/frontend
            packages/shared
          sparse-checkout-cone-mode: true
      - run: cd packages/frontend && npm ci && npm test

  backend-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          sparse-checkout: |
            packages/backend
            packages/shared
      - run: cd packages/backend && npm ci && npm test

# Manual sparse checkout in any CI
steps:
  - run: git clone --filter=blob:none --sparse $REPO_URL .
  - run: git sparse-checkout init --cone
  - run: git sparse-checkout set packages/frontend

04Path-Based CI Triggers: Skip Unnecessary Jobs

The fastest CI job is the one that never runs. Path-based triggers skip CI jobs entirely when relevant files have not changed.

GitHub uses paths: filter in workflow triggers. GitLab uses rules:changes: in job configuration.

If only the frontend changed, skip backend tests. If only docs changed, skip all tests. This saves massive CI minutes and reduces feedback time for developers.

# GitHub Actions: Path-based triggers
name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  frontend:
    runs-on: ubuntu-latest
    paths:
      - 'packages/frontend/**'
      - 'packages/shared/**'
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2  # Need 2 for diff
      - run: |
          if git diff --name-only HEAD~1 HEAD | grep -q "^packages/frontend/"; then
            cd packages/frontend && npm ci && npm test
          fi

  backend:
    runs-on: ubuntu-latest
    paths:
      - 'packages/backend/**'
      - 'packages/shared/**'
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - run: |
          if git diff --name-only HEAD~1 HEAD | grep -q "^packages/backend/"; then
            cd packages/backend && npm ci && npm test
          fi

  # Docs never trigger tests
  # Only 'docs/**' paths → skip all test jobs

05Caching Strategies for CI

Caching persists data between CI runs, avoiding redundant downloads. Cache node_modules, .git directory, and build artifacts.

GitHub Actions uses actions/cache with key based on lockfile hash. Cache hit = instant restore. Cache miss = download and save for next run.

npm ci is faster than npm install in CI because it uses the lockfile directly.

💡 Pro Tip: Use npm ci instead of npm install in CI. It deletes node_modules and installs exactly from package-lock.json, making builds deterministic and 2-3x faster. Never run npm install in CI — it can resolve different versions than your lockfile.

# GitHub Actions: Caching strategy
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      # Shallow clone
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1

      # Cache node_modules
      - uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

      # Cache .git (for incremental fetches)
      - uses: actions/cache@v3
        with:
          path: .git
          key: git-${{ github.sha }}

      # Use npm ci (deterministic, fast)
      - run: npm ci
      - run: npm test

# GitLab CI: Caching
cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/
    - .git/

The CI optimization hierarchy: 1) Do not run the job (path-based triggers) — 100% time saved. 2) Do not download what you do not need (shallow clone, sparse checkout) — 50-90% clone time saved. 3) Cache what you download (node_modules, .git) — 30-70% install time saved. Apply in this order for maximum impact.

Lo kar liya — Key Points:

✅ The git clone step is often the slowest part of CI for large repositories, sometimes taking 5-15 minutes
✅ Shallow clone (fetch-depth: 1) downloads only the latest commit, reducing clone time by 80-90%
✅ Sparse checkout in CI only checks out the directories needed for the current job, saving clone and build time
✅ Path-based triggers skip CI jobs entirely when relevant files haven't changed — the fastest job is one that never runs
✅ npm ci is faster and more deterministic than npm install in CI — it installs exactly from the lockfile
✅ Caching (node_modules, .git) persists data between runs, avoiding redundant downloads
✅ Apply optimizations in order: skip jobs → shallow clone → sparse checkout → cache → npm ci

Previous10.4 — Long Running PR Conflicts 📋 All Chapters Next Chapter10.6 — Git Submodules → →