Here's what our manual process was actually costing us:

The wake-up call: We calculated that manual deployments cost us $12,000/month in developer time alone. Not counting the cost of downtime, bugs reaching production, or developer sanity.

Remember that Friday database migration disaster? Here's what actually happened:

# What I meant to run
ALTER TABLE users ADD COLUMN last_login TIMESTAMP;

# What I actually ran (missing semicolon caused next command to concatenate)
ALTER TABLE users ADD COLUMN last_login TIMESTAMP
DROP TABLE sessions;

Result: All user sessions deleted. Everyone logged out. 4 hours to restore from backup.

Root cause: Manual typing in production terminal. No review. No testing. No rollback plan.

That's when I started researching CI/CD.

We chose GitHub Actions because:

• Our code was already on GitHub
• Free for private repos (up to 2,000 minutes/month)
• YAML configuration was straightforward
• Marketplace had everything we needed
• No infrastructure to maintain

Don't try to build the perfect CI/CD pipeline on day one. Start with the bare minimum.

Our first GitHub Actions workflow was literally 15 lines:

name: CI

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dependencies
        run: npm install
      - name: Run tests
        run: npm test

That's it. No deployment. No fancy caching. Just automated tests on every push.

Impact: We caught 3 bugs in the first week that would have reached production.

Before thinking about deployment, nail your CI. Here's how we evolved our pipeline:

Week 1: Basic Testing

• Run unit tests
• Check if code compiles

Week 2: Code Quality

• Linting (ESLint)
• Code formatting checks (Prettier)
• Type checking (TypeScript)

Week 3: Advanced Testing

• Integration tests
• Test coverage reporting
• Parallel test execution

Week 4: Performance

• Build caching
• Docker layer caching
• Dependency caching

Result after 1 month: Our CI pipeline went from 8 minutes to 2.5 minutes. Every PR got automated feedback within 3 minutes.

Once CI was solid, we tackled CD (Continuous Deployment). We tried multiple strategies:

Blue-Green Deployment (Our Choice)

Run two identical production environments. Deploy to inactive one, swap traffic.

Pros:

• Zero-downtime deployments
• Instant rollback (just swap back)
• Test in production-identical environment

Cons:

• Doubles infrastructure cost
• Database migrations tricky
• Requires load balancer

Canary Deployment

Deploy to small subset of servers/users first, gradually roll out.

Pros:

• Minimal risk
• Real user testing
• Easy to abort

Cons:

• Complex traffic routing
• Longer deployment time
• Monitoring overhead

Rolling Deployment

Update servers one at a time while others handle traffic.

Pros:

• No extra infrastructure
• Gradual rollout
• Simple implementation

Cons:

• Mixed versions in production
• Longer deployment window
• Complex rollbacks

We use Blue-Green for our main app and Rolling for microservices.

name: Production Deploy

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

What it does: Runs on every push to main and every PR targeting main.

Why this matters: PRs get tested before merge. Main branch deploys automatically.

jobs:
  test-and-deploy:
    runs-on: ubuntu-latest

    env:
      NODE_ENV: production
      DATABASE_URL: ${{ secrets.DATABASE_URL }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

Key points:

• Secrets stored in GitHub Settings, never in code
• Node version pinned (no surprises from version changes)
• NPM cache enabled (saves 30 seconds per build)

- name: Install dependencies
  run: npm ci

- name: Run linting
  run: npm run lint

- name: Run tests
  run: npm run test:ci

- name: Generate coverage
  run: npm run test:coverage

- name: Upload coverage
  uses: codecov/codecov-action@v3

Why npm ci instead of npm install?

• Faster (no package.json resolution)
• Deterministic (installs exact versions from package-lock.json)
• Fails if lock file is out of sync

- name: Build application
  run: npm run build

- name: Run build size check
  run: |
    SIZE=$(du -sh dist | cut -f1)
    echo "Build size: $SIZE"
    # Fail if over 10MB
    if [ $(du -s dist | cut -f1) -gt 10240 ]; then
      echo "Build too large!"
      exit 1
    fi

Smart checks we added:

• Build size monitoring
• Bundle analysis
• Dead code detection
• Asset optimization verification

The rule: CI pipeline must complete in under 5 minutes for PRs.

How we achieved it:

• Parallel test execution (4 workers)
• Selective test running (only affected tests)
• Fail-fast strategy (stop on first failure)
• Cached dependencies and build artifacts

We used to have 1,200 tests. Half were garbage.

Bad tests we removed:

• Flaky tests (failed randomly)
• Slow tests (>5 seconds each)
• Redundant tests (testing same thing 3 different ways)
• Tests that tested implementation, not behavior

Result: Down to 995 tests. CI time dropped 40%. Confidence increased.

Flaky tests will kill your CI/CD adoption. Here's how we handled them:

Our policy:

1. First flaky failure: Create issue
2. Second flaky failure: Quarantine test (run separately)
3. Third flaky failure: Delete test or fix immediately

No exceptions. A flaky test that blocks deployments is worse than no test.

Goal: Make staging identical to production.

What we did:

• Same infrastructure (via Infrastructure as Code)
• Same environment variables (different values, same keys)
• Same deployment process
• Same monitoring and logging

What broke parity:

• Database size (staging has subset of prod data)
• Third-party integrations (use sandbox APIs)
• Traffic volume (can't replicate that)

Never, ever, EVER commit secrets.

Our secrets hierarchy:

GitHub Secrets (encrypted at rest)
     ↓
CI/CD Pipeline (uses secrets as env vars)
     ↓
Deployment Script (passes to infrastructure)
     ↓
Application (reads from environment)

Tools we use:

• GitHub Secrets for CI/CD variables
• AWS Secrets Manager for application secrets
• .env.example (template with no actual values)
• Git hooks to prevent secret commits

Different configs for different environments:

# config/production.js
module.exports = {
  database: {
    host: process.env.DB_HOST,
    pool: { min: 10, max: 50 }
  },
  cache: {
    enabled: true,
    ttl: 3600
  },
  logging: {
    level: 'error'
  }
}

# config/development.js
module.exports = {
  database: {
    host: 'localhost',
    pool: { min: 2, max: 10 }
  },
  cache: {
    enabled: false
  },
  logging: {
    level: 'debug'
  }
}

Key insight: Prod should be optimized for performance. Dev should be optimized for debugging.

Our blue-green setup using AWS:

[Load Balancer]
     ↓
┌────────────┬────────────┐
│  Blue      │  Green     │
│  (Active)  │  (Idle)    │
└────────────┴────────────┘

Deployment process:

1. Deploy new version to Green environment
2. Run smoke tests against Green
3. Gradually shift traffic (10%, 25%, 50%, 100%)
4. Monitor error rates at each step
5. If errors spike, instant rollback to Blue
6. If all good, Green becomes active, Blue becomes idle

Actual deployment script snippet:

# Deploy to green environment
./deploy-to-green.sh

# Health check
if ! curl -f https://green.myapp.com/health; then
  echo "Green deployment failed health check"
  exit 1
fi

# Shift 10% traffic
aws elbv2 modify-target-group-weights --target-group-weight blue=90,green=10

# Wait and monitor
sleep 300
check_error_rates

# Continue if no errors...

The problem: Blue and Green both hit the same database. How do you handle schema changes?

Our solution: Backward-compatible migrations

Phase 1: Add new column (backward compatible)

ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT false;

Deploy new code. Old code ignores new column. New code uses it.

Phase 2: Migrate data (background job)

UPDATE users SET email_verified = true WHERE email_confirmed_at IS NOT NULL;

Phase 3: Remove old column (next deployment)

ALTER TABLE users DROP COLUMN email_confirmed_at;

Rule: Every migration must work with both old and new code for at least one deployment cycle.

Metrics we track per deployment:

Deployment started: 2024-01-15 14:32:18 UTC
Deployment completed: 2024-01-15 14:37:42 UTC
Duration: 5m 24s

Health checks: ✓ Passed
Error rate: 0.02% (within threshold)
Response time p95: 245ms (baseline: 230ms)
Traffic: 1,245 req/min

Status: SUCCESS

Automated alerts:

• Error rate >1% for 2 minutes
• Response time >500ms p95
• Health check failures
• Deployment duration >10 minutes

Mistake: Tried to implement CI/CD, switch cloud providers, and migrate to microservices simultaneously.

Result: 3 months of chaos. Nothing worked.

Lesson: One thing at a time. CI first. Then CD. Then optimization.

Mistake: Tested code extensively. Never tested deployment script.

Result: Deployment script had a bug. Deleted production database. (We had backups, but still...)

Lesson: Deployment scripts ARE code. Test them.

Mistake: Let CI pipeline grow to 15 minutes. "It's fine, it's automated!"

Result: Developers stopped making small commits. Batched changes. More bugs.

Lesson: Fast feedback is critical. Keep CI under 5 minutes or developers will avoid it.

Mistake: Assumed deployments would always work.

Result: When deployment failed at 3 PM on Friday, we had no rollback plan.

Lesson: Every deployment needs a rollback plan. Test it monthly.

• Day 1-2: Choose CI/CD platform
• Day 3-4: Set up basic pipeline (test only)
• Day 5: Add code quality checks (linting, formatting)

• Day 8-9: Integrate test suite into CI
• Day 10-11: Add test coverage reporting
• Day 12: Optimize for speed (caching, parallelization)

• Day 15-16: Set up staging environment
• Day 17-18: Implement automated deployment to staging
• Day 19: Test rollback procedures

• Day 22-23: Deploy to production (manual trigger)
• Day 24-25: Add monitoring and alerts
• Day 26: Document everything
• Day 29-30: Enable auto-deployment, celebrate!