Security Incident Response

Emergency procedures for when API keys leak - containment, cleanup and recovery

Last updated: 8/15/2025

When API keys leak, speed matters. This guide provides step-by-step procedures to contain damage and recover quickly.

Emergency Response Checklist

Phase 1: Immediate Containment (0-15 minutes)

🚨 Stop the bleeding first, investigate later

Identify the exposed key type
- Anon key: Limited damage if RLS is properly configured
- Service role key: Full database access - highest priority
Rotate the key immediately
- Supabase Dashboard → Settings → API → Generate new key
- Copy new key to secure location (password manager)
Remove from CI/CD systems
- GitHub Actions: Settings → Secrets → Delete/update
- Vercel: Project Settings → Environment Variables
- Other platforms: Update environment variables
Emergency deployment (if needed)
- Deploy with new keys or blank values to stop active usage
- Better to have a broken app than a compromised database

Phase 2: Git History Cleanup (15-60 minutes)

Clean your repository before attackers download it

Quick Detection

# Find commits containing the leaked key
git log --grep="eyJhbGciOiJIUzI1NiIs" --oneline
git log -S "your-leaked-key" --oneline

# Check current files
grep -r "your-leaked-key" . --exclude-dir=.git

Remove from History

Option A: BFG Repo-Cleaner (Recommended)

# Install BFG
brew install bfg

# Clone a fresh copy of your repo
git clone --mirror https://github.com/username/repo.git

# Remove the secret
bfg --replace-text passwords.txt repo.git
# where passwords.txt contains: your-leaked-key

# Clean up and force push
cd repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

Option B: git filter-repo

# Install git-filter-repo
pip install git-filter-repo

# Remove secret from all history
git filter-repo --replace-text <(echo "your-leaked-key==>REDACTED")

# Force push
git push --force --all

Update Team

# Notify all team members
echo "🚨 SECURITY INCIDENT: Force-pushed clean history. 
Please delete your local repo and re-clone:
rm -rf project-name
git clone https://github.com/username/repo.git"

Phase 3: Damage Assessment (Same hour)

Check what the attacker might have accessed

Supabase Logs Analysis

Dashboard → Logs → API
- Look for unusual traffic patterns during exposure window
- Check for bulk data exports or unexpected queries
- Note any unfamiliar IP addresses
Database Activity
- Check for new users created
- Look for modified data or deleted records
- Review any admin operations

Log Analysis Script

-- Check for suspicious recent activity
SELECT 
  created_at,
  event_type,
  ip_address,
  user_agent,
  COUNT(*) as request_count
FROM auth.audit_log_entries 
WHERE created_at >= '2025-01-15 10:00:00'  -- Adjust to exposure start time
GROUP BY created_at, event_type, ip_address, user_agent
ORDER BY created_at DESC;

-- Check for bulk operations
SELECT 
  table_name,
  operation,
  COUNT(*) as operation_count,
  MIN(created_at) as first_operation,
  MAX(created_at) as last_operation
FROM audit_logs 
WHERE created_at >= '2025-01-15 10:00:00'
GROUP BY table_name, operation
HAVING COUNT(*) > 100  -- Adjust threshold
ORDER BY operation_count DESC;

Phase 4: Recovery (Same day)

Update all systems with new credentials

Environment Updates

# Local development
cp .env.example .env.local
# Add new keys to .env.local

# Production deployment
# Update environment variables in hosting platform
# Redeploy application

# CI/CD systems
# Update secrets in GitHub Actions / GitLab CI / etc.

Verification Script

// scripts/verify-rotation.js
const testConnections = async () => {
  const environments = ['development', 'staging', 'production'];
  
  for (const env of environments) {
    try {
      const { createClient } = require('@supabase/supabase-js');
      const client = createClient(
        process.env[`${env.toUpperCase()}_SUPABASE_URL`],
        process.env[`${env.toUpperCase()}_SUPABASE_ANON_KEY`]
      );
      
      const { data, error } = await client
        .from('test_table')
        .select('count')
        .limit(1);
        
      console.log(`✅ ${env}: Connection successful`);
    } catch (error) {
      console.log(`❌ ${env}: Connection failed - ${error.message}`);
    }
  }
};

testConnections();

Advanced Response Procedures

Temporary Access Restriction

If you suspect active exploitation, temporarily lock down access:

-- Create emergency lockdown policies
ALTER TABLE sensitive_table ENABLE ROW LEVEL SECURITY;

-- Temporarily deny all access
CREATE POLICY "emergency_lockdown" ON sensitive_table
FOR ALL USING (false) WITH CHECK (false);

-- Allow only specific admin access
CREATE POLICY "admin_only" ON sensitive_table  
FOR ALL USING (
  auth.jwt() ->> 'email' = 'admin@yourcompany.com'
);

Data Integrity Verification

-- Check for unauthorized modifications
SELECT 
  table_name,
  COUNT(*) as total_records,
  MAX(updated_at) as last_modified,
  COUNT(CASE WHEN created_at > '2025-01-15 10:00:00' THEN 1 END) as new_records
FROM information_schema.tables t
JOIN your_audit_table a ON a.table_name = t.table_name
WHERE t.table_schema = 'public'
GROUP BY table_name
ORDER BY last_modified DESC;

User Account Audit

-- Check for suspicious user accounts
SELECT 
  id,
  email,
  created_at,
  email_confirmed_at,
  last_sign_in_at,
  sign_in_count
FROM auth.users 
WHERE created_at >= '2025-01-15 10:00:00'  -- Exposure window
ORDER BY created_at DESC;

-- Check for privilege escalations
SELECT 
  user_id,
  role,
  created_at
FROM user_roles 
WHERE created_at >= '2025-01-15 10:00:00'
AND role IN ('admin', 'owner');

Communication Templates

Internal Team Alert

🚨 SECURITY INCIDENT - ACTION REQUIRED

**What happened:** Supabase service role key exposed in commit abc123

**Immediate actions taken:**
- ✅ Key rotated in dashboard
- ✅ Removed from CI/CD systems  
- ✅ Emergency deployment with new keys

**Actions required from you:**
1. Delete your local repo: `rm -rf project-name`
2. Re-clone fresh copy: `git clone https://github.com/username/repo.git`
3. Get new .env.local from team lead
4. Verify your local development works

**Timeline:**
- 10:30 AM: Key exposed in commit
- 10:45 AM: Incident detected
- 10:50 AM: Key rotated
- 11:15 AM: Clean history force-pushed

**Next steps:**
- [ ] Complete damage assessment
- [ ] Review prevention measures
- [ ] Schedule post-incident review

Customer Communication (if needed)

**Security Update - No Action Required**

We recently identified and resolved a security issue that could have potentially affected some user data. We want to be transparent about what happened and what we've done to address it.

**What happened:**
A database access key was briefly exposed in our development tools.

**What we did:**
- Immediately revoked the exposed credentials
- Conducted a thorough security audit
- Found no evidence of unauthorized access to user data
- Implemented additional safeguards

**What this means for you:**
No action is required on your part. Your account remains secure and no user data was compromised.

**Our commitment:**
We take security seriously and are continuously improving our practices to protect your data.

If you have any questions, please contact security@yourcompany.com

Post-Incident Review

Root Cause Analysis

## Incident Post-Mortem

**Date:** 2025-01-15
**Duration:** 45 minutes (detection to full resolution)
**Severity:** High (service role key exposure)

### Timeline
- 10:30 AM: Developer committed .env file containing service role key
- 10:45 AM: GitHub secret scanning detected and alerted
- 10:50 AM: Key rotated in Supabase dashboard
- 11:00 AM: Git history cleaned and force-pushed
- 11:15 AM: All environments updated with new keys

### Root Causes
1. **Immediate:** .env file accidentally committed
2. **Contributing:** Pre-commit hooks not installed on developer machine
3. **Systemic:** Insufficient onboarding checklist for new developers

### What Went Well
- GitHub secret scanning caught the leak quickly
- Response team followed procedures correctly
- No evidence of data compromise found
- Communication was clear and timely

### What Could Be Improved
- Detection could be faster (15-minute delay)
- Pre-commit hooks should be mandatory
- Need better developer training on git practices

### Action Items
- [ ] Make pre-commit hooks mandatory in setup script
- [ ] Add security training to developer onboarding
- [ ] Implement daily secret scanning of all repositories
- [ ] Create automated incident response playbooks

Prevention Improvements

Enhanced Detection

# .github/workflows/security-scan.yml
name: Security Scan
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      
      - name: Run Gitleaks
        uses: zricethezav/gitleaks-action@v2
        with:
          config-path: .gitleaks.toml

Mandatory Security Setup

#!/bin/bash
# scripts/setup-security.sh

echo "🔒 Setting up security tools..."

# Install gitleaks
if ! command -v gitleaks &> /dev/null; then
    echo "Installing gitleaks..."
    brew install gitleaks
fi

# Setup pre-commit hooks
echo "Setting up pre-commit hooks..."
npx husky install
npx husky add .husky/pre-commit "gitleaks protect --staged"

# Verify .gitignore
if ! grep -q "\.env" .gitignore; then
    echo "Adding .env patterns to .gitignore..."
    echo -e "\n# Environment files\n.env*\n*.env" >> .gitignore
fi

# Create env template if missing
if [ ! -f .env.example ]; then
    echo "Creating .env.example template..."
    cat > .env.example << EOF
# Copy to .env.local and fill in real values
NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_anon_key_here
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key_here
EOF
fi

echo "✅ Security setup complete"
echo "Remember to copy .env.example to .env.local and add real values"

Emergency Contact List

Incident Response Team

## Emergency Contacts

### Primary Response Team
- **Security Lead:** security@company.com / +1-555-0123
- **DevOps Lead:** devops@company.com / +1-555-0124  
- **CTO:** cto@company.com / +1-555-0125

### External Contacts
- **Supabase Support:** Supabase Support
- **GitHub Security:** GitHub Security
- **Legal Counsel:** legal@company.com

### Communication Channels
- **Incident Channel:** #security-incidents (Slack)
- **Status Page:** Company Status Page
- **Customer Support:** support@company.com

Key Rotation Automation

Scheduled Rotation Script

// scripts/rotate-keys.js
const rotateSupabaseKeys = async () => {
  console.log('🔄 Starting key rotation...');
  
  // 1. Generate new keys in Supabase (manual step for now)
  console.log('1. Generate new keys in Supabase dashboard');
  
  // 2. Update CI/CD systems
  console.log('2. Update CI/CD environment variables');
  
  // 3. Deploy to all environments
  console.log('3. Deploy with new keys');
  
  // 4. Verify connections
  console.log('4. Verify all services connect successfully');
  
  // 5. Deactivate old keys
  console.log('5. Deactivate old keys in dashboard');
  
  console.log('✅ Key rotation complete');
};

// Run quarterly
rotateSupabaseKeys();

Rotation Tracking

-- Track key rotations
CREATE TABLE key_rotations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  key_type TEXT NOT NULL,
  rotated_at TIMESTAMPTZ DEFAULT NOW(),
  reason TEXT,
  rotated_by TEXT,
  old_key_prefix TEXT, -- First 8 chars for tracking
  new_key_prefix TEXT
);

-- Log rotation
INSERT INTO key_rotations (key_type, reason, rotated_by, old_key_prefix, new_key_prefix)
VALUES ('service_role', 'security_incident', 'security-team', 'eyJhbGci', 'eyJhbGci');

Legal and Compliance

Breach Notification Requirements

## Data Breach Assessment

### Questions to Answer
1. **Was personal data accessed?** 
   - Check logs for PII access during exposure window
   
2. **How many individuals affected?**
   - Count unique users in accessed data
   
3. **What type of data was exposed?**
   - Names, emails, payment info, etc.
   
4. **Legal obligations:**
   - GDPR: 72-hour notification requirement
   - CCPA: Reasonable security requirements
   - Industry-specific regulations

### Documentation Required
- [ ] Timeline of events
- [ ] Technical details of exposure
- [ ] Data types and volumes affected
- [ ] Remediation steps taken
- [ ] Impact assessment

Recovery Verification

System Health Check

#!/bin/bash
# scripts/post-incident-verify.sh

echo "🔍 Post-incident verification..."

# Test database connections
echo "Testing database connections..."
npm run test:db-connection

# Verify RLS policies
echo "Checking RLS policies..."
npm run test:rls-policies

# Check application functionality
echo "Testing critical user flows..."
npm run test:e2e:critical

# Verify monitoring systems
echo "Checking monitoring and alerts..."
curl -f https://your-monitoring-endpoint/health

# Test backup systems
echo "Verifying backup integrity..."
npm run test:backup-restore

echo "✅ Post-incident verification complete"

User Communication Follow-up

## Follow-up Communication (1 week later)

**Security Update Follow-up**

Last week, we informed you about a security issue we quickly identified and resolved. We wanted to provide an update on our investigation and the steps we've taken.

**Investigation Results:**
- No user data was accessed or compromised
- All systems remained secure throughout the incident
- Response time was under 1 hour from detection to resolution

**Additional Security Measures:**
- Enhanced monitoring systems
- Improved development security tools
- Additional team security training

**Your Account:**
No action is required. Your account and data remain fully secure.

Thank you for your trust in our platform.

Security incidents are stressful, but having a clear response plan makes all the difference. Practice these procedures during non-emergency times so your team can execute them smoothly when it matters most.

Learn