cloud-server/migrations/README.md

# Database Migrations

This directory contains SQL migration files for database schema changes.

## Automated Migration System

The project uses an **automated migration tracking system** that:
- ✅ Automatically detects and runs unapplied migrations
- ✅ Tracks migration history in the database
- ✅ Executes migrations in numerical order
- ✅ Safe to run multiple times (idempotent)
- ✅ Records execution time and errors

### Quick Start

```bash
# Check migration status
npm run db:migrate:status

# Run all pending migrations (local)
npm run db:migrate

# Run all pending migrations (remote)
npm run db:migrate:remote
```

### Migration Files

All migration files are named with a numeric prefix for ordering:
- `000_migration_history.sql` - Creates migration tracking table (always runs first)
- `002_add_composite_indexes.sql` - Query performance optimization
- `003_add_retail_pricing.sql` - Add retail pricing columns
- `004_anvil_tables.sql` - Anvil-branded product tables

## Migration Details

### 000_migration_history.sql
**Date**: 2026-01-25
**Purpose**: Create migration tracking system
- Creates `migration_history` table to track applied migrations
- Records execution time, success/failure status, and error messages

### 002_add_composite_indexes.sql
**Date**: 2026-01-21
**Purpose**: Add composite indexes to optimize query performance

**Indexes Added**:
1. `idx_instance_types_provider_family_specs` - Optimizes instance filtering by provider, family, and specs
2. `idx_pricing_instance_region_price` - Optimizes pricing queries with JOIN operations and sorting
3. `idx_regions_provider_code` - Optimizes region lookups by provider and region code

**Performance Impact**:
- Reduces query execution time for filtered instance searches
- Improves JOIN performance between instance_types, pricing, and regions tables
- Enables efficient ORDER BY on hourly_price without additional sort operations

### 003_add_retail_pricing.sql
**Date**: 2026-01-23
**Purpose**: Add retail pricing fields to all pricing tables
- Adds `hourly_price_retail` and `monthly_price_retail` columns
- Backfills existing data with 1.21x markup

### 004_anvil_tables.sql
**Date**: 2026-01-25
**Purpose**: Create Anvil-branded product tables
- `anvil_regions` - Anvil regional datacenters
- `anvil_instances` - Anvil instance specifications
- `anvil_pricing` - Anvil retail pricing with cost tracking
- `anvil_transfer_pricing` - Data transfer pricing

## Running Migrations

### Automated (Recommended)

```bash
# Check current status
npm run db:migrate:status              # Local database
npm run db:migrate:status:remote       # Remote database

# Run all pending migrations
npm run db:migrate                     # Local database
npm run db:migrate:remote              # Remote database
```

### Manual (Backward Compatibility)

Individual migration scripts are still available:

```bash
npm run db:migrate:002                 # Local
npm run db:migrate:002:remote          # Remote
npm run db:migrate:003                 # Local
npm run db:migrate:003:remote          # Remote
npm run db:migrate:004                 # Local
npm run db:migrate:004:remote          # Remote
```

## Migration Best Practices

1. **Idempotent Operations**: All migrations use `IF NOT EXISTS` to ensure safe re-execution
2. **Backwards Compatible**: New indexes don't break existing queries
3. **Performance Testing**: Test migration impact on query performance before deploying
4. **Rollback Plan**: Each migration includes rollback instructions in comments

## Query Optimization Details

### Main Query Pattern (query.ts)
```sql
SELECT ...
FROM instance_types it
JOIN providers p ON it.provider_id = p.id
JOIN pricing pr ON pr.instance_type_id = it.id
JOIN regions r ON pr.region_id = r.id
WHERE p.name = ?
  AND r.region_code = ?
  AND it.instance_family = ?
  AND it.vcpu >= ?
  AND it.memory_mb >= ?
ORDER BY pr.hourly_price
```

**Optimized By**:
- `idx_instance_types_provider_family_specs` - Covers WHERE conditions on instance_types
- `idx_pricing_instance_region_price` - Covers JOIN and ORDER BY on pricing
- `idx_regions_provider_code` - Covers JOIN conditions on regions

### Expected Performance Improvement
- **Before**: Full table scans on instance_types and pricing tables
- **After**: Index seeks with reduced disk I/O
- **Estimated Speedup**: 3-10x for filtered queries with sorting

## Verifying Index Usage

You can verify that indexes are being used with SQLite EXPLAIN QUERY PLAN:

```bash
# Check query execution plan
npm run db:query "EXPLAIN QUERY PLAN SELECT ... FROM instance_types it JOIN ..."
```

Look for "USING INDEX" in the output to confirm index usage.

## Notes

- SQLite automatically chooses the most efficient index for each query
- Composite indexes follow the "leftmost prefix" rule
- Indexes add minimal storage overhead but significantly improve read performance
- Write operations (INSERT/UPDATE) are slightly slower with more indexes, but read performance gains outweigh this for read-heavy workloads