refactor: comprehensive code review fixes (security, performance, QA)
## Security Improvements - Fix timing attack in verifyApiKey with fixed 256-byte buffer - Fix sortOrder SQL injection with whitelist validation - Fix rate limiting bypass for non-Cloudflare traffic (fail-closed) - Remove stack trace exposure in error responses - Add request_id for audit trail (X-Request-ID header) - Sanitize origin header to prevent log injection - Add content-length validation for /sync endpoint (10KB limit) - Replace Math.random() with crypto.randomUUID() for sync IDs - Expand sensitive data masking patterns (8 → 18) ## Performance Improvements - Reduce rate limiter KV reads from 3 to 1 per request (66% reduction) - Increase sync batch size from 100 to 500 (80% fewer batches) - Fix health check N+1 query with efficient JOINs - Fix COUNT(*) Cartesian product with COUNT(DISTINCT) - Implement shared logger cache pattern across repositories - Add CacheService singleton pattern in recommend.ts - Add composite index for recommendation queries - Implement Anvil pricing query batching (100 per chunk) ## QA Improvements - Add BATCH_SIZE bounds validation (1-1000) - Add pagination bounds (page >= 1, MAX_OFFSET = 100000) - Add min/max range consistency validation - Add DB reference validation for singleton services - Add type guards for database result validation - Add timeout mechanism for external API calls (10-60s) - Use SUPPORTED_PROVIDERS constant instead of hardcoded list ## Removed - Remove Vault integration (using Wrangler secrets) - Remove 6-hour pricing cron (daily sync only) ## Configuration - Add idx_instance_types_specs_filter composite index - Add CORS Access-Control-Expose-Headers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
kappa
@@ -2,249 +2,178 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Implemented complete database schema and repositories for Anvil-branded cloud products, including regions, instances, pricing, and transfer pricing.
|
||||
Anvil-branded cloud products with automated pricing sync from source providers (Linode, Vultr).
|
||||
|
||||
## Implementation Summary
|
||||
**Key Features:**
|
||||
- USD retail pricing only (simplified schema)
|
||||
- Automatic pricing sync: wholesale × 1.21 = retail
|
||||
- Source mapping via `source_instance_id` and `source_region_id`
|
||||
|
||||
### 1. Database Migration (004_anvil_tables.sql)
|
||||
## Database Schema
|
||||
|
||||
Created 4 new tables with proper indexes, triggers, and foreign key constraints:
|
||||
|
||||
#### anvil_regions
|
||||
- Maps Anvil-branded regions to source provider regions
|
||||
- Contains 6 initial regions (Tokyo 1-3, Osaka 1-2, Seoul 1)
|
||||
- Links to existing `regions` table via `source_region_id`
|
||||
### anvil_regions
|
||||
Maps Anvil regions to source provider regions.
|
||||
|
||||
**Columns:**
|
||||
- `id`, `name` (unique), `display_name`, `country_code`
|
||||
- `source_provider`, `source_region_code`, `source_region_id` (FK)
|
||||
- `source_provider` (linode, vultr)
|
||||
- `source_region_code`, `source_region_id` (FK to `regions.id`)
|
||||
- `active`, `created_at`, `updated_at`
|
||||
|
||||
**Initial Data:**
|
||||
```
|
||||
anvil-tyo1 → Linode ap-northeast (region_id: 26)
|
||||
anvil-tyo2 → Linode jp-tyo-3 (region_id: 3)
|
||||
anvil-tyo3 → Vultr nrt (region_id: 323)
|
||||
anvil-osa1 → Linode jp-osa (region_id: 13)
|
||||
anvil-osa2 → Vultr itm (region_id: 314)
|
||||
anvil-sel1 → Vultr icn (region_id: 313)
|
||||
```
|
||||
**Region Mappings:**
|
||||
| Anvil Region | Source Provider | Source Region |
|
||||
|-------------|-----------------|---------------|
|
||||
| anvil-tyo1 | Linode | ap-northeast |
|
||||
| anvil-tyo2 | Linode | jp-tyo-3 |
|
||||
| anvil-tyo3 | Vultr | nrt |
|
||||
| anvil-osa1 | Linode | jp-osa |
|
||||
| anvil-osa2 | Vultr | itm |
|
||||
| anvil-sel1 | Vultr | icn |
|
||||
|
||||
#### anvil_instances
|
||||
- Defines Anvil product specifications
|
||||
- Supports categories: vm, gpu, g8, vpu
|
||||
- GPU-specific fields: `gpu_model`, `gpu_vram_gb`
|
||||
### anvil_instances
|
||||
Anvil product specifications.
|
||||
|
||||
**Columns:**
|
||||
- `id`, `name` (unique), `display_name`, `category`
|
||||
- `id`, `name` (unique), `display_name`, `category` (vm, gpu, g8, vpu)
|
||||
- `vcpus`, `memory_gb`, `disk_gb`
|
||||
- `transfer_tb`, `network_gbps`
|
||||
- `gpu_model`, `gpu_vram_gb` (nullable)
|
||||
- `gpu_model`, `gpu_vram_gb` (nullable, for GPU instances)
|
||||
- `active`, `created_at`, `updated_at`
|
||||
|
||||
#### anvil_pricing
|
||||
- Retail pricing with cost tracking
|
||||
- Auto-calculates KRW prices using exchange rates
|
||||
- Links to both Anvil instances/regions and source instances
|
||||
### anvil_pricing
|
||||
Retail pricing in USD. Auto-synced from source provider pricing.
|
||||
|
||||
**Columns:**
|
||||
- `id`, `anvil_instance_id` (FK), `anvil_region_id` (FK)
|
||||
- `hourly_price`, `monthly_price` (retail USD)
|
||||
- `hourly_price_krw`, `monthly_price_krw` (retail KRW)
|
||||
- `cost_hourly`, `cost_monthly` (wholesale USD)
|
||||
- `source_instance_id` (optional FK to source tables)
|
||||
- `source_instance_id` (FK to `instance_types.id`)
|
||||
- `active`, `created_at`, `updated_at`
|
||||
- UNIQUE constraint on (anvil_instance_id, anvil_region_id)
|
||||
|
||||
#### anvil_transfer_pricing
|
||||
- Data transfer pricing per region
|
||||
- KRW conversion support
|
||||
### anvil_transfer_pricing
|
||||
Data transfer pricing per region.
|
||||
|
||||
**Columns:**
|
||||
- `id`, `anvil_region_id` (FK)
|
||||
- `price_per_gb` (USD), `price_per_gb_krw` (KRW)
|
||||
- `price_per_gb` (USD)
|
||||
- `included_tb` (reference only)
|
||||
- `active`, `created_at`, `updated_at`
|
||||
- UNIQUE constraint on anvil_region_id
|
||||
|
||||
### 2. TypeScript Types (src/types.ts)
|
||||
## Pricing Sync Flow
|
||||
|
||||
Added 8 new type definitions:
|
||||
Provider sync automatically updates Anvil pricing via `syncAnvilPricing()`.
|
||||
|
||||
### Flow Diagram
|
||||
```
|
||||
Provider API → pricing table (wholesale USD) → anvil_pricing (retail USD)
|
||||
↓
|
||||
syncProvider()
|
||||
↓
|
||||
syncAnvilPricing()
|
||||
↓
|
||||
retail = wholesale × 1.21
|
||||
```
|
||||
|
||||
### Retail Markup Calculation
|
||||
```typescript
|
||||
AnvilRegion
|
||||
AnvilInstance
|
||||
AnvilPricing
|
||||
AnvilTransferPricing
|
||||
AnvilRegionInput
|
||||
AnvilInstanceInput
|
||||
AnvilPricingInput
|
||||
AnvilTransferPricingInput
|
||||
// src/constants.ts
|
||||
export const USD_RETAIL_DEFAULTS = {
|
||||
MARGIN_MULTIPLIER: 1.1, // 10% margin
|
||||
VAT_MULTIPLIER: 1.1, // 10% VAT
|
||||
TOTAL_MULTIPLIER: 1.21, // Combined
|
||||
};
|
||||
|
||||
// Hourly: rounded to 4 decimal places
|
||||
calculateRetailHourly(0.0075) // → 0.0091
|
||||
|
||||
// Monthly: rounded to nearest $1
|
||||
calculateRetailMonthly(5) // → 6
|
||||
```
|
||||
|
||||
All Input types auto-derive from their base types, excluding `id`, `created_at`, and `updated_at`.
|
||||
### Source Mapping
|
||||
`anvil_pricing` links to source data via:
|
||||
- `source_instance_id` → `instance_types.id`
|
||||
- `anvil_region_id` → `anvil_regions.source_region_id` → `regions.id`
|
||||
|
||||
### 3. Repositories
|
||||
When sync runs, it:
|
||||
1. Finds anvil_pricing records with `source_instance_id` set
|
||||
2. Looks up wholesale price from `pricing` table using source_instance_id + source_region_id
|
||||
3. Calculates retail price (×1.21)
|
||||
4. Updates anvil_pricing with new retail prices
|
||||
|
||||
Created 4 repository classes following existing BaseRepository pattern:
|
||||
## API Authentication
|
||||
|
||||
#### AnvilRegionsRepository
|
||||
- `findByName(name: string)` - Find by Anvil region name
|
||||
- `findByCountry(countryCode: string)` - Get all regions in a country
|
||||
- `findBySourceRegion(sourceRegionId: number)` - Reverse lookup
|
||||
- `findActive()` - Get all active regions
|
||||
- `updateActive(id: number, active: boolean)` - Toggle active status
|
||||
- `upsertMany(regions: AnvilRegionInput[])` - Bulk insert/update
|
||||
API key stored in Vault:
|
||||
```
|
||||
Path: secret/data/cloud-instances-api
|
||||
Key: api_key
|
||||
```
|
||||
|
||||
#### AnvilInstancesRepository
|
||||
- `findByName(name: string)` - Find by instance name
|
||||
- `findByCategory(category)` - Filter by vm/gpu/g8/vpu
|
||||
- `findActive(category?)` - Get active instances with optional category filter
|
||||
- `searchByResources(minVcpus?, minMemoryGb?, minDiskGb?, category?)` - Resource-based search
|
||||
- `updateActive(id: number, active: boolean)` - Toggle active status
|
||||
- `upsertMany(instances: AnvilInstanceInput[])` - Bulk insert/update
|
||||
Set wrangler secret:
|
||||
```bash
|
||||
wrangler secret put API_KEY
|
||||
```
|
||||
|
||||
#### AnvilPricingRepository
|
||||
- `findByInstance(anvilInstanceId: number)` - All pricing for an instance
|
||||
- `findByRegion(anvilRegionId: number)` - All pricing in a region
|
||||
- `findByInstanceAndRegion(instanceId, regionId)` - Specific pricing record
|
||||
- `findActive(instanceId?, regionId?)` - Active pricing with optional filters
|
||||
- `searchByPriceRange(minHourly?, maxHourly?, minMonthly?, maxMonthly?)` - Price range search
|
||||
- `updateActive(id: number, active: boolean)` - Toggle active status
|
||||
- `upsertMany(pricing: AnvilPricingInput[])` - Bulk insert/update with auto KRW calculation
|
||||
|
||||
#### AnvilTransferPricingRepository
|
||||
- `findByRegion(anvilRegionId: number)` - Get transfer pricing for a region
|
||||
- `findActive()` - Get all active transfer pricing
|
||||
- `updateActive(id: number, active: boolean)` - Toggle active status
|
||||
- `upsertMany(pricing: AnvilTransferPricingInput[])` - Bulk insert/update with KRW conversion
|
||||
|
||||
### 4. Repository Factory Updates
|
||||
|
||||
Updated `RepositoryFactory` class to include:
|
||||
- `anvilRegions: AnvilRegionsRepository`
|
||||
- `anvilInstances: AnvilInstancesRepository`
|
||||
- `anvilPricing: AnvilPricingRepository`
|
||||
- `anvilTransferPricing: AnvilTransferPricingRepository`
|
||||
|
||||
All repositories use lazy singleton pattern for efficient caching.
|
||||
|
||||
### 5. KRW Pricing Support
|
||||
|
||||
Pricing repositories automatically calculate KRW prices using environment variables:
|
||||
- `KRW_EXCHANGE_RATE` - Base USD to KRW conversion rate (default: 1450)
|
||||
- `KRW_VAT_RATE` - VAT multiplier (default: 1.1 for 10% VAT)
|
||||
- `KRW_MARKUP_RATE` - Markup multiplier (default: 1.1 for 10% markup)
|
||||
|
||||
KRW prices are auto-calculated during `upsertMany()` operations.
|
||||
|
||||
## Database Schema
|
||||
## Database Relationships
|
||||
|
||||
```
|
||||
anvil_regions (1) ──< anvil_pricing (M)
|
||||
anvil_instances (1) ──< anvil_pricing (M)
|
||||
anvil_regions (1) ──< anvil_transfer_pricing (1)
|
||||
regions (1) ──< anvil_regions (M) [source mapping]
|
||||
providers (1) ──< instance_types (M) ──< pricing (M)
|
||||
↑ ↑
|
||||
source_instance_id source_region_id
|
||||
↓ ↓
|
||||
anvil_instances (1) ──< anvil_pricing (M) >── anvil_regions (1)
|
||||
```
|
||||
|
||||
## Deployment Status
|
||||
|
||||
✅ Migration 004 applied to remote database (2026-01-25)
|
||||
✅ 17 queries executed successfully
|
||||
✅ 6 initial regions inserted
|
||||
✅ All 4 tables created with indexes and triggers
|
||||
✅ TypeScript compilation successful
|
||||
✅ Deployed to production (Version: 5905e5df-7265-4872-b05e-e3fe4b7d0619)
|
||||
✅ Migration 004 applied to remote database
|
||||
✅ 6 Anvil regions configured
|
||||
✅ Source instance mappings for tyo1, tyo2, tyo3, osa1, osa2, sel1
|
||||
✅ Auto-sync working: retail = wholesale × 1.21
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Query Anvil Regions
|
||||
### Trigger Sync
|
||||
```bash
|
||||
curl -X POST "https://cloud-instances-api.kappa-d8e.workers.dev/sync" \
|
||||
-H "X-API-Key: YOUR_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"providers": ["linode", "vultr"]}'
|
||||
```
|
||||
|
||||
### Query Anvil Pricing
|
||||
```typescript
|
||||
const repos = new RepositoryFactory(env.DB, env);
|
||||
|
||||
// Get all Japan regions
|
||||
const jpRegions = await repos.anvilRegions.findByCountry('JP');
|
||||
// Returns: anvil-tyo1, anvil-tyo2, anvil-tyo3, anvil-osa1, anvil-osa2
|
||||
// Get pricing for a region
|
||||
const pricing = await repos.anvilPricing.findByRegion(regionId);
|
||||
|
||||
// Find specific region
|
||||
const tyo1 = await repos.anvilRegions.findByName('anvil-tyo1');
|
||||
// Get pricing for an instance
|
||||
const instancePricing = await repos.anvilPricing.findByInstance(instanceId);
|
||||
```
|
||||
|
||||
### Create Anvil Instance
|
||||
```typescript
|
||||
await repos.anvilInstances.create({
|
||||
name: 'anvil-1g-1c',
|
||||
display_name: 'Basic 1GB',
|
||||
category: 'vm',
|
||||
vcpus: 1,
|
||||
memory_gb: 1,
|
||||
disk_gb: 25,
|
||||
transfer_tb: 1,
|
||||
network_gbps: 1,
|
||||
gpu_model: null,
|
||||
gpu_vram_gb: null,
|
||||
active: 1,
|
||||
});
|
||||
### Add Source Mapping
|
||||
```sql
|
||||
-- Map anvil_pricing to source instance
|
||||
UPDATE anvil_pricing
|
||||
SET source_instance_id = (
|
||||
SELECT id FROM instance_types
|
||||
WHERE instance_id = 'vc2-1c-1gb' AND provider_id = 2
|
||||
)
|
||||
WHERE anvil_instance_id = 1 AND anvil_region_id = 3;
|
||||
```
|
||||
|
||||
### Set Pricing (Auto KRW Calculation)
|
||||
```typescript
|
||||
await repos.anvilPricing.upsertMany([
|
||||
{
|
||||
anvil_instance_id: 1,
|
||||
anvil_region_id: 1,
|
||||
hourly_price: 0.015,
|
||||
monthly_price: 10,
|
||||
cost_hourly: 0.012,
|
||||
cost_monthly: 8,
|
||||
source_instance_id: 123,
|
||||
active: 1,
|
||||
}
|
||||
]);
|
||||
// KRW prices auto-calculated:
|
||||
// hourly_price_krw = 0.015 × 1450 × 1.1 × 1.1 ≈ 26.37
|
||||
// monthly_price_krw = 10 × 1450 × 1.1 × 1.1 ≈ 17,545
|
||||
```
|
||||
## Files
|
||||
|
||||
### Search by Resources
|
||||
```typescript
|
||||
// Find instances with at least 2 vCPUs and 4GB RAM
|
||||
const instances = await repos.anvilInstances.searchByResources(
|
||||
2, // minVcpus
|
||||
4, // minMemoryGb
|
||||
null, // minDiskGb
|
||||
'vm' // category
|
||||
);
|
||||
```
|
||||
### Repositories
|
||||
- `/src/repositories/anvil-regions.ts`
|
||||
- `/src/repositories/anvil-instances.ts`
|
||||
- `/src/repositories/anvil-pricing.ts`
|
||||
- `/src/repositories/anvil-transfer-pricing.ts`
|
||||
|
||||
## Files Changed
|
||||
### Sync Service
|
||||
- `/src/services/sync.ts` - Contains `syncAnvilPricing()` method
|
||||
|
||||
### Created
|
||||
- `/migrations/004_anvil_tables.sql` - Database schema
|
||||
- `/src/repositories/anvil-regions.ts` - Regions repository
|
||||
- `/src/repositories/anvil-instances.ts` - Instances repository
|
||||
- `/src/repositories/anvil-pricing.ts` - Pricing repository
|
||||
- `/src/repositories/anvil-transfer-pricing.ts` - Transfer pricing repository
|
||||
- `/src/repositories/anvil-regions.test.ts` - Unit tests
|
||||
|
||||
### Modified
|
||||
- `/src/types.ts` - Added 8 new types
|
||||
- `/src/repositories/index.ts` - Added 4 new repository exports and factory getters
|
||||
|
||||
## Next Steps
|
||||
|
||||
To complete the Anvil product implementation:
|
||||
|
||||
1. **Populate Instance Data**: Create Anvil instance definitions (VM, GPU, G8, VPU tiers)
|
||||
2. **Set Pricing**: Map Anvil instances to source providers and set retail pricing
|
||||
3. **API Endpoints**: Create endpoints for querying Anvil products
|
||||
4. **Sync Service**: Implement sync logic to update costs from source providers
|
||||
5. **Documentation**: Add API documentation for Anvil endpoints
|
||||
|
||||
## Testing
|
||||
|
||||
Basic repository tests created in `anvil-regions.test.ts`. Additional test coverage recommended for:
|
||||
- Instance search and filtering
|
||||
- Pricing calculations and KRW conversion
|
||||
- Transfer pricing queries
|
||||
- Edge cases and error handling
|
||||
### Constants
|
||||
- `/src/constants.ts` - `USD_RETAIL_DEFAULTS`, `calculateRetailHourly()`, `calculateRetailMonthly()`
|
||||
|
||||
Reference in New Issue
Block a user