CI/CD Pipeline Guide

Jubiloop uses GitHub Actions for automated continuous integration and deployment workflows.

Overview

All deployment configuration is centralized in infra/deploy/env.deploy.yml. This single file defines:

• Infrastructure settings (Terraform variables, backend config)
• Environment variables for all services
• Secret references (auto-discovered by scripts)
• Variable references (non-sensitive config)

Configuration Generation Scripts

The deployment system is powered by two main scripts that dynamically process env.deploy.yml.

generate-env-files.js

Generates environment configuration for all services.

Usage:

# Generate .env files for server services (default)
node scripts/generate-env-files.js DEV

# Generate shell exports for web app
node scripts/generate-env-files.js DEV --web-app

# Generate shell exports for marketing
node scripts/generate-env-files.js PROD --marketing

# Preview without writing files
node scripts/generate-env-files.js DEV --dry-run

# List required secrets
node scripts/generate-env-files.js DEV --list-secrets

Features:

• Auto-discovers services from env.deploy.yml
• Handles from_secrets pattern (generates secret names like DEV_SERVER_APP_KEY)
• Supports explicit secrets (see Secret Management section)
• Validates environments and services

extract-config.js

Master configuration extractor for all deployment needs.

Usage:

# Generate Terraform variables
node scripts/extract-config.js terraform-vars --env dev-qa

# Generate Terraform backend config
node scripts/extract-config.js backend-hcl

# Generate secret export script
node scripts/extract-config.js secret-exports

# Generate Ansible inventory
node scripts/extract-config.js ansible-inventory --env dev-qa

# Generate documentation
node scripts/extract-config.js documentation

# List all secrets
node scripts/extract-config.js list-secrets

# List all variables
node scripts/extract-config.js list-variables

GitHub Actions Workflows

terraform-deploy.yml

• Purpose: Manages infrastructure with Terraform
• Trigger: Manual dispatch only
• Environments: dev-qa or prod (both enabled)
• Actions: plan or apply

deploy-server.yml

• Purpose: Builds and deploys AdonisJS backend
• Automatic triggers:
  • develop branch → dev environment
  • qa branch → qa environment
  • main branch → production environment
• Process: Builds Docker image, pushes to registry, deploys via Ansible

deploy-web-app.yml

• Purpose: Deploys React app to Cloudflare Pages
• Triggers: Changes in apps/webapp/ or manual dispatch
• Target: Cloudflare Pages

deploy-marketing.yml

• Purpose: Deploys Next.js marketing site to Cloudflare Pages
• Triggers: Changes in apps/marketing/ or manual dispatch
• Target: Cloudflare Pages

Secret Management

Two patterns for secrets:

1. Auto-generated: from_secrets creates names like {ENV}_{SERVICE}_{KEY}

   • Example: DEV.server.APP_KEY: from_secrets becomes DEV_SERVER_APP_KEY

2. Explicit: GitHub secrets syntax uses exact secret name

   • Example: PROD.server.DB_URL: '\$\{\{ secrets.NEON_DATABASE_URL }\}\'

Adding secrets:

1. Define in env.deploy.yml
2. Add to GitHub Settings → Secrets and variables → Actions
3. Verify with node scripts/extract-config.js list-secrets

Environment Structure

Example env.deploy.yml structure:

INFRASTRUCTURE:
  dev-qa:
    droplet_size: 's-1vcpu-1gb'
    region: 'nyc3'

DEV:
  server:
    PORT: 3333
    NODE_ENV: 'development'
    DATABASE_URL: from_secrets
  webapp:
    REACT_APP_API_URL: 'https://dev-api.jubiloop.ca'

PROD:
  server:
    PORT: 3333
    NODE_ENV: 'production'
    DATABASE_URL: '${{ secrets.NEON_DATABASE_URL }}' # Explicit secret reference

Deployment Process

1. Configuration: Define everything in env.deploy.yml
2. Secret Discovery: Scripts find all secret references
3. GitHub Actions: Workflows generate configs dynamically
4. Infrastructure: Terraform provisions resources
5. Deployment: Ansible deploys applications

Local Testing

cd infra/deploy

# Test server configs
node scripts/generate-env-files.js DEV --server --dry-run

# Check required secrets
node scripts/extract-config.js list-secrets

# Generate documentation
node scripts/extract-config.js documentation

Troubleshooting

Common issues:

1. Secret not found: Check list-secrets output matches GitHub secrets
2. Files not generated: Check YAML syntax, use --dry-run
3. Terraform issues: Verify R2 credentials and backend.hcl

Best Practices

1. Never hardcode secrets
2. Test with --dry-run first
3. Keep service names consistent
4. Use variables for non-sensitive config
5. All config in env.deploy.yml

Detailed Script Documentation

generate-env-files.js Features

The script supports multiple environments and modes:

Environment Support:

• Single environments: DEV, QA, PROD
• Combined environment: DEV-QA (generates files for both)

Service Discovery:

• Automatically finds all services in env.deploy.yml
• No hardcoded service lists
• Validates service availability per environment

Output Formats:

• Server mode: Creates .env.{service} files
• Web mode: Outputs shell export commands

Secret Handling:

• from_secrets: Auto-generates as {ENV}_{SERVICE}_{KEY}
• Explicit references: Uses exact secret name from GitHub

extract-config.js Commands

terraform-vars:

• Extracts INFRASTRUCTURE section from env.deploy.yml
• Creates terraform.tfvars with all variables
• Supports environment-specific configurations

backend-hcl:

• Generates Terraform backend configuration
• Configures Cloudflare R2 for state storage
• Uses credentials from env.deploy.yml

secret-exports:

• Discovers all secrets and variables
• Generates shell script for GitHub Actions
• Exports secrets dynamically during workflow

ansible-inventory:

• Reads Terraform state output
• Creates dynamic inventory with server IPs
• Groups servers by environment

documentation:

• Generates markdown documentation
• Lists all secrets and variables
• Shows environment configurations

Adding New Services

1. Update env.deploy.yml:

   DEV:
     my_new_service:
       PORT: 8080
       API_KEY: from_secrets

2. Add Docker Compose configuration

3. Test configuration:

   node scripts/generate-env-files.js DEV --dry-run
   node scripts/extract-config.js list-secrets

4. Add required secrets to GitHub

Script Documentation Details

How the Scripts Work Together

The deployment system is built around two core scripts that work in tandem:

1. extract-config.js reads env.deploy.yml and extracts specific configurations:

   • Infrastructure variables for Terraform
   • Secret and variable references for GitHub Actions
   • Dynamic inventory generation from Terraform state
   • Complete documentation of all settings

2. generate-env-files.js reads env.deploy.yml and generates environment files:

   • Discovers all services dynamically (no hardcoded lists)
   • Handles secret substitution from GitHub Actions environment
   • Outputs appropriate format based on deployment target

Script Execution in GitHub Actions

The typical workflow execution:

# 1. Export all secrets/variables to environment
./export_secrets.sh  # Generated by extract-config.js secret-exports

# 2. Generate Terraform variables
node scripts/extract-config.js terraform-vars --env $ENVIRONMENT

# 3. Apply infrastructure changes
terraform apply -auto-approve

# 4. Generate Ansible inventory from Terraform state
node scripts/extract-config.js ansible-inventory --env $ENVIRONMENT

# 5. Generate environment files for all services
node scripts/generate-env-files.js $ENVIRONMENT --server

# 6. Deploy with Ansible
ansible-playbook -i inventory.ini playbook.yml

Environment Variable Substitution

The scripts handle three types of values:

1. Static values: Copied as-is

   PORT: 3333 # Becomes PORT=3333

2. Auto-generated secrets: from_secrets

   APP_KEY: from_secrets # Becomes APP_KEY=$DEV_SERVER_APP_KEY

3. Explicit references: GitHub secrets/variables

   DATABASE_URL: '${{ secrets.PROD_DB_URL }}' # Becomes DATABASE_URL=$PROD_DB_URL

Multi-Environment Support

The scripts support both single and combined environments:

• Single: DEV, QA, PROD - generates files for one environment
• Combined: DEV-QA - generates files for both dev and qa simultaneously

This allows the shared dev-qa server to be deployed with a single command.

Support

For deployment issues:

1. Check generated configuration locally
2. Review GitHub Actions workflow logs
3. Contact team member directly
4. Check 1Password for secret documentation