Jubiloop Docs

Appearance

Terraform Setup ​

Standard Procedure: GitHub Actions ​

Use the manually dispatched Terraform Deploy workflow for every normal plan and apply.

  1. Confirm no workflow or operator is using the target environment state.
  2. Dispatch action=plan for dev-qa or prod and review the complete plan.
  3. Obtain the required approval for the environment.
  4. Dispatch action=apply with auto_approve=true.
  5. Review the fresh plan created by the apply run, its result, and Terraform outputs.

The apply workflow cannot pause interactively. It creates and applies a new saved plan in the same run rather than consuming the earlier plan artifact. GitHub environment protection must enforce any approval required before the job starts.

Manual Fallback ​

Local Terraform is only for approved diagnostics, isolated testing, or recovery. Coordinate with the team, confirm CI is idle, back up state before risky state operations, and record the intervention. Do not run a local apply merely because the workflow is inconvenient.

Directory Structure ​

infra/deploy/terraform/
├── dev-qa/          # Development and QA shared infrastructure
├── prod/            # Production infrastructure
└── modules/         # Reusable Terraform modules

Backend Configuration ​

Terraform state is stored remotely in Cloudflare R2 buckets for:

  • State persistence across team members
  • Encrypted storage at rest

The current R2 S3 backend does not configure DynamoDB, S3 lock files, or another lock service. GitHub Actions prevents two same-environment workflow runs from continuing concurrently, but its top-level concurrency group uses cancel-in-progress: true: dispatching another run can cancel an active plan or apply rather than queue behind it. Before dispatching or running Terraform locally, confirm that no run or operator is working on that environment. If a run is cancelled during apply, treat the environment as potentially changed: inspect the Actions log and remote state, then create a fresh plan before taking further action.

R2 Setup ​

  1. Create R2 Bucket:

    • Name: jubiloop-terraform-state
    • Region: Auto (managed by Cloudflare)

  2. Create R2 API Token:

    • Permission: Cloudflare R2:Edit
    • Save Access Key ID and Secret Access Key

  3. Configure Backend:

    hcl
    # backend.hcl
bucket = "jubiloop-terraform-state"
endpoints = {
  s3 = "https://ACCOUNT-ID.r2.cloudflarestorage.com"
}
access_key = "your-r2-access-key-id"
secret_key = "your-r2-secret-access-key"

Required Credentials ​

All credentials are stored in 1Password (Jubiloop group) and configured in GitHub Secrets:

DigitalOcean ​

  • API Token: Full access for resource management
  • Project ID: Optional for resource organization

Cloudflare ​

  • API Token: Custom token with permissions:
    • Zone:Zone:Read
    • Zone:DNS:Edit
  • Zone IDs: For jubiloop.ca
  • R2 Credentials: For state backend

SSH Access ​

  • Deploy Key: Ed25519 key for server access
  • Public Key: Added to droplets for deployment

Environment Configuration ​

Development & QA ​

Shared infrastructure configuration:

hcl
# terraform.tfvars
digital_ocean_region = "tor1"
droplet_size = "s-1vcpu-1gb"
deploy_user = "deploy"

Resources created:

  • Single droplet hosting both environments; droplet_size defaults to s-1vcpu-1gb and can be overridden by generated variables
  • Reserved IP for static addressing
  • Cloudflare DNS records:
    • dev-api.jubiloop.ca → Droplet IP
    • qa-api.jubiloop.ca → Droplet IP
  • Cloudflare Pages projects:
    • dev-app-jubiloop-ca → dev-app.jubiloop.ca
    • qa-app-jubiloop-ca → qa-app.jubiloop.ca
    • dev-jubiloop-ca → dev.jubiloop.ca; the marketing workflow deploys through OpenNext using this project name
    • qa-jubiloop-ca → qa.jubiloop.ca; the marketing workflow deploys through OpenNext using this project name
    • dev-docs-jubiloop-ca → dev-docs.jubiloop.ca
  • Cloudflare Zero Trust Access protection for all frontend apps
  • DigitalOcean firewall rules:
    • SSH: Open to all (configurable)
    • HTTP/HTTPS: Cloudflare IPs only

Production ​

Dedicated infrastructure configuration:

hcl
# terraform.tfvars
digital_ocean_region = "tor1"
droplet_size = "s-1vcpu-1gb"  # Same size as dev-qa but dedicated
deploy_user = "deploy"

Resources created:

  • Dedicated production droplet; droplet_size defaults to s-1vcpu-1gb and can be overridden by generated variables
  • Reserved IP for static addressing
  • Cloudflare DNS records:
    • api.jubiloop.ca → Droplet IP
  • Cloudflare Pages projects:
    • app-jubiloop-ca → app.jubiloop.ca (Protected with Zero Trust temporarily)
    • jubiloop-ca → jubiloop.ca and www.jubiloop.ca; the marketing workflow deploys through OpenNext using this project name
  • DigitalOcean Managed PostgreSQL 16 (db-s-1vcpu-1gb, with PgBouncer connection pooling)
  • Enhanced backup configuration enabled
  • Cloudflare Zero Trust Access for app.jubiloop.ca only (temporary)
  • DigitalOcean firewall rules:
    • SSH: Open to all (configurable)
    • HTTP/HTTPS: Cloudflare IP ranges only

Deployment Process ​

Initial Setup ​

bash
# Navigate to environment directory
cd infra/deploy/terraform/dev-qa  # or /prod for production

# Copy configuration files
cp terraform.tfvars.example terraform.tfvars
cp backend.hcl.example backend.hcl

# Fill in values from 1Password
# Edit terraform.tfvars and backend.hcl

# Initialize Terraform
terraform init -backend-config="backend.hcl"

SSH Key Configuration ​

The infrastructure uses separate SSH keys for security isolation:

Dev/QA Environments (shared key):

bash
# Generate key for dev/qa
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_jubiloop_dev_qa -C "devops@jubiloop.ca" -N ""

# Add to GitHub Secrets:
# - DEV_QA_SSH_PUBLIC_KEY: contents of ~/.ssh/id_ed25519_jubiloop_dev_qa.pub
# - DEV_QA_SSH_PRIVATE_KEY: contents of ~/.ssh/id_ed25519_jubiloop_dev_qa

Production Environment (separate key):

bash
# Generate key for production
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_jubiloop_prod -C "devops@jubiloop.ca" -N ""

# Add to GitHub Secrets:
# - PROD_SSH_PUBLIC_KEY: contents of ~/.ssh/id_ed25519_jubiloop_prod.pub
# - PROD_SSH_PRIVATE_KEY: contents of ~/.ssh/id_ed25519_jubiloop_prod

Planning Changes ​

bash
# Preview the current changes
terraform plan

# Save the exact plan for an approved manual fallback
terraform plan -out=tfplan

Applying Changes Manually ​

Approved fallback only

Confirm the target state is idle and that the local plan has approval. Prefer the GitHub Actions procedure above. Never use -auto-approve with an ad hoc local plan.

bash
# Apply the exact reviewed saved plan
terraform apply tfplan

The GitHub Actions workflow is manual. A plan-only run is an advisory preview: a later apply run recalculates the plan from the current configuration and state, then applies the plan generated in that same run. It does not reuse the earlier plan artifact. Before dispatching apply, confirm that the preview is acceptable, no other infrastructure run is active, and no relevant configuration or state has changed. Dispatch apply promptly and review its generated plan and final outputs.

Set auto_approve=true only when intentionally dispatching action=apply. With it set to false, the non-interactive job exits with an error; it does not pause for approval. Use GitHub environment controls to authorize apply runs, but do not treat that approval as approval of an exact earlier plan. The initial_apply option performs targeted bootstrap applies followed by a full apply and requires the same checks.

Viewing Outputs ​

bash
# Show all outputs
terraform output

# Get specific output
terraform output droplet_info
terraform output ssh_connection

Resource Management ​

Terraform Modules ​

The infrastructure is organized into reusable modules:

Droplet Module (modules/droplet/) ​

Creates standardized DigitalOcean droplets:

hcl
module "dev_qa_droplet" {
  source = "../modules/droplet"

  droplet_name         = "dev-qa-server"
  environment          = "dev-qa"
  digital_ocean_region = var.digital_ocean_region
  size                 = var.droplet_size

  ssh_key_ids           = [digitalocean_ssh_key.deploy_key.id]
  deploy_user           = var.deploy_user
  deploy_ssh_public_key = var.ssh_public_key

  use_reserved_ip = true
  enable_backups  = false  # Cost optimization

  tags = ["dev-qa", "jubiloop", "terraform-managed"]
}

Features:

  • Cloud-init configuration for initial setup
  • Firewall with Cloudflare IP whitelist
  • Optional reserved IP and volumes
  • Monitoring enabled by default
  • IPv6 support

DNS Module (modules/dns/) ​

Manages Cloudflare DNS records with support for apex domains:

hcl
# Regular subdomain
module "api_dns" {
  source           = "../modules/dns"
  zone_id          = var.cloudflare_zone_id_jubiloop_ca
  dns_content      = module.prod_droplet.public_ip
  ipv6_dns_content = module.prod_droplet.ipv6_address
  domain           = "jubiloop.ca"
  subdomain        = "api"        # Regular subdomain
  with_www         = false        # No www.api.jubiloop.ca
}

# Apex domain with www
module "marketing_dns" {
  source           = "../modules/dns"
  zone_id          = var.cloudflare_zone_id_jubiloop_ca
  dns_content      = cloudflare_pages_project.marketing.subdomain
  domain           = "jubiloop.ca"
  subdomain        = ""           # Empty for apex domain
  with_www         = true         # Also creates www.jubiloop.ca
}

Creates A and AAAA records with proxy enabled. Handles apex domains when subdomain is empty or "@".

Cloudflare Pages Module (modules/cloudflare_pages/) ​

Configures Pages projects with support for apex and www domains:

hcl
# Regular subdomain
module "app_pages" {
  source            = "../modules/cloudflare_pages"
  project_name      = "app-jubiloop-ca"
  production_branch = "main"
  zone_id           = var.cloudflare_zone_id_jubiloop_ca
  account_id        = var.cloudflare_account_id
  domain            = "jubiloop.ca"
  subdomain         = "app"
  with_www          = false  # No www.app.jubiloop.ca (free tier limit)
}

# Apex domain with www
module "marketing_pages" {
  source            = "../modules/cloudflare_pages"
  project_name      = "jubiloop-ca"
  production_branch = "main"
  zone_id           = var.cloudflare_zone_id_jubiloop_ca
  account_id        = var.cloudflare_account_id
  domain            = "jubiloop.ca"
  subdomain         = ""     # Empty for apex domain
  with_www          = true   # Creates both jubiloop.ca and www.jubiloop.ca
}

Droplets ​

Configuration includes:

  • Ubuntu 22.04 base image
  • Deploy user with SSH key
  • Docker-ready setup via cloud-init
  • Security hardening
  • Swap space configuration

Networking ​

  • Reserved IPs: Static addresses for DNS
  • Firewall Rules:
    • SSH (22): Configurable IP whitelist (default: open)
    • HTTP (80): Cloudflare IPs only
    • HTTPS (443): Cloudflare IPs only
  • Private Networking: Internal service communication

DNS Management ​

Automated DNS record creation:

  • API endpoints (proxied through Cloudflare)
  • Direct droplet access (when needed)
  • Automatic SSL/TLS via Cloudflare
  • IPv6 support

State Management ​

Viewing State ​

bash
# Show current state
terraform show

# List resources
terraform state list

# Show specific resource
terraform state show digitalocean_droplet.web

State Operations ​

bash
# Pull latest state
terraform state pull

# Move resources
terraform state mv old_name new_name

# Remove from state (careful!)
terraform state rm resource_name

Troubleshooting ​

Common Issues ​

Backend Initialization Failures:

  • Verify R2 credentials in backend.hcl
  • Check bucket exists and is accessible
  • Ensure account ID is correct

Resource Creation Failures:

  • Check API token permissions
  • Verify quota limits
  • Review error messages for specifics

DNS Issues:

  • Confirm Cloudflare API token permissions
  • Verify zone IDs are correct
  • Check domain ownership

Debug Mode ​

bash
# Enable debug logging for a plan without sharing credential values
export TF_LOG=DEBUG
terraform plan

# A targeted plan can help diagnose dependencies
terraform plan -target=digitalocean_droplet.web

Best Practices ​

  1. Always Plan First: Review changes before applying
  2. Use Environment Directories: Keep dev-qa and production state keys separate
  3. Coordinate Runs: Do not overlap manual operations with CI because the backend has no separate lock service
  4. Version Control: Track infrastructure changes
  5. Document Changes: Update README for significant changes
  6. Test First: Use dev-qa before production
  7. Protect State: R2 stores the current remote state; arrange a separate backup before risky state operations

CI/CD Integration ​

Infrastructure changes use the manually dispatched terraform-deploy.yml workflow:

  1. Terraform Plan: Select plan and the dev-qa or prod environment
  2. Terraform Apply: Start a separate apply run after reviewing the change
  3. Plan Freshness: The apply run generates its own plan; it does not reuse the plan artifact
  4. Concurrency: A new same-environment CI run can cancel the active run; the R2 backend has no lock, so confirm the environment is idle before dispatching or running Terraform locally
  5. Secret Handling: Via GitHub Secrets

Disaster Recovery ​

Backup Procedures ​

  1. State Storage: Current state is remote in R2; no separate state backup process is configured
  2. Configuration Backup: Git repository
  3. Secret Backup: 1Password

Recovery Steps ​

  1. Confirm the current R2 state and restore a separately retained copy only if one exists
  2. Run terraform plan to verify
  3. Apply to recreate infrastructure
  4. Redeploy applications via CI/CD

Terraform and application workflows do not automatically roll back failed changes. Inspect partial state, create a fresh plan, and use an approved forward fix or Git revert as appropriate.

Security Considerations ​

  • State Encryption: Encrypted at rest in R2
  • Access Control: Limited to authorized team members
  • Audit Trail: All changes tracked in Git
  • Secret Management: Never commit secrets
  • Network Security: Firewall rules enforced

Maintenance ​

Regular Tasks ​

  • Review and update provider versions
  • Audit resource usage and costs
  • Update security group rules
  • Clean up unused resources

Upgrade Procedures ​

  1. Test upgrades in dev-qa first
  2. Plan during maintenance windows
  3. Have rollback plan ready
  4. Document any breaking changes

Cloudflare Zero Trust Access ​

Overview ​

Cloudflare Zero Trust Access provides email-based authentication (OTP) to protect dev and QA frontend applications while keeping APIs accessible for existing session-based auth.

Protected Domains ​

Dev Environment ​

  • dev-app.jubiloop.ca - Web Application (Protected)
  • dev.jubiloop.ca - Marketing Site (Protected)
  • dev-api.jubiloop.ca - API Server (Not Protected)

QA Environment ​

  • qa-app.jubiloop.ca - Web Application (Protected)
  • qa.jubiloop.ca - Marketing Site (Protected)
  • qa-api.jubiloop.ca - API Server (Not Protected)

Production Environment ​

  • app.jubiloop.ca - Web Application (Protected - temporary until public launch)
  • jubiloop.ca - Marketing Site (Public)
  • www.jubiloop.ca - Marketing Site (Public)
  • api.jubiloop.ca - API Server (Not Protected)

Configuration ​

Email Allowlists ​

Configure allowed emails in your terraform.tfvars:

hcl
# Dev environment access
dev_allowed_emails = [
  "accounts@jubiloop.ca",
  "developer@example.com"
]

# QA environment access (can include clients)
qa_allowed_emails = [
  "team@jubiloop.ca",
  "client@example.com",
  "qa-tester@example.com",
  "accounts@jubiloop.ca"
]

# Production app access (temporary until public launch)
prod_allowed_emails = [
  "admin@jubiloop.ca",
  "team@jubiloop.ca"
]

# Session duration (default: 24h)
access_session_duration = "24h"

Environment Variables ​

You can also pass email lists via environment variables:

bash
# As comma-separated string
export TF_VAR_dev_allowed_emails="user1@example.com,user2@example.com"
export TF_VAR_qa_allowed_emails="user1@example.com,user2@example.com"

# Or as JSON array
export TF_VAR_dev_allowed_emails='["user1@example.com","user2@example.com"]'

Required Permissions ​

Your Cloudflare API token needs:

  • Account:Cloudflare Access: Edit
  • All existing permissions remain unchanged

User Experience ​

  1. User visits protected domain (e.g., dev-app.jubiloop.ca)
  2. Redirected to Cloudflare Access login page
  3. User enters their email address
  4. Receives 6-digit code via email
  5. Enters code and gains access for configured duration
  6. Can use the application normally with existing auth

Managing Access ​

Adding/Removing Users ​

  1. Update email lists in terraform.tfvars
  2. Run through GitHub Actions (or manually if emergency):
    bash
    cd infra/deploy/terraform/dev-qa  # or /prod for production
terraform plan
terraform apply
  3. Changes take effect immediately

Disabling Access Protection ​

If you need to disable Access:

  1. Comment out the Access module blocks in main.tf
  2. Run terraform apply
  3. Sites become publicly accessible immediately

API Access Strategy ​

APIs remain unprotected by Cloudflare Access to maintain compatibility with:

  • Existing session-based authentication
  • CORS policies configured in the application
  • Service-to-service communication

The frontend protection ensures only authorized users can access the web applications that communicate with the APIs.

Troubleshooting ​

Can't receive email codes ​

  • Check spam folder
  • Ensure email is in the allowlist
  • Try a different email provider

Session expires too quickly ​

  • Increase access_session_duration (max: 30d)
  • Consider using "remember me" in app auth

Built with ❤️ by the Jubiloop team

Copyright © Jubiloop