Web S3 Docker

A Terraform module, creating a self-hosted static web hosting solution in the Hetzner cloud, offering S3 object storage and a Docker registry.

  • Public and private S3 buckets
  • Custom domains for public S3 Buckets
  • Public and private Docker Registry

Prerequisites

The module will create DNS entries for the various service endpoints. This requires a valid DNS zone that is managed via the Hetzner Cloud API.

Usage Example

resource "tls_private_key" "ssh_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "hcloud_ssh_key" "ssh_key1" {
  name       = "ssh-key1"
  public_key = tls_private_key.ssh_key.public_key_openssh
}

module "web-s3-docker" {
  source   = "https://github.com/pellepelster/solidblocks/releases/download/v0.4.9/terraform-hcloud-blcks-web-s3-docker-v0.4.9.zip"
  name     = "server1"
  dns_zone = "blcks-test.de"

  ssh_keys = [hcloud_ssh_key.ssh_key1.id]

  s3_buckets = [
    {
      name                     = "bucket1",
      web_access_public_enable = true,
      web_access_domains       = ["blcks-test.de", "www.blcks-test.de"]
    },
    {
      name             = "bucket2",
      owner_key_id     = "cbeebb7f1fa4de50025a5c95",
      owner_secret_key = "f775d527e821ab035b5a874e6326f20c135b2d2fc903112fe768a17681f54043"
    },
  ]

}

Configuration

Name and DNS

The name and dns_zone will be used as base domain for all created endpoints using the pattern <name>.<dns_zone>.

S3 Buckets

The variable s3_buckets can be used to auto-create S3 buckets during provisioning.

Minimal Bucket configuration

module "web_s3_docker" {
  # ...
  s3_buckets = [{ name = "bucket1" }]
}

will create a single S3 bucket named bucket1 with autogenerated access keys for the bucket owner, a read-write rw as well as a read-only ro user. The keys can be retrieved using the module output s3_buckets.

Example output for s3_buckets

[
  {
    "name": "bucket1",
    "owner_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "owner_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "ro_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "ro_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "rw_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "rw_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    //...
  }
]

Together with the base S3 api address from the output s3_host the bucket can be used with S3 tools like s3cmd like this

export S3_HOST=$(terraform output -raw s3_host)
export ACCESS_KEY=$(terraform output -raw s3_host | jq '.[0].owner_key_id')
export SECRET_KEY=$(terraform output -raw s3_host | jq '.[0].owner_secret_key')

s3cmd --host-bucket ${S3_HOST} --host ${S3_HOST} --access_key ${ACCESS_KEY} --secret_key ${SECRET_KEY} ls s3://bucket1

Custom access keys

You can override the access key ids as well as the secrets by providing your own. To do this simply provide them in the bucket config

module "web_s3_docker" {
  //...
  s3_buckets = [
    {
      name             = "bucket2",
      owner_key_id     = "cbeebb7f1fa4de50025a5c95",
      owner_secret_key = "f775d527e821ab035b5a874e6326f20c135b2d2fc903112fe768a17681f54043"
    },
  ]
}

[owner|rw|ro]_key_id has to be 24 characters long, and [owner|rw|ro]_secret_key 64. You can either provide both, just one or none, the missing parts will be automatically generated. The module takes care to add the GK prefix which is needed by the underlying S3 server.

Static web hosting

By setting the bucket configuration web_access_public_enable to true, static web hosting can be enabled for the Bucket, making all its content available on the internet. The list web_access_domains enables access to this bucket using custom domains.

Tip

DNS entries for all domains that are within the dns_zone are auto-created. If the provided web_access_domains is not part of the dns_zone the A respectively AAAA DNS entries need to be created manually, the IPv4 and IPv6 addresses are available through the module output ipv4_address and ipv6_address.

The created web endpoints again can be retrieved via the s3_buckets output.

Example output for a bucket with web_access_public_enable

[
  {
    "name": "bucket1",
    //...
    "web_access_public_enable": true,
    "web_access_addresses": [
      "https://bucket1.s3-web.web-s3-docker.blcks-test.de",
      "https://blcks-test.de",
      "https://www.blcks-test.de"
    ]
  }
]
Tip

When uploading content to be served it is important to use the correct mime-types, when using s3cmd to upload content this can be achieved like this

s3cmd sync --no-mime-magic --guess-mime-type ./contnt/* s3://<bucket_name>

Docker

By setting docker_enable to true the service will also expose a docker registry. The creation of credentials follows the pattern of the S3 bucket access key creation. If no user is specified a read-write as well as a read-only user will be autogenerated. Those can be retrieved via the outputs docker_rw_users respectively docker_ro_users.

Example output for docker_rw_users

[
  {
    "username": "61008df60fd0abac80a5cba645056a80",
    "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
]

using the private docker endpoint that can be retrieved from docker_host_private, docker images can be pushed like this

export DOCKER_HOST_PRIVATE=$(terraform output -raw docker_host_private)
export DOCKER_USERNAME=$(terraform output -raw docker_rw_users | jq '.[0].username')
export DOCKER_PASSWORD=$(terraform output -raw docker_rw_users | jq '.[0].password')

echo ${DOCKER_PASSWORD} | docker login ${DOCKER_HOST_PRIVATE} --username ${DOCKER_USERNAME} --password-stdin
docker push ${DOCKER_HOST_PRIVATE}/my-image:latest

Public registry

When docker_public_enable is set to true all images that are pushed to the private endpoint (docker_host_private) cn be pulled without authentication from the public endpoint

export DOCKER_HOST_PUBLIC=$(terraform output -raw docker_host_public)

docker pull ${DOCKER_HOST_PUBLIC}/my-image:latest

Endpoints

endpoint description address
S3 S3 api endpoint <bucket>.s3.<name>.<dns_zone>
S3 Web public accessible endpoint if web_access_public_enable is set to true <bucket>.s3-web.<name>.<dns_zone>
S3 Admin API API endpoint for the GarageFS S3 admin API. Adress can be as well as the access token can be retrieved from the output garage_admin_address and garage_admin_token s3-admin.<name>.<dns_zone>
private docker registry endpoint for publishing and pulling Docker images usein credentials docker-private.s3-web.<name>.<dns_zone>
public docker registry endpoint unauthorized pulling if enabled via docker_public_enable docker-public.s3-web.<name>.<dns_zone>

Terraform Module

Requirements

Name Version
hcloud >=1.56.0
http >= 3.3.0

Providers

Name Version
hcloud >=1.56.0
random n/a

Resources

Name Type
hcloud_primary_ip.ipv4 resource
hcloud_primary_ip.ipv6 resource
hcloud_server.server resource
hcloud_volume.data resource
hcloud_volume_attachment.data resource
hcloud_zone_rrset.root_domain_catchall_ipv4 resource
hcloud_zone_rrset.root_domain_ipv4 resource
hcloud_zone_rrset.web_access_domains_ipv4 resource
random_bytes.admin_token resource
random_bytes.docker_ro_default_password resource
random_bytes.docker_ro_default_user resource
random_bytes.docker_ro_password resource
random_bytes.docker_rw_default_password resource
random_bytes.docker_rw_default_user resource
random_bytes.docker_rw_password resource
random_bytes.metrics_token resource
random_bytes.owner_key_ids resource
random_bytes.owner_secret_keys resource
random_bytes.ro_key_ids resource
random_bytes.ro_secret_keys resource
random_bytes.rpc_secret resource
random_bytes.rw_key_ids resource
random_bytes.rw_secret_keys resource

Inputs

Name Description Type Default Required
data_volume_size data volume size in GB number 64 no
datacenter n/a string "nbg1-dc3" no
dns_zone DNS zone to use for hostname and DNs entries string n/a yes
docker_enable Enable Docker registry bool false no
docker_public_enable Enable public anonymous access to Docker registry, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details bool false no
docker_ro_users Docker read-write users to provision. If no users is given a user will be auto.created, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details 
list(object({
    username : string,
    password : optional(string)
  }))
 [] no
docker_rw_users Docker read-write users to provision. If no users is given a user will be auto.created, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details 
list(object({
    username : string,
    password : optional(string)
  }))
 [] no
labels A list of labels to be attached to the server instance. map(any) {} no
location Hetzner location to use for provisioned resources string "nbg1" no
name Unique name for the server instance string n/a yes
s3_buckets S3 buckets to provision, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#s3-buckets for details 
list(object({
    name                     = string
    owner_key_id             = optional(string)
    owner_secret_key         = optional(string)
    ro_key_id                = optional(string)
    ro_secret_key            = optional(string)
    rw_key_id                = optional(string)
    rw_secret_key            = optional(string)
    web_access_public_enable = optional(bool, false)
    web_access_domains       = optional(list(string))
  }))
 [] no
server_type Hetzner cloud server type, supports x86 and ARM architectures string "cx23" no
ssh_keys ssh keys to provision for instance access list(number) n/a yes

Outputs

Name Description
docker_host_private fully qualified domain for the private docker registry
docker_host_public fully qualified domain for the public docker registry if enabled
docker_ro_users readonly users for the docker registry
docker_rw_users write users for the docker registry
garage_admin_address address for the GarageFS admin endpoint
garage_admin_token token for the GarageFS admin endpoint
ipv4_address IPv4 address of the created server
ipv6_address IPv6 address of the created server
s3_buckets the created S3 bucket with access credentials and public endpoints if available
s3_host fully qualified for the s3 endpoint
server_id Hetzner ID of the created server