Web S3 Docker

A Terraform module, creating a self-hosted static web hosting solution in the Hetzner cloud, offering S3 object storage and a Docker registry.

Public and private S3 buckets
Custom domains for public S3 Buckets
Public and private Docker Registry

Prerequisites

The module will create DNS entries for the various service endpoints. This requires a valid DNS zone that is managed via the Hetzner Cloud API.

Usage Example

resource "tls_private_key" "ssh_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "hcloud_ssh_key" "ssh_key1" {
  name       = "ssh-key1"
  public_key = tls_private_key.ssh_key.public_key_openssh
}

module "web-s3-docker" {
  source   = "https://github.com/pellepelster/solidblocks/releases/download/v0.4.12/blcks-terraform-hcloud-web-s3-docker-v0.4.12.zip"
  name     = "server1"
  dns_zone = "blcks-test.de"

  ssh_keys = [hcloud_ssh_key.ssh_key1.id]

  s3_buckets = [
    {
      name                     = "bucket1",
      web_access_public_enable = true,
      web_access_domains       = ["blcks-test.de", "www.blcks-test.de"]
    },
    {
      name             = "bucket2",
      owner_key_id     = "cbeebb7f1fa4de50025a5c95",
      owner_secret_key = "f775d527e821ab035b5a874e6326f20c135b2d2fc903112fe768a17681f54043"
    },
  ]

  disable_volume_delete_protection = true
}

Configuration

Name and DNS

The name and dns_zone will be used as base domain for all created endpoints using the pattern <name>.<dns_zone>.

S3 Buckets

The variable s3_buckets can be used to auto-create S3 buckets during provisioning.

Minimal Bucket configuration

module "web_s3_docker" {
  # ...
  s3_buckets = [{ name = "bucket1" }]
}

will create a single S3 bucket named bucket1 with autogenerated access keys for the bucket owner, a read-write rw as well as a read-only ro user. The keys can be retrieved using the module output s3_buckets.

Example output for `s3_buckets`

[
  {
    "name": "bucket1",
    "owner_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "owner_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "ro_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "ro_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "rw_key_id": "GKxxxxxxxxxxxxxxxxxxxxxxxx",
    "rw_secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    //...
  }
]

Together with the base S3 api address from the output s3_host the bucket can be used with S3 tools like s3cmd like this

export S3_HOST=$(terraform output -raw s3_host)
export ACCESS_KEY=$(terraform output -raw s3_host | jq '.[0].owner_key_id')
export SECRET_KEY=$(terraform output -raw s3_host | jq '.[0].owner_secret_key')

s3cmd --host-bucket ${S3_HOST} --host ${S3_HOST} --access_key ${ACCESS_KEY} --secret_key ${SECRET_KEY} ls s3://bucket1

Custom access keys

You can override the access key ids as well as the secrets by providing your own. To do this simply provide them in the bucket config

module "web_s3_docker" {
  //...
  s3_buckets = [
    {
      name             = "bucket2",
      owner_key_id     = "cbeebb7f1fa4de50025a5c95",
      owner_secret_key = "f775d527e821ab035b5a874e6326f20c135b2d2fc903112fe768a17681f54043"
    },
  ]
}

[owner|rw|ro]_key_id has to be 24 characters long, and [owner|rw|ro]_secret_key 64. You can either provide both, just one or none, the missing parts will be automatically generated. The module takes care to add the GK prefix which is needed by the underlying S3 server.

Static web hosting

By setting the bucket configuration web_access_public_enable to true, static web hosting can be enabled for the Bucket, making all its content available on the internet. The list web_access_domains enables access to this bucket using custom domains.

Tip

DNS entries for all domains that are within the dns_zone are auto-created. If the provided web_access_domains is not part of the dns_zone the A respectively AAAA DNS entries need to be created manually, the IPv4 and IPv6 addresses are available through the module output ipv4_address and ipv6_address.

The created web endpoints again can be retrieved via the s3_buckets output.

Example output for a bucket with `web_access_public_enable`

[
  {
    "name": "bucket1",
    //...
    "web_access_public_enable": true,
    "web_access_addresses": [
      "https://bucket1.s3-web.web-s3-docker.blcks-test.de",
      "https://blcks-test.de",
      "https://www.blcks-test.de"
    ]
  }
]

Tip

When uploading content to be served it is important to use the correct mime-types, when using s3cmd to upload content this can be achieved like this

s3cmd sync --no-mime-magic --guess-mime-type ./contnt/* s3://<bucket_name>

Docker

By setting docker_enable to true the service will also expose a docker registry. The creation of credentials follows the pattern of the S3 bucket access key creation. If no user is specified a read-write as well as a read-only user will be autogenerated. Those can be retrieved via the outputs docker_rw_users respectively docker_ro_users.

Example output for `docker_rw_users`

[
  {
    "username": "61008df60fd0abac80a5cba645056a80",
    "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
]

using the private docker endpoint that can be retrieved from docker_host_private, docker images can be pushed like this

export DOCKER_HOST_PRIVATE=$(terraform output -raw docker_host_private)
export DOCKER_USERNAME=$(terraform output -raw docker_rw_users | jq '.[0].username')
export DOCKER_PASSWORD=$(terraform output -raw docker_rw_users | jq '.[0].password')

echo ${DOCKER_PASSWORD} | docker login ${DOCKER_HOST_PRIVATE} --username ${DOCKER_USERNAME} --password-stdin
docker push ${DOCKER_HOST_PRIVATE}/my-image:latest

Public registry

When docker_public_enable is set to true all images that are pushed to the private endpoint (docker_host_private) cn be pulled without authentication from the public endpoint

export DOCKER_HOST_PUBLIC=$(terraform output -raw docker_host_public)

docker pull ${DOCKER_HOST_PUBLIC}/my-image:latest

Endpoints

endpoint	description	address
S3	S3 api endpoint	<bucket>.s3.<name>.<dns_zone>
S3 Web	public accessible endpoint if `web_access_public_enable` is set to `true`	<bucket>.s3-web.<name>.<dns_zone>
S3 Admin API	API endpoint for the GarageFS S3 admin API. Adress can be as well as the access token can be retrieved from the output `garage_admin_address` and `garage_admin_token`	s3-admin.<name>.<dns_zone>
private docker registry	endpoint for publishing and pulling Docker images usein credentials	docker-private.s3-web.<name>.<dns_zone>
public docker registry	endpoint unauthorized pulling if enabled via `docker_public_enable`	docker-public.s3-web.<name>.<dns_zone>

Terraform Module

Requirements

Name	Version
hcloud	>=1.56.0
http	>= 3.3.0

Providers

Name	Version
hcloud	>=1.56.0
random	n/a

Resources

Name	Type
hcloud_primary_ip.ipv4	resource
hcloud_primary_ip.ipv6	resource
hcloud_server.server	resource
hcloud_volume.data	resource
hcloud_volume_attachment.data	resource
hcloud_zone_rrset.root_domain_catchall_ipv4	resource
hcloud_zone_rrset.root_domain_ipv4	resource
hcloud_zone_rrset.web_access_domains_ipv4	resource
random_bytes.admin_token	resource
random_bytes.docker_ro_default_password	resource
random_bytes.docker_ro_default_user	resource
random_bytes.docker_ro_password	resource
random_bytes.docker_rw_default_password	resource
random_bytes.docker_rw_default_user	resource
random_bytes.docker_rw_password	resource
random_bytes.metrics_token	resource
random_bytes.owner_key_ids	resource
random_bytes.owner_secret_keys	resource
random_bytes.ro_key_ids	resource
random_bytes.ro_secret_keys	resource
random_bytes.rpc_secret	resource
random_bytes.rw_key_ids	resource
random_bytes.rw_secret_keys	resource

Inputs

Name	Description	Type	Default	Required
data_volume_size	data volume size in GB	`number`	`64`	no
datacenter	n/a	`string`	`"nbg1-dc3"`	no
disable_volume_delete_protection	n/a	`bool`	`false`	no
dns_zone	DNS zone to use for hostname and DNs entries	`string`	n/a	yes
docker_enable	Enable Docker registry	`bool`	`false`	no
docker_public_enable	Enable public anonymous access to Docker registry, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details	`bool`	`false`	no
docker_ro_users	Docker read-write users to provision. If no users is given a user will be auto.created, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details	list(object({ username : string, password : optional(string) }))	`[]`	no
docker_rw_users	Docker read-write users to provision. If no users is given a user will be auto.created, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#docker for details	list(object({ username : string, password : optional(string) }))	`[]`	no
labels	A list of labels to be attached to the server instance.	`map(any)`	`{}`	no
location	Hetzner location to use for provisioned resources	`string`	`"nbg1"`	no
name	Unique name for the server instance	`string`	n/a	yes
s3_buckets	S3 buckets to provision, see https://pellepelster.github.io/solidblocks/hetzner/web-s3-docker/#s3-buckets for details	list(object({ name = string owner_key_id = optional(string) owner_secret_key = optional(string) ro_key_id = optional(string) ro_secret_key = optional(string) rw_key_id = optional(string) rw_secret_key = optional(string) web_access_public_enable = optional(bool, false) web_access_domains = optional(list(string)) }))	`[]`	no
server_type	Hetzner cloud server type, supports x86 and ARM architectures	`string`	`"cx23"`	no
ssh_host_cert_ecdsa	override generated ssh host ed25519 certificate, must be set alongside with ‘ssh_host_key_ecdsa’	`string`	`""`	no
ssh_host_cert_ed25519	override generated ssh host ed25519 certificate, must be set alongside with ‘ssh_host_key_ed25519’	`string`	`""`	no
ssh_host_cert_rsa	override generated ssh host ed25519 certificate, must be set alongside with ‘ssh_host_key_rsa’	`string`	`""`	no
ssh_host_key_ecdsa	override generated ssh host ed25519 key, must be set alongside with ‘ssh_host_cert_ecdsa’	`string`	`""`	no
ssh_host_key_ed25519	override generated ssh host ed25519 key, must be set alongside with ‘ssh_host_cert_ed25519’	`string`	`""`	no
ssh_host_key_rsa	override generated ssh host ed25519 key, must be set alongside with ‘ssh_host_cert_rsa’	`string`	`""`	no
ssh_keys	ssh keys to provision for instance access	`list(number)`	n/a	yes

Outputs

Name	Description
docker_host_private	fully qualified domain for the private docker registry
docker_host_public	fully qualified domain for the public docker registry if enabled
docker_ro_users	readonly users for the docker registry
docker_rw_users	write users for the docker registry
garage_admin_address	address for the GarageFS admin endpoint
garage_admin_token	token for the GarageFS admin endpoint
ipv4_address	IPv4 address of the created server
ipv6_address	IPv6 address of the created server
s3_buckets	the created S3 bucket with access credentials and public endpoints if available
s3_host	fully qualified for the s3 endpoint
server_id	Hetzner ID of the created server