How to deploy Drone 2.16.0 in docker swarm behind Caddy v2.6.4

Drone is a self-service Continuous Integration platform for busy development teams.

In this post, I am going to show you how to deploy Drone 2.16.0, a Container-Native, Continuous Delivery Platform in Docker Swarm Cluster using the Docker Compose tool behind Caddy 2.6.4

Drone is a self-service Continuous Integration platform for busy development teams

If you want to learn more about Drone, please go through the below links.

1. Drone website
2. Official documentation
3. GitHub repository

Let’s start with actual deployment…

Prerequisites

Please make sure you should fulfill the below requirements before proceeding to the actual deployment.

1. Docker Swarm Cluster with GlusterFS as persistent tool.
2. Caddy as reverse proxy to expose micro-services to external.

Introduction

Drone is a continuous delivery system built on container technology. Drone uses a simple YAML build file, to define and execute build pipelines inside Docker Containers.

Drone uses build file (.drone.yml) that contains a single pipeline or multiple pipelines to build an application and publish the docker container to a registry of your choice.

Pipelines are configured using YML with a simple, easy to read file that we commit to our git repository. Each pipeline step is executed inside an isolated Docker Container that is automatically downloaded at runtime.

Drone Features

Drone has below features

Any Source Code ManagerAny PlatformAny Language

• We can integrate Drone with multiple source code systems
• GitHub, GitLab, Bitbucket and Gitea
• Drone supports multiple operating systems and architectures
• Linux x64, ARM, ARM64 and Windows x64
• Drone works with any language, database or service
• Drone supports anything that runs inside a Docker Container

Persist Drone Data

Containers are fast to deploy and make efficient use of system resources. Developers get application portability and programmable image management and the operations team gets standard run time units of deployment and management.

With all the known benefits of containers, there is one common misperception that the containers are ephemeral, which means if we restart the container or in case of any issues with it, we lose all the data for that particular container. They are only good for stateless micro-service applications and that it’s not possible to containerize stateful applications.

I am going to use GlusterFS to overcome the ephemeral behavior of Containers.

I already set up a replicated GlusterFS volume to have data replicated throughout the cluster if I would like to have some persistent data.

The below diagram explains how the replicated volume works.

Volume will be mounted on all the nodes, and when a file is written to the /mnt partition, data will be replicated to all the nodes in the Cluster

Persistent application state or data needs to survive application restarts and outages. We are storing the data or state in GlusterFS and had periodic backups performed on it.

Drone service should be available if something goes wrong with any of the nodes on our Docker Swarm Cluster. The data will be available to all the nodes in the cluster because of GlusterFS Replicated Volume.

I am going to create a folder dronedata in /mnt directory to map container volume /data

cd /mnt

sudo mkdir -p dronedata

Prepare Drone Environment

I am going to use docker-compose to prepare the environment file for deploying Mattermost. The compose file is known as YAML ( YAML stands for Yet Another Markup Language) and has extension .yml or .yaml

I am going to create application folders in /opt directory on manager node in our docker swarm cluster to store configuration files, nothing but docker compose files (.yml or .yaml).

Also, I am going to use the caddy overlay network created in the previous Caddy post.

Now it’s time to create a folder, drone in /opt directory to place configuration file, i.e., .yml file for drone.

Use the below commands to create the folder.

Go to /opt directory by typing cd /opt in Ubuntu console

make a folder, drone in /opt with sudo mkdir -p drone

Let’s get into drone folder by typing cd drone

Now create a docker-compose file inside the drone folder using sudo touch drone.yml

Open drone.yml docker-compose file with nano editor using sudo nano drone.yml and copy and paste the below code in it.

Drone Docker Compose

Here is the docker-compose file for Drone. I am going to utilize SQLite as a back-end database for it.

version: "3.7"

services:
  drone:
    image: drone/drone:latest
    volumes:
      - /mnt/dronedata:/data
    environment:
      - DRONE_DATABASE_DRIVER=sqlite3
      - DRONE_DATABASE_DATASOURCE=/data/database.sqlite
      - DRONE_RPC_SECRET=drone-client-secret
      - DRONE_COOKIE_TIMEOUT=720h
      - DRONE_CRON_DISABLED=true
      - DRONE_OPEN=true
      - DRONE_NETWORK=caddy
      - DRONE_ADMIN=rajasekhar
      - DRONE_USER_CREATE=username:rajasekhar,admin:true
      - DRONE_SERVER_PORT=:80
      - DRONE_GIT_ALWAYS_AUTH=true
      - DRONE_SERVER_HOST=drone.example.com
      - DRONE_HOST=https://drone.example.com
      - DRONE_SERVER_PROTO=https
      - DRONE_TLS_AUTOCERT=false
      - DRONE_AGENTS_ENABLED=true
      - DRONE_GITEA_SERVER=https://gitea.example.com
      - DRONE_GITEA_CLIENT_ID=gitea-client-id
      - DRONE_GITEA_CLIENT_SECRET=gitea-client-secret
    networks:
      - caddy
    ports:
      - "8005:80"
    deploy:
      placement:
        constraints: [node.role == worker]
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure  
    
  runner:
    image: drone/drone-runner-docker:latest
    command: agent
    depends_on:
      - drone
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_SERVER=https://drone.example.com
      - DRONE_RPC_HOST=drone.example.com
      - DRONE_RPC_PROTO=https
      - DRONE_RPC_SECRET=drone-client-secret
      - DRONE_RUNNER_CAPACITY=10
      - DRONE_RUNNER_NETWORKS=caddy
      - DRONE_RUNNER_NETWORK_OPTS=com.docker.network.drive.mtu:1442
    networks:
      - caddy
    deploy:
      placement:
        constraints: [node.role == worker]
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure

volumes:
  dronedata:
    driver: "local"
networks:
  caddy:
    external: true
    attachable: true

Please find documentation below to learn how to configure drone based on the selected git provider. I am using Gitea as a provider.

https://docs.drone.io/server/provider/gitea/

Caddyfile - Drone

The Caddyfile is a convenient Caddy configuration format for humans.

Caddyfile is easy to write, easy to understand, and expressive enough for most use cases.

Please find Production-ready Caddyfile for Drone.

Learn more about Caddyfile here to get familiar with it.

{
    email you@example.com
    default_sni drone
    cert_issuer acme
    # Production acme directory
    acme_ca https://acme-v02.api.letsencrypt.org/directory
    # Staging acme directory
    #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    servers {
        protocol {
            experimental_http3
            allow_h2c
            strict_sni_host
        }
        timeouts {
            read_body   10s
            read_header 10s
            write       10s
            idle        2m
        }
        max_header_size 16384
    }
}
drone.example.com {
    log {
        output file /var/log/caddy/drone.log {
        roll_size 20mb
        roll_keep 2
        roll_keep_for 6h
        }
        format console
        level error
    }
    encode gzip zstd
    reverse_proxy drone:80
}

Please go to Caddy Post to get more insight to deploy it in the docker swarm cluster.

Final Drone Docker Compose (Including caddy server configuration)

Please find the full docker-compose file below. You can deploy as many sites as you want using it.

Don’t forget to map site data directories like /mnt/dronedata:/data in Caddy configuration caddy.yml.

I already wrote an article Caddy in Docker Swarm. Please go through if you want to learn more.

version: "3.7"

services:
  caddy:
    image: tuneitme/caddy
    ports:
      - "80:80"
      - "443:443"
    networks:
      - caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - /mnt/caddydata:/data
      - /mnt/caddyconfig:/config
      - /mnt/caddylogs:/var/log/caddy
      - /mnt/gitea:/data
    deploy:
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
  drone:
    image: drone/drone:latest
    volumes:
      - /mnt/dronedata:/data
    environment:
      - DRONE_DATABASE_DRIVER=sqlite3
      - DRONE_DATABASE_DATASOURCE=/data/database.sqlite
      - DRONE_RPC_SECRET=drone-client-secret
      - DRONE_COOKIE_TIMEOUT=720h
      - DRONE_CRON_DISABLED=true
      - DRONE_OPEN=true
      - DRONE_NETWORK=caddy
      - DRONE_ADMIN=rajasekhar
      - DRONE_USER_CREATE=username:rajasekhar,admin:true
      - DRONE_SERVER_PORT=:80
      - DRONE_GIT_ALWAYS_AUTH=true
      - DRONE_SERVER_HOST=drone.example.com
      - DRONE_HOST=https://drone.example.com
      - DRONE_SERVER_PROTO=https
      - DRONE_TLS_AUTOCERT=false
      - DRONE_AGENTS_ENABLED=true
      - DRONE_GITEA_SERVER=https://gitea.example.com
      - DRONE_GITEA_CLIENT_ID=gitea-client-id
      - DRONE_GITEA_CLIENT_SECRET=gitea-client-secret
    networks:
      - caddy
    ports:
      - "8005:80"
    deploy:
      placement:
        constraints: [node.role == worker]
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure    
  runner:
    image: drone/drone-runner-docker:latest
    command: agent
    depends_on:
      - drone
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_SERVER=https://drone.example.com
      - DRONE_RPC_HOST=drone.example.com
      - DRONE_RPC_PROTO=https
      - DRONE_RPC_SECRET=drone-client-secret
      - DRONE_RUNNER_CAPACITY=10
      - DRONE_RUNNER_NETWORKS=caddy
      - DRONE_RUNNER_NETWORK_OPTS=com.docker.network.drive.mtu:1442
    networks:
      - caddy
    deploy:
      placement:
        constraints: [node.role == worker]
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure

volumes:
  dronedata:
    driver: "local"
networks:
  caddy:
    external: true
    attachable: true

Here I used a custom Caddy docker container with plugins, like Cloudflare DNS, Caddy Auth Portal etc…

Please find the custom caddy docker image below.

Tuneit Caddy Docker Image

Deploy Drone Stack using Docker Compose

Now it’s time to deploy our docker-compose file above, gitea.yml using the below command

docker stack deploy --compose-file drone.yml drone

In the above command, you have to replace drone.yml with your docker-compose file name and drone with whatever name you want to call this particular application.

With docker compose in docker swarm what ever we are deploying is called as docker stack and it has multiple services in it as per the requirement.

As mentioned earlier I named my docker-compose as drone.yml and named my application stack as drone

Check the status of the stack by using docker stack ps drone

Check drone stack logs using docker service logs drone_drone

One thing we observe is that it automatically re-directs to https with Letsencrypt generated certificate. The information is stored in /data a directory.

I will be using this caddy stack as a reverse proxy / load balancer for the applications I am going to deploy to Docker Swarm Cluster.

Also I use docker network caddy to access the applications externally.

Access / Configure Drone

Now open any browser and type drone.example.com to access the site. it will automatically be redirected to https://drone.example.com/welcome ( Be sure to replace example.com with your actual domain name).

Make sure that you have DNS entry for your application (drone.example.com) in your DNS Management Application.

Please find below images for your reference.

Deployment of Drone behind Caddy in our Docker Swarm is successful

If you enjoyed this tutorial, please give your input/thought on it by commenting below. It would help me to bring more articles that focus on Open Source to self-host.

Stay tuned for other deployments in coming posts…🙄