Selfhosting a cloud. Part 1: Reverse Proxy

Selfhosting a cloud

  1. Basic setup
  2. Reverse Proxy with Traefik v2 (this post)
  3. Secure your services with HTTPS
  4. Nextcloud - everything you need and more
  5. Some nice services you might want
  6. Use keycloak for user management and SSO

Disclaimer

Due to how disorganised I am, there will be a lot of tangents and (more or less) off-topic paragraphs.

Any paragraph that looks like this is one of those paragraphs. You can ignore them and information on this page will still make sense.

I will sometimes use these paragraphs to explain something in more details.

This post assumes you read the previous post

If you want to skip my rambling and go straight to installing and configuring Traefik click here

What is a reverse proxy? Do I need it?

From Wikipedia:

In computer networks, a reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned...

This doesn't tell you much, does it. Let me try illustrate it a bit better.

Let's assume that you want to host multiple applications, like a file server, search engine, blog, password manager or anything else. They all allow you to access them with your browser via HTTP(S). HTTP uses TCP port 80 (port 443 for HTTPS).

HTTPS adds an extra layer of security by encrypting all HTTP traffic using SSL certificates. I'm not an expert on encryption or network security, so I won't be explaining how that works. You can find more information about that on Youtube

A TCP (or UDP) port is an identificator for a protocol. Ports from 0 to 1023 are reserved for specific protocols. DNS uses 53, SSH uses 22 etc. Any application can use any port, as long as it's not already in use.

A port can only be used by one application at a time. However if you have a bunch of web applications they will all want to use port 80. This obviously wouldn't work.

One way to "fix" this is to assign a different port per application. Since we're using docker, it's really easy to do that by just changing the ports: section in docker-compose.yml.

The obvious drawback here is that in order to reach these applications, you have to specify the port along with URL. So instead of opening app.example.com in your browser, you have to open example.com:1234

Reverse Proxy to the rescue

A reverse proxy such as Traefik or HAproxy (and some web servers like Nginx and Apache) sit between you and your apps. It's the only application that uses port 80 on your server. It listens to all HTTP requests and redirects them to your apps according to rules you specify. For example: every HTTP request where the domain name is blog.draganczuk.me is redirected to a docker container with my blog.

Using Traefik v2 as a proxy

What is Traefik

Traefik is a piece of software written in Go specifically to act as a reverse proxy. It has some nice features, like automatically acquiring SSL certificates from Let's Encrypt, very advanced routing rules, and it can react to changes in docker containers on the fly (mostly)

Why not Nginx/Apache/HAProxy/Caddy/anything else?

Until very recently I was using Nginx as my reverse proxy. It was working great, especially since I was using linuxserver/letsencrypt as my reverse proxy container. The problem was when I wanted to secure some of my apps to require logging in using Keycloak as a backend (I'll cover that in a later post).

Turns out that in order to do it you either need a paid version of Nginx, or some kind of custom distribution with extra lua support. Both of these options required quite a lot of work to integrate with the container I was using. After some googling I found that Traefik can be integrated relatively easily with any OpenID Connect provider, like Keycloak.

I was also getting annoyed with having to restart my whole proxy container every time I wanted to add, remove or change an application.

Installing and configuring Traefik

Create docker-compose.yml

Since we're using docker and docker-compose for everything, our proxy will also be defined as a docker-compose stack. Here's a starting docker-compose.yml file:


version: "3"
services:
    traefik:
        image: traefik
        container_name: traefik
        restart: unless-stopped
        ports:
            - 80:80
            - 443:443
Create a new folder (like proxy) in your main servers folder and paste that to docker-compose.yml. Right now everything in this file was explained earlier.

Adding some basic configuration

Docker volumes

We need to provide some files to Traefik. These files are:

We do that by adding the following section to service definition in docker-compose.yml:

volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ./traefik.yml:/etc/traefik/traefik.yml
    - ./data/extra-conf:/extra-conf
The syntax is pretty simple. It's local file : container file. Prepend "./" to local file to use relative path.

Proxy network for containers

Since we want our applications to be able to talk to each other, they need to be on the same network. By default docker-compose creates a new network for every docker-compose.yml file. This means that if we follow best practices and don't put everything in a single file, we won't be able to proxy anything.

To fix this, we need to create a new network which will connect all the services:


docker network create proxy
Now we just need to tell docker-compose to connect traefik to that network by adding this section to service definition:

networks:
    - proxy
As you can see from the syntax, we can specify multiple networks, though we won't do it now.

The last thing to do in docker-compose right now is to tell docker-compose that it's allowed to use that network. Add this section to the end of file, but make sure that "networks:" is NOT indented at all:


networks:
    proxy:
        external: true
This block should have the same indentation level as "services" and "version".

Traefik.yml config file

Although we can provide all this configuration in docker-compose.yml, it's recommended to put it in separate file. This file can be in either YAML or TOML format.

I personally don't like TOML, which looks like a needlessly complicated version of the old MS-INI format.

Here's a basic traefik.yml file configured for usage with docker:



# What ports should we listen on
entrypoints:
    # You can replace "http" here with "web" or any other name you want
    http:
        address: ":80"
    https:
        address: ":443"
# You'll often find these entrypoints called "web" and "web-secure"

# Where will other information be available
providers:
    # Docker containers can set information about themselves
    # This allows traefik to read that information
    docker:
        # Docker socket
        endpoint: "unix:///var/run/docker.sock"
        # We don't want *every* service (like databases) to be exposed
        exposedByDefault: false
        # Access containers with this network
        network: proxy
    # Extra configuration can be stored in this folder
    file:
        directory: /extra-conf
        # watch for changes and reload configuration as needed
        watch: true
I've put some comments to make it easier to understand.

That's about everything you need for a very basic installation of Traefik. Now let's test it with our whoami container from last time.

Testing

Modify the docker-compose for whoami service. Remove the "ports" section and add the exact same "networks:" sections like we did above. Remember to add both service "networks" and global "networks".

Now we need to add traefik configuration to whoami container. We do it by adding "labels" to the container. Add following lines to whoami service definition:


labels:
    # Enable traefik for this container
    - traefik.enable: true
    # We want to route traffic through entrypoint "http" (defined in traefik.yml)
    - traefik.http.routers.whoami.entrypoints: http
    # What URL do we want this app to be available at?
    # Replace with your domain
    - traefik.http.routers.whoami.rule: Host(`example.com`)
    # What port is the container normally using?
    # Whoami listens on port 80. It's not exposed outside,
    # but traefik can route to it
    - traefik.http.services.whoami.loadbalancer.server.port: 80
Now you can docker-compose up -d both whoami and traefik. When you navigate to http://example.com you should see the same whoami information as before.

You should probably save the "label" section (without the comments). You'll be adding it to multiple containers to get traefik to recognise them.

That's it

Right now you should have the exact same whoami functionality on your server as you did before. The difference is that you have a whole lot of complexity that you might think is unnecessary. You'll see why it's useful later, once we start adding services.

Also, you will not need the whoami service after the next post. It's great for testing, but you probably want something more useful on your server, like a portfolio website or a blog.

In the next post I'll explain how to set up HTTPS with SSL certificates from Let's Encrypt. That should be a short post, so it shouldn't take as long to write as this post.