Tutorial Installing Caddy reverse proxy

[spindrift]

Hi there! This is my first post, and SNB has been massively helpful in my decision to get an ASUS router (AX86U) as well as integrating it into my home network.
I know that nginx is available as a Merlin add-on, but I've become quite smitten with caddy as it automatically generates certs and I generally find the config syntax a lot easier.
I wanted to see if I could install caddy on my router, and surprisingly, it worked! Here are the steps I used for anyone else interested. (If you see anything incorrect or inadvisable, please do let me know, though I only ask you do so constructively!)

Assumptions/Requirements

Requirement	Rationale
JFFS is on and persistent	If you want to keep Go on your system, you'll need to export variables in /jffs/scripts/init-start.
USB drive is mounted	tmpfs will run out of memory if we try to build XCaddy inside it.
EntWare is installed on USB at /opt	We'll be using the EntWare folder to store Caddy, and the instructions assume the default install path of /opt.
Web Access from WAN is disabled	Caddy needs to be the only thing listening for web connections over WAN.
Local Access Config has ports changed away from 80/443	Not necessary if you want Caddy to use different ports.
Dynamic DNS enabled	Any reverse proxy requires a way for WAN traffic to be forwarded to it.
Port Forwarding set to forward 80/443 to router	(Or whatever ports you want Caddy to accept connections on.)

All steps below assume you're logged into the router over SSH.

Install Go

opkg install go

Folders and variables: Go
By default, Go sets itself up in $HOME, which by default is /root on tmpfs. We need to relocate it to the USB drive, and set environment variables so the Go build system knows where to find itself and its build system.

mv $HOME/go /opt/home # So we don't run out of space
export GOROOT=/opt/bin/go # Go is here
export GOPATH=/opt/home/go # XCaddy will go here
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin # So we can access both
export TMPDIR=/opt/tmp # Temporary build folder for Go

Folders and variables: Caddy
Caddy uses $XDG_DATA_HOME and $XDG_CONFIG_HOME to set its own environment. We want to keep it on /opt, and make sure it knows that.

mkdir /opt/var/log/caddy # Caddy logs will go here (need to be referenced in Caddyfile)
mkdir /opt/share/caddy # Caddy certs will go here
mkdir /opt/etc/caddy # Caddyfile will go here
# This sets Caddy's root
cat > /opt/etc/caddy/Caddyfile << EOF
{
  storage file_system {
    root /opt/share/caddy
  }
}
EOF

export XDG_DATA_HOME=/opt/share
export XDG_CONFIG_HOME=/opt/etc

Installing XCaddy and building Caddy

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

# Replace with your own desired plugins, or none. Only --output is required.
# transform-encoder is required to make Caddy output common_log, which is required for Fail2Ban etc.
# And if you want Caddy to generate certs automatically, you DO need a plugin for a supported DNS handler.[/SIZE]

xcaddy build --output /opt/bin --with github.com/caddy-dns/cloudflare --with github.com/caddyserver/transform-encoder

Installing as a service

We'll use the EntWare init script format to let us easily run Caddy in the background, restart, and check on it.

cat > /opt/etc/init.d/S98caddy << EOF
#!/bin/sh

ENABLED="yes"
PROCS="caddy"
ARGS="start --environ --config $XDG_CONFIG_HOME/caddy/Caddyfile"
WORK_DIR="$XDG_DATA_HOME/caddy"
DESC=$PROCS
PREARGS=""
PRECMD=""
POSTCMD=""

. /opt/etc/init.d/rc.func
EOF

Persisting variables

cat >> /jffs/scripts/init-start << EOF
# For Caddy
export XDG_DATA_HOME=/opt/share
export XDG_CONFIG_HOME=/opt/etc

# Only if you want to keep the Go build system around
export GOROOT=/opt/bin/go
export GOPATH=/opt/home/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
export TMPDIR=/opt/tmp
EOF

Starting Caddy

Before you start it, you'll want to make sure you build your Caddyfile and check it with caddy fmt (while in the same folder as the Caddyfile).

/opt/etc/init.d/S98caddy start

Troubleshooting

caddy start runs Caddy in the background, which keeps us from seeing its runtime logs (which only seem to show up on stdout).
If Caddy is dying unexpectedly, or something otherwise seems off, run /opt/etc/init.d/S98caddy stop (or killall caddy) to quit the service, then manually run caddy run --config /opt/etc/caddy/Caddyfile to see what it's doing. When you're satisfied, you can ^C and go back to running it through the init script.

If Caddy isn't working and caddy fmt says your config file is off, but it looks okay, try checking for random spaces around the {} brackets and deleting them.

Example Caddyfile

Just for fun, here's an example Caddyfile with DNS settings, logging, and a couple reverse proxy entries, one of which has CORS set and the other does not.
It's recommended to use wildcard domains for reverse proxying, so you don't leak your servers to SSL registrars, although sometimes that also means you have to set up CORS for things like server dashboards that use IFrames.
Also note that whatever you're using for your DNS, you have to set up a CNAME entry that points to your DDNS domain for every subdomain you wish to reverse proxy.

{
        storage file_system {
                root /opt/share/caddy
        }
}
*.domain.com {
        tls {
                dns cloudflare <cloudflare DNS key>
        }
        log {
                output file /opt/var/log/caddy/<domain>-access.log
                format transform "{common_log}"
        }
        @app host app.domain.com
        handle @app {
                reverse_proxy <server IP>:<port> {
                        header_down Access-Control-Allow-Origin *
                }
        }
        @another_app host anotherapp.domain.com
        handle @another_app {
                reverse_proxy <server IP>:<port>
        }
}

Hope this helps! As I mentioned, I'm very new to working in AsusWRT, so please be kind.

 

[Jeffrey Young]

Welcome to the forums and thank for your contribution.

 

[L&LD]

Welcome to the forums @spindrift.

A kind thank you from me as well. Looking eagerly to your future contributions here too!

 

[hexcallm]

Just wanted to add a tip in case anyone else is hung up by this. If you have an older model (AC88U in my case), you may need to install go_nohf:

opkg install go_nohf

Try it if you're getting "illegal instruction" when running go.

Edit:

Of course I immediately hit another compatibility issue. The current @latest version of xcaddy (v0.3.5 from August 2023) is not compatible with the version of go (1.20.7) that is available in the entware go_nohf package. This is due to Go 1.21 (released August 2023) introducing the 'toolchain' directive, which caddy appears to use. After a bumbling attempt to builddirectly using go, I tried an older version of xcaddy.

To make it work, use a prior version tag in the command outined by spindrift above to install xcaddy.

go install github.com/caddyserver/xcaddy/cmd/xcaddy@v0.3.4

In my case, v0.3.4 worked (of course it took 1.5 hours to build and interrupted my connectivity--makes me think I need an armv5 container/vm running on a more robust machine if I have to do this again). For reference, the xcaddy releases page: https://github.com/caddyserver/xcaddy/releases

PS, thanks to spindrift. (And side note, I also enjoy your sparkling water.)