Setting up Prometheus

I’ve been building a toy app for fun and games the past two weekends, and one thing I wanted to learn was setting up a simple self hosted metrics platform. I’m running two Golang APIs on a digital ocean VPS: one is an at Proto personal data server, and the other is an at Proto firehose filter app with a websocket pub/sub functionality that is hit by a react SPA hosted on Netlify.

I chose as the metrics I wanted to track messages consumed from firehose, messages sent to sockets, active socket connections and messages by keyword filter. This would form a nice basis for basic usage of the API - I chose Prometheus for its ease of setup within the go ecosystem and paired it with a grafana dashboard for visualizations.

I’ll skip the instrumentation of the app as that is fairly straightforward, involving adding a metrics endpoint and registering prometheus counters and gauges. The only change structurally in the app itself involves adding a metrics endpoint on a separate port from the rest of the app.

On the VPS reverse proxy is handled by caddy with a firewall in front of the unit as well. I combined both apps into a single docker network, allowing caddy to proxy requests to either of them, as well as the websocket calls.

DNS entries were made for two new metrics and grafana subdomains, those proxy to the docker services as well.

Several gotchas which came up during this process:

By moving the caddy proxy into docker the ports had to be changed from their exposed external ports to their internal docker ports

CORS preflight requests had to be handled and headers forwarded to the containers. The caddy syntax for this is a pain in the ass.

Basic Auth was added to the new subdomains. This is a precheck prior to the user auth of the grafana dash itself.

TLS on demand settings kept breaking on Caddyfile so I eventually settled on just providing a TLS directive.

In the end it’s all up and running and I now have a fancy dashboard that tells me nobody is using my site 😂