File based Service Discovery with Prometheus

Prometheus does provide many configs for service discovery. It’s very easy to discover the targets from consul, kubernetes & mesos, etc.However, there are certain use cases where we maybe need to add static configs. With growing infrastructure and monitoring targets, it may be possible that the static configs will become hard to maintain. Because of this file-based discovery is a convenient way to manage such configs.

Its very simple and only three-step is needed to configure this:

1. Create a YAML or JSON file which consist of the target’s
2. Configure the Prometheus to scrape the targets from a file
3. Update the file whenever you want to add or replace the targets

Create YAML or JSON file :
Folder Structure:

# Prometheus Installed location
$ /opt/prometheus
# Location for the file 
$ /opt/prometheus/sd_configs/dev/logstash.json

logstash.json

[
  {
    "labels": {
      "job": "logstash1",
      "env": "dev"
    },
    "targets": [
      "localhost:9501",
      "localhost:9502",
      "localhost:9503"
   ]
  }
]

Let’s understand this file:

Labels: First it’s possible to add labels that are custom and Prometheus will ingest that into metrics.

Targets: Multiple targets can be added that are to be scraped. However here Prometheus will look for “/metrics” endpoints of the targets. So Prometheus will scrape from an example: http://localhost:9501/metrics

Configure the Prometheus to scrape:

scrape_configs:
- job_name: 'logstash-fd'
  file_sd_configs:
  - files:
    - 'sd_configs/dev/*.json' #This path is relative

Once configured Prometheus configuration reloaded if Prometheus is already running or started if it’s stopped.

If additional targets to be added then it will only need to update the file with potential targets. Prometheus will reload the file config when it discovers file content is changed.

Pros of using File-Based Discovery:

1. No need to reload Prometheus like static configs. Especially this is really in the Production environment.

2. Custom labels can be added with a file.

3. Single point of file to be maintained. And it is recommended if git is used for maintaining those files.

Cons of using File-Based Discovery:
1. In container based environment volume is needed to add to maintain the file.

Recommendation: In production always carefully update the file content. Before update validate the JSON or YAML file.

Do check different tips on prometheus here: https://kuberneteslab.com and also here on medium.