Thursday, June 1, 2017

Kicking the tires of Prometheus using Docker on Fedora

Straight from the Prometheus documentation: "Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud."

I haven't had a chance to even take a look at using Prometheus.  Here I'll go over the steps I had to follow to get a working local Prometheus install monitoring my local Docker daemon so I could see metrics through the Prometheus dashboard.

First things first, here are the versions of what I am using (eh, until we find out what the problem is (listed below)):
  • Fedora 25
  • Docker
    • docker-1.12.6-6.gitae7d637.fc25.x86_64
    • docker-common-1.12.6-6.gitae7d637.fc25.x86_64
    • docker-latest-1.12.6-2.git51ef5a8.fc25.x86_64
    • Prometheus
      • prom/prometheus b0195cb1a666
    So, there were a couple of places I went for documentation to get started:



    So, following those docs, I tried to use the default Fedora Docker configuration.  That did not work.  The Docker documentation was off, at least for the version of Docker I am using.  By default, in Fedora, you get a Docker package that is a bit out of date.  Here are the steps I took and what I had to do as a workaround.

    First, grab the latest version of Prometheus from Docker hub.

    $ sudo docker pull prom/prometheus

    Start the Prometheus container and test dashboard access.

    Create a /tmp/prometheus.yml file with the following contents:

    # my global config
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
      # Attach these labels to any time series or alerts when communicating with
      # external systems (federation, remote storage, Alertmanager).
          monitor: 'codelab-monitor'
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
      # - "first.rules"
      # - "second.rules"
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
          - targets: ['localhost:9090']
      - job_name: 'docker'
             # metrics_path defaults to '/metrics'
             # scheme defaults to 'http'.
          - targets: ['localhost:9323']

    Run the Docker container

    $ sudo docker run -dt -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml

    The next thing to do is enable the Docker daemon to expose and endpoint that Prometheus can monitor.  This is where the Docker docs are off a bit.  They suggest that you create a /etc/docker/daemon.json with the following configuration.

      "metrics-addr" : "",
      "experimental" : true

    The issue with that is, Docker doesn't respect that configuration.  So, instead, I chose to use the latest version of Docker in Fedora.  To do this, I went to the Docker website for instructions. Following those got me to these versions:

    $ dnf list docker-ce.x86_64  --showduplicates | sort -r
    Last metadata expiration check: 0:35:28 ago on Thu Jun  1 16:20:35 2017.
    Installed Packages
    docker-ce.x86_64               17.05.0.ce-1.fc25                docker-ce-edge 
    docker-ce.x86_64               17.05.0.ce-1.fc25                @docker-ce-edge
    docker-ce.x86_64               17.05.0.ce-1.fc25                @docker-ce-edge
    docker-ce.x86_64               17.04.0.ce-1.fc25                docker-ce-edge 
    docker-ce.x86_64               17.03.1.ce-1.fc25                docker-ce-stable
    docker-ce.x86_64               17.03.0.ce-1.fc25                docker-ce-stable
    Available Packages

    Now that I had those, I could go back to the previous documentation from Docker to get started with Prometheus.  That link is here: and for the most part it worked.  The issue was, I could start the container, but when I tried to hit the Prometheus target of http://localhost:9090/targets, I received the following error:

    Get http://localhost:9323/metrics: dial tcp getsockopt: connection refused

    It has the "Docker" target in a "Down" state.  Through a bit more investigation, I was able to change 2 things to enable it:

    1. Change the /etc/docker/daemon.json from to
    2. Restart the Prometheus image with --network=host

    Now I have the target as up, and a graph with Docker attributes that I can start monitoring.  Check out the graph below.

    At this point, I can now start evaluating Prometheus and start learning more about it.

    Thanks for hanging in there.  If you have any questions or comments, please do engage below.

    No comments:

    Post a Comment