Set up a Linux server monitoring system using Grafana & Prometheus

Shashank Srivastava

5 min readOct 28, 2021

Follow these easy steps & learn how to monitor your Linux server metrics using Grafana & Prometheus.

Introduction.

Before I actually start this tutorial, I’d like to start with a brief description of the system & its components.

Monitoring system.

Our Linux monitoring system consists of 3 components — Prometheus, Exporter & Grafana. We will use this system to monitor our Linux server metrics.

I have explained these components briefly below.

What is Prometheus?

From Prometheus's official website, it is an open-source system monitoring and alerting toolkit. Prometheus collects and stores its metrics as time-series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

What is Grafana?

As the name suggests, Grafana is the dashboard component of this monitoring system. With Grafana you can create, explore and share all of your data through beautiful, flexible dashboards (taken from the official website).

In the nutshell, Grafana collects the data (metrics) from Prometheus & displays that in form of graphs & charts.

What is an exporter?

An exporter is a library that exports metrics from 3rd party systems, such as Linux servers, to Prometheus. To monitor the Linux servers, we’d need Node Exporter.

What will we monitor?

Using this system, we’ll monitor our Linux server metrics such as CPU, RAM, Disk, Network, etc. These metrics will be displayed on a Grafana dashboard. This dashboard can be set to refresh automatically after a fixed time.

Requirements.

Prometheus & Grafana pre-installed. These 2 can be installed on the same server or host. I installed these on MacBook Pro.
A Linux VM or EC2 instance.

Below is what my setup looks like.

Prometheus version — v 2.30.3
Grafana version — v 8.2.2
Node Exporter — v.1.2.2
Host — macOS Big Sur.
Linux VM — Amazon Linux 2

You can see that I am monitoring a Linux EC2 instance using Prometheus & Grafana set up on macOS.

Steps to perform.

1. Start Prometheus & Grafana service.

If the services are not running, please start the service. The method depends on the OS of your host.

On macOS with Homebrew, you can start the services using…

shashanksrivastava@MBP ~> brew services start grafana
shashanksrivastava@MBP ~> brew services start prometheus
==> Successfully started `prometheus` (label: homebrew.mxcl.prometheus)

2. Install `node_exporter` on the Linux server.

Now, download & install node_exporter on the Linux server you want to monitor.

To install the latest version…

[ec2-user@linux-vm ~] wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz[ec2-user@linux-vm ~] cd node_exporter-1.2.2.linux-amd64/[ec2-user@linux-vm ~] chmod +x node_exporter

3. Install `node_exporter` service.

While you can simply type ./node_exporter inside the Node Exporter directory to start the process, it is better to install the service which can be controlled via systemctl.

You need to do it either as root or a user with sudo access.

Create the below file (node-exporter.service) under /etc/systemd/system directory.

vim /etc/systemd/system/node-exporter.service

Add the following content.

[Unit]
Description=Node Exporter Agent[Service]
User=ec2-user
ExecStart=/home/ec2-user/node_exporter-1.2.2.linux-amd64/node_exporter
Restart=always[Install]
WantedBy=multi-user.target

Pay attention to the Service section. You need to enter the path of node_exporter binary file & specify the user who will own this service.

Save the file & run the below command to register this new service.

systemctl daemon-reload

Start the service.

systemctl start node-exporter.service

Check the status of this service.

[root@linux-vm ec2-user]# systemctl status node-exporter.service
● node-exporter.service - Node Exporter Agent
   Loaded: loaded (/etc/systemd/system/node-exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-10-28 05:46:57 UTC; 1h 42min ago
 Main PID: 2480 (node_exporter)
   CGroup: /system.slice/node-exporter.service
           └─2480 /home/ec2-user/node_exporter-1.2.2.linux-amd64/node_exporter

You should be able to see the collected metrics by executing curl http://localhost:9100/metrics.

4. Configure Prometheus.

Now, edit prometheus.yml file to include Node Exporter metrics endpoint.

The location of this file depends on the OS. On macOS with Homebrew, the location is.

/opt/homebrew/etc/prometheus.yml

Add a new job.

- job_name: Linux_Node
    static_configs:
    - targets: ['IP_of_Linux_VM:9100']

The complete file should look like this.

global:
  scrape_interval: 15sscrape_configs:
  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]- job_name: Linux_Node
    static_configs:
    - targets: ['IP_of_Linux_VM:9100']

Restart prometheus service to load the changes.

shashanksrivastava@MBP ~> brew services restart prometheus
Stopping `prometheus`... (might take a while)
==> Successfully stopped `prometheus` (label: homebrew.mxcl.prometheus)
==> Successfully started `prometheus` (label: homebrew.mxcl.prometheus)

5. Configure Grafana.

If you haven’t already added Prometheus data source in Grafana, please do it now. Go to http://localhost:3000/datasources and click the Add data source button.

6. Load Linux metrics dashboard.

Here comes the last & the most exciting part. There’s already a ready-made dashboard that can be imported to Grafana.

To import it, go to http://localhost:3000/dashboard/import & enter 10180 in the text box. Click the Load button. Choose Prometheus data store & click Import.

7. Explore the metrics.

See Grafana in action. Select the dashboard you just created, wait for a few minutes so that it can draw the graphs/charts & you should be able to visualize your Linux server metrics.

Below are the screenshots from my setup.