Prometheus 學習筆記-2

Devops > Prometheus .NetCore Prometheus
發布時間 :
字數:1.2k 觀看數 : 評論:
  1. 前言
  2. 作業環境
  3. PushGateway 介紹
  4. 設定說明
  5. .Net Core 專案設定
  6. 操作流程
  7. 刪除 PushGateway Metrics
  8. 參考文件

前言

延續上一篇 w4560000 - Prometheus 學習筆記-1

本篇記錄如何透過 .Net Core 主動發送 Metrics 至 PushGateway
而 Prometheus 再去 Pull PushGateway 的 Metrics

作業環境

Windows 10 Professional (22H2)
Docker Desktop
Docker Compose

Dotnet 版本: .Net Core 3.1
Nuget 套件: prometheus-net.AspNetCore 4.1.1

PushGateway 介紹

以往 Prometheus 是定時透過 Pull 的方式，由 Prometheus 到各個站台去撈取 Metrics
但有時若有網路層級問題，或者站台本身執行或存活時間極短， 導致 Prometheus 沒辦法及時連線到各個站台撈取 Metrics時，
這時就會需要 PushGateway 來當作中間層協助暫存資料

PushGateway 是 Prometheus 的一個組件，可由各個站台主動發送 Metrics 至 PushGateway 來暫存資料
而 Promethes 再定期 Pull PushGateway 的 Metrics 即可

但仍有一點需注意
舊數據若無需使用，需手動清理

設定說明

可參考原始碼 Github - w4560000
原始碼資料夾樹狀結構如下

.
├── docker-compose.yml
├── telegraf/
│   ├── telegraf.d
│   │   ├── amazon.conf
│   │   └── github.conf
│   └── telegraf.conf
└── prometheus/
    ├── prometheus.yml
    └── dotnet-core-exporter/  (新建立的 .net core 3.1 專案)

docker-compose.yml
新增 dotnet-core-exporter、Promethus pushgateway 兩個服務

version: "3.0"

services:
  telegraf:
    image: telegraf:1.16.0
    restart: always
    container_name: telegraf
    hostname: telegraf
    ports:
      - 9273:9273
    volumes:
      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf
      - ./telegraf/telegraf.d/:/etc/telegraf/telegraf.d/
    command: telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d

  prometheus:
    image: prom/prometheus:latest
    restart: always
    container_name: prometheus
    hostname: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    command: --config.file=/etc/prometheus/prometheus.yml

+  dotnet-core-exporter:
+    build:
+      context: ./prometheus/dotnet-core-exporter
+      args:
+        ProjectName: dotnet-core-exporter
+    restart: always
+    container_name: dotnet-core-exporter
+    hostname: dotnet-core-exporter
+    ports:
+      - 8787:80
+
+  pushgateway:
+    image: prom/pushgateway:latest
+    restart: always
+    container_name: pushgateway
+    hostname: pushgateway
+    ports:
+      - 9091:9091

prometheus.yml

新增 Prometheus 排程去抓取 PushGateway 的 Metrics

因要主動去抓 PushGateway 的 Metrics，故 dotnet-core-exporter 就先不主動 Pull 了，避免 Metrics 重複

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "telegraf"
    static_configs:
      - targets: ["telegraf:9273"]

  # - job_name: "dotnet-core-exporter"
  #   static_configs:
  #     - targets: ["dotnet-core-exporter:80"]

+  - job_name: "pushgateway"
+    static_configs:
+      - targets: ["pushgateway:9091"]

.Net Core 專案設定

  1. 建置一個 .Net Core 3.1 Web API 專案
  2. 安裝 Nuget 套件 prometheus-net.AspNetCore 4.1.1
  3. 調整 Startup.cs
...略
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

+   app.UseMetricServer();
+   app.UseHttpMetrics();

    app.UseHttpsRedirection();

    app.UseRouting();

    app.UseAuthorization();

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
    });
}
...略
  1. 調整 WeatherForecastController.cs
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using Prometheus;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

namespace dotnet_core_exporter.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private readonly Counter count = Metrics.CreateCounter("weatherforecast_get_request_count", "Total number of requests");

        private static readonly string[] Summaries = new[]
        {
            "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
        };

        private readonly ILogger<WeatherForecastController> _logger;

        public WeatherForecastController(ILogger<WeatherForecastController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IEnumerable<WeatherForecast> Get()
        {
            count.Inc();
            var rng = new Random();
            return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateTime.Now.AddDays(index),
                TemperatureC = rng.Next(-20, 55),
                Summary = Summaries[rng.Next(Summaries.Length)]
            })
            .ToArray();
        }

        [HttpGet("PushGateway")]
        public async Task<string> PushGateway()
        {
            string msg = "";
            try
            {
                string job = "job test";
                using (var st = new MemoryStream())
                {
                    await Metrics.DefaultRegistry.CollectAndExportAsTextAsync(st);

                    string metricsText = Encoding.UTF8.GetString(st.ToArray());

                    var myCounterText = GetMetricText(metricsText, "weatherforecast_get_request_count");

                    using (var httpClient = new HttpClient())
                    {
                        var pushUrl = $"http://pushgateway:9091/metrics/job/{job}/instance/dotnet-core-exporter";
                        // prometheus pushgateway 傳送的參數 最後要斷行
                        // 否則會回傳 text format parsing error in line 3: unexpected end of input stream 的錯誤
                        var content = new StringContent(myCounterText + "\n");

                        var response = httpClient.PostAsync(pushUrl, content).Result;

                        if (response.IsSuccessStatusCode)
                        {
                            Console.WriteLine("weatherforecast_get_request_count pushed successfully!");
                        }
                        else
                        {
                            Console.WriteLine($"Failed to push weatherforecast_get_request_count. Status code: {response.StatusCode}");
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                msg = ex.ToString();
            }

            return string.IsNullOrEmpty(msg) ? "Success" : msg;
        }

        private static string GetMetricText(string allMetricsText, string metricName)
        {
            List<string> metricsList = allMetricsText.Split('\n').ToList();

            return string.Join('\n', metricsList.Where(x => x.Contains(metricName)));
        }
    }
}

操作流程

docker-compose up -d

# 確認一下服務
docker-compose ps

# 輸出
CONTAINER ID   IMAGE                                    COMMAND                   CREATED          STATUS          PORTS                                                  NAMES
05792a3d7fed   prometheusgrafana-dotnet-core-exporter   "/bin/sh -c 'dotnet …"   15 seconds ago   Up 11 seconds   0.0.0.0:8787->80/tcp                                   dotnet-core-exporter
fb244eaee4fd   prom/prometheus:latest                   "/bin/prometheus --c…"   15 seconds ago   Up 13 seconds   0.0.0.0:9090->9090/tcp                                 prometheus
57a97429b0d9   telegraf:1.16.0                          "/entrypoint.sh tele…"   15 seconds ago   Up 11 seconds   8092/udp, 8125/udp, 8094/tcp, 0.0.0.0:9273->9273/tcp   telegraf
a6e2b159e35a   prom/pushgateway:latest                  "/bin/pushgateway"       15 seconds ago   Up 12 seconds   0.0.0.0:9091->9091/tcp                                 pushgateway

查看 Prometheus Target 狀況，確認服務正常後即可開始測試流程
http://localhost:9090/targets

  1. 呼叫 http://localhost:8787/WeatherForecast
    讓測試Metrics weatherforecast_get_request_count 自行加總

  2. 呼叫 http://localhost:8787/WeatherForecast/PushGateway
    手動拋送 weatherforecast_get_request_count 的 Metrics 至 PushGateway

  3. 確認 PushGateway Metrics 是否有收到
    http://localhost:9091/metrics

  4. 確認 Prometheus 是否有接收到來自 PushGateway 的 Metrics
    http://localhost:9090/

  5. 至此測試結束

刪除 PushGateway Metrics

刪除所有 {job=”some_job”,instance=”some_instance”} 的 Metrics

curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

刪除所有 {job=”some_job”} 的 Metrics

備註: 並不包含 {job=”some_job”,instance=”some_instance”} 的 Metrics

  curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job

參考文件

Github - Prometheus PushGateway

轉載請註明來源，若有任何錯誤或表達不清楚的地方，歡迎在下方評論區留言，也可以來信至 leozheng0621@gmail.com
如果文章對您有幫助，歡迎斗內(donate)，請我喝杯咖啡

斗內💰

©2022-2022 LeoZheng

Built with Hexo and 3-hexo theme

×

歡迎斗內

github