- Learn about monitoring & observability concepts.
- Learn about Prometheus, Loki, Grafana, Promtail.
We need to make our system more observable, to do so we will start by setting up a PLG(Promtail, Loki, Grafana) stack along with Prometheus for metrics monitoring.
We will monitor our application along with its dependent services. For monitoring DB we will use the appropriate DB metrics exporter, and configure Prometheus to scrape metrics from the DB metrics exporter.
We will also monitor latency, uptime, etc. for endpoints such as the ArgoCD server, Hashicorp Vault, and our REST API.
The following expectations should be met to complete this milestone.
- Prometheus, Loki, and Grafana should be deployed on the
dependent_servicesnode in observability namespace. Here’s the deployment diagram for reference.
- Promtail should be configured to send only application logs to Loki.
- DB metrics exporter should be run and configured to monitor DB.
- Blackbox exporter should be run and configured to monitor internal endpoints.
- Prometheus should also scrape kube-state metrics along with node metrics.
- Grafana should have two data sources for Loki (logs) and Prometheus (metrics).
- Helm charts and configs for deploying the observability stack should be committed in the same GitHub repository at the proper filepath.
- README.md should be updated with appropriate instructions to set up these components in a Kubernetes cluster.