Repository navigation
Update setting up Prometheus docs #5217
Description
Activity
- changed the title
[-]Connection refused to control plane 9090 port[/-][+]Connection refused to control plane metrics port[/+]on Sep 14, 2022 related issue: knative/eventing#6509
resolved by editing configmap
apiVersion: v1 kind: ConfigMap metadata: name: config-observability namespace: knative-serving labels: app.kubernetes.io/name: knative-serving app.kubernetes.io/component: observability app.kubernetes.io/version: "1.7.1" annotations: knative.dev/example-checksum: "fed4756e" data: metrics.backend-destination: prometheus metrics.request-metrics-backend-destination: prometheus
- changed the title
[-]Connection refused to control plane metrics port[/-][+]Prometheus metrics not exported by default[/+]on Sep 14, 2022 I think we need to add below block to setting up prometheus part.
https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus
kubectl patch --namespace knative-serving configmap/config-observability \ --type merge \ --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}' kubectl patch --namespace knative-serving configmap/config-observability \ --type merge \ --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'@DevSusu: The label(s)
area/monitoringcannot be applied, because the repository doesn't have them.Details
In response to this:
/area monitoring
What version of Knative?
1.7.1(installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)Expected Behavior
- install knative-serving
- port-fowarded one of them to debug on prometheus metrics
- pods report metrics described in https://knative.dev/docs/serving/observability/metrics/serving-metrics/
Actual Behavior
kubectl port-forward service/activator-service -n knative-serving 9090:9090 Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 Handling connection for 9090 E0914 16:42:51.809345 3007207 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4f48d8d6708f2f6cc464957b6718d2ec89088951c6430acfb0fb04857e595a94, uid : exit status 1: 2022/09/14 07:42:51 socat[13441] E connect(5, AF=2 127.0.0.1:9090, 16): Connection refused
I had to specify the
metrics.backend-destinationandmetrics.request-metrics-backend-destinationto get this working.Steps to Reproduce the Problem
- follow instructions on https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#prerequisites
- port-forward one of the components with 9090 port exposed
- curl localhost:9090 (same error with curl localhost:9090/metrics)
apiVersion: v1 kind: ConfigMap metadata: name: config-observability namespace: knative-serving labels: app.kubernetes.io/name: knative-serving app.kubernetes.io/component: observability app.kubernetes.io/version: "1.7.1" annotations: knative.dev/example-checksum: "fed4756e" data: {}
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
- changed the title
[-]Prometheus metrics not exported by default[/-][+]Update setting up Prometheus docs[/+]on Sep 14, 2022 10 remaining items
Can I help?
/assign
I think we need to add below block to setting up prometheus part.
https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus
kubectl patch --namespace knative-serving configmap/config-observability \ --type merge \ --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}' kubectl patch --namespace knative-serving configmap/config-observability \ --type merge \ --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'The config map is to be set before we apply the ServiceMonitors/PodMonitors to collect metrics from Knative, right?
/assign
- added a commit that references this issue
on Oct 13, 2023 Please assign me this issue
Hi, I'd like to work on this issue if it is still available. I plan to verify the current Prometheus setup instructions and update the docs with the required config-observability settings for Knative Serving, keeping the change scoped to the documentation and current release behavior. I'll open a PR with validation notes and links to the relevant docs.
psschwei: originally opened in serving and moved here. updates needed for docs can be found in this comment: #5217 (comment)
/area monitoring
What version of Knative?
1.7.1(installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)Expected Behavior
Actual Behavior
I had to specify the
metrics.backend-destinationandmetrics.request-metrics-backend-destinationto get this working.Steps to Reproduce the Problem