Create api-availability measurement #1096

Activity

1. k8s-ci-robot commented on Mar 3, 2020

   

   k8s-ci-robot

   on Mar 3, 2020

   Contributor

   More actions

   @mm4tt:
   This request has been marked as suitable for new contributors.

   Please ensure the request meets the requirements listed here.

   If this request no longer meets these requirements, the label can be removed
   by commenting with the /remove-good-first-issue command.

   Details

   In response to this:

   Justification

   See this comment - #1086 (comment)

   Milestones

   V0

   1. Create a new "ApiAvailability" measurement
   2. This measurement should periodically probe the apiserver's /healthz endpoint and record whether api was available or not
   3. The measurement should output the following stats
      1. availability percentage (e.g. % of OK responses over all responses)
      2. Longest consecutively unavailability period
      3. ...

   V1

   1. Make the measurement fail if apiserver is consecutively not available in the last XX min (e.g. 30min).
   2. Make the error critical to make sure the test execution is stopped in such case

   V2

   1. Come up with exact SLI/SLO definition, make it available in perf-dash for further analysis
   2. Make it a WIP Scalability SLO
   3. Evaluate and promote to official SLO

   /good-first-issue

   Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
2. k8s-ci-robot

   added

   good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.Denotes an issue ready for a new contributor, according to the "help wanted" guidelines.

   help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

   on Mar 3, 2020
3. vamossagar12 commented on Mar 13, 2020

   

   vamossagar12

   on Mar 13, 2020

   Last edited by vamossagar12

   Contributor

   More actions

   /assign
   I would like to take this up. Can I expect more details?
4. k8s-ci-robot

   assigned

   vamossagar12

   on Mar 13, 2020
5. mm4tt commented on Mar 18, 2020

   

   mm4tt

   on Mar 18, 2020

   ContributorAuthor

   More actions

   Great, thanks @vamossagar12

   Let's start with the V0. Could you let me know which part of the description is not clear or requires further details?
6. vamossagar12 commented on Mar 18, 2020

   

   vamossagar12

   on Mar 18, 2020

   Contributor

   More actions

   thanks @mm4tt Couple of things:

   1. how frequently should the probe happen? Would it be configurable or we can define it to be fixed?
   2. The availability percentage should be wrt the entire perf test run and we have to keep track of percentage and longest unavailable period?

   As an outside question, typically if the health end point is down for a configured time, response would be to scale up or something like that. Just wanted to know the rationale behind adding this particular measurement. Thanks!
7. wojtek-t commented on Mar 19, 2020

   

   wojtek-t

   on Mar 19, 2020

   Member

   More actions

   how frequently should the probe happen? Would it be configurable or we can define it to be fixed?

   Seems like implementation detail. We can make it configurable, but I doubt we will be using different values once we agree on sth.

   The availability percentage should be wrt the entire perf test run and we have to keep track of percentage and longest unavailable period?

   Correct

   As an outside question, typically if the health end point is down for a configured time, response would be to scale up or something like that. Just wanted to know the rationale behind adding this particular measurement. Thanks!

   There are couple points:

   • we want to ensure that we understand how availability looks like (I can imgine all other SLOs being fine, but cluster being periodicall unavailable)
   • we can use it as optimization - if cluster is down for X minutes, the it probably won't get up ever, and we can shut down the test and save money

   👍React with 👍1Reacted by Matt Matejczyk
8. vamossagar12 commented on Mar 21, 2020

   

   vamossagar12

   on Mar 21, 2020

   Contributor

   More actions

   Thanks for the updates @wojtek-t . I will start with this issue now. Pretty sure there would be a few more questions along the way though :)
9. vamossagar12 commented on Mar 26, 2020

   

   vamossagar12

   on Mar 26, 2020

   Last edited by vamossagar12

   Contributor

   More actions

   hi. Since I last commented, I didn't get a chance to look at it. Will start over the next couple of days..

   👍React with 👍1Reacted by Matt Matejczyk
10. vamossagar12 commented on Mar 29, 2020

    

    vamossagar12

    on Mar 29, 2020

    Contributor

    More actions

    hi so i started looking at this today. One of the things is that there's already a measurement metrics_for_e2e which, among other things, also fetches the metrics from api-server.

    So, the new measurement that needs to be created, can work along the same lines just that instead of hitting the /metrics end-point, it can hit the /healthz end-point of api-server. Thats' from an implementation standpoint.

    The other question that I had is that the measurements that we defines are within a Step and a group of Steps forms a single test, so when we say that this new measurement measures the health of the api-server for the duration of a test(mentioned above), so what exactly does a test mean in this case? A Step which houses the measurement(s) or the overall test?
11. mm4tt commented on Mar 30, 2020

    

    mm4tt

    on Mar 30, 2020

    ContributorAuthor

    More actions

    Hey, @vamossagar12

    It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

    Regarding your second question. Usually a measurement has two actions: start and gather. This one should work the same. The start action should start a goroutine that will be continuously pinging /healthz endpoint and the gather will stop this goroutine and wrap everything up. Does it make sense?
12. wojtek-t commented on Mar 30, 2020

    

    wojtek-t

    on Mar 30, 2020

    Member

    More actions

    It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

    Not sure I understand - for metrics_for_e2e IIRC we're fetching the metrics once at the end of test. Assuming that the above is true - it can't really work the same way...
13. mm4tt commented on Mar 30, 2020

    

    mm4tt

    on Mar 30, 2020

    ContributorAuthor

    More actions

    It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

    Not sure I understand - for metrics_for_e2e IIRC we're fetching the metrics once at the end of test. Assuming that the above is true - it can't really work the same way...

    Good point, I should have checked how the metrics_for_e2e works :) Still you should be able to take some inspiration from that measurement. Let me know if you have more questions.
14. vamossagar12 commented on Mar 30, 2020

    

    vamossagar12

    on Mar 30, 2020

    Contributor

    More actions

    Actually I meant to use the metrics_e2e as only a baseline of how to interact with the api server. I hadn't looked at the internals so what @wojtek-t told is even more valuable :)
15. 34 remaining items

    Load more

16. k8s-triage-robot commented on May 22, 2022

    

    k8s-triage-robot

    on May 22, 2022

    More actions

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle stale
    • Mark this issue or PR as rotten with /lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale
17. k8s-ci-robot

    added

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.

    on May 22, 2022
18. wojtek-t commented on May 23, 2022

    

    wojtek-t

    on May 23, 2022

    Member

    More actions

    /remove-lifecycle stale
19. k8s-ci-robot

    removed

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.

    on May 23, 2022
20. k8s-triage-robot commented on Aug 21, 2022

    

    k8s-triage-robot

    on Aug 21, 2022

    More actions

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle stale
    • Mark this issue or PR as rotten with /lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale
21. k8s-ci-robot

    added

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.

    on Aug 21, 2022
22. k8s-triage-robot commented on Sep 20, 2022

    

    k8s-triage-robot

    on Sep 20, 2022

    More actions

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle rotten
23. k8s-ci-robot

    added

    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.Denotes an issue or PR that has aged beyond stale and will be auto-closed.

    and removed

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.

    on Sep 20, 2022
24. wojtek-t commented on Sep 22, 2022

    

    wojtek-t

    on Sep 22, 2022

    Member

    More actions

    /remove-lifecycle rotten
25. k8s-ci-robot

    removed

    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.Denotes an issue or PR that has aged beyond stale and will be auto-closed.

    on Sep 22, 2022
26. k8s-triage-robot commented on Feb 8, 2023

    

    k8s-triage-robot

    on Feb 8, 2023

    More actions

    The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

    This bot triages PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

    You can:

    • Mark this PR as fresh with /remove-lifecycle stale
    • Close this PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale
27. k8s-ci-robot

    added

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.

    on Feb 8, 2023
28. wojtek-t commented on Feb 8, 2023

    

    wojtek-t

    on Feb 8, 2023

    Member

    More actions

    /remove-lifecycle rotten
29. vamossagar12

    removed their assignment

    on Jun 2, 2023