Designing lean operators

What is the URL of the document?

https://sdk.operatorframework.io/docs/best-practices/common-recommendation/

Which section(s) is the issue in?

This is about adding a new section

What needs fixing?

One of the pitfalls that many operators are failing into is that they watch resources with high cardinality like secrets possibly in all namespaces. Besides the security aspect, which may be worth mentioning this has a massive impact on the memory used by the controller on big clusters. Such resources should either be filtered by label or possibly avoided by for instance using a custom resource instead.

Also it may be worth mentioning ResyncPeriod. Setting it to low would also cause the unnecessary consumption of CPU cycles.

Additional context

This is a follow up of: https://github.com/operator-framework/operator-sdk/pull/5353/files#r753112161

This sounds like a great idea to add to the best practices section in the docs. Would you be interested in contributing this?

Related info on the c-r:
https://github.com/kubernetes-sigs/controller-runtime/blob/master/designs/use-selectors-at-cache.md

@fgiloux

@fgiloux just wanted to follow up on this issue. Would you like to contribute on this, or have any suggestions/tips which we could add in the documentation.

@varshaprasad96

@varshaprasad96 yes I would like to contribute. I have however a couple of things higher on my priority list right now, so that I am also ok with another person taking the lead on the subject if someone happens to have more free cycles.
For providing good guidance I would need to experiment, which requires time.

The link shared by @tlwu2013 is really to the point.

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

/remove-lifecycle rotten

/remove-lifecycle stale

/remove-lifecycle rotten

Based on the discussion, the required task for this issue to include docs on how to filter caches using labels in controller-runtime. The original design doc is here: https://github.com/kubernetes-sigs/controller-runtime/blob/master/designs/use-selectors-at-cache.md

This slide from Kubecon may be helpful

@rocrisp

I've taken this as assigned on behalf of @rocrisp who's not yet a member of the org.

@rocrisp

Hey @rocrisp,

We are experiencing high memory usage and are hoping that this guide helps resolve that. I've read https://sdk.operatorframework.io/docs/best-practices/designing-lean-operators/ and I'm bit unclear on a few points.

My use case is an operator with multiple controllers for several Custom Resources (Environments and Sites) that are in multiple namespaces across the cluster.

Controller A is configured For(Environments) with Owns(Secrets) and Owns(ConfigMaps)
Controller B is configured For(Sites) with Owns(Secrets) and Owns(ConfigMaps)

The guide says we can override the default cache with NewCache and the example shows &corev1.Secret{}

Does the new cache behave like the default cache but with an override for secrets or does it only cache secrets and the developer must add every resource the controller watches?
The Secrets and Configmaps mentioned in my example above have ownerReferences but it looks like only Labels and Fields are supported. Is the recommendation to add a label to all Secrets and Configmaps owned by the operator or is there a way to use the ownerReference with cache.SelectorsByObject?

And thanks for the guide! It's very helpful ⭐

@blkperl

@blkperl Yes, the new cache behaves like the default cache but with an override for secrets per labels you create. The watch is in the controller code so not the same as the filter code in main.go when you setup a newManager.
In essence, the controller and the filter filters are two different things. no?
In theory, you should be able to add ownerRefence to the secret label.
Thank you for your feedback on the doc and with your help we can update the doc to make it more clear.

@asmacdo

@asmacdo Can you shed more light on the question above? Thank you.

I meant utilizing the existing ownerReference as a field rather than copying it to a label. It's tricky though because it doesn't seem queryable in golang.

k get secret env-config -o json | jq '.metadata.ownerReferences[0].kind'
"Environment"

The approach that I think we might take is using labels.NewRequirement since we have an existing label with an internal id and we don't care about the value.

	envIdReq, err := labels.NewRequirement("environment-id", selection.Exists, []string{})
	if err != nil {
		// handle error
	}
	envIdSelector := labels.NewSelector()
	envIdSelector = envIdSelector.Add(*envIdReq)

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        // other options ....
	NewCache: cache.BuilderWithOptions(cache.Options{
			SelectorsByObject: cache.SelectorsByObject{
				&corev1.ConfigMap{}: {
					Label: envIdSelector,
				},
			},
	}),
})

@blkperl

@blkperl this approach looks right to me.
If it doesn't work come by the channel and someone will help.

If it works, please consider adding anything to the docs that would have helped you :)

Navigation Menu

Repository navigation

Designing lean operators #5382

What is the URL of the document?

Which section(s) is the issue in?

What needs fixing?

Additional context

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Designing lean operators #5382

Description

What is the URL of the document?

Which section(s) is the issue in?

What needs fixing?

Additional context

Activity

jberkhahn commented on Nov 22, 2021

tlwu2013 commented on Nov 22, 2021

varshaprasad96 commented on Nov 29, 2021

fgiloux commented on Nov 29, 2021

openshift-bot commented on Feb 27, 2022

openshift-bot commented on Mar 29, 2022

fgiloux commented on Mar 30, 2022

camilamacedo86 commented on Mar 30, 2022

fgiloux commented on Mar 31, 2022

16 remaining items

varshaprasad96 commented on Oct 12, 2022

asmacdo commented on Oct 31, 2022

asmacdo commented on Nov 2, 2022

blkperl commented on Jan 20, 2023

rocrisp commented on Jan 20, 2023

rocrisp commented on Jan 20, 2023

blkperl commented on Jan 20, 2023

asmacdo commented on Jan 21, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions