Add ability to set tolerations and affinities on resources generated by Karmada resources

[brandonrjacobs]

What would you like to be added:
I would like to add the ability to add add explicit tolerations and affinity blocks to the underlying resources for karmada when creating a CR.

Examples of said additions could look like the following:

spec:
  components:
    etcd:
      local:
        tolerations: 
        - effect: NoSchedule
          operator: Exists
        affinity: {} # Add affinity block here 
        imagePullPolicy: IfNotPresent
        imageRepository: registry.k8s.io/etcd
        imageTag: 3.5.16-0
        priorityClassName: system-node-critical
        replicas: 1
        resources: {}
        volumeData:
          volumeClaim:
            metadata:
              name: etcd-data
            spec:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 3Gi
    karmadaScheduler:
      imagePullPolicy: IfNotPresent
      imageRepository: docker.io/karmada/karmada-scheduler
      imageTag: v1.13.0
      priorityClassName: system-node-critical
      replicas: 1
      resources: {}
      tolerations: 
      - effect: NoSchedule
        operator: Exists
      affinity: {} # Add affinity block here 

Why is this needed:
In some environments, when installing the control plane we have specific taints / settings on our nodes that require adding additional settings. We could solve this by injecting mutatingwebhook policies but that seems like a lot of overhead for a simple ask of supporting traditional kubernetes resource fields.

[zhzhuang-zju]

In some environments, when installing the control plane we have specific taints / settings on our nodes that require adding additional settings. We could solve this by injecting mutatingwebhook policies but that seems like a lot of overhead for a simple ask of supporting traditional kubernetes resource fields.

@brandonrjacobs Thanks, that's a reasonable usage scenario. It can be implemented by extending the CommonSettings of the Karmada API. BTW, may I ask what you are using the karmada-operator for?

[brandonrjacobs]

@zhzhuang-zju I'm attempting use the operator to manage the Karmada Control Plane within a given host cluster. Using the classic Kubernetes within Kubernetes approach for deployment.

[brandonrjacobs]

I will attempt to PR this change in the coming weeks.

[zhzhuang-zju]

I will attempt to PR this change in the coming weeks.

Great! Take your time and feel free to ask if you have any other issues.

[RainbowMango]

Thanks for the spotting. @brandonrjacobs It sounds a reasonable feature to me.
I understand you want to configure the tolerations and affinity for all components installed by karmada-operator, isn't that right?

also, cc @jabellard to take a look.

[jabellard]

Thanks for the spotting. @brandonrjacobs It sounds a reasonable feature to me.
I understand you want to configure the tolerations and affinity for all components installed by karmada-operator, isn't that right?

also, cc @jabellard to take a look.

I think this will be a useful addition.