unable to modify the nlb idle-timeout using annotation

[L040639]

i can able to deploy the private Nlb, but want to change the idle timeout of the nlb to some custom value using the below annotation, which is not working for me . Any idea here anything i am missing still here ??

service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=4000
below all are my kubernetes service annotations .

annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=4000

NLB has TCP listener at 9094 port and We are using redhat aws load balancer operator to handle these load balancer deployments . version is: 1.0.1 provided by Red Hat Inc also updated the iam role policy still the attributes modifying is not working . My full config is below

# namespace name: kafka-streams-dev
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: <XXX>
  namespace: <XXX>
spec:
  kafka:
    authorization:
      type: opa
      url: https://<XXX>:8181/v1/data/kafka/allow
      allowOnError: false
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.9'
      connections.max.idle.ms: 1800000
    storage:
      type: persistent-claim
      size: 100Gi
      class: gp3
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-dev-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-dev-cert-secret
          bootstrap:
            alternativeNames:
            - <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.9.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-streams-dev-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=4000
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=4000
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-streams-dev-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-streams-dev-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      class: gp3
      size: 10Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-streams-dev-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-streams-dev-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-streams-dev-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[shraddhabang]

@L040639 I have tried your manifest, it seems to be working for me. Do you see any errors in the logs for modifying listener attributes calls? Also can you share please the model that is built for your svc by controller? You should be able to find that in the logs too.

[wweiwei-li]

Looks like you created TLS listener instead of TCP ?