Skip to content

chore(tracking): Release Notes for SDP 25.3.0 #723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Mar 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
453b6cc
chore: Update release template
NickLarsenNZ Mar 26, 2025
7c092fe
chore: Add 25.3.0 release notes partial
NickLarsenNZ Mar 26, 2025
f731c5e
chore: Add link to 25.3.0 release notes partial
NickLarsenNZ Mar 26, 2025
d3f762f
chore: Update release template
NickLarsenNZ Mar 26, 2025
5e0fc56
chore(25.3.0): Add and link product version changes
NickLarsenNZ Mar 26, 2025
3169f48
chore(25.3.0): Add upgrade intructions using stackablectl
Techassi Mar 26, 2025
196bb71
chore(25.3.0): Add upgrade intructions using helm
Techassi Mar 26, 2025
bc6999b
chore: Add upgrade instructions to release template
Techassi Mar 26, 2025
bafc9bd
chore(25.3.0): Add supported K8s and OpenShift versions
Techassi Mar 26, 2025
673acb0
chore: Add supported K8s and OpenShift section to release template
Techassi Mar 26, 2025
f0c14b4
chore: Update ui submodule
Techassi Mar 26, 2025
ae8fa7f
chore(25.3.0): Add Trino breaking change
NickLarsenNZ Mar 26, 2025
456e119
chore(25.3.0): Add platform feature for Additional trust roots
NickLarsenNZ Mar 26, 2025
b462c5f
chore(25.3.0): Add s3 region breaking change
NickLarsenNZ Mar 26, 2025
4778de3
chore(25.3.0): Add OPA rego changes
NickLarsenNZ Mar 26, 2025
a378e19
chore(25.3.0): Add Druid changes
NickLarsenNZ Mar 26, 2025
3458f69
chore(25.3.0): Add Listener volume bug fix
NickLarsenNZ Mar 26, 2025
24324dc
chore(25.3.0): Add experimentalCertManager key length feature
NickLarsenNZ Mar 26, 2025
97c7c2d
chore(25.3.0): Add TLS certificate configurable lifetime feature
NickLarsenNZ Mar 26, 2025
6af4ef9
chore(25.3.0): Add containerdebug diagnostics feature
NickLarsenNZ Mar 26, 2025
a42882c
chore(25.3.0): Add airflow breaking change for dagsGitSync[].wait Dur…
NickLarsenNZ Mar 26, 2025
025a7ff
chore(25.3.0): Add Hive healthcheck known issue
NickLarsenNZ Mar 26, 2025
ec67eec
chore(25.3.0): Add Hive memory reservation increase
Techassi Mar 26, 2025
47c4218
chore(25.3.0): Add OCI registry migration section
Techassi Mar 26, 2025
1b01286
chore(25.3.0): Add checksum/config annotation section
Techassi Mar 26, 2025
85a17e5
chore(25.3.0): Add Kafka -nodeport CM deprecation section
Techassi Mar 26, 2025
f673b70
chore(25.3.0): Add NiFi Hadoop library change
Techassi Mar 26, 2025
471f5eb
chore(25.3.0): Add DNS lookup perfomance improvements
Techassi Mar 26, 2025
7c0d9bb
chore: Update headlines
Techassi Mar 26, 2025
d080420
chore(25.3.0): Add stackablectl and demo section
Techassi Mar 27, 2025
457a934
chore(25.3.0): Add fixed vulnerabilities
Techassi Mar 27, 2025
8fc09f0
chore(25.3.0): Add jmx_exporter bug fix
NickLarsenNZ Mar 26, 2025
a6a9732
chore: Merge branch 'main' into docs/release-notes-25.3.0
Techassi Mar 27, 2025
400d908
chore: Sigh, Antora...
Techassi Mar 27, 2025
7c603d6
chore(25.3.0): Add Airflow and Superset OPA auth integration
Techassi Mar 27, 2025
e13eeb2
chore(25.3.0): Add JVM argument override support
Techassi Mar 27, 2025
098969e
fix(25.3.0): Add .adoc extension to xref
Techassi Mar 27, 2025
e7b1f5b
chore: Use updated heading sizes
Techassi Mar 27, 2025
afaf600
chore(25.3.0): Add more links
Techassi Mar 27, 2025
c730be5
chore(25.3.0): Remove leftover section from template
Techassi Mar 27, 2025
1b680d7
chore(25.3.0): Improve druid-opa-authorizer xref
Techassi Mar 27, 2025
e0713c6
chore: Update release and product versions in getting_started page
NickLarsenNZ Mar 27, 2025
cd57e0e
chore(25.3.0): Put each sentence on a new line
NickLarsenNZ Mar 27, 2025
e359753
chore: Remove mention of airflowdbs crd
NickLarsenNZ Mar 27, 2025
7ca71a1
chore(25.3.0): Fix spelling errors
Techassi Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions modules/ROOT/pages/getting-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ Install the Stackable command line utility xref:management:stackablectl:index.ad

The Stackable operators are components that translate the service definitions deployed via Kubernetes into deploy services on the worker nodes. These can be installed on any node that has access to the Kubernetes control plane. In this example we will install them on the controller node.

Stackable operators can be installed using `stackablectl`. Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 24.11 release.
Stackable operators can be installed using `stackablectl`. Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 25.3 release.

[source,bash]
----
stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 24.11
stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 25.3
----

.Using Helm instead
Expand All @@ -50,12 +50,12 @@ Install the operators:

[source,bash]
----
helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=24.11.1
helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=24.11.1
helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=24.11.1
helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=24.11.1
helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=24.11.1
helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=24.11.1
helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=25.3.0
helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=25.3.0
helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=25.3.0
helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=25.3.0
helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=25.3.0
helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=25.3.0
----
====

Expand All @@ -64,12 +64,12 @@ You can check which operators are installed using `stackablectl operator install
[source,console]
----
OPERATOR VERSION NAMESPACE STATUS LAST UPDATED
commons 24.11.1 default deployed 2024-11-30 17:58:32.916032854 +0100 CET
kafka 24.11.1 default deployed 2024-11-30 17:58:55.036115353 +0100 CET
listener 24.11.1 default deployed 2024-11-30 17:59:18.136775259 +0100 CET
nifi 24.11.1 default deployed 2024-11-30 17:59:51.927081648 +0100 CET
secret 24.11.1 default deployed 2024-11-30 18:00:05.060241771 +0100 CET
zookeeper 24.11.1 default deployed 2024-11-30 18:00:08.425686918 +0100 CET
commons 25.3.0 default deployed 2024-11-30 17:58:32.916032854 +0100 CET
kafka 25.3.0 default deployed 2024-11-30 17:58:55.036115353 +0100 CET
listener 25.3.0 default deployed 2024-11-30 17:59:18.136775259 +0100 CET
nifi 25.3.0 default deployed 2024-11-30 17:59:51.927081648 +0100 CET
secret 25.3.0 default deployed 2024-11-30 18:00:05.060241771 +0100 CET
zookeeper 25.3.0 default deployed 2024-11-30 18:00:08.425686918 +0100 CET
----

== Deploying Stackable Services
Expand All @@ -90,7 +90,7 @@ metadata:
name: simple-zk
spec:
image:
productVersion: 3.9.2
productVersion: 3.9.3
clusterConfig:
tls:
serverSecretClass: null
Expand Down Expand Up @@ -278,7 +278,7 @@ To get the IP address we need to connect to (in this case `172.18.0.2`), run:
----
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
quickstart-control-plane Ready control-plane 4d18h v1.30.0 172.18.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.11.3 containerd://1.7.15
quickstart-control-plane Ready control-plane 4d18h v1.32.0 172.18.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.13.2 containerd://1.7.24
----


Expand Down
12 changes: 12 additions & 0 deletions modules/ROOT/pages/release-notes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,25 @@
The Stackable platform consists of multiple operators that work together. Periodically a platform release is made,
including all components of the platform at a specific version.

// WARNING: Please keep the empty newlines, otherwise headings are broken.
include::partial$release-notes/release-25.3.adoc[]

include::partial$release-notes/release-24.11.adoc[]

include::partial$release-notes/release-24.7.adoc[]

include::partial$release-notes/release-24.3.adoc[]

include::partial$release-notes/release-23.11.adoc[]

include::partial$release-notes/release-23.7.adoc[]

include::partial$release-notes/release-23.4.adoc[]

include::partial$release-notes/release-23.1.adoc[]

include::partial$release-notes/release-22.11.adoc[]

include::partial$release-notes/release-22.9.adoc[]

include::partial$release-notes/release-22.6.adoc[]
8 changes: 4 additions & 4 deletions modules/ROOT/partials/release-notes/release-22.11.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This is the third release of the Stackable Data Platform, which this time focuse

The following new major platform features were added:

CPU and memory limits configurable::
===== CPU and memory limits configurable

The operators now https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[request] resources from Kubernetes for the products and required CPU and memory can now also be configured for all products.
If your product instances are less performant after the update, the new defaults might be set too low and we recommend to xref:kafka:usage-guide/storage-resources.adoc[set custom requests] for your cluster.
Expand All @@ -24,7 +24,7 @@ If your product instances are less performant after the update, the new defaults
* https://github.com/stackabletech/airflow-operator/pull/167[Apache Airflow]
* https://github.com/stackabletech/superset-operator/pull/273[Apache Superset]

Orphaned Resources::
===== Orphaned Resources

The operators now properly clean up after scaling down products.
This means for example deleting StatefulSets that were left over after scaling down.
Expand All @@ -37,7 +37,7 @@ This means for example deleting StatefulSets that were left over after scaling d
* https://github.com/stackabletech/trino-operator/pull/310[Trino]
* https://github.com/stackabletech/airflow-operator/pull/174[Apache Airflow]

New Versions::
===== New Versions

New product versions are supported.

Expand All @@ -47,7 +47,7 @@ New product versions are supported.
* https://github.com/stackabletech/druid-operator/pull/317[Apache Druid 24.0.0]
* https://github.com/stackabletech/airflow-operator/pull/179[Apache Airflow 2.4.1]

Product features::
===== Product features

Additionally there are some individual product features that are noteworthy

Expand Down
14 changes: 7 additions & 7 deletions modules/ROOT/partials/release-notes/release-22.6.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,37 +13,37 @@ While we are very proud of this release it is our first one and we'll add new fe

==== Platform features

Easily install production ready data applications::
===== Easily install production ready data applications

Using a familiar declarative approach, users can easily install data applications such as Apache Kafka or Trino across multiple cloud Kubernetes providers or on their own data centers.
The installation process is fully automated while also providing the flexibility for the user to tune relevant aspects of each application.

Monitoring::
===== Monitoring

All products have monitoring with prometheus enabled.
xref:operators:monitoring.adoc[Learn more]

Service discovery::
===== Service discovery

Products on the Stackable platform use service discovery to easily interconnect with each other.
xref:concepts:service_discovery.adoc[Learn more]

Configuration overrides::
===== Configuration overrides

All operators support configuration overrides, these are documented in the specific operator documentation pages.

Common S3 configuration::
===== Common S3 configuration

Many products support connecting to S3 to load and/or store data.
There is a common resource for S3 connections and buckets across all operators that can be reused.
xref:concepts:s3.adoc[Learn more]

Roles and role groups::
===== Roles and role groups

To support hybrid hardware clusters, the Stackable platform uses the concept of role groups.
Services and applications can be configured to maximize hardware efficiency.

Standardized::
===== Standardized

Learn once reuse everywhere.
We use the same conventions in all our operators.
Expand Down
6 changes: 3 additions & 3 deletions modules/ROOT/partials/release-notes/release-22.9.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The main features focus on OpenShift support and security.

The following new major platform features were added:

OpenShift compatibility::
===== OpenShift compatibility

We have made continued progress towards OpenShift compability, and the following operators can now be previewed on OpenShift.
Further improvements are expected in future releases, but no stability or compatibility guarantees are currently made for OpenShift clusters.
Expand All @@ -20,15 +20,15 @@ Further improvements are expected in future releases, but no stability or compat
* https://github.com/stackabletech/hdfs-operator/pull/225[Apache HDFS]
* https://github.com/stackabletech/spark-k8s-operator/pull/126[Apache Spark on K8s]

Support for internal and external TLS::
===== Support for internal and external TLS

The following operators support operating the products at a maximal level of transport security by using TLS certificates to secure internal and external communication:

* https://github.com/stackabletech/trino-operator/pull/244[Trino]
* https://github.com/stackabletech/kafka-operator/pull/442[Apache Kafka]
* https://github.com/stackabletech/zookeeper-operator/pull/479[Apache ZooKeeper]

LDAP authentication::
===== LDAP authentication

Use a central LDAP server to manage all of your user identities in a single place.
The following operators added support for LDAP authentication:
Expand Down
10 changes: 5 additions & 5 deletions modules/ROOT/partials/release-notes/release-23.1.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The focus in this platform release is on the support of offline (or on-premise)

The following new major platform features were added:

Product image selection::
===== Product image selection

Product image selection has been expanded to cover different scenarios:

Expand All @@ -32,7 +32,7 @@ These options are described in more detail xref:contributor:adr/ADR023-product-i

*N.B.* this is a breaking change across all operators as `spec.version` has been replaced by `spec.image`.

Logging Aggregation::
===== Logging Aggregation

Component activity within the platform is logged in a way that makes it difficult to find, persist and consolidate this information.
Log configuration is also a challenge.
Expand All @@ -47,19 +47,19 @@ In this release this has been added to the following components:

Support for other products will be added in future releases.

New Versions::
===== New Versions

The following new product version is now supported:

* https://github.com/stackabletech/trino-operator/pull/358[Trino 403]

Deprecated Versions::
===== Deprecated Versions

The following product version is no longer supported:

* https://github.com/stackabletech/druid-operator/pull/339[Druid 0.22.1]

Product features::
===== Product features

Additionally, there are some individual product features that are noteworthy

Expand Down
21 changes: 10 additions & 11 deletions modules/ROOT/partials/release-notes/release-23.11.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,45 +11,45 @@ Released 2023-11-30.

The following new major platform features were added:

PodDisruptionBudgets::
===== PodDisruptionBudgets
Kubernetes has mechanisms to ensure minimal planned downtime.
Our product operators deploy so-called PodDisruptionBudget (PDB) resources alongside the products.
For every role that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created.
This will determine the extent to which roles for a given application may be inactive at any given time.
See xref:concepts:operations/pod_disruptions.adoc[the documentation] for more details.

Graceful shutdown::
===== Graceful shutdown
Graceful shutdown refers to the managed, controlled shutdown of service instances in the manner intended by the software authors.
Typically, an instance will receive a signal indicating the intent for the server to shut down, and it will initiate a controlled shutdown.
Our operators configure a sensible amount of time Pods are granted to properly shut down without disrupting the availability of the product.
See xref:concepts:operations/graceful_shutdown.adoc[the documentation] for more details.

Signed SDP product images::
===== Signed SDP product images
As of this release all Stackable product images are signed (the signing of operator images was delivered in SDP 23.7).
Please see this xref:guides:enabling-verification-of-image-signatures.adoc[tutorial] for more information.

Airflow KubernetesExecutor::
===== Airflow KubernetesExecutor
Airflow clusters can now be configured to use Kubernetes executors, whereby pods are spun up for job tasks and terminated when complete, thus offering an alternative way to use resources without the need for job queuing.

Overridable Java security settings::
===== Overridable Java security settings
For JVM-based products (i.e. Druid, HBase, HDFS, Hive, Kafka, NiFi, Spark, Trino and ZooKeeper) it is now possible to provide custom security settings that override the default values.
This allows the user to control things such as DNS lookup caches.

Stackable Cockpit::
===== Stackable Cockpit
This release includes a very early preview version of Stackable Cockpit, a browser-based management tool which interacts with the Stackable data platform to display e.g. deployed stacklets and their status.

stackablectl::
===== stackablectl
Our command line tool has been re-worked to use the same backbone as Stackable Cockpit: you can find out about the recent enhancements by visiting the online xref:management:stackablectl:index.adoc[documentation].

Listener operator::
===== Listener operator
The listener-operator was introduced in release 23.1 and the associated ServiceType field in 23.4.
In this release we introduce configurable ListenerClass _presets_ that map to the service types appropriate for different environments.
This is discussed in more detail in the xref:listener-operator:listenerclass.adoc[documentation].

Openshift certification::
===== Openshift certification
All Stackable operators in the 23.11 release have been certified for Openshift versions 4.11-4.13 and can be installed directly from the OperatorHub UI: image:openshift_operatorhub.png[OperatorHub in Openshift portal]

Product features::
===== Product features

Additionally, there are some other individual product features that are noteworthy:

Expand Down Expand Up @@ -692,7 +692,6 @@ kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-ope
[source,console]
----
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...
Expand Down
16 changes: 8 additions & 8 deletions modules/ROOT/partials/release-notes/release-23.4.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,19 @@ It is recommended to install <<Release 23.4.1>> instead, as it contains relevant

The following new major platform features were added:

Cluster Operation::
===== Cluster Operation

The first part of xref:concepts:operations/cluster_operations.adoc[Cluster operations] was rolled out in every applicable Stackable Operator.
This supports pausing the cluster reconciliation and stopping the cluster completely.
Pausing reconciliation will not apply any changes to the Kubernetes resources (e.g. when changing the custom resource).
Stopping the cluster will set all replicas of StatefulSets, Deployments or DaemonSets to zero and therefore result in the deletion of all Pods belonging to that cluster (not the PVCs).

Status Field::
===== Status Field

Operators of the Stackable Data Platform create, manage and delete Kubernetes resources: in order to easily query the health state of the products - and react accordingly - Stackable Operators use several predefined condition types to capture different aspects of a product's availability.
See this xref:contributor:adr/ADR027-status[ADR] for more information.

Default / Custom Affinities::
===== Default / Custom Affinities

In Kubernetes there are different ways to influence how Pods are assigned to Nodes.
In some cases it makes sense to co-locate certain services that communicate a lot with each other, such as HBase regionservers with HDFS datanodes.
Expand All @@ -50,29 +50,29 @@ There may also be additional requirements e.g. placing important services - such
This release implements default affinities that should suffice for many scenarios out-of-the box, while also allowing for custom affinity rules at a role and/or role-group level.
See this xref:contributor:adr/ADR026-affinities.adoc[ADR] for more information.

Log Aggregation::
===== Log Aggregation

The logging framework (added to the platform in Release 23.1) offers a consistent custom resource configuration and a separate, persisted sink (defaulting to OpenSearch).
This has now been rolled out across all products.
See this xref:contributor:adr/adr025-logging_architecture[ADR] and this xref:concepts:logging.adoc[concepts page] for more information.

Service Type::
===== Service Type

The Service type can now be specified in all products.
This currently differentiates between the internal ClusterIP and the external NodePort and is forward compatible with the xref:listener-operator:listenerclass.adoc[ListenerClass] for the automatic exposure of Services via the Listener Operator.
This change is not backwards compatible with older platform releases.
For security reasons, the default is set to the cluster-internal (ClusterIP) ListenerClass.
A cluster can be exposed outside of Kubernetes by setting clusterConfig.listenerClass to external-unstable (NodePort) or external-stable (LoadBalancer).

New Versions::
===== New Versions

No new product versions are supported in this platform release.

Deprecated Versions::
===== Deprecated Versions

No product versions have been deprecated in this platform release.

Product features::
===== Product features

Additionally, there are some individual product features that are noteworthy:

Expand Down
Loading