Skip to content

Commit ba76694

Browse files
rickReamer
rick
authored andcommitted
[ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes cluster
### What is this PR for? Currently, when zeppelin server is running outside of k8s cluster, users cannot access the spark UI directly from the web page of Zeppelin. So this PR creates ingress resource in `k8s/interpreter/100-interpreter-spec.yaml` and enables the `SPARK JOB` button to jump directly to the URL defined by ingress. If user does not specify the property `zeppelin.spark.uiWebUrl`, it will be automatically generated according to the template: `{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}`, which is also the URL that the `SPARK JOB` button points to. So this PR also configured the ingress.host by this URL. This PR also adds a property `zeppelin.k8s.spark.useIngress` to let user choose whether to create ingress. Example usage: ``` zeppelin.k8s.spark.useIngress true ``` ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * <https://issues.apache.org/jira/browse/ZEPPELIN-5445> ### How should this be tested? * CI pass and manually tested ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: rick <rick@rickdeMacBook-Pro.local> Closes apache#4176 from rickchengx/ZEPPELIN-5445 and squashes the following commits: fdfe89a [rick] update docs 1773d06 [rick] [ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes cluster
1 parent 979820e commit ba76694

File tree

3 files changed

+48
-0
lines changed

3 files changed

+48
-0
lines changed

docs/quickstart/kubernetes.md

+4
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,7 @@ to customize,
245245
4. Run a paragraph will create an interpreter using modified yaml files.
246246
247247
The interpreter pod can also be customized through the interpreter settings. Here are some of the properties:
248+
248249
| Property Name | Default Value | Description |
249250
| ----- | ----- | ----- |
250251
| `zeppelin.k8s.namespace` | `default` | The Kubernetes namespace to use. |
@@ -256,6 +257,9 @@ The interpreter pod can also be customized through the interpreter settings. Her
256257
| `zeppelin.k8s.interpreter.imagePullSecrets` | (optional) | Set the comma-separated list of Kubernetes secrets while pulling images, e.g., `mysecret1,mysecret2` |
257258
| `zeppelin.k8s.interpreter.container.imagePullPolicy` | (optional) | Set the pull policy of the interpreter image, e.g., `Always` |
258259
| `zeppelin.k8s.spark.container.imagePullPolicy` | (optional) | Set the pull policy of the spark image, e.g., `Always` |
260+
| `zeppelin.spark.uiWebUrl` | `//{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}` | The URL for user to access Spark UI. The default value is a [jinjava](https://github.com/HubSpot/jinjava) template that contains three variables. |
261+
| `zeppelin.k8s.spark.useIngress` | (optional) | If true, the [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) will be created when creating the spark interpreter. So users can access the Spark UI through Ingress. |
262+
| `zeppelin.k8s.spark.ingress.host` | `{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}` | If `zeppelin.k8s.spark.useIngress` is `true`, it configures the `host` value of the Ingress. The default value is a [jinjava](https://github.com/HubSpot/jinjava) template that contains three variables. Users can access the Spark UI through a customized `zeppelin.k8s.spark.ingress.host`. |
259263
260264
## Future work
261265

k8s/interpreter/100-interpreter-spec.yaml

+30
Original file line numberDiff line numberDiff line change
@@ -180,4 +180,34 @@ roleRef:
180180
kind: Role
181181
name: {{zeppelin.k8s.interpreter.pod.name}}
182182
apiGroup: rbac.authorization.k8s.io
183+
{% if zeppelin.k8s.spark.useIngress is defined and zeppelin.k8s.spark.useIngress == "true" %}
184+
---
185+
# create ingress of spark UI
186+
apiVersion: networking.k8s.io/v1
187+
kind: Ingress
188+
metadata:
189+
name: spark-ui-{{zeppelin.k8s.interpreter.pod.name}}
190+
namespace: {{zeppelin.k8s.namespace}}
191+
{% if zeppelin.k8s.server.uid is defined %}
192+
ownerReferences:
193+
- apiVersion: v1
194+
controller: false
195+
blockOwnerDeletion: false
196+
kind: Pod
197+
name: {{zeppelin.k8s.server.pod.name}}
198+
uid: {{zeppelin.k8s.server.uid}}
199+
{% endif %}
200+
spec:
201+
rules:
202+
- host: {{zeppelin.k8s.spark.ingress.host}}
203+
http:
204+
paths:
205+
- pathType: Prefix
206+
path: "/"
207+
backend:
208+
service:
209+
name: {{zeppelin.k8s.interpreter.pod.name}}
210+
port:
211+
number: 4040
212+
{% endif %}
183213
{% endif %}

zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java

+14
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,20 @@ Properties getTemplateBindings(String userName) {
318318
getPodName(),
319319
k8sEnv.get(ENV_SERVICE_DOMAIN)
320320
));
321+
322+
// configure interpreter property "zeppelin.k8s.spark.ingress.host" if not defined, to enable spark ui through ingress
323+
String ingressHost = (String) properties.get("zeppelin.k8s.spark.ingress.host");
324+
if (StringUtils.isBlank(ingressHost)) {
325+
ingressHost = "{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}";
326+
}
327+
properties.put("zeppelin.k8s.spark.ingress.host",
328+
sparkUiWebUrlFromTemplate(
329+
ingressHost,
330+
webUiPort,
331+
getPodName(),
332+
k8sEnv.get(ENV_SERVICE_DOMAIN)
333+
));
334+
321335
// Resources of Interpreter Pod
322336
if (properties.containsKey(SPARK_DRIVER_MEMORY)) {
323337
String memory;

0 commit comments

Comments
 (0)