You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[v2-10-test] Update XCom docs around containers/helm (#44570) (#44573)
This removes the whole section about helm, as it really isn't directly related to the XCom concept at all. I also simplified the section about containers as well - this one is a bit more practical, so I've left it.
(cherry picked from commit 3747c91)
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Copy file name to clipboardExpand all lines: docs/apache-airflow/core-concepts/xcoms.rst
+3-24
Original file line number
Diff line number
Diff line change
@@ -98,36 +98,15 @@ There is also an ``orm_deserialize_value`` method that is called whenever the XC
98
98
99
99
You can also override the ``clear`` method and use it when clearing results for given DAGs and tasks. This allows the custom XCom backend to process the data lifecycle easier.
100
100
101
-
Working with Custom XCom Backends in Containers
102
-
-----------------------------------------------
101
+
Verifying Custom XCom Backend usage in Containers
102
+
-------------------------------------------------
103
103
104
104
Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
105
105
106
-
Firstly, if you can exec into a terminal in the containerthen you should be able to do:
106
+
If you can exec into a terminal in an Airflow container, you can then print out the actual XCom class that is being used:
107
107
108
108
.. code-block:: python
109
109
110
110
from airflow.models.xcom import XCom
111
111
112
112
print(XCom.__name__)
113
-
114
-
which will print the actual class that is being used.
115
-
116
-
You can also examine Airflow's configuration:
117
-
118
-
.. code-block:: python
119
-
120
-
from airflow.settings import conf
121
-
122
-
conf.get("core", "xcom_backend")
123
-
124
-
Working with Custom Backends in K8s via Helm
125
-
--------------------------------------------
126
-
127
-
Running custom XCom backends in K8s will introduce even more complexity to your Airflow deployment. Put simply, sometimes things go wrong which can be difficult to debug.
128
-
129
-
For example, if you define a custom XCom backend in the Chart ``values.yaml`` (via the ``xcom_backend`` configuration) and Airflow fails to load the class, the entire Chart deployment will fail with each pod container attempting to restart time and time again.
130
-
131
-
When deploying in K8s your custom XCom backend needs to be reside in a ``config`` directory otherwise it cannot be located during Chart deployment.
132
-
133
-
An observed problem is that it is very difficult to acquire logs from the container because there is a very small window of availability where the trace can be obtained. The only way you can determine the root cause is if you are fortunate enough to query and acquire the container logs at the right time. This in turn prevents the entire Helm chart from deploying successfully.
0 commit comments