Introduced in Airflow 2.0, the TaskFlow API abstracts the backend plumbing. Returning a value from a @task decorated function automatically creates an XCom, and passing that function's output as an argument to another task automatically performs an xcom_pull .
Cloud buckets can leverage lifecycle policies to automatically delete old XCom data after 30 days, preventing data bloat.
XCom (short for ) lets tasks exchange small pieces of data.
Tasks use xcom_pull to retrieve values from previous tasks. You can filter these requests by: Specify which task the data came from. Keys: Filter for specific identifiers. DAG IDs: Pull from different DAGs if necessary. Best Practices and Limitations airflow xcom exclusive
Includes metadata like the task_id , dag_id , and a creation timestamp. How to Use XComs
: Airflow must serialize data to JSON or strings, causing high CPU utilization for complex Python objects.
Your (Astronomer, MWAA, Helm chart, Docker Compose) Introduced in Airflow 2
Which and deployment environment (e.g., MWAA, Astro, local Docker) are you running?
@task def extract_data(): # Dangerous: Automatically pushes a massive list to the metadata DB return ["row1", "row2", "row3", ... , "row1000000"] Use code with caution.
The and type of data you need to pass between your tasks. XCom (short for ) lets tasks exchange small pieces of data
What if your pipeline inherently requires passing larger, complex objects (like dataframes, custom model objects, or encrypted configurations) between tasks seamlessly?
You can view, debug, and clear XComs directly in the Airflow UI: Go to the . Click on the specific Task Instance that pushed the data.
AIRFLOW__CORE__XCOM_BACKEND=path.to.your.custom_module.S3XComBackend Use code with caution. XCom Security and Data Lifecycle Governance