No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
-
Updated
Apr 15, 2025 - Python
#
etl-pipeline
Here are 877 public repositories matching this topic...
Implementing best practices for PySpark ETL jobs and applications.
-
Updated
Jan 1, 2023 - Python
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
infrastructure aws postgres data airflow cloudformation cassandra cluster aws-s3 aws-sdk data-warehouse data-engineering data-lake aws-ec2 postgresql-database data-modeling cassandra-database etl-pipeline data-engineering-pipeline airflow-operators
-
Updated
Aug 26, 2022 - Python
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
python data-science machine-learning etl numpy pandas data-engineering data-platform software-engineering feature-engineering dataframe dag hamiltonian etl-framework hamilton featurization etl-pipeline stitch-fix
-
Updated
Jul 3, 2023 - Python
Enterprise-grade and API-first LLM workspace for unstructured documents, including data extraction, redaction, rights management, prompt playground, and more!
-
Updated
Apr 9, 2025 - Python
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
python ai concurrency ai-agents etl-pipeline llm llm-agent ai-agents-framework agentic-ai-development
-
Updated
Mar 10, 2025 - Python
Pythonic Stream-like manipulation of iterables
python data etl iterator python3 streams immutability data-engineering fluent-interface collections asyncio visitor-pattern lazy-evaluation threads iterable decorator-pattern method-chaining iterator-pattern etl-pipeline reverse-etl
-
Updated
Apr 13, 2025 - Python
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
docker big-data cassandra apache-spark data-storage postgresql data-engineering apache-kafka data-processing data-pipeline real-time-analytics containerization apache-zookeeper apache-airflow etl-pipeline
-
Updated
Feb 14, 2025 - Python
Data pipelines from re-usable components
data-science sql etl pipelines immutability data-engineering functional-reactive-programming data-analysis data-pipelines data-pipeline etl-framework etl-pipeline etl-pipelines
-
Updated
Mar 30, 2023 - Python
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
sql database spark hive hadoop etl pyspark data-engineering spark-streaming data-analysis databricks datalake spark-sql timetravel apachespark etl-pipeline deltalake
-
Updated
Jul 28, 2024 - Python
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
python data etl openai datapipeline etl-framework unstructured pinecone cohere etl-pipeline weaviate vector-database qdrant qdrant-vector-database
-
Updated
Oct 5, 2024 - Python
Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
-
Updated
Apr 28, 2022 - Python
Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
python bigquery postgres data-science data machine-learning pipeline etl snowflake orchestration data-engineering data-analysis data-integration redshift dbt trino etl-pipeline data-orc
-
Updated
Nov 25, 2024 - Python
Near real time ETL to populate a dashboard.
-
Updated
Jun 17, 2024 - Python
Conductor OSS SDK for Python programming language
-
Updated
Mar 25, 2025 - Python
Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.
-
Updated
May 19, 2021 - Python
implementing an end-to-end tweets ETL/Analysis pipeline.
tweets analysis twitter-api multithreading api-client datawarehousing datawarehouse web-crawling ssis google-api-client etl-pipeline tweets-classification cube-analysis powerbi-report ssas-multidimensional multi-dimensional-analysis tweets-scraper
-
Updated
Dec 8, 2022 - Python
🚹 💾 Script to import issues from a JIRA instance into a database.
-
Updated
Dec 8, 2022 - Python
Improve this page
Add a description, image, and links to the etl-pipeline topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the etl-pipeline topic, visit your repo's landing page and select "manage topics."