List: Build Data Pipelines | Curated by Sai Parvathaneni

Feb 17, 2025
22 stories
4 saves
Build Data Pipelines
In
Towards Dev
by
Sai Parvathaneni
Building a Data Modeling Pipeline - dbt, Snowflake, and AirflowIn this article, we’ll build a Data Modeling pipeline using dbt, Snowflake, and Airflow. By the end, you’ll have a good understanding of…
Aug 25, 2024
Aug 25, 2024
In
Towards Dev
by
Sai Parvathaneni
Building a Log Analysis Data Pipeline Using Kafka, Elasticsearch, Logstash, and Kibana — ELK StackWhen it comes to analyzing logs, having a real-time, centralized, and automated solution is a game changer. Instead of sifting through…
Sep 11, 2024
Sep 11, 2024
In
Towards Dev
by
Sai Parvathaneni
Building a Streaming Data Pipeline: Spark vs. Flink Comparison with Kafka IntegrationWhen handling streams of data, two prominent frameworks that often come into play are Apache Spark and Apache Flink. Both are commonly used…
Sep 13, 2024
Sep 13, 2024
In
Towards Dev
by
Sai Parvathaneni
CI/CD for Modern Data EngineeringIn this article, we’ll walk through setting up a CI/CD pipeline tailored for modern data engineering. We’ll use a simple ETL pipeline as…
Aug 19, 2024
4
Aug 19, 2024
4
In
Towards Dev
by
Sai Parvathaneni
Data Engineering with Kubernetes: End-to-End Data PipelineIn this article, we dive into setting up Kubernetes for modern data engineering tasks. We’ll cover everything from installation to running…
Jul 27, 2024
4
Jul 27, 2024
4
In
Towards Dev
by
Burak Uğur
Kafka + NiFi + Clickhouse in Docker  PART-IIn this article, we will process real-time car entry/exit data from 5 different parking lots using big data tools and Apache NiFi, and we…
Mar 20, 2024
1
Mar 20, 2024
1
In
Towards Dev
by
Burak Uğur
Kafka + NiFi + Clickhouse in Docker PART-IIIn the first part of this article, I touched on the following topics:
Mar 31, 2024
2
Mar 31, 2024
2
Sai Parvathaneni
Automate Email Alerts for Critical Errors in Log Files— Code, Execute, and AutomateIntroduction:
Mar 20, 2023
Mar 20, 2023
In
Towards Dev
by
Sai Parvathaneni
Apache Airflow for Beginners: Build your first  DAGApache Airflow, explained like you are 10.
Apr 7, 2023
2
Apr 7, 2023
2
Sai Parvathaneni
Building a Data Stream Pipeline — Part 1 with Crypto API, Python, Airflow, and KafkaIntroduction:
Apr 16, 2023
Apr 16, 2023
Sai Parvathaneni
Building an End-to-End Data Pipeline — Part 2 with AWS ServicesIntroduction:
Apr 16, 2023
Apr 16, 2023
Sai Parvathaneni
Building a Data Stream Pipeline —  Part 2 with Kafka Streams and CassandraIntroduction:
Apr 17, 2023
Apr 17, 2023
In
Towards Dev
by
Sai Parvathaneni
Snowflake for Beginners: Build your first SnowpipeLearn how to set up a Snowflake account, understand the architecture, and terminologies, and build your first Snowpipe for loading data…
May 10, 2023
1
May 10, 2023
1
In
Towards Dev
by
Sai Parvathaneni
dbt (Data Build Tool) for Beginners: Build Your First Data Modeldbt is the ‘T’ in ELT
May 28, 2023
2
May 28, 2023
2
In
Towards Dev
by
Sai Parvathaneni
Building a Real-time Log Monitoring System with Kafka and Spark StreamingBy leveraging Kafka for data streaming and integration, and Spark for real-time data processing and analysis, organizations can achieve a…
Jun 6, 2023
1
Jun 6, 2023
1
In
Towards Dev
by
Sai Parvathaneni
Building a Real-time Sensor Data Analysis Pipeline — AWS Kinesis and DynamoDBIntroduction
Jun 22, 2023
Jun 22, 2023
In
Towards Dev
by
Sai Parvathaneni
Kafka Connect: Achieving Change Data Capture — Part 1: SourceIntroduction
Jul 18, 2023
Jul 18, 2023
Sai Parvathaneni
Kafka Connect: Achieving Change Data Capture — Part 2: SinkIntroduction
Jul 22, 2023
Jul 22, 2023
In
Towards Dev
by
Sai Parvathaneni
Building an End-to-End Data Pipeline with Delta Lake and DatabricksIntroduction
Aug 4, 2023
4
Aug 4, 2023
4
In
Towards Dev
by
Sai Parvathaneni
Building an End-to-End Data Pipeline — Part 1 with Airflow and PythonIntroduction
Apr 9, 2023
7
Apr 9, 2023
7