In this tutorial, we will guide you through the process of creating data pipelines using various AWS services. We will be specifically using Amazon S3, Amazon Glue, Amazon EMR, and Amazon Redshift.
<h1>What is Data Pipeline?</h1>
A data pipeline is a system or framework designed to automate the flow of data from various sources to their desired destinations. It involves a series of interconnected steps that extract, transform, and load (ETL) data, enabling organizations to efficiently process, transform, and analyze large volumes of data. Data pipelines play a crucial role in modern data architectures, as they enable the movement and transformation of data across different systems, databases, and applications. They ensure data consistency, reliability, and timeliness by providing mechanisms for data validation, cleansing, enrichment, and aggregation. Data pipelines can be used for a variety of purposes, including data integration, data warehousing, data migration, real-time streaming analytics, and machine learning. By automating the data flow and transformation process, data pipelines help organizations achieve faster and more accurate insights, improve data quality, and streamline their data operations.
<h1>What is AWS Glue?</h1>
AWS Glue is a fully managed extract, transform, and load (ETL) service provided by Amazon Web Services (AWS). It simplifies the process of preparing and transforming data for analytics and data processing. AWS Glue allows users to discover, catalog, and transform data from various sources like databases, data lakes, and data warehouses. It automates data discovery, schema inference, and schema evolution, making it easier to manage changing data structures. With AWS Glue, users can generate ETL code in Python or Scala and customize the transformations as needed. It integrates seamlessly with other AWS services like Amazon S3, Amazon RDS, Amazon Redshift, and Amazon Athena, enabling smooth data integration and analysis. By leveraging AWS Glue, organizations can accelerate their data integration and processing tasks while reducing the operational overhead of managing infrastructure and ETL workflows.
It is important to note Glue runs your ETL jobs on Apache Spark serverless environment. It runs these jobs on virtual resources that it provisions and manages in its own service account. This can help us Data Engineer to eliminate maintain spark clusters and reduce operational overload while still easily taking benefit of Spark’s scalability
<h1>Why is AWS Glue so useful for Data Engineers?</h1>
As a data engineer, AWS Glue offers a range of features that significantly enhance your data integration and processing capabilities. One key benefit is its automation prowess. AWS Glue automates the laborious task of data discovery, cataloging, and schema inference, saving you valuable time and effort. Additionally, it generates ETL code in Python or Scala, which can be customized to fit your specific requirements. This automation allows data engineers to focus on the more critical aspects of their work, such as data analysis and transformation, leading to increased productivity and faster insights.
Scalability and flexibility are vital considerations for data engineers, and AWS Glue excels in these areas. As a fully managed service, AWS Glue takes care of the underlying infrastructure, allowing you to scale resources seamlessly based on demand. Whether you&#8217;re dealing with small datasets or massive volumes of data, AWS Glue can handle it. Its serverless architecture also ensures cost-efficiency, as you only pay for the resources you utilize. Additionally, AWS Glue integrates seamlessly with other AWS services like Amazon S3, Amazon RDS, and Amazon Redshift, simplifying data integration across your entire data ecosystem. With its scalability, flexibility, and seamless integration capabilities, AWS Glue empowers data engineers to tackle complex data challenges with ease.
<h1>Fun Part</h1>
Alright, enough with the overview. Let&#8217;s dive into the exciting part. In this tutorial, we will create a data pipeline that reads data from S3, transforms it, and writes the processed data back to S3. AWS Glue offers several features, and we will explore each of them step by step.
<h2>Create an IAM Role:</h2>
We will create an IAM role for AWS Glue to for now add access to S3, Redshift, and EMR. We may add more access to more services if we need them later. Follow these steps to create an IAM role.
<ol>
<li>Go to AWS Management Console and navigate to the IAM service.</li>
<li>Choose &#8220;Roles&#8221; from the left-hand menu and click on the &#8220;Create role&#8221; button.</li>
<li>Select &#8220;Glue&#8221; as the trusted entity, attach policies for S3, Redshift, and EMR access, and provide a name for the IAM role. Finally, review and create the role.</li>
</ol>
By following these steps, you can create an IAM role that grants AWS Glue the necessary permissions to interact with S3, Redshift, and EMR services.
<img loading="lazy" decoding="async" class="aligncenter wp-image-246 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-1-1024x225.png" alt="" width="769" height="169" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-1-1024x225.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-1-300x66.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-1-768x169.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-1.png 1322w" sizes="(max-width: 769px) 100vw, 769px" />
<h2>Upload Source Data to S3: </h2>
To upload data to S3 using the AWS Console, follow these steps:
<ol>
<li>On your AWS Management Console, navigate to the S3 service.</li>
<li>Create a new S3 bucket or choose the S3 bucket where you want to upload the data or create a new bucket.</li>
<li>Click on the &#8220;Upload&#8221; button, select the files or folders you want to upload from your local machine and follow the prompts to complete the upload process.</li>
</ol>
We will create two folders named inputs and outputs, and upload publicly available data of Electric Vehicle Population Data in the input folder. Also, note the Copy S3 URI button that will help you to get the S3 path if you are not sure and we will need it in the later step
<img loading="lazy" decoding="async" class="aligncenter wp-image-258 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-1024x335.png" alt="" width="767" height="251" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-1024x335.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-300x98.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-768x251.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-1536x503.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download.png 1549w" sizes="(max-width: 767px) 100vw, 767px" />
<h2>Glue Setup: Visual Approach</h2>
There are multiple ways to create pipelines using Glue and we will explore some of the common approaches in this blog. Let&#8217;s start with the Visual Approach.
Step 1: Select Visual with a source and target option and click on create
<img loading="lazy" decoding="async" class="aligncenter wp-image-247 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-2-1024x461.png" alt="" width="762" height="343" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-2-1024x461.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-2-300x135.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-2-768x346.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-2-1536x692.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-2.png 1627w" sizes="(max-width: 762px) 100vw, 762px" />
Step 2: Add the source S3 input file URL and use the same configurations shown in the screenshot below. If you are using your own data that is different, you obviously need to verify the structure of the data and change the configuration appropriately
<img loading="lazy" decoding="async" class="aligncenter wp-image-248 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-3-1024x694.png" alt="" width="752" height="510" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-3-1024x694.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-3-300x203.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-3-768x521.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-3-1536x1041.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-3.png 1602w" sizes="(max-width: 752px) 100vw, 752px" />
Step 3: We will explore the Apply Mapping node. Here, we can try to delete a few unwanted columns using this.
<img loading="lazy" decoding="async" class="aligncenter wp-image-249 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-4-1024x721.png" alt="" width="753" height="530" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-4-1024x721.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-4-300x211.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-4-768x541.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-4-1536x1082.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-4.png 1589w" sizes="(max-width: 753px) 100vw, 753px" />
Step 4: Now let&#8217;s add a new custom node. To do this, first click on + button on top left of the visual diagram and select any appropriate transform that you would like to do. We will go with filter transform for now
Step 5:
<img loading="lazy" decoding="async" class="aligncenter wp-image-250 size-large" src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-5-404x1024.png" alt="" width="404" height="1024" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-5-404x1024.png 404w, https://data.cogtix.com/wp-content/uploads/2023/05/download-5.png 427w" sizes="(max-width: 404px) 100vw, 404px" />
Step 6: Node Mapping and Filters
<ol>
<li>Select the new node and add the parent node of ApplyMapping to it.</li>
<li>Additionally, you can modify the parent node of the output S3 bucket node and assign the new node as its parent.</li>
</ol>
These steps will allow you to establish the desired node hierarchy and achieve the required configuration.
<img loading="lazy" decoding="async" class="aligncenter wp-image-251 size-large" src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-6-1024x594.png" alt="" width="1024" height="594" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-6-1024x594.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-6-300x174.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-6-768x445.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-6-1536x890.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-6.png 1946w" sizes="(max-width: 1024px) 100vw, 1024px" />
Step 7:
Add the output directory to the same output folder that we created in the above S3 section
Step 8:
Attach the IAM role that we created above and select appropriate configs. For this demo, we will go with the most basic infrastructure.
<img loading="lazy" decoding="async" class="aligncenter wp-image-252 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-7-806x1024.png" alt="" width="699" height="888" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-7-806x1024.png 806w, https://data.cogtix.com/wp-content/uploads/2023/05/download-7-236x300.png 236w, https://data.cogtix.com/wp-content/uploads/2023/05/download-7-768x976.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-7.png 868w" sizes="(max-width: 699px) 100vw, 699px" />
Step 9:
Save the pipeline and click on run to execute it
<img loading="lazy" decoding="async" class="aligncenter wp-image-253 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-8-1024x159.png" alt="" width="717" height="111" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-8-1024x159.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-300x47.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-768x119.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-1536x238.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8.png 1857w" sizes="(max-width: 717px) 100vw, 717px" />
Step 10:
Once the job is completed, we can verify the output file at created and saved at provided output directory in Glue
<img loading="lazy" decoding="async" class="aligncenter wp-image-254 size-large" src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-9-1024x295.png" alt="" width="1024" height="295" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-9-1024x295.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-9-300x86.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-9-768x221.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-9-1536x442.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-9.png 1584w" sizes="(max-width: 1024px) 100vw, 1024px" />
<img loading="lazy" decoding="async" class="size-medium wp-image-257 aligncenter" src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-290x300.jpeg" alt="" width="290" height="300" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-290x300.jpeg 290w, https://data.cogtix.com/wp-content/uploads/2023/05/download.jpeg 387w" sizes="(max-width: 290px) 100vw, 290px" />
Step 11: Automate and create a schedule
Now let&#8217;s imagine your S3 data is changing every now and then and you would like to trigger the pipeline either using schedules or crawlers. AWS offers both options and is explained below
Schedules:
To create schedules, you can follow these steps
<ol>
<li>Click on &#8220;Schedules&#8221; on the job edit page</li>
<li>Set preferred cron schedule</li>
<li>Create schedule</li>
</ol>
<img loading="lazy" decoding="async" class="aligncenter wp-image-255 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-10.png" alt="" width="758" height="537" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-10.png 933w, https://data.cogtix.com/wp-content/uploads/2023/05/download-10-300x213.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-10-768x544.png 768w" sizes="(max-width: 758px) 100vw, 758px" />
Crawlers:
Alternatively, we can also create crawlers to trigger the pipeline when the input file updates
<ol>
<li>On AWS Management Console and navigate to the AWS Glue service.</li>
<li>Select &#8220;Crawlers&#8221; from the left-hand menu and click on the &#8220;Add crawler&#8221; button.</li>
<li>Provide a name for the crawler and select the data source type as &#8220;S3.&#8221;</li>
<li>In the configuration settings, specify the S3 path where your files are located and enable the option for &#8220;Update all new and existing files.&#8221;</li>
<li>Configure any additional settings or permissions as required, and then start the crawler to begin the process of scanning and cataloging the S3 data.</li>
</ol>
By following these steps, you can create an S3 crawler in AWS Glue that is triggered on file updates, ensuring that any changes in the specified S3 path are automatically detected and updated in the Glue Data Catalog.
<img loading="lazy" decoding="async" class="aligncenter wp-image-256 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-11-1024x290.png" alt="" width="798" height="226" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-11-1024x290.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-11-300x85.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-11-768x217.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-11-1536x435.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-11.png 1919w" sizes="(max-width: 798px) 100vw, 798px" />
<img loading="lazy" decoding="async" class="aligncenter wp-image-253 " src="https://serene-boyd.162-222-205-61.plesk.page/root/wp-content/uploads/2023/05/download-8-1024x159.png" alt="" width="807" height="125" srcset="https://data.cogtix.com/wp-content/uploads/2023/05/download-8-1024x159.png 1024w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-300x47.png 300w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-768x119.png 768w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8-1536x238.png 1536w, https://data.cogtix.com/wp-content/uploads/2023/05/download-8.png 1857w" sizes="(max-width: 807px) 100vw, 807px" />
Step 12:
If you have no plans to continue using this pipeline, feel free to delete it, along with any created schedules and crawlers, to avoid incurring any additional costs.
Congratulations! We have successfully created our very own ETL pipeline using Glue Studio. It reads data from S3, performs transformations, and writes the processed data back to S3. Additionally, we have set up schedules and crawler triggers, which greatly assist us in a production system where data frequently changes and requires the ETL pipeline to be triggered for processing. In the upcoming section, we will delve into the creation of an AWS Glue pipeline and learn how to utilize Apache Spark within it. Furthermore, we will write PySpark code to carry out data transformations and execute them on Spark clusters with the support of AWS Glue.