The Delta Lake format is a powerful data storage format that combines the reliability of ACID transactions, schema enforcement, and data versioning for big data workloads. It is designed to address common data lake challenges such as data quality, data reliability, and data lifecycle management.<\/p>\n
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics. It supports various data formats, including the Delta Lake format.<\/p>\n
To read the Delta Lake format in AWS Glue, follow these steps:<\/p>\n
1. Set up an AWS Glue job:<\/p>\n
Log in to the AWS Management Console and navigate to the AWS Glue service. Click on ‘ETL Jobs’ in the navigation pane and then click on ‘Author code with a script editor’, choose engine as ‘Spark’ then hit ‘Create Script’.<\/p>\n
2. Specify the Delta Lake format as the data source:<\/p>\n
In the ‘Job details’ section, provide a name for the job and select the ‘IAM Role’. Under ‘Job parameters’, choose ‘–datalake-formats’ as key and value as ‘delta’. Create another key named ‘–conf’ \u00a0and set it to the following value.<\/p>\n
spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore<\/pre>\n3. Configure the job script to read the Delta Lake format via glue libraries or using spark dataframe :
\nUsing Glue libraries :<\/p>\nimport sys\nfrom awsglue.transforms import *\nfrom awsglue.utils import getResolvedOptions\nfrom pyspark.context import SparkContext\nfrom awsglue.context import GlueContext\nfrom awsglue.job import Job\n\n## @params: [JOB_NAME]\nargs = getResolvedOptions(sys.argv, ['JOB_NAME'])\n\nsc = SparkContext()\nglueContext = GlueContext(sc)\nspark = glueContext.spark_session\njob = Job(glueContext)\njob.init(args['JOB_NAME'], args)\n\ndf = glueContext.create_data_frame.from_catalog(\ndatabase=\"<your_database_name>\",\ntable_name=\"<your_table_name>\",\nadditional_options=additional_options\n)\ndf.show()\njob.commit()<\/pre>\nUsing the Spark dataframe :<\/p>\n
deltaDF = spark.read.format(\"delta\").load(\"<s3path for delta table>\")\ndeltaDF.show()<\/pre>\n4. Run the AWS Glue job:<\/p>\n
Click on ‘Save’ to proceed to the ‘Job details’ section. Review and modify any necessary job parameters, such as the number of concurrent workers and the maximum capacity.<\/p>\n
Finally, click on ‘Run job’ to start reading the data from the S3 DeltaLake.<\/p>\n
Conclusion<\/h2>\n
The Delta Lake format in AWS Glue provides a reliable and efficient way to read and process big data workloads. By following the step-by-step guide mentioned above, you can easily set up an AWS Glue job to read the Delta Lake format and leverage its benefits for your data analytics needs.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction to Delta Lake Format The Delta Lake format is a powerful data storage format that combines the reliability of ACID transactions, schema enforcement, and data versioning for big data workloads. It is designed to address common data lake challenges such as data quality, data reliability, and data lifecycle management. Reading Delta Lake Format in AWS Glue AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics. It supports various data formats, including the Delta Lake format. To read the Delta Lake format in AWS Glue, follow these steps: Set up an AWS Glue job in the AWS Glue Studio. Specify the Delta Lake format as the data source in the job configuration. Configure the job to read the Delta Lake format using the appropriate input options. Run the AWS Glue job to read the data from the Delta Lake format. Step-by-Step Guide 1. Set up an AWS Glue job: Log in to the AWS Management Console and navigate to the AWS Glue service. Click on ‘ETL Jobs’ in the navigation pane and then click on ‘Author code with a script editor’, choose engine as ‘Spark’ then hit ‘Create Script’. 2. Specify the Delta Lake format as the data source: In the ‘Job details’ section, provide a name for the job and select the ‘IAM Role’. Under ‘Job parameters’, choose ‘–datalake-formats’ as key and value as ‘delta’. Create another key named ‘–conf’ \u00a0and set it to the following value. spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension –conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog –conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore 3. Configure the job script to read the Delta Lake format via glue libraries or using spark dataframe : Using Glue libraries : import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job ## @params: [JOB_NAME] args = getResolvedOptions(sys.argv, [‘JOB_NAME’]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args[‘JOB_NAME’], args) df = glueContext.create_data_frame.from_catalog( database=”<your_database_name>”, table_name=”<your_table_name>”, additional_options=additional_options ) df.show() job.commit() Using the Spark dataframe : deltaDF = spark.read.format(“delta”).load(“<s3path for delta table>”) deltaDF.show() 4. Run the AWS Glue job: Click on ‘Save’ to proceed to the ‘Job details’ section. Review and modify any necessary job parameters, such as the number of concurrent workers and the maximum capacity. Finally, click on ‘Run job’ to start reading the data from the S3 DeltaLake. Conclusion The Delta Lake format in AWS Glue provides a reliable and efficient way to read and process big data workloads. By following the step-by-step guide mentioned above, you can easily set up an AWS Glue job to read the Delta Lake format and leverage its benefits for your data analytics needs.<\/p>\n","protected":false},"author":8,"featured_media":76,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[4],"tags":[6,7,5],"class_list":["post-51","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","tag-aws-glue","tag-data-analytics","tag-delta-lake"],"_links":{"self":[{"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/posts\/51","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/comments?post=51"}],"version-history":[{"count":3,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/posts\/51\/revisions"}],"predecessor-version":[{"id":79,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/posts\/51\/revisions\/79"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/media\/76"}],"wp:attachment":[{"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/media?parent=51"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/categories?post=51"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataaiguru.com\/index.php\/wp-json\/wp\/v2\/tags?post=51"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}