Digital Event Horizon
AWS, the world's largest cloud infrastructure provider, has declared its full support for Apache Iceberg, solidifying its position as the preferred standard for analytics and storage across its platform. This move is driven by customer demand and demonstrates AWS's commitment to innovation, emphasizing a unified and flexible approach to data storage, analytics, and machine learning.
AWS has declared full support for Apache Iceberg open table format (OTF), solidifying its position as the preferred standard for analytics and storage across its platform. The decision is driven by customer demand, with many of AWS's largest analytics customers already leveraging Iceberg on its S3 object storage. AWS has established core committers on the open-source stack to shape APIs and ensure seamless integration across various projects. The move underscores AWS's efforts to provide a unified and flexible platform for data storage, analytics, and machine learning. The acquisition of Tabular by Databricks highlights Iceberg's growing recognition and need for a standardized approach in data storage and analytics. AWS's existing Redshift offerings can now work with any Iceberg storage, enabling greater flexibility and access to data.
AWS, the world's largest cloud infrastructure provider, has made a significant announcement regarding its stance on the Apache Iceberg open table format (OTF). According to recent reports, AWS has declared its full support for Iceberg, solidifying its position as the preferred standard for analytics and storage across its platform. This move is seen as a strategic response to customer demand, with many of AWS's largest analytics customers already leveraging Iceberg on its S3 object storage.
In an interview with The Register, Andy Warfield, Vice President and Distinguished Engineer at AWS, explained that the company's decision to adopt Iceberg was driven by the growing consensus among its customers. Warfield stated that "we are working directly with Iceberg" and have established core committers on the open-source stack. This collaboration enables AWS to shape APIs and work closely with other contributors to ensure seamless integration across various projects.
The significance of this move cannot be overstated, particularly given the market share dominance of S3 in the global enterprise data storage software market (23%). With annual revenue projected to reach $105 billion, making it the largest cloud infrastructure provider by some margin, AWS's commitment to Iceberg sends a strong signal about its dedication to customer needs. Warfield acknowledged that "all of this stuff is just really being driven by the resounding voice of lot of our customers who are doing analytics." This emphasis on customer-driven innovation underscores AWS's efforts to provide a unified and flexible platform for data storage, analytics, and machine learning.
The decision also marks an important milestone in the evolution of Iceberg, which has gained significant traction among industry players. Databricks, the company behind Delta Lake, recently acquired Tabular, the original authors of Iceberg, for an estimated $1 billion to $2 billion without gaining control over the technology. This acquisition underscores the growing recognition of Iceberg's potential and the need for a standardized approach in data storage and analytics.
Warfield noted that AWS has been working closely with Databricks since it builds systems on top of S3, ensuring seamless integration between their platforms. However, the ball now lies firmly in Databricks' court, as the company must navigate the future of Delta Lake in relation to Iceberg. As Russel Spitzer, an Iceberg committer and PMC member, recently stated, "I hope vendors would all use Iceberg under the hood to eliminate table formats as a design point." This sentiment resonates with Warfield's comments about AWS aiming to provide "a really attractive direction in terms of its design" for building structured support on storage.
Furthermore, this development has implications for AWS's existing data warehouse offerings, particularly Redshift. With the introduction of Iceberg REST Catalog support inside the Sagemaker Lakehouse Catalog, Redshift can now work with any Iceberg storage, enabling greater flexibility and access to data. Conversely, Redshift through the Iceberg REST Catalog can also be accessed by other analytics platforms.
As the cloud computing landscape continues to evolve, AWS's commitment to Apache Iceberg serves as a significant catalyst for industry growth and innovation. With its vast resources and customer base, AWS has positioned itself at the forefront of this initiative, driving toward a future where data storage and analytics engines are unified under a single standard. As Warfield aptly put it, "we will obviously explore adding support for those things [alternatives]...but for now, Iceberg has emerged as a really attractive direction in terms of its design."
Related Information:
https://go.theregister.com/feed/www.theregister.com/2025/01/20/aws_iceberg_support/
Published: Mon Jan 20 18:35:55 2025 by llama3.2 3B Q4_K_M
