Parquet

Parquet is an open-source type-aware columnar data storage format that can store nested data in a flat columnar format. You can use Parquet files (ending in the file extension .parquet) in the Apache Spark and Hadoop ecosystems, as they are compatible with large data streaming and processing workflows. Parquet is highly structured meaning it stores the schema and data type of each column with the data files. To learn more about Parquet data format, see the Parquet documentation. To learn more about using Parquet files with Spark SQL, see Spark's documentation on the Parquet data source.

Note:

It is recommended to use GeoParquet to write DataFrames containing geometry data to Parquet format. GeoParquet's structure enables interoperability between any system that reads or writes spatial data in Parquet format. Use the Parquet data source described in this topic if your DataFrame does not contain any geometry columns.

After you load one or more Parquet files as a Spark DataFrame, you can create a geometry column and define its spatial reference using GeoAnalytics for Microsoft Fabric SQL functions. For example, if you had polygons stored as WKT strings you could call ST_PointFromText to create a point column from a string column and set its spatial reference. For more information see Geometry.

After creating the geometry column and defining its spatial reference, you can perform spatial analysis and visualization using the SQL functions and tools available in GeoAnalytics for Microsoft Fabric. You can also export a Spark DataFrame to Parquet files for data storage or export to other systems.

To add data that is stored in Delta Parquet format to a map, export it to your lakehouse as a CSV, GeoJSON, or Parquet.

Learn more about Parquet

GeoParquet

GeoParquet is a standardized open-source columnar storage format that extends Parquet by defining how to store geospatial data, including the representation of geometries and the required additional metadata. Parquet is highly structured meaning it stores the schema and data type of each column with the data files. GeoParquet's structure enables interoperability between any system that reads or writes spatial data in Parquet format. To write to Parquet format without geometry data, see the Parquet data source included with Apache Spark.

Learn more about GeoParquet.

Spatially optimized parquet

Esri leveraged the efficiency and scalability of Parquet data and extended it to support faster spatial queries and visualization through the development of the Spatially optimized Parquet format.

The spatial optimization includes the addition of two key characteristics:

Geographically adjacent data storage – Features that are geographically close are stored near one another in the file, allowing spatial queries to access nearby features without searching the entire dataset. This reduces the query response time.

Multiscale geometry – Generalized geometries are stored in separate columns, each optimized for a specific map scale based on the geographic extent and number of features. Clients can use simplified geometry at coarser scales for faster visualization and detailed geometry at finer scales.

Spatially optimized Parquet layers are currently built on the open GeoParquet format, aligning with the broader Parquet standard to promote interoperability across systems.

You can create spatially optimized Parquet layers using the st.with_geodisplay() DataFrame extension in ArcGIS GeoAnalytics for Fabric. This extension generates a GeoDisplay column using a geometry column in your DataFrame. You can add the output layer to ArcGIS Maps for Fabric from your lakehouse.

Learn more about Spatially optimized Parquet.