Menu

📰
0

Writing Parquet Files Using Haskell

Hacker News·about 10 hours ago
#4bXInWiw
Reading 0:00
15s threshold

We implemented a parquet writer in DataHaskell/Dataframe . Using it is simple; you need only pass your dataframe into the writeParquet function which writes a parquet file with sane defaults for row group and page sizes. An example: import qualified DataFrame as D import qualified DataFrame.Functions as F import DataFrame ( as , ( |> )) main = do sales <- D . readParquet "sales_data.parquet" sales |> D . groupBy [ "product" ] |> D . aggregate [ F . sum ( F . col @ Int "amount" ) ` as ` "total" , F . count ( F . col @ Int "amount" ) ` as ` "orders" ] |> D . writeParquet "total_orders.parquet" If you need more fine-grained control over the parquet file you’ll want to use writeParquetWithOptions . Read on to see what those options are and how they affect the final file. For Haskell to interoperate with the data ecosystem, it must be able to understand the standard formats in use by that ecosystem.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More