Skip to content
Large-Scale Data Engineering in Cloud

Performance Tuning, Cost Optimization / Internals, Research

  • About
  • About
  • ETL,  Hive,  Presto

    Presto vs Hive – SLA Risks for Long Running ETL – Failures and Retries Due to Node Loss

    December 4, 2019

    Presto is an extremely powerful distributed SQL query engine, so at some point you may consider using it to replace SQL-based ETL processes that you currently run on Apache Hive.

    Although it is completely possible, you should be aware of some limitations that may affect your SLAs.

    Read More
    dmtolpeko
  • ETL,  Snowflake

    Snowflake – Reloading Data from Stage – TRUNCATE, DELETE, COPY and Transactions

    May 6, 2019

    Sometimes you need to reload the entire data set from the source storage into Snowflake. For example, you may want to fully refresh a quite large lookup table (2 GB compressed) without keeping the history. Let’s see how to do this in Snowflake and what issues you need to take into account.

    Read More
    dmtolpeko

Recent Posts

  • Aug 30, 2022 Spark 2.4 – Slow Performance on Writing into Partitions – Why Sorting Involved
  • Aug 30, 2022 Spark – Create Multiple Output Files per Task using spark.sql.files.maxRecordsPerFile
  • Aug 29, 2022 EMR Spark – Initial Number of Executors and spark.dynamicAllocation.enabled
  • Aug 26, 2022 EMR Spark – Much Larger Executors are Created than Requested
  • Apr 20, 2022 Amazon EMR Spark – Ignoring Partition Filter and Listing All Partitions When Reading from S3A

Archives

  • August 2022 (4)
  • April 2022 (1)
  • March 2021 (2)
  • January 2021 (2)
  • June 2020 (4)
  • May 2020 (8)
  • April 2020 (3)
  • February 2020 (3)
  • December 2019 (5)
  • November 2019 (4)
  • October 2019 (1)
  • September 2019 (2)
  • August 2019 (1)
  • May 2019 (9)
  • April 2019 (2)
  • January 2019 (3)
  • December 2018 (4)
  • November 2018 (1)
  • October 2018 (6)
  • September 2018 (2)

Categories

  • Amazon (14)
  • Auto Scaling (1)
  • AWS (28)
  • Cost Optimization (1)
  • CPU (2)
  • Data Skew (1)
  • Distributed (1)
  • EC2 (1)
  • EMR (13)
  • ETL (2)
  • Flink (5)
  • Hadoop (14)
  • Hive (17)
  • Hue (1)
  • I/O (22)
  • JVM (3)
  • Kinesis (1)
  • Logs (1)
  • Memory (7)
  • Monitoring (4)
  • ORC (5)
  • Parquet (7)
  • Pig (2)
  • Presto (3)
  • Qubole (2)
  • RDS (1)
  • S3 (18)
  • Snowflake (6)
  • Spark (9)
  • Storage (14)
  • Tez (10)
  • YARN (18)

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org
Savona Theme by Optima Themes