Large-Scale Analytics and Data Engineering in Cloud

Hive, Presto, Spark, Tez – Performance Tuning and Cost Optimization

  • About
  • About
  • AWS,  EMR,  Hadoop,  YARN

    Amazon EMR – Downscaling and Ghost (Impaired) Nodes

    August 27, 2019

    I already wrote about recovering the ghost nodes in Amazon EMR clusters, but in this article I am going to extend this information for EMR clusters configured for auto-scaling.

    Read More
    dmtolpeko

Recent Posts

  • Nov 16, 2019 Apache Hive – Monitoring Progress of Long-Running Reducers – hive.log.every.n.records Option
  • Nov 01, 2019 Apache Hive/Pig on Tez – Long Running Tasks and Their Failed Attempts – Analyzing Performance and Finding Bottlenecks (Insufficient Parallelism) using Application Master Logs
  • Oct 18, 2019 Sqoop Import from Amazon RDS Read Replica – ERROR: Canceling Statement due to Conflict with Recovery
  • Sep 11, 2019 Hadoop YARN – Calculating Per Second Utilization of Cluster using Resource Manager Logs
  • Sep 04, 2019 Tuning Hadoop YARN – Boosting Memory Settings Beyond the Limits to Increase Cluster Capacity and Utilization

Archives

  • November 2019 (2)
  • October 2019 (1)
  • September 2019 (2)
  • August 2019 (1)
  • May 2019 (9)
  • April 2019 (2)
  • January 2019 (3)
  • December 2018 (4)
  • November 2018 (1)
  • October 2018 (6)
  • September 2018 (2)

Categories

  • Amazon (10)
  • AWS (14)
  • Cost Optimization (1)
  • Data Skew (1)
  • Distributed (1)
  • EMR (9)
  • ETL (1)
  • Hadoop (8)
  • Hive (11)
  • Hue (1)
  • I/O (10)
  • Logs (1)
  • Memory (3)
  • Monitoring (4)
  • ORC (5)
  • Pig (2)
  • Qubole (1)
  • RDS (1)
  • S3 (7)
  • Snowflake (5)
  • Spark (1)
  • Storage (9)
  • Tez (7)
  • YARN (10)

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org
Savona Theme by Optima Themes