• AWS,  EMR,  Hadoop,  YARN

    Amazon EMR - Recovering Ghost Nodes

    May 23, 2019

    In a Hadoop cluster besides Active nodes you may also have Unhealthy, Lost and Decommissioned nodes. Unhealthy nodes are running but just excluded from scheduling the tasks because they, for example, do not have enough disk space. Lost nodes are nodes that are not reachable anymore. The decommissioned nodes are nodes that successfully terminated and left the cluster.

    But all these nodes are known to the Hadoop YARN cluster and you can see their details such as IP addresses, last health updates and so on.

    At the same time there can be also Ghost nodes i.e. nodes that are running by Amazon EMR services but Hadoop itself does not know anything about their existence. Let’s see how you can find them.

    Read More
  • AWS,  EMR,  Hadoop,  YARN

    Hadoop YARN - Collecting Utilization Metrics from Multiple Clusters

    May 15, 2019

    When you run many Hadoop clusters it is useful to automatically collect metrics from all clusters in a single place (Hive table i.e.).

    This allows you to perform any advanced and custom analysis of your clusters workload and not be limited to the features provided by Hadoop Administration UI tools that often offer only per cluster view so it is hard to see the whole picture of your data platform.

    Read More
  • I/O,  Snowflake,  Storage

    Snowflake - Monitoring Data Ingestion using QUERY_HISTORY and COPY_HISTORY - Single Large File vs Multiple Small Files

    May 14, 2019

    Snowflake provides various options to monitor data ingestion from external storage such as Amazon S3. In this article I am going to review QUERY_HISTORY and COPY_HISTORY table functions.

    The COPY commands are widely used to move data into Snowflake on a time-interval basis, and we can monitor their execution accessing the query history with query_type = 'COPY' filter.

    Read More
  • Cost Optimization,  Snowflake

    Snowflake - Data Ingestion - Cluster Utilization, Idle Time and Compute Cost

    May 11, 2019

    Snowflake separates compute and storage, so it is typical to have a dedicated compute cluster (virtual warehouse) to handle data ingestion into Snowflake (if you do not use Snowpipe).

    Like reporting and Ad-hoc SQL, data ingestion has some specifics. Usually there are many tables (data sources) that have own schedule (daily, hourly, every 5, 10, 15 minutes etc.) for periodic data transfer. ETL processes can be overlapped, and can have spikes followed by idle time and so on.

    Read More
  • I/O,  Snowflake

    Snowflake - Remote Disk I/O, Local Disk Cache - Capacity, Utilization and Transfer Rate

    May 4, 2019

    Snowflake uses a cloud storage service such as Amazon S3 as permanent storage for data (Remote Disk in terms of Snowflake), but it can also use Local Disk (SSD) to temporarily cache data used by SQL queries. Let’s test Remote and Local I/O performance by executing a sample SQL query multiple times on X-Large and Medium size Snowflake warehouses:

    SELECT MIN(event_hour), MAX(event_hour) FROM events WHERE event_name = 'LOGIN';

    Note that you should disable the Result Cache for queries in your session to perform such tests, otherwise Snowflake will just return the cached result immediately after the first attempt:

    alter session set USE_CACHED_RESULT = FALSE;
    Read More