Performance Tuning, Cost Optimization / Internals, Research by Dmitry Tolpeko
In cloud it is typical to run multiple compute clusters, so browsing the Web UI for every cluster to check the current resource consumption by applications is not always easy and convenient especially if YARN clusters are managed by different Hadoop distributions (Amazon EMR, Cloudera, Qubole etc.).
Let’s see how you can automate this process and find out how many applications are running and which resources they are consuming (containers, memory and CPU).
It is quite common today to convert incoming JSON data into Parquet format to improve the performance of analytical queries.
When JSON data has an arbitrary schema i.e. different records can contain different key-value pairs, it is common to parse such JSON payloads into a map column in Parquet.
How is it stored? What read performance can you expect? Will json_map["key"] read only data for key or the entire JSON?
It is quite common to have a streaming Flink application that reads incoming data and puts them into Parquet files with low latency (a couple of minutes) for analysts to be able to run both near-realtime and historical ad-hoc analysis mostly using SQL queries.
But let’s review write patterns and problems that can appear for such applications at scale.
Amazon S3 is highly scalable distributed system that can handle extremely large volumes of data, can adapt to an increasing workload and provide quite good performance as a file storage.
But sometimes you have to tweak it to run faster that can be especially important for latency-sensitive applications.