We’re excited to launch Unified Profiling for PySpark Person-Outlined Capabilities (UDFs) as a part of Databricks Runtime 17.0 (launch notes). Unified Profiling for PySpark UDFs lets builders profile the efficiency and reminiscence utilization of their PySpark UDFs, together with monitoring operate calls, execution time, reminiscence utilization, and different metrics. This allows PySpark builders to simply determine and handle bottlenecks, resulting in quicker and extra resource-efficient UDFs.
The unified profilers could be enabled by setting the Runtime SQL configuration “spark.sql.pyspark.udf.profiler” to “perf” or “reminiscence” to allow the efficiency or reminiscence profiler, respectively, as proven under.
Substitute for Legacy Profiling
Legacy profiling [1, 2] was applied on the SparkContext degree and, thus, didn’t work with Spark Join. The brand new profiling is SparkSession-based, applies to Spark Join, and could be enabled or disabled at runtime. It maximizes API parity with legacy profiling by offering “present” and “dump” instructions to visualise profile outcomes and save them to a workspace folder. Moreover, it affords comfort APIs to assist handle and reset profile outcomes on demand. Lastly, it helps registered UDFs, which weren’t supported by the legacy profiling.
PySpark Efficiency Profiler
The PySpark efficiency profiler leverages Python’s built-in profilers to increase profiling capabilities to the motive force and UDFs executed on executors in a distributed method.
Let’s dive into an instance to see the PySpark efficiency profiler in motion. We run the next code on Databricks Runtime 17.0 notebooks.
The added.present() command shows efficiency profiling outcomes as proven under.
The output consists of info such because the variety of operate calls, whole time spent within the given operate, and the filename, together with the road quantity to help navigation. This info is important for figuring out tight loops in your PySpark applications and enabling you to make choices to enhance efficiency.
It is essential to notice that the UDF id in these outcomes straight correlates with the one discovered within the Spark plan, by observing the “ArrowEvalPython [add1(…)#50L]”, which is revealed when calling the clarify technique on the dataframe.
Lastly, we are able to dump the profiling outcomes to a folder and clear the outcome profiles as proven under.
PySpark Reminiscence Profiler
It’s based mostly on memory-profiler, which may profile the motive force, as seen right here. PySpark has expanded its utilization to incorporate profiling UDFs, that are executed on executors in a distributed method.
To allow reminiscence profiling on a cluster, we must always set up the memory-profiler on the cluster as proven under.
The above instance modifies the final two strains by:
Then we get hold of reminiscence profiling outcomes as proven under.
The output consists of a number of columns that offer you a complete view of how your code performs by way of reminiscence utilization. “Mem utilization” reveals the reminiscence utilization after executing that line. “Increment” particulars the change in reminiscence utilization from the earlier line, serving to you notice the place reminiscence utilization spikes. “Occurrences” signifies what number of occasions every line was executed.
The UDF id in these outcomes additionally straight correlates with the one discovered within the Spark plan, the identical as efficiency profiling outcomes, by observing the “ArrowEvalPython [add1(…)#4L]”, which is revealed when calling the clarify technique on the dataframe as proven under.
Please be aware that for this performance to work, the memory-profiler bundle should be put in in your cluster.
Conclusion
PySpark Unified Profiling, which incorporates efficiency and reminiscence profiling for UDFs, is offered in Databricks Runtime 17.0. Unified Profiling offers a streamlined technique for observing essential elements reminiscent of operate name frequency, execution durations, and reminiscence consumption. It simplifies the method of pinpointing and resolving bottlenecks, paving the way in which for the event of quicker and extra resource-efficient UDFs.
Able to discover extra? Try the PySpark API documentation for detailed guides and examples.
