Unity Catalog Python user-defined capabilities (UC Python UDFs) are more and more utilized in trendy knowledge warehousing, operating tens of millions of queries every day throughout hundreds of organizations. These capabilities enable customers to harness the total energy of Python from any Unity Catalog-enabled compute, together with clusters, SQL warehouses and DLT.
We’re excited to announce a number of enhancements to UC Python UDFs that at the moment are obtainable in Public Preview on AWS, Azure, and GCP with Unity Catalog clusters operating Databricks Runtime 16.3, SQL warehouses (2025.15), and Serverless notebooks and workflows:
- Help for customized Python dependencies, put in from Unity Catalog Volumes or exterior sources.
- Batch enter mode, providing extra flexibility and improved efficiency.
- Safe entry to exterior cloud providers utilizing Unity Catalog Service Credentials.
Every of those options unlocks new potentialities for working with knowledge and exterior programs straight from SQL. Beneath, we’ll stroll by way of the small print and examples.
Utilizing customized dependencies in UC Python UDFs
Customers can now set up and use customized Python dependencies in UC Python UDFs. You’ll be able to set up these packages from PyPI, Unity Catalog Volumes, and blob storage. The instance perform beneath installs the pycryptodome from PyPI to return SHA3-256 hashes:
With this characteristic, you may outline steady Python environments, keep away from boilerplate code, and convey the capabilities of UC Python UDFs nearer to session-based PySpark UDFs. Dependency installations can be found beginning with Databricks Runtime 16.3, on SQL warehouses, and in Serverless notebooks and workflows.
Introducing Batch UC Python UDFs
UC Python UDFs now enable capabilities to function on batches of knowledge, just like vectorized Python UDFs in PySpark. The brand new perform interface provides enhanced flexibility and offers a number of advantages:
- The batched execution offers customers extra flexibility: UDFs can hold state between batches, i.e., carry out costly initialization work as soon as on startup.
- UDFs leveraging vectorized operations on pandas collection can enhance efficiency in comparison with row-at-a-time execution.
- As proven within the cloud perform name instance beneath, sending batched knowledge to cloud providers could be more cost effective than invoking them one row at a time.
Batch UC Python UDFs, now obtainable on AWS, Azure, and GCP, are also called Pandas UDFs or Vectorized Python UDFs. They’re launched by marking a UC Python UDF with PARAMETER STYLE PANDAS and specifying a HANDLER perform to be known as by identify. The handler perform is a Python perform that receives an iterator of pandas Sequence, the place every pandas Sequence corresponds to 1 batch. The handler capabilities are appropriate with the pandas_udf API.
For instance, take into account the beneath UDF that calculates the inhabitants by state, based mostly on a JSON object mapping that it downloaded on startup:
Unity Catalog Service Credential entry
Customers can now leverage Unity Catalog service credentials in Batch UC Python UDFs to effectively and securely entry exterior cloud providers. This performance permits customers to work together with cloud providers straight from SQL.
UC Service Credentials are ruled objects in Unity Catalog. They will present entry to any cloud service, resembling key-value shops, key administration providers, or cloud capabilities. UC Service credentials can be found in all main clouds and are at the moment accessible from Batch UC Python UDFs. Help for regular UC Python UDFs will observe sooner or later.
Service credentials can be found to Batch UC Python UDFs utilizing the CREDENTIALS clause within the UDF definition (AWS, Azure, GCP).
Instance: Calling a cloud perform from Batch UC Python UDFs
In our instance, we’ll name a cloud perform from a Batch UC Python UDF. This performance permits for seamless integration with present capabilities and permits using any base container, programming language, or surroundings.
With Unity Catalog, we will implement efficient governance of each Service Credential and UDF objects. Within the determine above, Alice is the proprietor and definer of the UDF. Alice can grant EXECUTE permission for the UDF to Bob. When Bob calls the UDF, Unity Catalog Lakeguard will run the UDF with Alice’s service credential permissions whereas guaranteeing that Bob can’t entry the service credential straight. UDFs will use the defining consumer’s permissions to entry the credentials.
Whereas all three main clouds are supported, we’ll concentrate on AWS on this instance. Within the following, we’ll stroll by way of the steps to create and name the Lambda perform.
Making a UC service credential
As a prerequisite, we should arrange a UC Service Credential with the suitable permissions to execute Lambda capabilities. For this, we observe the directions to arrange a service credential known as mycredential. Moreover, we enable our function to invoke capabilities by attaching the AWSLambdaRole coverage.
Making a Lambda perform
Within the second step, we create an AWS Lambda perform by way of the AWS UI. Our instance Lambda HashValuesFunctionNode runs in nodejs20.x and computes a hash of its enter knowledge:
Invoking a Lambda from a Batch UC Python UDFs
Within the third step, we will now write a Batch UC Python UDF that calls the Lambda perform. The UDF beneath makes the service credentials obtainable by specifying them within the CREDENTIALS clause. The UDF invokes the Lambda perform for every enter batch, calling cloud capabilities with a whole batch of knowledge could be extra cost-efficient than calling them row-wise. The instance additionally demonstrates how one can ahead the invoking consumer’s identify from Spark’s TaskContext to the Lambda perform, which could be helpful for attribution:
Get began in the present day
Check out the Public Preview of Enhanced Python UDFs in Unity Catalog – to put in dependencies, to leverage the batched enter mode, or to make use of UC service credentials!
Be a part of the UC Compute and Spark product and engineering workforce on the Information + AI Summit, June 9–12 on the Moscone Heart in San Francisco! Get a primary have a look at the most recent improvements in knowledge and AI governance and safety. Register now to safe your spot!
