How to Access Azure Cosmos DB from Databricks Workspace

Azure Cosmos DB is a fully managed and serverless NoSQL and vector database solution from Microsoft. Azure Databricks is a service jointly developed by Microsoft and Databricks (the creators of Apache Spark) for data engineering, analytics, and machine learning. You can also find Databricks on AWS and Databricks on Google Cloud. The screenshots below illustrate a Cosmos DB for NoSQL and Azure Databricks workspace. In the Cosmos DB, we have a database called SampleDB and a container called SampleContainer with some data. In the workspace, we spin up a compute, SampleComute, to run the Python code in the workspace notebooks.

Cosmos DB with data
Compute in Databricks workspace
Notebook in Databricks workspace

There are two ways to access the data in Cosmos DB from Databricks workspace notebooks:

  1. Using Cosmos DB Spark 3 Connector with DB Key
  2. Using CosmosClient with role-based access

Using Cosmos DB Spark 3 Connector with DB Key

This way is more straightforward, but we will need the DB key. First, we install the Maven package azure-cosmos-spark from Maven Central on the compute.

Installing azure-cosmos-spark from Maven Central
Installing azure-cosmos-spark from Maven Central
azure-cosmos-spark installed

Next, we collect the master key, database name, and container name from the Cosmos DB, and put them in the config. You can refer to azure-cosmos-spark documentation for more optional configurations.

config = {
  "spark.cosmos.accountEndpoint": "https://<cosmosdb-name>.documents.azure.com:443/",
  "spark.cosmos.accountKey": "<cosmos-account-key>",
  "spark.cosmos.database": "<db-name>",
  "spark.cosmos.container": "<container-name>"
}
Cosmos DB Keys

You can also choose to store the account key in Secret Scope to avoid plain text in the code snippet.  For example, if the account key is stored in Secret Scope named <scope-name> with the key name <key-name>, it can be retrieved with dbutils.secrets.get():

"spark.cosmos.accountKey": dbutils.secrets.get(scope="<scope-name>", key="<key-name>")

Finally, we use Cosmos DB Spark 3 Connector to read the data as a Pandas Dataframe:

df = spark.read.format("cosmos.oltp").options(**config)
  .option("spark.cosmos.read.customQuery", "SELECT * FROM c OFFSET 0 LIMIT 3").load()
df.head()

The screenshot below shows that the first three rows are retrieved

Data retrieved from Cosmos DB

References:

Using CosmosClient with Role-Based Access

Alternatively, we can leverage role-based access to avoid using the DB key. This way, we assign a role to the Service Principal (SP) on the Cosmos and use CosmosClient to access the data. First, we collect the object ID of the SP for role assignment. Make sure to use the Object ID in the Enterprise Applications section.

Object ID of Service Principal in Enterprise Applications

We want to assign the Cosmos DB Built-in Data Contributor role to the SP. From this document, we know that the ID of this role is 00000000-0000-0000-0000-000000000002. Run the commands below with Azure CLI for role assignment. You can find PowerShell commands or other languages in the reference at the end.

$resourceGroupName="<resource-group-name>"
$accountName="<cosmosdb-name>"
$DataContributorRoleDefinitionId="00000000-0000-0000-0000-000000000002"
# Ensure to use the Object ID in the Enterprise applications of the Azure Active Directory
$principalId="<sp-object-id>"

az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName --scope "/" --principal-id $principalId --role-definition-id $DataContributorRoleDefinitionId
Role assigned to the SP on the Cosmos

Next, we install PyPI packages: azure-cosmos and azure-identity on the compute.

Installing azure-cosmos and azure-identity from PyPI
azure-cosmos and azure-identity installed

Before creating the CosmosClient, we need the SP’s tenant ID, client ID, and client secret. You can find them in the App Registration from Azure Portal.

App registration of the SP
Create the client secret for App Registration (Service Principal)

We can then create a CosmosClient and use it to access CosmosDB with the Python code below:

from azure.identity import ClientSecretCredential
from azure.cosmos import CosmosClient

ENDPOINT = "https://<cosmosdb-name>.documents.azure.com:443/"
TENANT_ID = "<tenant-id>"
CLIENT_ID = "<client-id>"
CLIENT_SECRET = "<client-secret>"
credential = ClientSecretCredential(
    tenant_id=TENANT_ID, client_id=CLIENT_ID, client_secret=CLIENT_SECRET
)

client = CosmosClient(ENDPOINT, credential)
db = client.get_database_client("<db-name>")
container = db.get_container_client("<container-name>")
result = container.query_items(
  query="SELECT * FROM c OFFSET 0 LIMIT 3",
  enable_cross_partition_query=True
)
print(list(result))
Data retrieved from Cosmos DB with CosmosClient

References:

Access Troubleshooting

Network

If the Cosmos DB only allows selected networks, the notebook code may return firewall-related errors, such as:

# Public IP blocked
{"innerErrorMessage": "Request originated from IP X.X.X.X through public internet. This is blocked by your Cosmos DB account firewall settings."}

# VNet blocked
{"Errors": "Request originated from VNET through service endpoint. This is blocked by your Cosmos DB account firewall settings"}

To resolve the issue, you must add the public IP or the VNets used by Databricks (more precisely, by the compute) to the Cosmos, as shown in the screenshot below.

Whitelist VNets or Public IPs on Cosmos

The VNets used by Databricks are displayed in the Overview section with names Private Subnet Name and Public Subnet Name.

The VNets used by Databricks

Py4JJavaError when Calling Pandas Methods

I encountered a weird error: raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) when calling df.head(). The root cause was that the version of azure-cosmos-spark was incompatible with the compute’s Spark runtime version. For instance, the error happened when the compute’s Databricks Runtime Version was 15.4 LTS (Spark 3.5.0), and I used com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12. Downgrading the runtime version to 13.3 LTS (Spark 3.4.1) resolved the issue.

Edit Databricks runtime version for the compute
Jim
Jim

Site reliability engineer working in a global gaming and entertainment company

Articles: 18

Leave a Reply

Your email address will not be published. Required fields are marked *