Azure Cosmos DB is a fully managed and serverless NoSQL and vector database solution from Microsoft. Azure Databricks is a service jointly developed by Microsoft and Databricks (the creators of Apache Spark) for data engineering, analytics, and machine learning. You can also find Databricks on AWS and Databricks on Google Cloud. The screenshots below illustrate a Cosmos DB for NoSQL and Azure Databricks workspace. In the Cosmos DB, we have a database called SampleDB and a container called SampleContainer with some data. In the workspace, we spin up a compute, SampleComute, to run the Python code in the workspace notebooks.



There are two ways to access the data in Cosmos DB from Databricks workspace notebooks:
- Using Cosmos DB Spark 3 Connector with DB Key
- Using CosmosClient with role-based access
Using Cosmos DB Spark 3 Connector with DB Key
This way is more straightforward, but we will need the DB key. First, we install the Maven package azure-cosmos-spark from Maven Central on the compute.



Next, we collect the master key, database name, and container name from the Cosmos DB, and put them in the config. You can refer to azure-cosmos-spark documentation for more optional configurations.
config = {
"spark.cosmos.accountEndpoint": "https://<cosmosdb-name>.documents.azure.com:443/",
"spark.cosmos.accountKey": "<cosmos-account-key>",
"spark.cosmos.database": "<db-name>",
"spark.cosmos.container": "<container-name>"
}
You can also choose to store the account key in Secret Scope to avoid plain text in the code snippet. For example, if the account key is stored in Secret Scope named <scope-name> with the key name <key-name>, it can be retrieved with dbutils.secrets.get():
"spark.cosmos.accountKey": dbutils.secrets.get(scope="<scope-name>", key="<key-name>")Finally, we use Cosmos DB Spark 3 Connector to read the data as a Pandas Dataframe:
df = spark.read.format("cosmos.oltp").options(**config)
.option("spark.cosmos.read.customQuery", "SELECT * FROM c OFFSET 0 LIMIT 3").load()
df.head()The screenshot below shows that the first three rows are retrieved

References:
- Tutorial: Connect by using Spark – Azure Cosmos DB for NoSQL | Microsoft Learn
- Tutorial – Implement Azure Databricks with an Azure Cosmos DB endpoint | Microsoft Learn
- Azure/azure-cosmosdb-spark: Apache Spark Connector for Azure Cosmos DB
Using CosmosClient with Role-Based Access
Alternatively, we can leverage role-based access to avoid using the DB key. This way, we assign a role to the Service Principal (SP) on the Cosmos and use CosmosClient to access the data. First, we collect the object ID of the SP for role assignment. Make sure to use the Object ID in the Enterprise Applications section.

We want to assign the Cosmos DB Built-in Data Contributor role to the SP. From this document, we know that the ID of this role is 00000000-0000-0000-0000-000000000002. Run the commands below with Azure CLI for role assignment. You can find PowerShell commands or other languages in the reference at the end.
$resourceGroupName="<resource-group-name>"
$accountName="<cosmosdb-name>"
$DataContributorRoleDefinitionId="00000000-0000-0000-0000-000000000002"
# Ensure to use the Object ID in the Enterprise applications of the Azure Active Directory
$principalId="<sp-object-id>"
az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName --scope "/" --principal-id $principalId --role-definition-id $DataContributorRoleDefinitionId

Next, we install PyPI packages: azure-cosmos and azure-identity on the compute.


Before creating the CosmosClient, we need the SP’s tenant ID, client ID, and client secret. You can find them in the App Registration from Azure Portal.


We can then create a CosmosClient and use it to access CosmosDB with the Python code below:
from azure.identity import ClientSecretCredential
from azure.cosmos import CosmosClient
ENDPOINT = "https://<cosmosdb-name>.documents.azure.com:443/"
TENANT_ID = "<tenant-id>"
CLIENT_ID = "<client-id>"
CLIENT_SECRET = "<client-secret>"
credential = ClientSecretCredential(
tenant_id=TENANT_ID, client_id=CLIENT_ID, client_secret=CLIENT_SECRET
)
client = CosmosClient(ENDPOINT, credential)
db = client.get_database_client("<db-name>")
container = db.get_container_client("<container-name>")
result = container.query_items(
query="SELECT * FROM c OFFSET 0 LIMIT 3",
enable_cross_partition_query=True
)
print(list(result))
References:
- Use data plane role-based access control – Azure Cosmos DB for NoSQL | Microsoft Learn
- Get started with Azure Cosmos DB for NoSQL using Python | Microsoft Learn
Access Troubleshooting
Network
If the Cosmos DB only allows selected networks, the notebook code may return firewall-related errors, such as:
# Public IP blocked
{"innerErrorMessage": "Request originated from IP X.X.X.X through public internet. This is blocked by your Cosmos DB account firewall settings."}
# VNet blocked
{"Errors": "Request originated from VNET through service endpoint. This is blocked by your Cosmos DB account firewall settings"}To resolve the issue, you must add the public IP or the VNets used by Databricks (more precisely, by the compute) to the Cosmos, as shown in the screenshot below.

The VNets used by Databricks are displayed in the Overview section with names Private Subnet Name and Public Subnet Name.

Py4JJavaError when Calling Pandas Methods
I encountered a weird error: raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) when calling df.head(). The root cause was that the version of azure-cosmos-spark was incompatible with the compute’s Spark runtime version. For instance, the error happened when the compute’s Databricks Runtime Version was 15.4 LTS (Spark 3.5.0), and I used com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12. Downgrading the runtime version to 13.3 LTS (Spark 3.4.1) resolved the issue.

