Talascend is currently seeking a Senior Azure Databricks Engineer for a contract opportunity with our client in Glendale, Arizona.
Overview
This role involves designing, implementing, and maintaining enterprise data platform solutions utilizing Azure Databricks and related Azure services. The engineer will work on data engineering frameworks, security architecture, data governance, workload optimization, and platform monitoring to support enterprise analytics and data management needs.
Responsibilities
- Collaborate with data platform and integration teams to architect and implement scalable enterprise data solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and other Azure services
- Design and develop data engineering frameworks for batch, streaming, ELT/ETL, and real-time workloads
- Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace governance, and secret management
- Implement and oversee enterprise security architecture, including identity integration, RBAC, Privileged Identity Management, data encryption, private networking, and Key Vault integration
- Establish data governance standards utilizing Unity Catalog, including data lineage, data quality, metadata management, and access policies
- Optimize Databricks workloads through performance tuning, Spark optimization, caching, partitioning, and cost management
- Develop operational dashboards, logging, alerting, and platform health monitoring with Azure Monitor, Log Analytics, and other tools
- Create reusable architecture patterns, standards, and documentation for platform support and migration activities
- Support migration of legacy data warehouses, ETL solutions, and reporting platforms to Azure Databricks and Lakehouse architecture
- Provide technical troubleshooting and support for production incidents involving Azure Databricks and data pipelines
Qualifications
- 8-10 years of experience in designing and implementing enterprise data platforms
- 5+ years of hands-on experience with Azure Databricks in enterprise environments
- Experience architecting Lakehouse solutions using Azure Databricks and Delta Lake
- Proficiency in administering Azure Databricks workspaces, clusters, and security features
- Experience developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL)
- Knowledge of data governance using Unity Catalog, data lineage, and access controls
- Strong understanding of enterprise security architectures including RBAC, managed identities, and network security
- Experience with data quality frameworks, observability, dashboards, and monitoring
- Solid knowledge of data modeling techniques such as dimensional modeling, Data Vault, and medallion architecture
- Proficiency in Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Azure AD, and networking
- Strong SQL, Python, and PySpark programming skills
- Experience with CI/CD pipelines using Azure DevOps, automation, and deployment management
Preferred Qualifications
- Experience in enterprise data warehouse concepts and migration projects
This position does not offer remote work and requires physical presence at the Glendale, Arizona location.
We thank all applicants for their interest. However, only those qualified individuals who closely meet the qualifications of the position will be contacted. The details of the position are only a summary, other duties may be assigned as necessary.
Background Check and Drug Screen may be required.
Talascend is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.