Specialist - Data Engineering

Primary skills are Primary skills are Azure Databricks and

Secondary Skills are API proxies Environment Management Open AI Swagger RESTful API

Location India MultiNational FMCG

FunctionPortfolio Application Landscape Digital Customer Experience Engineering Platforms

Employment Type Fulltime

Level Developer Senior Developer

Cloud Landscape Hyperscalerfirst strong enterprise investment in Azure and cloud platforms

Role Summary

We are looking for a Developer Senior Developer with strong expertise in Azure Databricks to build and operate scalable data engineering and analytics solutions for a global FMCG enterprise Beyond core Databricks delivery you will work with API proxies RESTful APIs OpenAI integrations and Swaggerbased API documentation enabling intelligent datadriven capabilities across the organizations digital and customer experience application landscape

Key Responsibilities

Azure Databricks Engineering

o Design develop and optimize data pipelines and transformation workflows using Azure Databricks PySpark Spark SQL Delta Lake

o Implement Delta Lake patterns ACID transactions schema evolution time travel and mergeupsert operations

o Build and manage Databricks Workflows Jobs Tasks Clusters for orchestration and scheduling

o Optimize Databricks workloads for performance and cost cluster sizing caching partitioning and autoscaling

o Implement Unity Catalog governance patterns data access lineage and classification where applicable

API Integration Proxy Management

o Develop and consume RESTful APIs to integrate Databricksprocessed data with downstream applications and digital platforms

o Work with API proxies eg Azure API ManagementAPIM for routing throttling versioning and securing API endpoints

o Manage environment configurations devteststagingprod for API deployments and Databricks workspaces

o Document APIs using Swagger OpenAPI Specification for developerfriendly standardized API contracts

OpenAI GenAI Integration

o Integrate Azure OpenAI APIs into data and application workflows for intelligent processing use cases eg text summarization classification enrichment or conversational interfaces

o Implement prompt engineering patterns structured outputs and guardrails for enterprisegrade AI features

o Build scalable secure connectors between Databricks data layers and OpenAI endpoints

Environment Lifecycle Management

o Manage multienvironment Databricks workspace configurations devtestprod and deployment pipelines

o Implement CICD for Databricks notebooksrepos using Azure DevOps or GitHub Actions

o Ensure security access control and compliance across environments

Senior Developer Specifics

o Lead technical design for Databricks solutions and API integration patterns

o Define standards for API documentation versioning and governance

o Mentor junior engineers and drive production readiness reviews

Primary Skills MustHave

Strong handson experience with Azure Databricks PySpark Spark SQL Delta Lake Workflows

Solid understanding of data engineering fundamentals batchstreaming pipelines data modeling and performance optimization

Experience with Python for data processing utilities and API integration

Strong SQL skills for complex transformations and data validation

Secondary Skills GoodtoHave

API Proxies Experience with Azure API Management APIM or equivalent API gatewayproxy solutions

Environment Management Multienvironment deployment and configuration management for Databricks and APIs

OpenAI Handson experience integrating Azure OpenAI APIs into data or application workflows

Swagger Experience authoring and consuming OpenAPISwagger specifications for RESTful APIs

RESTful API Strong understanding of REST principles HTTP standards authentication OAuth2JWT and API versioning