Overview:
Must Have Technical / Functional Skills
Strong core data engineering competencies with expertise in data concepts and data modelling
Experience with Big Data platforms and scalable data pipeline design
Hands-on experience with Microsoft Fabric Data Agents and Azure AI Services
Knowledge of AI/ML integration within enterprise data platforms
Strong analytical and problem-solving capabilities
Expertise in performance tuning, monitoring, and optimization
Proficiency in PySpark for large-scale data processing
Ecommerce domain knowledge with customer behavior analytics experience
Experience with Adobe Analytics, Adobe Customer Journey Analytics (CJA), and clickstream data analysis
Roles & Responsibilities
Experimentation Data Enablement (Silver Layer Ownership)
Design, build, and maintain curated Silver-layer datasets in Microsoft Fabric for experimentation reporting and analytics
Collaborate with BI/Data Reporting teams to define dimensions, metrics, and joins such as visitor/session, variant, campaign, geo, device, channel, funnel steps, and conversion events
Develop reusable and standardized data products including tables and views for dashboards, scorecards, and ad hoc reporting
Ensure Silver-layer datasets are clean, conformed, deduplicated, and aligned with agreed business definitions
Data Gap Analysis & Assessment
Conduct regular gap assessments across experimentation requirements, existing Silver-layer data, and upstream telemetry/source systems
Identify missing fields, inconsistent definitions, data latency issues, and join-key mismatches
Document business impact, severity, remediation plans, timelines, and dependencies for identified issues
Recommend improvements in data models, including facts/dimensions, surrogate keys, grain definition, and conformance rules
Gold Layer Requirements & Stakeholder Management
Lead workshops with experimentation, BI, measurement, and engineering teams to define Gold-layer reporting requirements
Define KPI calculations, attribution rules, scorecard structures, segmentation requirements, governance standards, and refresh SLAs
Prepare functional and technical documentation including source-to-target mappings, data dictionaries, validation rules, and acceptance criteria
Ensure alignment on single-source-of-truth definitions across CJA, Power BI, and scorecards
Data Pipeline Engineering
Build and maintain robust pipelines using Microsoft Fabric Pipelines and Azure Data Factory (ADF)
Work with 1DS telemetry pipelines or equivalent systems to ensure accurate event and attribute flow into Fabric
Implement orchestration, incremental loads, monitoring, and error-handling mechanisms to meet reporting timelines
Data Validation & Reconciliation
Perform reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA)
Validate event counts, session/user logic, experiment attribution, conversions, and time-window consistency
Build automated checks for missing data, duplicate events, schema drift, and metric anomalies
Coordinate issue resolution with telemetry, tagging, product engineering, and reporting teams
Experimentation Lifecycle Support
Ensure datasets are ready for pre-launch checks, measurement, scorecard generation, health checks, and post-test analysis
Curate experiment metadata including test IDs, allocation details, start/end dates, KPI metrics, and slicing dimensions
Support consistent and reliable experimentation scorecard generation
AI Agent Design & Development
Design and develop AI-powered agents using Fabric Data Agents, Copilot, and Azure OpenAI
Enable automation for scorecard creation, narrative summaries, self-service analytics, anomaly investigation, and metric definition assistance
Define agent scope, personas, grounding datasets, RBAC/security models, and evaluation metrics
Partner with experimentation and reporting teams for pilot implementation, feedback gathering, and production rollout
Documentation, Governance & Operational Excellence
Maintain detailed documentation for datasets, transformation logic, metric definitions, pipelines, validation rules, and operational runbooks
Establish standards for naming conventions, semantic consistency, versioning, backward compatibility, and performance optimization
Provide operational support including monitoring, troubleshooting, incident management, and continuous improvement initiatives