Specialist - Software Engineering

Senior Data Engineer – GCP 

We are seeking a highly skilled Data Engineer with strong experience in Google Cloud Platform (GCP), particularly BigQuery, to design, build, and maintain scalable data pipelines. The ideal candidate will have hands-on experience with Python, SQL, Airflow/Composer, and CI/CD practices, and will play a key role in expanding and migrating Adobe-based data pipelines while integrating and operationalizing Orion datasets across hybrid cloud environments.

Key Responsibilities

• Design, build, and maintain ETL/ELT data pipelines using Python, SQL, Airflow, and GCP Composer.
• Support the migration and expansion of Adobe-based data pipelines.
• Integrate Orion datasets into existing enterprise datasets and data platforms.
• Develop new data pipelines for Orion event data collection.
• Implement and manage data ingestion, transformation, and orchestration workflows in BigQuery.
• Optimize pipeline performance, including runtime, compute usage, storage tiering, and query costs, with a strong focus on BigQuery cost optimization.
• Establish data quality checks, validation logic, and reconciliation processes to ensure data accuracy and reliability.
• Work across hybrid data environments, supporting data movement and transformation across AWS, GCP, and Snowflake.
• Implement and maintain CI/CD pipelines using GitHub and enterprise DevOps practices.
• Create and maintain clear, comprehensive documentation for all data pipelines, integrations, and operational processes.
• Collaborate with cross-functional teams to support analytics, reporting, and downstream data consumers.

Required Skills and Qualifications

• Strong hands-on experience with Google Cloud Platform (GCP), particularly BigQuery.
• Proficiency in Python and SQL for data engineering use cases.
• Experience with Apache Airflow and Cloud Composer.
• Hands-on experience with GitHub and CI/CD pipelines.
• Solid understanding of data warehousing concepts, including:
  • Star schemas
  • Dimensional modeling
  • OLTP vs. OLAP architectures
• Experience designing and supporting ETL/ELT pipelines in large-scale data environments.
• Familiarity with Adobe data platforms and event-based data pipelines.
• Experience working in multi-cloud and hybrid environments (AWS, GCP, Snowflake).
• Strong problem-solving skills and attention to data quality and performance.

Nice to Have

• Experience with large-scale event data and real-time/near-real-time pipelines.
• Prior experience supporting marketing or analytics platforms (Adobe ecosystem).
• Exposure to cost optimization strategies in cloud-based data platforms.
• Strong documentation and stakeholder communication skills.