Share this Job
Start applying with LinkedIn
Date: Mar 26, 2023
Location: Edison, NJ, US
Company: LTIMindtree
Mandotory Certification (any one) :
Cloudera - CCA Spark and Hadoop Developer (CCA175)
Databricks Certified Developer: Apache Spark 2.X
Hortonworks - HDP Certified Apache Spark Developer
Senior Data Engineer Specialist (Hands-on) – VP:
Job Description
The TTS Core Accounts team is building the next generation platform and needs to build key integrations with Big Data platforms and other downstream consumers using Kafka events and streaming. There is also a need to build an operational data store for Operations reporting needs, which will require cutting-edge technologies for data injection, transformation and event notification to the clients. Data can range from transactional SQL server or NoSQL data, streaming data, and batch data.
The ideal candidate will have an eye for building and optimizing real-time data and reporting systems and will work closely with the Data team to help direct the flow of data within the pipeline and ensure consistency of data delivery and notification across TTS business.
Responsibilities:
• Create and maintain optimal data pipeline architecture. Deliver data pipeline (ingestion, data quality, transformation and reporting) for both real time and batch based use cases
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure ‘big data’ technologies.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Design key data elements, working with data streaming or processing frameworks such as Apache Kafka or similar technologies and manipulating data in Java, Python, or ETL tools like Talend etc.
• Formulate approaches to data into big data platforms such as Hadoop or Hive on a continual basis
• Has the ability to operate with a limited level of direct supervision.
• Act as technical advisor or coach to junior members in the team.
• Manage processes like cross border clearance (CBAT), enterprise architecture review (CART) for data use cases
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Technology Skills:
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases.
• Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Experience with stream-processing systems: Storm, Spark-Streaming, etc.
• Experience with object-oriented/object function scripting languages: Python, R, Java, C++, Scala, etc.
• Experience with stream-processing systems (Spark, Kafka etc)
• Preferred Experience in Hadoop, Hive, Impala
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Proven history of manipulating, processing and extracting value from large, disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Strong computer science fundamental (data structures and algorithms)
• Familiar with Data warehousing concepts, BI environment
• Familiar with Dimensional modeling, database structures, and query optimization
• Proven experience with ETL tools like Talend, Ab Initio etc.
• Experience with design patterns for cloud services such as data storage and retrieval, data security managem
Nearest Major Market: New Jersey
Job Segment:
Information Technology, IT Architecture, SQL, Database, Computer Science, Technology
