- Name of the position: Data Engineer
- Location: Remote
- No.of resources needed for this position: 01
- Mode : Contract
- Years of experience: 10+ Years
We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate should have a robust background in the various phases of ETL data applications, including data ingestion, preprocessing, and transformation. This role is perfect for someone who thrives in a fast-paced environment and is passionate about leveraging data to drive business success.
Key Responsibilities:
- Design and implement efficient data ingestion pipelines to collect and process large volumes of data from various sources.
- Hands-on experience with AWS Database Migration Service for seamless data migration between databases.
- Develop and maintain scalable data processing systems, ensuring high performance and reliability.
- Utilize advanced data transformation techniques to prepare and enrich data for analytical purposes.
- Collaborate with cross-functional teams to understand data needs and deliver solutions that meet business requirements.
- Manage and optimize cloud-based infrastructure, particularly within the AWS ecosystem, including services such as S3, Step-Function, EC2, and IAM.
- Experience with cloud platforms and understanding of cloud architecture.
- Knowledge of SQL and NoSQL databases, data modeling, and data warehousing principles.
- Familiarity with programming languages such as Python or Java.
- Implement security and compliance measures to safeguard data integrity and privacy.
- Monitor and tune the performance of data processing systems to ensure optimal efficiency.
- Stay updated with emerging trends and technologies in data engineering and propose adaptations to existing systems as needed.
- Proficient in AWS Glue for ETL (Extract, Transform, Load) processes and data cataloging.
- Hands-on experience with AWS Lambda for serverless computing in data workflows.
- Knowledge of AWS Glue Crawler Kinesis RDS for batch / real-time data streaming
- Familiarity with AWS Redshift for large-scale data warehousing and analytics.
- Skillful in implementing data lakes using AWS Lake Formation for efficient storage and retrieval of diverse datasets.
- Experience with AWS Data Pipeline for orchestrating and automating data workflows.
- In-depth understanding of AWS CloudFormation for infrastructure as code (IaC) deployment.
- Proficient in AWS CloudWatch for monitoring and logging data processing workflows.
- Familiarity with AWS Glue DataBrew for visual data preparation and cleaning.
- Expertise in optimizing data storage costs through AWS Glacier and other cost-effective storage solutions.
- Hands-on experience with AWS DMS (Database Migration Service) for seamless data migration between different databases.
- Knowledge of AWS Athena for interactive query processing on data stored in Amazon S3.
- Experience with AWS AppSync for building scalable and secure GraphQL APIs.
Qualifications:
- A minimum of 10 years of experience in data engineering or a related field.
- Strong background in big data application phases, including data ingestion, preprocessing, and transformation.
Education:
- Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field. A Master’s degree is preferred.
India
