Responsibilities
- Liaising with coworkers and clients to elucidate the requirements for each task.
- Conceptualizing and generating infrastructure that allows big data to be accessed and analyzed.
- Reformulating existing frameworks to optimize their functioning.
- Testing such structures to ensure that they are fit for use.
- Building a data pipeline from different data sources using different data types like API, CSV, JSON, etc.
- Preparing raw data for manipulation by Data Scientists.
- Implementing proper data validation and data reconciliation methodologies.
- Ensuring that your work remains backed up and readily accessible to relevant coworkers.
- Remaining up-to-date with industry standards and technological advancements that will improve the quality of your outputs.
Requirements
- Bachelor's degree in Data Engineering, Big Data Analytics, Computer Engineering, or related field.
- A Master's degree in a relevant field is an added advantage.
- 3+ years of Python, Java or any programming language development experience
- 3+ years of SQL No-SQL experience (Snowflake Cloud DW MongoDB experience is a plus)
- 3+ years of experience with schema design and dimensional data modeling
- Expert proficiency in SQL, NoSQL, Python, C++, Java, R.
- Expert with building Data Lake, Data Warehouse or suitable equivalent.
- Expert in AWS Cloud.
- Excellent analytical and problem-solving skills.
- A knack for independence and group work.
- Capacity to successfully manage a pipeline of duties with minimal supervision.