Big Data Engineer

About the role

AvePoint, a global leader in data management and data governance, is looking for a highly skilled and experienced Big Data Engineer. The successful candidate will design, develop and maintain data ingestion and processing jobs according to business requirements, and integrate data sets so users have seamless access to data. The role suits consultants or senior consultants with at least 4 years of experience managing data engineering jobs in a big data environment such as Cloudera Data Platform.

Key responsibilities

• Analyse the Authority's data needs and document the requirements.
• Refine data collection and consumption by migrating data collection to more efficient channels.
• Plan, design and implement data engineering jobs and reporting solutions to meet analytical needs.
• Develop test plans and scripts for system testing and support user acceptance testing.
• Work with the Authority's technical teams to ensure smooth deployment and adoption of new solutions.
• Ensure the smooth operation and service levels of IT solutions and support production issues.

Required profile

• Minimum 4 years of experience managing data engineering jobs in a big data environment; overall experience of 4 to 10 years.
• Consultant or senior consultant preferred.
• Good understanding of project delivery using waterfall or Agile methodology.
• Ability to troubleshoot complex issues, from system resources to application stack traces.
• Strong SQL, data modelling and data analysis skills.
• Hands-on experience with Python, PySpark, Linux and ETL tools such as Informatica.
• Track record implementing systems with high availability, high performance and high security.
• Good written and verbal communication and interpersonal skills, able to understand business requirements and talk confidently with stakeholders.

Required skills

• Preferred: hands-on experience with Hive, Impala and Cloudera Data Platform.
• Preferred: hands-on experience with DevOps deployment and data virtualization tools such as Denodo.
• Beneficial but not required: reporting or visualization tools such as SAP BO and Tableau.