Responsibilities
● Design, develop, and maintain end-to-end ETL pipelines, encompassing data extraction, transformation, and loading processes to ensure the availability and accuracy of data for analytics and machine learning.
● Collaborate closely with data scientists, machine learning engineers, and other stakeholders to understand requirements and implement effective solutions that cater to their needs.
● Build and maintain scalable, high-performance services using Python to serve machine learning application results, ensuring reliability, low latency, and optimal performance.
● Apply best practices in software development to ensure code quality, maintainability, and scalability of ETL pipelines and services.
● Utilize Python expertise to optimize data processing and transformation tasks, implementing efficient algorithms and techniques.
● Develop and maintain documentation for ETL processes, data flow, and service architecture to ensure clear communication and knowledge sharing within the team.
● Monitor and troubleshoot ETL pipelines and services to identify and resolve issues promptly, ensuring minimal disruption to operations.
● Stay updated with the latest industry trends, tools, and technologies related to data engineering and machine learning, and actively apply this knowledge to enhance our processes and systems.