Principal Software Engineer - OpenShift AI Model Training

About the job

Do you want to design and build the tools that enable creation of the next generation of Foundation Models and Large Language Models (LLMs)? The Red Hat OpenShift AI (RHOAI) engineering team is seeking a Principal Software Engineer with Kubernetes and MLOps experience to join our rapidly growing Distributed Model Training platform team. Our team creates the tools necessary for enterprise data science teams to leverage distributed compute infrastructure across the hybrid cloud when training today’s most complex ML models.

In this role, you'll be contributing as an expert on ML model training and the tools necessary to support it at enterprise scale. Your contributions will enable bleeding edge platform capabilities around distributed computation and model training, model hyperparameter tuning, kubernetes-native job scheduling, and kubernetes-native resource optimization. This is an extraordinary opportunity to contribute to the development of the RHOAI family of products, participate in open source communities such as Kubeflow, Kubernetes, Ray, and Kueue, and be at the forefront of the exciting evolution of AI.

The future of the AI industry is open with extensive opportunities, and RHOAI is a strategic investment area for Red Hat. You'll join an ecosystem that fosters continuous learning, career growth, and professional development. This hands-on experience is a great way for you to get first-hand exposure to the AI landscape.

Are you ready to start developing new solutions that combine open source, hybrid cloud, and AI? Join the Red Hat OpenShift AI team!

What you will do

• Lead Red Hat’s participation in machine learning related upstream communities to ensure the technologies work on OpenShift and can be integrated with RHOAI
• Architect and lead implementation of scalable open source solutions for Data Scientists to leverage distributed computing capabilities to train their Machine Learning models, running on OpenShift
• Act as a MLOps SME within Red Hat by supporting customer facing discussions, presenting at technical conferences, and evangelizing OpenShift AI within the internal community of practice
• Architect and design new features for open source communities such as Kueue, KubeRay, PyTorch, KubeFlow, and CodeFlare
• Provide technical vision and leadership on critical and high impact projects
• Mentor, influence, and coach a distributed team of engineers
• Present at OpenShift/Kubernetes, and AI/ML related technology conferences and internally within the AI/ML communities of practice

What you will bring

• An existing contributor in one or more MLOps open source projects such as Ray/KubeRay, KubeFlow, Pytorch, Katib
• Experience training and tuning ML models using tools like Ray, Kubeflow training operator, Katib, MLFlow, or similar
• Advanced level of experience with Kubernetes
• Advanced level knowledge and experience in development in Go or Python
• Excellent system understanding and troubleshooting capabilities
• Solid innovation skills and a passion for technology
• Technical leadership acumen in a global team environment & mentorship experience
• Passion for writing and maintaining reliable code
• Excellent written and verbal communication skills; good English language skills

The following will be considered a plus:

• Bachelor's degree in statistics, mathematics, computer science, operations research, or a related quantitative field, or equivalent expertise; Master’s or PhD
• Experience in engineering, consulting or another field related to distributed model training or data processing in a customer environment or supporting a data science team
• Highly experienced in OpenShift