At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.
With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.
We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.
To move beyond traditional QA by owning the "Evaluation Harness," ensuring the AI agents are safe, compliant, and accurate through rigorous automated testing of non-deterministic outputs.
Key Responsibilities
Own and manage the Langfuse eval harness, including the creation of "Golden Paths" and adversarial test scenarios.
Validate regulatory quality gates (SR 26-2, NYDFS Reg 187) and run shadow mode validation to compare AI performance against human benchmarks.
Test for data quality degradation, population segmentation errors, and human-in-the-loop (HITL) checkpoint reliability.
Verify that the Designer’s output standards (explanation blocks, signal types) are maintained in all evaluation scenarios.
Required Skills and Qualifications
Advanced proficiency in Python 3.11+, Pytest, and automated testing frameworks.
Direct experience with AI Evaluation tools (Langfuse, RAGAS, or similar) and LLM-as-a-judge patterns.
Knowledge of model risk management and regulatory compliance testing.
Experience with "Shadow Mode" deployment strategies and statistical validation.
Soft Skills
A "skeptical" and adversarial mindset to find edge cases.
Uncompromising attention to detail regarding regulatory evidence.
Collaborative approach to working with engineers to fix logic failures.
Our benefits:
-Health and dental insurance-Meal and food allowance-Childcare assistance-Extended paternity leave-Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;-Profit Sharing and Results Participation (PLR);-Life insurance-Continuous learning platform (CI&T University);-Discount club-Free online platform dedicated to physical, mental, and overall well-being-Pregnancy and responsible parenting course-Partnerships with online learning platforms-Language learning platformAnd many more!More details about our benefits here: At CI&T, inclusion starts at the first contact. If you are a person with a disability, it is important to present your assessment during the selection process. See which data needs to be included in the report by clicking here.This way, we can ensure the support and accommodations that you deserve. If you do not yet have the assessment, don't worry: we can support you in obtaining it. We have a dedicated Health and Well-being team, inclusion specialists, and affinity groups who will be with you at every stage. Count on us to make this journey side by side.
CI&T is a technology transformation specialist that combines human expertise with artificial intelligence to create scalable tech solutions for enterprise clients. With over 8,000 employees worldwide, the company has built partnerships with more than 1,000 clients over its 30-year history. CI&T specializes in designing and implementing large-scale B2B applications and system architectures, with a focus on integrating AI-driven strategies into technical solutions. The company serves as a strategic technical partner for global enterprises, helping them understand technology implications and build long-term technical roadmaps.
