**\#JoinLuby** ---------------- **Here’s what you’ll get to do:** * **Grow in a high-performance environment:** A culture where continuous development is the norm — with autonomy and involvement in national and international projects that truly challenge your skills and unlock your potential. * **Join a team passionate about technology:** You’ll collaborate alongside experienced talent in a collaborative, disruptive environment, using the most advanced technologies available in the market. * **Make your mark on the world:** Our digital solutions directly impact the lives of thousands of people — and this is your chance to actively shape a more digital and connected future. * **Work remotely and flexibly:** No matter where you are, we’re ready to welcome you onto our team! **Your Mission:** We’re looking for a Senior RLHF and Preference Learning Engineer to end-to-end own the feedback loop enabling AI agents on a healthcare platform to continuously evolve based on clinicians’, support staff’s, and patients’ preferences. You’ll be responsible for the infrastructure collecting human preference signals, the pipelines transforming those signals into training data, and the reward modeling infrastructure shaping agent behavior — all within a regulated environment handling sensitive health data (PHI) and carrying direct clinical safety implications. In this role, you’ll define the weight of each quality signal and own the full cycle: annotation interface, training pipeline, evaluation framework, and production rollout. **Key Responsibilities:** * Design and maintain preference data collection infrastructure: human review UIs, clinical reviewer workflows, and pairwise preference annotation pipelines. * Build and maintain reward modeling infrastructure: preference datasets, reward model training pipelines and evaluation methodology — including reward hacking detection. * Own the RLHF/RLAIF learning loop for core agents: inbox automation, medical record generation, clinical documentation, and referral follow-up tracking. * Define and maintain the hierarchy of quality signals: which feedback signals require clinical specialist review, which can be handled by support reviewers, and which can be automated via RLAIF. * Build RLAIF pipelines where human annotation doesn’t scale: Constitutional AI approaches, LLM-as-judge calibration against clinical ground truth, and automated pairwise preference generation. * Partner with intelligence-layer engineering on fine-tuning strategy: when to apply fine-tuning, prompt engineering, or RAG — and how preference learning interacts with each path. * Ensure PHI compliance across all feedback pipelines: annotation interfaces that never expose raw PHI, data residency requirements (PIPEDA/PHIPA), and logged-in, auditable reviewer access. **Essential Requirements:** * Master’s degree in Computer Science, Data Science, or equivalent experience. * At least 8 years of experience in Data Science or ML Engineering, including a minimum of 3 years in preference learning, RLHF, or reward modeling in production. * Hands-on experience with RLHF/RLAIF: preference learning pipelines already deployed in production, with real-world exposure to reward hacking and practical design of mitigation strategies. * Expertise in reward modeling: building preference datasets, training reward models, evaluation methodology, and stability under distribution shift. * Strong foundations in ML engineering: training pipelines, experiment tracking (MLflow or equivalent), model versioning, and evaluation frameworks running in production CI. * Proficiency in Python and ML frameworks — PyTorch preferred — and familiarity with efficient fine-tuning (PEFT/LoRA). * Experience in regulated domains, understanding why clinical AI feedback loops carry distinct risk profiles compared to consumer AI, and designing annotation workflows accordingly. * GCP Machine Learning Engineer certification (or equivalent cloud ML certification). * Fluent/advanced English. **Nice-to-Have:** * Experience with annotation workflows in clinical AI: clinical reviewer dashboards, annotation quality control, and inter-annotator agreement measurement in ambiguous clinical domains. * Familiarity with Constitutional AI / large-scale AI feedback (e.g., Anthropic’s RLAIF approach or equivalent). * Experience with GCP/Vertex AI for fine-tuning: Vertex AI managed fine-tuning and Vertex AI Pipelines for training orchestration. * Knowledge of clinical AI safety frameworks: model cards, bias audits, and distribution shift monitoring for clinical outcomes. **What We Offer:** * **Remote Work:** Enjoy the flexibility to work from anywhere in Brazil — with full autonomy and self-organization. * **Health & Well-being:** Health and dental insurance, life insurance, and Wellhub (Gympass). * **\+Education:** Discounts on FIAP courses and USP Esalq MBA programs — so you can keep learning and growing. * **Exclusive Perks:** Discounts at Multilaser stores, referral bonuses for professionals and business leads, flexible working hours, and annual bonus. ### ***Learn more about our culture, benefits, and what it’s like to be a \#Luber on our*** ***Careers Page!***
João Silva
Indeed · HR
João Silva
Indeed · HR
