Data Engineer (AWS/LLMs) | SR (Remote)

Description

. **Responsibilities and Assignments** * Define and evolve cloud-native data architecture on AWS; * Design data models optimized for operational queries (APIs) and consumption by conversational agents; * Lead migration of historical data from mainframe; * Ensure consistency, integrity, and traceability; * Work with large volumes and complex historical data; * Build and maintain ingestion and transformation pipelines; * Integrate existing data flows (PeopleSoft, mainframe, RDS); * Optimize PostgreSQL queries; * Implement partitioning, indexing, and caching strategies (Redis); * Reduce latency for frequent queries; * Ensure compliance with LGPD and data retention policies; * Implement data access controls, masking / anonymization when required, auditing, and lineage tracking; * Structure data for consumption by LLMs; * Support creation of semantic layers; * Work with embeddings / semantic indexing (differentiator); * Implement data quality validations; * Monitor pipelines and data consistency; **Requirements and Qualifications** * Experience in Data Engineering on AWS; * Building and maintaining data pipelines (ingestion and transformation); * Integration of data from legacy and enterprise systems (mainframe, PeopleSoft, RDS); * Experience in data modeling for diverse use cases (APIs and analytics/AI); * Strong expertise in PostgreSQL (query optimization, performance tuning, indexing, and partitioning); * Knowledge of caching (Redis); * Experience handling large-scale data and migrating legacy systems; * Data governance practices (LGPD, auditing, lineage, access control, and quality); * Structuring data for LLM consumption (semantic layers and embeddings – differentiator); Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI-driven tools to design and build digital-native platforms, integrating cutting-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI-driven revolution.**