Senior Big Data Engineer

Description

Job Summary: Cidacs is seeking a Senior Big Data Engineer to join a multidisciplinary team, responsible for designing, implementing, and optimizing Big Data solutions with autonomy and leadership potential. Key Highlights: 1. Autonomous work and leadership potential in Big Data 2. Design and implementation of complex Big Data solutions 3. A welcoming environment conducive to development and innovation Are you a professional with over 5 years of experience in Big Data Engineering, capable of performing tasks with high autonomy, possessing leadership potential, and skilled at handling highly complex tasks, solving intricate challenges, and making well-considered decisions backed by your expertise and professional trajectory? This position may be right for you! At the Center for Integration of Data and Knowledge for Health (Cidacs), we are seeking a Senior Big Data Engineer to join the multidisciplinary team of our Data Platform. This professional will be responsible for designing and implementing solutions, optimizing performance, and supporting Big Data infrastructure and services. The objective is to build documented, technology-based strategies, processes, and products to meet business needs and solve business problems. Work will primarily be on-site at Cidacs/Fiocruz Bahia, with the possibility of hybrid work arrangements. **Responsibilities and Duties** **RESPONSIBILITIES AND DUTIES:** * **Design and Implementation of Big Data Solutions:** Design, build, and implement Big Data solutions to meet requirements for storage, processing, and analysis of large volumes of data. * **Infrastructure Maintenance:** Ensure Big Data systems remain operational, optimized, and performant. * **Performance Optimization:** Monitor and optimize Big Data system performance, ensuring fast and efficient data processing. * **Data Security:** Implement and maintain security standards and protocols to protect data and ensure privacy and compliance, especially in regulated contexts. * **Interdisciplinary Collaboration:** Collaborate closely with data scientists, analysts, and other stakeholders to understand their needs and translate them into technical solutions. * **Scalability:** Design scalable solutions capable of handling increasingly larger data volumes without compromising performance. * **Implementation of Big Data Tools:** Work with a variety of Big Data technologies to implement solutions aligned with specific needs. * **Data Pipeline Management:** Create and manage end-to-end data pipelines, from ingestion through storage, processing, and analysis. * **Automation:** Implement automated solutions for recurring tasks, ensuring efficiency and reducing manual errors. * **Training and Leadership:** As a senior professional, this individual is expected to mentor junior team members, sharing knowledge and providing technical guidance. * **Evaluation of New Technologies:** Stay current with the latest Big Data technologies and trends, evaluating and recommending new tools and approaches that may benefit the organization. * **Documentation:** Produce robust technical documentation for implemented solutions, ensuring other engineers and stakeholders understand the solutions and can work with them in the future. **Requirements and Qualifications** **QUALIFICATIONS AND REQUIREMENTS:** ***EDUCATION:*** * Degree in Computer Science, Computer Engineering, Information Systems, or related fields. ***PRACTICAL EXPERIENCE:*** * Minimum of 5 years in roles related to Big Data, with emphasis on infrastructure and services; * Proven experience in designing, implementing, operating, and optimizing scalable Big Data solutions; * Track record of successfully delivered projects in complex, high-availability environments. ***TECHNICAL SKILLS:*** * In-depth knowledge and hands-on experience with Big Data technologies such as Spark, Elasticsearch, Kubernetes, Hadoop, and other Big Data solutions; * Experience with distributed storage systems such as Yugabyte/Postgres; * Experience with S3-like object storage databases, such as MinIO and Ceph RGW; * Experience with data lake and data lakehouse solutions, such as Delta Lake, Apache Iceberg, and Trino; * Experience with Linux, Shell Scripting, and Linux service configuration; * Experience with NoSQL databases such as MongoDB and Redis; * Experience with Infrastructure-as-Code (IoC) tools such as Ansible and Terraform; * Proficiency in Java or Python programming languages; * Knowledge of Big Data processing engines such as Spark and Elasticsearch; * Understanding of Big Data performance optimization techniques and tools; * Experience with data workflow orchestration tools such as Apache Airflow or NiFi; * Familiarity with on-premise data center architecture and operations; * Familiarity with software architecture best practices. ***INTERPERSONAL SKILLS:*** * Excellent communication skills, both for explaining technical complexities to non-technical stakeholders and for collaborating effectively with the team; * Leadership and mentoring ability to develop junior team members; * Strong analytical skills and problem-oriented approach. ***OTHER QUALIFICATIONS:*** * Knowledge of Big Data security practices and regulatory compliance; * Experience with containers and orchestration; * Intermediate English proficiency, especially for technical reading and writing; * Willingness to stay updated in a rapidly evolving technology field. ***PREFERRED QUALIFICATIONS (not mandatory but desirable):*** * Professional certifications in Big Data or related technologies; * Postgraduate degree in fields relevant to this position; * Knowledge of High-Performance Computing (HPC) environments; * Knowledge of data integration and interoperability; * Prior experience in health-related or research sectors; * Experience with Kubernetes orchestrators and tools such as Kubesphere, Kubeflow, Rancher, and others; * Knowledge of health-related standards and regulations, such as Brazil’s General Data Protection Law (LGPD), GDPR, and HIPAA; * Technical knowledge of Information Security and Privacy (Privacy by Design); * Familiarity with Cloud Computing solutions such as AWS, Google Cloud, or Azure, and their respective Big Data services; * Familiarity with DevOps and CI/CD practices. **Additional Information** **ABOUT CIDACS:** CIDACS is a pioneering center for data integration, focused on understanding, studying, and evaluating the health conditions of Brazil’s population using Big Data. Our work supports public managers, researchers, and the community, contributing to transforming lives. If, like us, you are passionate about technology and innovation, join our team. Here, you will find a welcoming environment conducive to development and innovation, filled with professionals from diverse academic backgrounds united by a shared purpose. Bring your talent and commitment—and become part of this team! For more information about Cidacs, visit: www.cidacs.bahia.fiocruz.br Cidacs conducts studies and research based on interdisciplinary projects stemming from the linkage of large-scale datasets to expand understanding of social and environmental determinants and policies affecting population health.