Data Engineer

Description

Job Summary: We are seeking a Senior Data Engineer to design, build, and evolve our data platform, ensuring scalability, quality, reliability, and availability of information. Key Highlights: 1. Evolve the company's data platform architecture 2. Develop and maintain ETL/ELT pipelines 3. Work with microservices on AWS and asynchronous processing **Role:** Senior Data Engineer **Work Model:** 100% remote **Employment Type:** Independent Contractor (PJ) **Compensation:** USD 2,500\.00 **About the Company** HW Publishing is a U.S.-based holding company specializing in direct response marketing and affiliate marketing, with strong presence in the nutraceuticals market. Founded by renowned digital marketing professionals, HW quickly established itself as a global leader in the industry. In its first year, it achieved leadership in Latin America, and in Q1 2025, became the global leader in this segment. At HW, we empower our partners to become successful digital entrepreneurs within a collaborative, professional, and results-oriented environment — bringing together some of the top talents in direct marketing. **About the Role** We are looking for a Senior Data Engineer to design, build, and evolve our data platform, ensuring scalability, quality, reliability, and availability of information used by products, operations, and business teams. Our environment consists of AWS-based microservices, asynchronous processing using message queues, relational databases, Redis, and event-driven applications. Currently, the platform generates millions of records per day — including navigation/tracking events, webhooks, payments, affiliates, products, orders, logs, operational metrics, and financial data — processed via AWS microservices using PostgreSQL, Redis, Amazon SQS, and Lambda. The goal is to evolve this architecture into a modern data platform capable of delivering reliable, real-time insights for dashboards, operational analytics, business intelligence, and future Machine Learning initiatives. **Key Responsibilities** * Design, build, and evolve the company's data platform architecture. * Develop and maintain ETL/ELT pipelines, including API extraction, data transformations, incremental loading, and CDC (Change Data Capture). * Model data for Data Warehouse, Data Lake, and Data Marts using dimensional modeling (Star Schema, Snowflake Schema). * Implement batch and streaming processing with event-driven architecture and asynchronous processing. * Ensure database performance and reliability: relational modeling, indexing, partitioning, query optimization, and replication. * Establish Data Quality, Data Lineage, Data Catalog, and Data Governance practices. * Orchestrate pipelines and handle historical data with versioning and traceability. * Implement pipeline observability: logs, metrics, monitoring, and alerts. * Deliver reliable, real-time data for dashboards, operational analytics, and business intelligence. * Prepare the data foundation for future Machine Learning initiatives. * Automate processes and deployments using GitHub Actions and Docker. * Collaborate with product, engineering, and business teams to define requirements and KPIs. **Requirements** * **Languages:** Advanced Python and SQL. * **Databases:** PostgreSQL and MySQL; relational modeling, indexing, partitioning, query optimization, replication, and performance tuning. * **Data Modeling:** Data Warehouse, Data Lake, Star Schema, Snowflake Schema, Data Mart, dimensional modeling, normalization, and denormalization. * **ETL / ELT:** Pipeline development, API extraction, data transformation, incremental loading, and CDC (Change Data Capture). * **Data Processing:** Batch processing, streaming, asynchronous processing, and Event-Driven Architecture. * **AWS:** S3, Lambda, SQS, RDS, CloudWatch, and IAM. * **Messaging:** Amazon SQS. * **Version Control:** Git and GitHub. * **Observability:** Logs, metrics, pipeline monitoring, and alerts. * **Automation:** GitHub Actions. * **Containers:** Docker. * **Data Engineering:** Data Quality, Data Lineage, Data Catalog, Data Governance, data versioning, pipeline orchestration, and historical data handling. **Nice-to-Haves** * **Languages:** TypeScript and Node.js. * **AWS Analytics:** Glue, Athena, EMR, Kinesis, and Redshift. * **Messaging:** Kafka and RabbitMQ. * **Orchestration/Automation:** Apache Airflow and Prefect. * **Data Visualization:** Metabase, Power BI, and Looker Studio. * **Containers:** Kubernetes. * **Relevant Experience:** Fintechs or payment companies, high-volume event systems, real-time analytics platforms, Machine Learning Pipelines, Feature Store, Data Lakehouse, Apache Spark, Apache Iceberg, or Delta Lake.