Data Engineer

Job Description

As a Data Engineer, you’ll work with large-scale, heterogeneous datasets and hybrid cloud architectures to support analytics and AI solutions. Collaborate with data scientists, infra engineers, sales specialists, and stakeholders to ensure data quality, build scalable pipelines, and optimize performance. Your work will integrate telco data with other verticals (retail, healthcare), automate DataOps/MLOps/LLMOps workflows, and deliver production-grade systems.

As a Data Engineer, you will:

• Ensure Data Quality & Consistency
• Validate, clean, and standardize data (e.g., geolocation attributes) to maintain integrity.
• Define and implement data quality metrics (completeness, uniqueness, accuracy) with automated checks and reporting.

• Build & Maintain Data Pipelines
• Develop ETL/ELT workflows (PySpark, Airflow) to ingest, transform, and load data into warehouses (S3, Postgres, Redshift, MongoDB).
• Automate DataOps/MLOps/LLMOps pipelines with CI/CD (Airflow, GitLab CI/CD, Jenkins), including model training, deployment, and monitoring.

• Design Data Models & Schemas 
• Translate requirements into normalized/denormalized structures, star/snowflake schemas, or data vaults.
• Optimize storage (tables, indexes, partitions, materialized views, columnar encodings) and tune queries (sort/distribution keys, vacuum).

• Integrate & Enrich Telco Data
• Map 4G/5G infrastructure metadata to geospatial context, augment 5G metrics with legacy 4G, and create unified time-series datasets.
• Consume analytics/ML endpoints and real-time streams (Kafka, Kinesis), designing aggregated-data APIs with proper versioning (Swagger/OpenAPI).

• Manage Cloud Infrastructure
• Provision and configure resources (AWS S3, EMR, Redshift, RDS) using IaC (Terraform, CloudFormation), ensuring security (IAM, VPC, encryption).
• Monitor performance (CloudWatch, Prometheus, Grafana), define SLAs for data freshness and system uptime, and automate backups/DR processes.

• Collaborate Cross-Functionally & Document
• Clarify objectives with data owners, data scientists, and stakeholders; partner with infra and security teams to maintain compliance (PDPA, GDPR).
• Document schemas, ETL procedures, and runbooks; enforce version control and mentor junior engineers on best practices.

Requirements

• 2+ years of experience in data engineering and related DevOps roles for data platforms
• Proficient in Python for ETL/data engineering and Spark (PySpark) for building large-scale data pipelines
• Hands-on experience with Big Data and SQL frameworks such as Spark SQL, Redshift, and PostgreSQL, and familiarity with NoSQL systems like MongoDB for data modeling, indexing, partitioning, and schema evolutio
• Skilled in workflow orchestration using Airflow (or equivalent) and pipeline automation with GitLab CI/CD or Jenkins
• Experience with cloud and hybrid infrastructures, including AWS (S3, EMR, Glue, Redshift), on-prem clusters, and containerized environments (Docker, Kubernetes)
• Knowledge of Infrastructure as Code (IaC) using Terraform or CloudFormation for provisioning and drift detection
• Proven ability to design and optimize scalable storage solutions & tables, partitions, indexes, materialized views, and columnar encodings
• Skilled in query optimization and performance tuning, including execution plan analysis, key selection, and cost optimization (e.g., vacuum maintenance, cluster resizing, Spectrum)
• Knowledge of MLOps/LLMOps practices such as auto-scaling ML systems, model registry management, and CI/CD for model deployment
• Strong problem-solving, attention to detail, and ability to collaborate with cross-functional teams in agile environments.

Nice to Have

• Exposure to serverless architectures (AWS Lambda) for event-driven pipelines
• Familiarity with vector databases, data mesh, or lakehouse architectures
• Experience using BI/visualization tools (Tableau, QuickSight, Grafana) for data quality dashboards
• Hands-on with data quality frameworks (Deequ) or LLM-based data applications (NL-->SQL generation)
• Participation in GenAI POCs (RAG pipelines, Agentic AI demos, geomobility analytics)
• Client-facing or stakeholder-management experience in data-driven/AI projects