Senior Data Engineer

We're standing up a new AWS-based data platform for a workers' insurance fund and need a hands-on Senior Data Engineer to own the entire pipeline stack end to end. Source data is messy CSVs; everything is batch for this phase. The mandate: land → clean → model (with SCD history) → load into Amazon Redshift, with productionized orchestration, monitoring, and basic

cost controls—all within a two-week window.

**Outcomes you'll deliver (within 2 weeks)**

* Ingestion layer: Robust S3 landing/raw/processed zone layout; Glue crawlers where useful; schema drift-tolerant ingestion for dirty CSVs.

* Transformations: PySpark/Spark and/or SQL jobs to cleanse/standardize data; conforming dimensions and facts for workers' comp domain.

* Historical tracking: Implement Slowly Changing Dimensions (Type 2) for key entities; maintain audit columns and metadata.

* Warehouse: Provision and tune Amazon Redshift (WLM/concurrency scaling/sort & dist keys) with core star/snowflake models.

* Orchestration: Productionized DAGs in Airflow (MWAA) or AWS-native orchestration to run batch pipelines on schedule.

* Monitoring: CloudWatch metrics, logs, and alarms for pipeline health and job failures.

* DR & backups: Cross-region replication for critical S3 buckets and Redshift snapshots.

* Cost controls: Budgets, cost allocation tags, and basic guardrails to prevent runaway spend.

* Docs & handoff: Runbooks, architecture diagram, and quickstart ops guide.

**Responsibilities**

* Own design and implementation of the end-to-end batch data pipeline on AWS.

* Stand up S3 + Glue + EMR/Spark + Athena + Redshift + Lake Formation components needed for secure, governed access.

* Author PySpark/Spark and SQL transformations; handle schema drift gracefully (e.g., optional columns, late-arriving fields).

* Model data for analytics in Redshift (dimensions/facts, SCD2, surrogate keys, snapshot tables as needed).

* Build and deploy DAGs in Airflow (or AWS-native orchestration) with retries, SLAs, and notifications.

* Implement CloudWatch logging/metrics and alerting for jobs, EMR clusters, and Redshift.

* Configure Terraform modules for reproducible infra and CI/CD hooks (format, validate,plan, apply).

* Apply Lake Formation/IAM for least-privilege access; set up KMS encryption and VPC endpoints where applicable.

* Tune performance (partitioning, file sizing, compression, columnar formats where appropriate).

* Partner with stakeholders to validate outputs and deliver a clean handoff package.

**Required qualifications**

* 5–8+ years in data engineering with recent, deep AWS experience.

* Proven delivery owning the full pipeline: ingestion → transform → model → serve on S3/Glue/EMR/Athena/Redshift.

* Strong Python and PySpark/Spark; excellent SQL (DDL/DML, performance tuning).

* Hands-on Terraform for AWS data stacks.

* Implemented SCD Type 2 and historical strategies in production.

* Comfortable wrangling dirty CSVs and handling schema drift at scale.

* Built Airflow DAGs with monitoring and alerting (CloudWatch).

* Practical experience with cost management (budgets, tags, optimization tactics).

* Clear documentation and fast, pragmatic delivery in a compressed timeline.

**Nice to have**

* AWS Certified Data Analytics – Specialty (preferred).

* Familiarity with Athena/Presto, Iceberg/Delta/Hudi table formats and lakehouse patterns.

* Domain exposure to insurance / workers' comp datasets.

* Redshift automation/tuning experience (Spectrum, RA3, Vacuum/Analyze strategies).

**Tech stack you'll use here**

* AWS: S3, Glue (Jobs/Crawlers/Catalog), EMR, Athena, Redshift, Lake Formation, CloudWatch

* Orchestration: Airflow (MWAA) or AWS-native orchestration

* Languages/Frameworks: Python, PySpark/Spark, SQL

* IaC: Terraform

* Data formats: CSV (ingest), Parquet/ORC (processed, if introduced), Redshift tables

* Security/DR: IAM/Lake Formation, KMS, cross-region replication, Redshift snapshots

**Day-by-day sample plan (guide, not rigid)**

* Days 1–2: Infra bootstrap (Terraform), S3 zones, IAM/LF, networking basics, cost tags/budgets; sample CSV profiling.

* Days 3–5: Ingestion jobs (schema drift handling), baseline DQ checks (nulls, types), Parquet conversion if helpful.

* Days 6–8: Dim/fact modeling in Redshift; implement SCD2; load patterns; initial performance passes.

* Days 9–10: Orchestration in Airflow; retries/alerts; CloudWatch dashboards/alarms.

* Days 11–12: DR setup (CRR for S3, Redshift snapshot policy), optimize file sizing/partitioning.

* Days 13–14: UAT, documentation, runbooks, knowledge transfer & sign-off.

Contract details

● Type: Contract (1 month, full-time effort)

● Location: Remote (US time zone overlap preferred)

● Start date: ASAP

● Team: Solo contributor (reports to project lead; partners with stakeholders ad hoc)

● Compensation: ₦2,500,000 (Two Million Five Hundred Thousand Naira) for the one-

month engagement