The problem
A multi-service data ingestion pipeline was running on a manually configured EC2 box driving a docker-compose stack — no isolation between environments, no reproducible state, and a deploy meant SSHing in. It needed to become real infrastructure before it could scale.
Infrastructure as code
AWS infrastructure for the pipeline — EKS, RDS PostgreSQL, S3, SQS, ECR and VPC — provisioned through 11 reusable Terraform modules, with dev and prod isolated into separate state and environments.
GitOps migration
Deployment moved from CI-driven kubectl set image to full GitOps with ArgoCD: 7 backend microservices restructured into an app-of-apps hierarchy of 12 Applications with selfHeal and auto-sync, so cluster state is always derived from Git and rollback is a git revert.
CI/CD & security
The GitHub Actions pipeline was redesigned around keyless IAM OIDC authentication and commit-SHA-tagged, immutable ECR images, with a cross-repository update-manifest job closing the loop from code push to running pod — zero manual kubectl access for CI.
Observability
Loki/Fluent Bit for centralized logging and kube-prometheus-stack (Prometheus, node-exporter, kube-state-metrics) for infra and app metrics, running on a Grafana instance isolated from client-facing dashboards and gated by AWS WAFv2 IP-allowlist rules on the ALB.
Tracing a hidden bug
A Go microservice pipeline (SQS producer/consumer, S3 PUT/GET calls) was traced end-to-end from source to diagnose a hidden duplicated S3 write per uploaded file — redesigning the flow to cut per-file storage operations by 50%.
What shipped
- Provisioned AWS infrastructure for a multi-service data ingestion pipeline (EKS, RDS PostgreSQL, S3, SQS, ECR, VPC) through 11 reusable Terraform modules, isolating dev and prod into separate state/environments and replacing a manually configured EC2 docker-compose stack.
- Migrated deployment from CI-driven kubectl set image to full GitOps with ArgoCD, restructuring 7 backend microservices into an app-of-apps hierarchy (12 Applications with selfHeal/auto-sync) so cluster state is always derived from Git and rollback is a git revert.
- Redesigned the GitHub Actions pipeline around keyless IAM OIDC authentication and commit-SHA-tagged, immutable ECR images, adding a cross-repository update-manifest job that closes the loop from code push to running pod with zero manual kubectl access for CI.
- Stood up an internal observability stack — Loki/Fluent Bit for centralized logging and kube-prometheus-stack (Prometheus, node-exporter, kube-state-metrics) for infra/app metrics — on a Grafana instance isolated from client-facing dashboards and gated by AWS WAFv2 IP-allowlist rules on the ALB.
- Traced a Go microservice pipeline (SQS producer/consumer, S3 PUT/GET calls) end-to-end from source to diagnose a hidden duplicated S3 write per uploaded file, redesigning the flow to cut per-file storage operations by 50%.