Data Engineer

• Key Responsibilities
• Design, build, and maintain automated, scalable data pipelines for performance and reporting datasets.
• Refactor legacy pipelines to meet enterprise data standards and improve performance.
• Implement robust data quality routines (validation, anomaly detection, schema checks).
• Enhance orchestration and error handling for workflow reliability.
• Support infrastructure tasks like environment refreshes, upgrades, and disaster recovery testing.
• Ensure compliance with metadata, lineage, classification, and documentation standards.
• Collaborate on datasets for executive reporting and strategic planning across the Personal & Commercial Operations (PCO), ODIA (Operational Data, Information & AI).

• 3+ years in hands-on data engineering
• Tools & Languages:
• PySpark, Python, SQL
• Hadoop ecosystem: Spark, Hive, HDFS
• Shell scripting, Linux environments
• Query tuning with Spark SQL, Trino, Hive
• Deployment experience across dev/test/prod
• Working with partitioned datasets, Parquet, columnar storage
• Strong grasp of data governance, metadata, and validation

• Experience with event-driven or real-time processing (e.g., Kafka)
• Familiarity with vector databases, LLM-supportive architectures
• Tools: Apache Airflow, OpenShift (OCP4), AWS (for orchestration/scheduling)

We welcome candidates with the skillset above to immediately drop an email to nisha@talentspothub.com for more information and confidential conversation.