TQuanta Technologies Private Limited jobs - Spark Scala Developer

Work Experience Level: A – 3 to 6 Yrs and SA 6 to 9 Yrs

Work Model: Hybrid model

Interested candidates please share your resume 

Responsibilities:

Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently.

Implement data transformation logic to cleanse, normalize, and aggregate data from various sources, ensuring high data quality and consistency.

Develop Web services on Scala using various libraries like Akka HTTP, Play Framework etc.

Develop efficient and scalable Kafka messaging solutions, configure and manage Kafka clusters, and ensure reliable data ingestion and processing.

Integrate data from different systems and sources, including structured and unstructured data, databases, and Mainframe systems, ensuring seamless data flow and compatibility.

Implement data validation and quality checks within data pipelines to ensure data accuracy, consistency, and integrity, and handle exception handling and error logging.

Develop and execute unit tests, integration tests, and end-to-end tests to ensure the reliability and correctness of Spark applications. Debug and troubleshoot issues during development and production phases.

Contribute to the overall DevOps for builds, application deployment stages and releases.

Designing, Implementing, and maintain Continuous Integration and Continuous Deployment pipelines. Automate/Optimize the build, test, and deployment processes to enable rapid and reliable software delivery.

Collaborate with cross-functional teams, including Product managers, Architects, and other developers, to understand data requirements and deliver optimal solutions. Document data engineering processes, data flows, and technical specifications.

Keep up-to-date with the latest advancements in Spark, Scala, and related technologies, frameworks, and tools. Continuously improve skills and knowledge through self-learning and professional development.