Industry - Manufacturing
Category - IT & Systems
Job Type - Permanent
- Responsible for design and development of Big Data analytical applications and continuously enhance the project codebase
- Needs to have strong Big Data and Spark Optimization knowledge
- Help build high performance and flexible streaming pipelines that can rapidly evolve to handle new technologies, techniques, and modeling approaches
- Research and evaluate current and upcoming technologies and frameworks.
- Buildout of data ingestion framework/pipelines and troubleshoot complex issues in a distributed platform
- Write complex ETL processes and framework using cloud native services like Azure Data Factory, Azure Functions, Azure batch
- Implement large-scale near real-time streaming data processing pipelines
- Identify and assist in implementation of automation for the validation and approval of design specifications.
- Work with infrastructure engineers to ensure smooth deployments and sustainable performance for the platform.
- 12-15 Years of IT Experience.
- Should have experience with distributed systems (Spark, HDFS, ADLS, Kafka, Solr, Elastic Search, Kerberos, Oauth2, Security- Authorization & Authentication)
- Prior experience working on stream processing systems in production such as enabling real time data analytics at scale using technologies like Apache Kafka, Flink, Spark, Druid, Cassandra, Elasticsearch etc.
- Systems Engineering - you've built meaningful pieces of infrastructure in a cloud computing environment. Bonus if those were data processing systems or distributed systems
- Minimum of 5 years experienced working in cloud native environment (Azure, AWS, GCP)
- Hands on experience developing in Spark, Java, Spring Boot, Python
Didn’t find the job appropriate? Report this Job