Kafka Data Engineer
2T Consulting
Washington Crossing, Pennsylvania
Compensation: 110000 - 120000
We are seeking an experienced Kafka/Spark Data Engineer with strong expertise in real-time data processing, streaming technologies, and distributed systems. The role will focus on Kafka, Spark, Redis, Python, and cloud-based Big Data solutions.
Roles and Responsibilities
- Design, develop, and support real-time data streaming and Big Data solutions.
- Develop streaming applications using Apache Kafka, KStreams, and KTables.
- Work with Apache Spark for large-scale data processing.
- Develop applications using Python, Scala, Java, and Bash/Shell scripting.
- Work with NoSQL databases such as Redis, MongoDB, and HBase.
- Design scalable, distributed, and fault-tolerant data architectures.
- Implement data integration, security, and authentication solutions, including Kerberos.
- Work with streaming technologies such as Flink or Storm.
- Support cloud-based data solutions across AWS, Azure, or GCP.
- Apply data analysis, statistical methods, and machine learning techniques where required.
- Create and support data visualizations using Tableau.
- Troubleshoot business-critical applications and provide effective technical solutions.
Required Qualifications
- 9+ years of overall IT/industry experience, with strong Big Data and real-time processing experience.
- Bachelor's or Master's degree in Computer Science, Engineering, Science, or related field.
- Strong hands-on experience with Kafka and real-time streaming.
- Experience with Spark, Redis, and Python.
- Strong understanding of distributed systems, data partitioning, and fault-tolerant architectures.
- Experience with NoSQL databases and cloud platforms.
- Strong problem-solving, analytical, and communication skills.
Preferred Qualifications
- Experience with Flink, Storm, Scala, or Java.
- Knowledge of machine learning and predictive analytics.
- Experience with Tableau and real-time data visualization.
MilitaryX
