Hadoop Developer with Python

Hadoop Developer with Python Only W2 (Citizen,TN, GC, GC EAD and H4EAD)
Location: Chicago, IL / Denver, CO

Duration: 12+ Months

Must-Have Skills:

Hadoop: Extensive experience with Hadoop ecosystem and related technologies

Programming: Proficiency in Python (or Scala)

SparkSQL: Strong skills in Spark SQL for data processing and analysis

SQL: Proficiency in at least one of the following: MySQL, Hive, Impala

Data Ingestion: Experience with ingesting data from various sources (message queues, file shares, REST APIs, relational databases) and handling different data formats (JSON, CSV, XML).

Spark Structured Streaming: Experience working with Spark Structured Streaming for real-time data processing.

Responsibilities:

Design, develop, and maintain data pipelines using Hadoop, Spark, and Python/Scala

Ingest data from various sources and transform it into usable formats

Optimize Spark/MapReduce and SQL jobs for performance

Work in an Agile development environment, collaborating with cross-functional teams

Utilize version control systems like SVN, Git, or Bitbucket

Deploy and manage applications using Jenkins and handle Jar management

Qualifications:

Strong understanding of Big Data concepts and technologies

Hands-on experience with Hadoop, Spark, Sqoop, Kafka, MapReduce, and NoSQL databases (HBase, Solr, etc.)

Proficiency in Python or Scala for data processing and scripting

Experience with Linux operating system

Familiarity with data visualization tools like Elastic Search and Kibana

Ability to work independently and as part of a team

Excellent communication and problem-solving skills

Adaptable and eager to learn new technologies