Data Architect - Palo Alto, CA (Hybrid)

Data Architect

Location: Palo Alto, CA (Hybrid)

Duration: Long Term

Rate: DOE

Key Responsibilities:

Data Orchestration:

Design, implement, and manage data workflows using Airflow to automate and orchestrate data processing tasks.

Optimize Airflow DAGs (Directed Acyclic Graphs) for performance and scalability.

Task Management:

Develop and maintain distributed task processing using Celery and ensure robust task queue management with Redis or RabbitMQ.

Database Management:

Design and manage databases using Cosmos DB, MongoDB, and PostgreSQL.

Develop and maintain efficient data models and ensure data consistency and integrity.

API and Webhooks:

Implement and manage FastAPI webhooks to handle data ingestion and integration tasks.

Develop and maintain Azure Functions to support webhook operations and integrate with cloud services.

Streaming Data:

Implement and manage Kafka Streams to handle real-time data processing and streaming requirements.

Data Lake Management:

Work with Iceberg to manage and optimize large-scale data lake storage and querying.

Collaboration and Communication:

Collaborate with data scientists, engineers, and business analysts to understand data requirements and provide technical solutions.

Document processes, architectures, and configurations to ensure knowledge sharing and compliance with best practices.

Required Skills and Qualifications:

Experience and Knowledge:

Proven experience with Airflow for data orchestration and workflow management.

Hands-on experience with Celery for task management and Redis or RabbitMQ for messaging.

Proficiency with Cosmos DB, MongoDB, and PostgreSQL for data storage and management.

Experience developing and managing webhooks using FastAPI and integrating with Azure Functions.

Knowledge of Kafka Streams for real-time data processing.

Familiarity with Iceberg for data lake management and optimization.

Healthcare domain experience is good to have .

Technical Skills:

Strong understanding of data pipelines, ETL processes, and data integration.

Proficient in Python, with experience in building and maintaining data-oriented applications.

Ability to work with large datasets and optimize performance across distributed systems

Soft Skills:

Excellent problem-solving and analytical skills.

Strong communication and collaboration skills.

Ability to work independently and manage multiple priorities in a fast-paced environment.