Data Integration Engineer

About the Role

As a Data Integration Engineer at Virtusa, you will be responsible for designing and building connectors between enterprise security platforms and Databricks, establishing secure API connections, and implementing data extraction logic using REST APIs, SDKs, and native connectors.

You will work with a team to develop automated Python scripts for continuous data ingestion, build orchestration workflows using Apache Airflow, and implement error handling, retry mechanisms, and monitoring for pipeline reliability.

Key Responsibilities

• Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys, CrowdStrike, Okta, AWS CloudTrail) and Databricks.
• Establish secure API connections and authentication mechanisms with each source system.
• Implement data extraction logic using REST APIs, SDKs, and native connectors.
• Develop automated Python scripts for continuous data ingestion.
• Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools.
• Implement error handling, retry mechanisms, and monitoring for pipeline reliability.
• Configure S3 bucket architecture and data partitioning strategy.

Skills & Qualifications

• Advanced Python programming skills with experience in libraries like pandas, requests, and boto3.
• API Integration skills with knowledge of REST APIs, authentication (OAuth, API keys), and error handling.
• Cloud Platforms skills with proficiency in AWS (S3, EC2, IAM) and Databricks/Spark (PySpark, Delta Lake, SQL optimization).
• Experience with data pipeline tools like Airflow, Databricks Workflows, or equivalent orchestration.
• SQL skills with knowledge of complex queries, window functions, and performance tuning.
• Version control skills with experience in Git.
• Bachelor's degree in Computer Science, Information Technology, or related field.

What You'll Learn

As a Data Integration Engineer at Virtusa, you will gain hands-on experience in designing and building data pipelines, working with cloud platforms like AWS and Databricks, and implementing secure API connections and authentication mechanisms.

You will also learn about data quality and validation frameworks, performance tuning and optimization, and CI/CD pipeline familiarity.

Resume Tip

When applying for this role, make sure to highlight your experience with Python, API integration, and cloud platforms like AWS and Databricks.

Include specific examples of data pipelines you have built, and highlight your skills in data quality and validation, performance tuning, and CI/CD pipeline familiarity.