Oracle-Azure Devops Engineer
Job Description
Responsibilities
As a Databricks Data Engineer, you will:
- Design, construct, install, test, and maintain highly scalable data management systems using Azure Databricks.
- Develop data processing pipelines using Databricks Notebooks, Spark SQL, and other Databricks utilities.
- Integrate Azure Databricks with other Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB.
- Ensure optimal performance of data processing and extraction, transformation, and loading (ETL) processes.
- Collaborate with data architects to implement effective data models and schemas within the Databricks environment.
- Monitor data reliability, efficiency, and quality, and implement improvements as needed.
- Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
- Develop and maintain documentation related to data pipeline architecture, development processes, and data governance.
- Stay current with industry trends and developments in cloud computing, big data, and Azure Databricks.
Skills
- Azure certifications such as Azure Data Engineer Associate or Azure Data Scientist Associate.
- Experience with machine learning frameworks and libraries.
- Knowledge of additional big data tools and platforms.
- Familiarity with data visualization tools and techniques.
Qualification
- Bachelor’s degree in Engineering or a related field.
- Proven experience as a Data Engineer with a focus on Azure Databricks and related Azure services.
- Strong programming skills in languages such as Python, Scala, or Java.
- Experience with Apache Spark and its integration with Databricks.
- Proficiency in SQL and experience with data modeling and data warehousing concepts.
- Familiarity with DevOps practices and CI/CD pipelines.
- Excellent analytical and problem-solving abilities.
- Effective communication skills and the ability to work collaboratively in a team environment.