Data Engineer

Profile

Responsibilities: Engineering and software development skills in the delivery of full data lifecycle frameworks and patterns including data acquisition, migration, storage, transformation, prep, and consumption to support a ground up Enterprise Data Lake build for use in analytics solutions at different maturity levels between data science and operational teams Hands-on data engineering design and implementations including data ingestion, data models, data structures, data storage, high-throughput data processing, data pipelines, and data monitoring at scale Follow data engineering best practices with considerations for high data availability, computational efficiency, cost, and quality Build and maintain environments, processes, functionalities, and tools to improve all stages of data lake implementation and analytics solution development, e.g., proof of concepts, prototypes, and production Deliver capability for repeatable, configuration driven automation throughout the data lifecycle based on designated approaches Maintain awareness of relevant technology trends and product updates (i.e. AWS Services) Skills implementing general IT concepts, st***gies, methodologies, modern application and data engineering architectures and approaches including cloud, streaming, event based, IoT data, and edge server capability Write complex programs, enabling automation in cloud environments Implement configuration driven, reusable, automation frameworks consistent with designed approaches Apply DevOps best practices, CI/CD processes and tools, testing frameworks Optimize data solutions for multi-petabyte data systems in batch, streaming, and event approaches Work with agile methodologies, cross-functional teams (Product Owners, Scrum Masters, Developers, Test Engineers), backlog grooming, and tooling Ensure alignment and best practices for data governance Qualifications 3 years of professional experience in data engineering related roles - full software development lifecycle, DBA, data architect, data engineer 1 years specific experience in modern data engineering and cloud data practices including various data lake ingestion techniques, ETL/ELT, consumption, and operations Working experience in data analytics (data wrangling, mining, integration, analysis, visualization, data modeling, analysis/analytics, and reporting) using BI (Business Intelligence) tools Expertise in AWS services including S3, EC2, SQS, EMR, Lambda, Step Functions, Terraform, Glue, Redshift, Athena, DynamoDB, Cloudwatch, and IAM; Programming background and ability to utilize a variety of software/languages/tools, e.g., Python, Scala, Java, Spark, Hive, SQL; Linux including Korn shell, scripting, and regex Degree in Computer Science or equivalent work experience Required Skills : Amazon Web Services (AWS) Expertise in AWS services including S3, EC2, SQS, EMR, Lambda Must be able to build out data lakes

Company info

Sign Up Now - EngineeringCrossing.com