(At least modules 1-3 required + specialization for specific platform such as GCP, AWS, Azure)

A Data Engineer ensures that the data is properly collected, stored, and made accessible to the users, typically, data scientists and data analysts. Data Engineers are first of all strong software engineers, who are capable of constructing and maintaining performant, stable, and secure data ingestion and transformation pipelines.

Skills

Software engineering

Data modeling

Data pipeline management and automation

Data governance

Technologies

SQL and relational databases

Python (sometimes Scala or Java)

Data warehouses (GCP BigQuery, AWS Redshift, Azure Synapse Analytics)

Big data systems (e.g. Spark, Amazon EMR, Google BigQuery, Google Dataproc)

Orchestrators (Airflow, Dagster, Prefect)

(At least modules 1-3 required + specialization for specific platform such as GCP, AWS, Azure)

A Data Platform Engineer builds and maintains the data platform. Data Platform Engineers architect data platforms that are secure, reliable, and have the features that the platform users need to do their jobs.

Skills

Software engineering

Data platform management

Data pipeline management and automation

Data governance

Technologies

SQL and relational databases

Python (sometimes Scala or Java)

Big data systems (e.g. Spark, Amazon EMR, Google BigQuery, Google Dataproc)

Streaming systems (e.g. Flink, Apache Beam, Google Dataflow)

Containerization (Docker, Kubernetes)

Kafka, Pub/Sub, Kinesis

(At least modules 1-3 required + specialization for specific platform such as GCP, AWS, Azure)

An Analytics Engineer is a master of data modeling and transformation. Analytics Engineers make sure that the end users are provided with clean and reliable data. Moreover, they model data so that it empowers end users to turn the data into insights.

Skills

Data modeling

Writing performant data transformations

Data testing

Technologies

SQL and relational databases

dbt

Orchestrators (Airflow, Dagster, Prefect)

(At least modules 1-3 required + specialization for specific platform such as GCP, AWS, Azure)

An MLOps engineer builds and maintains the machine learning platform. MLOps engineers enable organizations to streamline their machine learning workflows, focusing on the end-to-end lifecycle of ML models. The MLOps Engineerings are experts on creating CI/CD pipelines, understanding big data infrastructure and security.

Skills

Software engineering

Machine lerning platform management

Model governance

Technologies

Python (sometimes Scala or Java)

ML Platforms (AWS SageMaker, Azure ML, GCP Vertex AI)

Containerization (Docker, Kubernetes)