What is this Book?
How to Contribute
YouTube
Twitter
Amazon Shop
If You Like This Book & Need More Help
Check out my Data Engineering Academy at LearnDataEngineering.com trusted by almost 2,000 students!
Visit learndataengineering.com: Click Here
- Learn Data Engineering with our online Academy
- Perfect for becoming a Data Engineer or add Data Engineering to your skillset
- Proven process based on years of experience and hundreds of hours of personal coaching
- Over 30 prepared courses on the most important techniques, fundamental tools and platforms plus our
- Associate Data Engineer Certification
- Academy Discord server with over 1,000 members
Support This Book For Free!
- Amazon: Click Here buy whatever you like from Amazon using this link* (Also check out my complete podcast gear and books)
Here's what's new:
Find the change log with all recent updates here: SEE UPDATES
Building a Portfolio Project?
Use my project README template to document it like a professional would: Get the template
Contents:
- Introduction
- Basic Engineering Skills
- Advanced Engineering Skills
- Free Hands On Courses / Tutorials‚
- Case Studies
- Best Practices Cloud Platforms
- 130+ Data Sources for Portfolio Projects
- Project README Template
- 1001 Interview Questions
- Recommended Books, Courses, and Podcasts
- Updates
- How To Contribute
- Support What You Like
- Important Links
Full Table Of Contents:
Introduction
- What is this Cookbook
- Data Engineers
- My Data Science Platform Blueprint
- Connect
- Buffer
- Processing Framework
- Store
- Visualize
- Who Companies Need
- How to Learn Data Engineering
- Andreas on the Super Data Science Podcast
- Building Blocks to Learn Data Engineering
- Roadmap for Beginners
- Roadmap for Data Analysts
- Roadmap for Data Scientists
- Roadmap for Software Engineers
- Data Engineers Skills Matrix
- How to Become a Senior Data Engineer
Basic Engineering Skills
- Learn To Code
- Get Familiar With Git
- Agile Development
- Why is agile so important?
- Agile rules I learned over the years
- Agile Frameworks
- Software Engineering Culture
- Learn how a Computer Works
- Data Network Transmission
- Security and Privacy
- SSL Public and Private Key Certificates
- JSON Web Tokens
- GDPR regulations
- Linux
- OS Basics
- Shell scripting
- Cron Jobs
- Packet Management
- Docker
- What is Docker and How it Works
- Don't Mess Up Your System
- Preconfigured Images
- Take it With You
- Kubernetes Container Deployment
- How to Create Start and Stop a Container
- Docker Micro Services
- Kubernetes
- Why and How To Do Docker Container Orchestration
- Userful Docker Commands
- The Cloud
- IaaS vs PaaS vs SaaS
- AWS Azure IBM Google IBM
- Cloud vs On-Premises
- Security
- Hybrid Clouds
- Security Zone Design
- How to secure a multi layered application
- Cluster security with Kerberos
Advanced Engineering Skills
- Data Science Platform
- Why a Good Data Platform Is Important
- Big Data vs Data Science and Analytics
- The 4 Vs of Big Data
- Why Big Data
- Planning is Everything
- The Problem with ETL
- Scaling Up
- Scaling Out
- When not to Do Big Data
- 81 Platform & Pipeline Design Questions
- Data Source Questions
- Goals and Destination Questions
- Connect
- REST APIs
- API Design
- Implemenation Frameworks
- Security
- Apache Nifi
- Logstash
- Buffer
- Apache Kafka
- Why a Message Queue Tool?
- Kafka Architecture
- Kafka Topics
- Kafka and Zookeeper
- How to Produce and Consume Messages
- Kafka Commands
- Apache Redis Pub-Sub
- AWS Kinesis
- Google Cloud PubSub
- Processing Frameworks
- Lambda and Kappa Architecture
- Batch Processing
- Stream Processing
- Three Methods of Streaming
- At Least Once
- At Most Once
- Exactly Once
- Check The Tools
- Should You do Stream or Batch Processing
- Is ETL still relevant for Analytics?
- MapReduce
- How Does MapReduce Work
- MapReduce
- MapReduce Example
- MapReduce Limitations
- Apache Spark
- What is the Difference to MapReduce?
- How Spark Fits to Hadoop
- Spark vs Hadoop
- Spark and Hadoop a Perfect Fit
- Spark on YARn
- My Simple Rule of Thumb
- Available Languages
- Spark Driver Executor and SparkContext
- Spark Batch vs Stream processing
- How Spark uses Data From Hadoop
- What are RDDs and How to Use Them
- SparkSQL How and Why to Use It
- What are Dataframes and How to Use Them
- Machine Learning on Spark (TensorFlow)
- MLlib
- Spark Setup
- Spark Resource Management
- AWS Lambda
- Apache Flink
- Elasticsearch
- Apache Drill
- StreamSets
- Store
- Analytical Data Stores
- Data Warehouse vs Data Lake
- Snowflake and dbt
- Transactional Data Stores
- SQL Databases
- PostgreSQL DB
- Database Design
- SQL Queries
- Stored Procedures
- ODBC/JDBC Server Connections
- NoSQL Stores
- HBase KeyValue Store
- HDFS Document Store
- MongoDB Document Store
- Elasticsearch Document Store
- Hive Warehouse
- Impala
- Kudu
- Apache Druid
- InfluxDB Time Series Database
- Greenplum MPP Database
- Visualize
- Android and IOS
- API Design for Mobile Apps
- Dashboards
- Webservers
- Tomcat
- Jetty
- NodeRED
- React
- Business Intelligence Tools
- Tableau
- Power BI
- Quliksense
- Identity & Device Management
- What Is A Digital Twin
- Active Directory
- Machine Learning
- How to do Machine Learning in production
- Why machine learning in production is harder then you think
- Models Do Not Work Forever
- Where are The Platforms That Support Machine Learning
- Training Parameter Management
- How to Convince People That Machine Learning Works
- No Rules No Physical Models
- You Have The Data. Use It!
- Data is Stronger Than Opinions
- AWS Sagemaker
Hands On Course
- Free Data Engineering Course with AWS, TDengine, Docker and Grafana
- Monitor your data in dbt & detect quality issues with Elementary
- Solving Engineers 4 Biggest Airflow Problems
- The best alternative to Airlfow? Mage.ai
Case Studies
- Data Science @Airbnb
- Data Science @Amazon
- Data Science @Baidu
- Data Science @Blackrock
- Data Science @BMW
- Data Science @Booking.com
- Data Science @CERN
- Data Science @Disney
- Data Science @DLR
- Data Science @Drivetribe
- Data Science @Dropbox
- Data Science @Ebay
- Data Science @Expedia
- Data Science @Facebook
- Data Science @Google
- Data Science @Grammarly
- Data Science @ING Fraud
- Data Science @Instagram
- Data Science @LinkedIn
- Data Science @Lyft
- Data Science @NASA
- Data Science @Netflix
- Data Science @OLX
- Data Science @OTTO
- Data Science @Paypal
- Data Science @Pinterest
- Data Science @Salesforce
- Data Science @Siemens Mindsphere
- Data Science @Slack
- Data Science @Spotify
- Data Science @Symantec
- Data Science @Tinder
- Data Science @Twitter
- Data Science @Uber
- Data Science @Upwork
- Data Science @Woot
- Data Science @Zalando
Best Practices Cloud Platforms
- Amazon Web Services (AWS)
- Connect
- Buffer
- Processing
- Store
- Visualize
- Containerization
- Best Practices
- More Details
- Microsoft Azure
- Connect
- Buffer
- Processing
- Store
- Visualize
- Containerization
- Best Practices
- Google Cloud Platform (GCP)
- Connect
- Buffer
- Processing
- Store
- Visualize
- Containerization
- Best Practices
130+ Data Sources for Portfolio Projects
- Student Favorites
- General And Academic
- Content Marketing
- Crime
- Drugs
- Education
- Entertainment
- Environmental And Weather Data
- Financial And Economic Data
- Government And World
- Health
- Human Rights
- Labor And Employment Data
- Politics
- Retail
- Social
- Travel And Transportation
- Various Portals
- Source Articles and Blog Posts
- Free Data Sources Data Science
1001 Interview Questions
Recommended Books, Courses, and Podcasts
- About Books and Courses
- Books
- Languages
- Data Tools & Platforms
- Business
- Community Recommendations
- Online Courses
- Preparation courses
- Data engineering courses
- Certifications
- Podcasts
- Super Data Science
- Data Skeptic
- Data Engineering Podcast
- Roaring Elephant BiteSized Big Tech
- SQL Data Partners Podcast
How To Contribute
If you have some cool links or topics for the cookbook, please become a contributor.
Simply pull the repo, add your ideas and create a pull request.
You can also open an issue and put your thoughts there.
Please use the "Issues" function for comments.
Important Links
Subscribe to my YouTube channel for regular updates:
Link to YouTube
I have a Medium publication where you can publish your data engineer articles to reach more people:
Medium publication
*(As an Amazon Associate I earn from qualifying purchases from Amazon
This is free of charge for you, but super helpful for supporting this channel)
