BEGINNER LEVEL QUESTIONS
1. What is Databricks?
Answer:
Databricks is a cloud-based data platform built on Apache Spark. It is used for data engineering, data science, machine learning, and big data analytics.
It allows teams to:
- Process large amounts of data
- Build machine learning models
- Collaborate using notebooks
- Run Spark without managing infrastructure
In simple words, Databricks helps companies analyze big data faster and more easily.
2. What is Apache Spark, and how is it related to Databricks?
Answer:
Apache Spark is an open-source distributed computing system used for big data processing.
Databricks is built on top of Apache Spark. It makes Spark easier to use by:
- Managing clusters automatically
- Providing notebooks
- Offering performance optimizations
So, Spark is the engine, and Databricks is the platform that makes it powerful and user-friendly.
3. What are the main components of Databricks?
Answer:
The main components are:
- Workspace – Where users create notebooks and collaborate.
- Clusters – Computing resources used to run code.
- Jobs – Used to schedule and automate tasks.
- DBFS (Databricks File System) – Storage layer.
- Delta Lake – Storage layer that supports ACID transactions.
4. What is a Databricks Cluster?
Answer:
A cluster is a group of virtual machines that work together to process data.
There are two types:
- All-purpose cluster – Used for interactive analysis.
- Job cluster – Created automatically to run a specific job.
Clusters can auto-scale depending on workload.
5. What is DBFS?
Answer:
DBFS (Databricks File System) is a distributed file system in Databricks.
It allows users to:
- Store data
- Read/write files
- Mount cloud storage like AWS S3 or Azure Data Lake
It works like a bridge between Databricks and cloud storage.
6. What is a Databricks Notebook?
Answer:
A notebook is an interactive environment where users write and execute code.
Databricks notebooks support:
- Python
- SQL
- Scala
- R
They are used for:
- Data analysis
- Machine learning
- Visualization
- Collaboration
7. What languages are supported in Databricks?
Answer:
Databricks supports:
- Python
- SQL
- Scala
- R
You can even switch between languages inside the same notebook.
8. What is Delta Lake?
Answer:
Delta Lake is a storage layer built on top of data lakes.
It provides:
- ACID transactions
- Schema enforcement
- Time travel
- Better performance
It solves common data lake problems like dirty data and inconsistent reads.
9. What is ACID in Delta Lake?
Answer:
ACID stands for:
- Atomicity
- Consistency
- Isolation
- Durability
Delta Lake ensures that data operations are reliable and safe, even if there is failure during processing.
10. What is Auto Scaling in Databricks?
Answer:
Auto Scaling automatically increases or decreases the number of cluster nodes based on workload.
Benefits:
- Saves cost
- Improves performance
- No manual intervention required
INTERMEDIATE LEVEL QUESTIONS
11. What is the difference between RDD, DataFrame, and Dataset?
Answer:
- RDD – Low-level distributed collection of objects.
- DataFrame – Structured data with columns (like a table).
- Dataset – Strongly typed version of DataFrame (mostly in Scala).
DataFrames are most commonly used because they are optimized and easy to use.
12. What is Lazy Evaluation in Spark?
Answer:
Spark does not execute transformations immediately.
It builds a plan and only executes when an action (like show() or count()) is called.
This improves performance by optimizing the execution plan.
13. What are Transformations and Actions?
Answer:
Transformations – Modify data (e.g., filter, select, groupBy).
Actions – Trigger execution (e.g., count, collect, display).
Spark only runs when an action is called.
14. What is a Spark Job?
Answer:
A Spark job is triggered when an action is executed.
One job can have:
- Multiple stages
- Multiple tasks
It represents a full data processing task.
15. What is Caching in Databricks?
Answer:
Caching stores data in memory for faster reuse.
If you use the same DataFrame multiple times, caching avoids recalculating it.
Example:
df.cache()
16. What is Partitioning in Spark?
Answer:
Partitioning divides data into smaller chunks across nodes.
Benefits:
- Parallel processing
- Faster execution
Too many or too few partitions can affect performance.
17. What is a Broadcast Join?
Answer:
In broadcast join, a small table is sent to all worker nodes to join with a large table.
This improves join performance.
Used when one dataset is small.
18. What is a UDF in Databricks?
Answer:
UDF (User Defined Function) allows you to write custom functions in Spark.
Used when built-in functions are not enough.
However, UDFs may reduce performance compared to built-in functions.
19. What is Structured Streaming?
Answer:
Structured Streaming is Spark’s stream processing engine.
It allows processing real-time data like:
- Logs
- IoT data
- Clickstream data
It treats streaming data like a continuously updating table.
20. What is the Medallion Architecture?
Answer:
Medallion architecture organizes data into 3 layers:
- Bronze – Raw data
- Silver – Cleaned and transformed data
- Gold – Business-level aggregated data
It improves data quality and organization.
ADVANCED LEVEL QUESTIONS
21. How does Delta Lake handle schema evolution?
Answer:
Delta Lake allows adding new columns automatically using schema evolution.
It prevents incorrect schema changes and ensures data consistency.
22. What is Time Travel in Delta Lake?
Answer:
Time travel allows accessing previous versions of data.
Example:
SELECT * FROM table VERSION AS OF 3
Useful for:
- Auditing
- Debugging
- Recovering deleted data
23. What is Z-Ordering?
Answer:
Z-Ordering improves query performance by colocating related data in storage.
It helps in faster filtering of large datasets.
24. What is the OPTIMIZE command in Delta?
Answer:
OPTIMIZE compacts small files into larger ones.
Benefits:
- Faster queries
- Reduced file overhead
25. What is Data Skew?
Answer:
Data skew happens when one partition has much more data than the others.
It causes:
- Slow performance
- Uneven workload
Solutions:
- Salting
- Repartitioning
- Broadcast joins
26. What is Photon in Databricks?
Answer:
Photon is a high-performance query engine in Databricks.
It improves SQL and Delta performance significantly.
27. What is Unity Catalog?
Answer:
Unity Catalog is a data governance solution.
It provides:
- Centralized access control
- Data lineage
- Auditing
28. What are Jobs in Databricks?
Answer:
Jobs allow scheduling notebooks or scripts.
Features:
- Scheduling
- Monitoring
- Retry on failure
- Email alerts
29. How do you optimize a slow Databricks job?
Answer:
Steps:
- Check Spark UI
- Increase partitions
- Use broadcast joins
- Avoid UDFs
- Cache reusable data
- Use OPTIMIZE and Z-ORDER
30. What are common real-world use cases of Databricks?
Answer:
- Data warehousing
- ETL pipelines
- Machine learning
- Real-time analytics
- Fraud detection
- Recommendation systems
