https://bricksnotes.com/blog Practical writing on Databricks, Spark, Delta Lake, and the craft of data engineering. en Thu, 20 Aug 2026 13:30:43 GMT https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings Thu, 20 Aug 2026 13:30:43 GMT BricksNotes Editorial A boundary instead of a study plan: two evenings, 100 messy orders, one gold table, and a pipeline that runs twice without breaking. Here is the full walkthrough you can repeat tonight in Databricks Free Edition. Databricks Data Engineering PySpark Getting Started 2026 https://bricksnotes.com/blog/databricks-late-arriving-data https://bricksnotes.com/blog/databricks-late-arriving-data Thu, 20 Aug 2026 01:50:50 GMT BricksNotes Editorial A phone was offline, a partner file was late, a queue backed up. Here is how to keep event time honest and stop numbers from drifting quietly. Databricks Streaming Delta Lake Data Quality https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream Wed, 19 Aug 2026 01:51:14 GMT BricksNotes Editorial Reloading March means touching a table six dashboards already read. Here is the plan that makes a backfill boring instead of frightening. Databricks Delta Lake Backfill Pipelines https://bricksnotes.com/blog/agent-discovers-agent-applies https://bricksnotes.com/blog/agent-discovers-agent-applies Tue, 18 Aug 2026 13:38:17 GMT BrickNotes Editorial An agent found the table, read the schema, and proposed the merge in forty seconds. Everything it got right came from work a data engineer had already done. AI Agents Unity Catalog Data Quality Delta Lake https://bricksnotes.com/blog/schema-evolution-databricks-pipelines https://bricksnotes.com/blog/schema-evolution-databricks-pipelines Tue, 18 Aug 2026 01:46:27 GMT BrickNotes Editorial A source team added one column and renamed another, and the nightly job failed. Here is how schema change actually works in Databricks, and how to handle each case calmly. Databricks Delta Lake Schema Evolution Auto Loader https://bricksnotes.com/blog/how-to-become-ai-data-engineer-with-bricksnotes https://bricksnotes.com/blog/how-to-become-ai-data-engineer-with-bricksnotes Mon, 17 Aug 2026 00:43:50 GMT BrickNotes Editorial A complete path for a fresh data engineering aspirant. Free Edition setup, the lessons in order, the habits that matter, and how to grow into AI ready work. Career Beginners Learning Path AI Data Engineer https://bricksnotes.com/blog/auto-loader-vs-copy-into-databricks https://bricksnotes.com/blog/auto-loader-vs-copy-into-databricks Sun, 16 Aug 2026 04:27:00 GMT BrickNotes Editorial A daily CSV drop grew to seven hundred files and turned a five minute job into forty. Here is how COPY INTO and Auto Loader remember what they already loaded, and how to choose. Databricks Auto Loader Ingestion Delta Lake https://bricksnotes.com/blog/idempotent-pipelines-safe-reruns-databricks https://bricksnotes.com/blog/idempotent-pipelines-safe-reruns-databricks Sat, 15 Aug 2026 03:02:12 GMT BrickNotes Editorial A nightly load failed at 2am, got rerun at 7am, and quietly doubled a revenue number for a week. Here is what idempotency really means, and the four write patterns that make a Databricks pipeline safe to run twice. Databricks Delta Lake Pipelines Data Quality https://bricksnotes.com/blog/databricks-190-billion-journey https://bricksnotes.com/blog/databricks-190-billion-journey Fri, 14 Aug 2026 02:37:26 GMT BrickNotes Editorial Databricks closed a $5 billion round at a $190 billion valuation in August 2026. Here is the full thirteen-year story behind that number, from Spark at Berkeley to Delta Lake, Unity Catalog, Lakeflow and the agent era, and what it means for data engineers. Databricks Story Lakehouse Unity Catalog 2026 https://bricksnotes.com/blog/streaming-table-vs-materialized-view-databricks https://bricksnotes.com/blog/streaming-table-vs-materialized-view-databricks Thu, 13 Aug 2026 03:54:48 GMT BrickNotes Editorial A nightly silver job that used to finish in twenty minutes started running past breakfast. Nothing was wrong with the code. It was rebuilding the whole table every night. Here is how to decide between a streaming table and a materialized view in Databricks, with examples that run on Free Edition. Lakeflow Streaming Materialized Views Databricks https://bricksnotes.com/blog/delta-lake-zorder-vs-liquid-clustering https://bricksnotes.com/blog/delta-lake-zorder-vs-liquid-clustering Wed, 12 Aug 2026 03:14:25 GMT BrickNotes Editorial Your dashboard slowed down and the query never changed. Most Databricks performance problems are layout problems. A practical guide to partitioning, Z-ORDER, and liquid clustering in 2026. Delta Lake Performance Liquid Clustering Databricks https://bricksnotes.com/blog/context-engineering-skill-for-data-engineers https://bricksnotes.com/blog/context-engineering-skill-for-data-engineers Mon, 10 Aug 2026 12:41:12 GMT BrickNotes Editorial An agent answered a simple question confidently and wrongly. Nobody had decided what it was allowed to know. That decision is now a job, and data engineers are the natural owners of it. Context Engineering AI Agents Databricks Governance context-advantage https://bricksnotes.com/blog/agentic-data-stack-built-for-ai-agents https://bricksnotes.com/blog/agentic-data-stack-built-for-ai-agents Sat, 08 Aug 2026 15:50:08 GMT BrickNotes Editorial Data platforms assumed a human was always in the loop. Agents break that assumption. Here is what context, tool access, observability, cost, and approval look like now. AI Agents Databricks Governance Context Engineering https://bricksnotes.com/blog/databricks-lakemeter-cost-estimation https://bricksnotes.com/blog/databricks-lakemeter-cost-estimation Fri, 07 Aug 2026 01:39:08 GMT BrickNotes Editorial Someone asks what your pipeline will cost. Lakemeter, the open source tool from Databricks Labs, turns that shrug into an estimate you can defend. Databricks Cost Open Source https://bricksnotes.com/blog/databricks-unity-ai-gateway-ga-explained https://bricksnotes.com/blog/databricks-unity-ai-gateway-ga-explained Wed, 05 Aug 2026 19:48:57 GMT BrickNotes Editorial Databricks made Unity AI Gateway generally available on August 4, 2026. Here is what it actually does, why cost and control matter more than model choice, and what still belongs to you. Databricks AI Governance Data Engineering https://bricksnotes.com/blog/databricks-variant-data-type-explained https://bricksnotes.com/blog/databricks-variant-data-type-explained Wed, 05 Aug 2026 03:00:00 GMT BrickNotes Editorial Databricks Variant is now generally available. It is a native SQL type for semi-structured data that solves the old tradeoff between flexibility and query performance. Here is what it does, how Variant Shredding works, and how to start using it with real code. Databricks Delta Lake Data Engineering https://bricksnotes.com/blog/100000-data-professionals-bricksnotes-milestone https://bricksnotes.com/blog/100000-data-professionals-bricksnotes-milestone Fri, 31 Jul 2026 03:43:02 GMT BrickNotes Editorial More than 100,000 unique data professionals have visited BricksNotes, and over 25,000 practice exams have been completed across Associate, Professional, and Analyst certifications. What these numbers mean, and where we go next. Community Milestone https://bricksnotes.com/blog/databricks-genie-code-sql-migration-explained https://bricksnotes.com/blog/databricks-genie-code-sql-migration-explained Fri, 31 Jul 2026 03:26:56 GMT BricksNotes Editorial Databricks announced an agentic code converter in Genie Code that translates proprietary SQL dialects to ANSI SQL. Here is what it does, what it does not do, and why open SQL matters for your career. databricks sql-migration genie-code ansi-sql unity-catalog https://bricksnotes.com/blog/databricks-context-engineer-associate-explained https://bricksnotes.com/blog/databricks-context-engineer-associate-explained Thu, 30 Jul 2026 02:04:01 GMT BricksNotes Editorial Databricks has a new certification that is not about models. It is about the information you put around them. Here is what the Context Engineer Associate covers, and why data engineers are closer to it than they think. databricks certification context-engineering ai https://bricksnotes.com/blog/databricks-free-edition-playbook-2026 https://bricksnotes.com/blog/databricks-free-edition-playbook-2026 Tue, 28 Jul 2026 02:35:10 GMT BricksNotes Editorial A calm, practical seven-day path for learning Databricks on Free Edition in 2026. What you get, what you do not, and the exact steps to build real skills. Databricks Free Edition Learning https://bricksnotes.com/blog/databricks-previews-late-july-2026 https://bricksnotes.com/blog/databricks-previews-late-july-2026 Mon, 27 Jul 2026 13:40:34 GMT BricksNotes Editorial A calm read of the Databricks previews from mid to late July 2026. Genie Agent API, agents that can read files in Unity Catalog Volumes, Lakebridge, pipeline unit tests, and a few smaller wins. databricks previews genie lakeflow unity-catalog https://bricksnotes.com/blog/ai-data-engineer-roadmap-2026 https://bricksnotes.com/blog/ai-data-engineer-roadmap-2026 Sun, 26 Jul 2026 21:00:00 GMT BricksNotes Editorial A calm, stage-based roadmap to becoming an AI data engineer in 2026. Databricks foundation, lakehouse pipelines, AI applications, and agent-ready systems. ai-data-engineer career roadmap context-advantage databricks https://bricksnotes.com/blog/failed-databricks-associate-exam-twice-what-fixed-it https://bricksnotes.com/blog/failed-databricks-associate-exam-twice-what-fixed-it Sun, 26 Jul 2026 03:26:07 GMT BricksNotes Editorial Two failed attempts taught me more than any cheat sheet. Here is the calm, five-part routine and two-week plan that finally worked. certification associate-exam study-guide https://bricksnotes.com/blog/apache-spark-42-defining-truth-age-of-ai https://bricksnotes.com/blog/apache-spark-42-defining-truth-age-of-ai Sat, 25 Jul 2026 02:52:39 GMT BricksNotes Editorial Apache Spark 4.2 shipped in July 2026. Metric Views, mature Spark Connect, vector search in SQL, and Auto CDC change where truth lives, how agents talk to Spark, and how streaming catches up to reality. Spark Databricks AI https://bricksnotes.com/blog/design-beautiful-aibi-dashboards https://bricksnotes.com/blog/design-beautiful-aibi-dashboards Tue, 21 Jul 2026 02:42:59 GMT BricksNotes Editorial Themes, layout, and the small design habits that turn a Databricks AI/BI dashboard from noisy to trustworthy. https://bricksnotes.com/blog/agent-bricks-ga-ship-without-regret https://bricksnotes.com/blog/agent-bricks-ga-ship-without-regret Tue, 21 Jul 2026 02:39:25 GMT BricksNotes Editorial Agent Bricks is now generally available. Here is how to take an agent from demo to production without it going off the rails. https://bricksnotes.com/blog/lakebase-ga-do-you-still-need-postgres https://bricksnotes.com/blog/lakebase-ga-do-you-still-need-postgres Sun, 19 Jul 2026 02:03:59 GMT BricksNotes Editorial Lakebase is generally available. Do you still need a separate Postgres next to your lakehouse? A calm, honest answer. lakebase postgres unity-catalog architecture databricks 2026 https://bricksnotes.com/blog/data-foundation-for-ai-agents-databricks https://bricksnotes.com/blog/data-foundation-for-ai-agents-databricks Sat, 18 Jul 2026 04:05:07 GMT BricksNotes Editorial Before you plug an agent into your warehouse, ask four quiet questions about the data underneath it. ai-agents data-quality unity-catalog medallion databricks https://bricksnotes.com/blog/context-control-cost-choice-data-engineer-guide https://bricksnotes.com/blog/context-control-cost-choice-data-engineer-guide Fri, 17 Jul 2026 03:25:14 GMT BricksNotes Editorial The interesting question is not which model to pick. It is what the model sees, what it is allowed to do, what one answer costs, and how locked in you become. context-advantage ai-agents career databricks https://bricksnotes.com/blog/genie-one-mobile-what-it-means https://bricksnotes.com/blog/genie-one-mobile-what-it-means Thu, 16 Jul 2026 15:05:29 GMT BricksNotes Editorial Genie One is now a mobile app. A quiet shift in how business users meet their data, and a bigger one than it looks. databricks genie mobile ai unity-catalog https://bricksnotes.com/blog/databricks-data-analyst-associate-practice-exam-live https://bricksnotes.com/blog/databricks-data-analyst-associate-practice-exam-live Tue, 14 Jul 2026 19:35:35 GMT BricksNotes Editorial A free, full-length practice exam for the Databricks Data Analyst Associate certification is now live. Try it, miss it, review it, and try again. databricks certification data-analyst-associate free-edition https://bricksnotes.com/blog/what-data-engineers-really-do https://bricksnotes.com/blog/what-data-engineers-really-do Sun, 12 Jul 2026 04:33:18 GMT BricksNotes Editorial Data engineers help data move safely from where it is created to where it becomes useful. A calm, beginner-friendly guide to the real job. data-engineering data-pipelines beginners databricks https://bricksnotes.com/blog/free-edition-refresh-2026 https://bricksnotes.com/blog/free-edition-refresh-2026 Fri, 10 Jul 2026 03:13:04 GMT BricksNotes Editorial We reviewed every chapter for accuracy, tidied the writing, and updated the steps for how Databricks Free Edition works today, including serverless compute and the newer chapters on pipelines, Apps, and Genie. databricks free-edition update data-engineering https://bricksnotes.com/blog/innovation-data-ai-people-together https://bricksnotes.com/blog/innovation-data-ai-people-together Sun, 05 Jul 2026 23:37:13 GMT Team BricksNotes A retail forecast that failed, a hospital model, a fraud check, and a logistics team. Real stories about why innovation lives where data, AI, and people meet. data-and-ai innovation data-engineering https://bricksnotes.com/blog/databricks-builders-advantage-data-ai https://bricksnotes.com/blog/databricks-builders-advantage-data-ai Thu, 02 Jul 2026 12:55:18 GMT Team BricksNotes The people who lead Databricks are the same people who built the underlying technology. Here is why that matters, and why data and AI is a direction, not a trend. databricks data-and-ai leadership https://bricksnotes.com/blog/databricks-data-engineer-associate-sample-questions https://bricksnotes.com/blog/databricks-data-engineer-associate-sample-questions Wed, 01 Jul 2026 03:43:59 GMT BricksNotes Editorial A set of Databricks Data Engineer Associate sample questions with clear explanations, so you learn how to reason, not just memorize answers. databricks certification data-engineer-associate sample-questions https://bricksnotes.com/blog/databricks-data-engineer-associate-exam-study-plan https://bricksnotes.com/blog/databricks-data-engineer-associate-exam-study-plan Wed, 01 Jul 2026 03:43:59 GMT BricksNotes Editorial A simple, week-by-week study plan for the Databricks Certified Data Engineer Associate exam, built on practice instead of memorization. databricks certification data-engineer-associate study-plan https://bricksnotes.com/blog/databricks-data-engineer-associate-common-mistakes https://bricksnotes.com/blog/databricks-data-engineer-associate-common-mistakes Wed, 01 Jul 2026 03:43:59 GMT BricksNotes Editorial The most common mistakes people make on the Databricks Data Engineer Associate exam, and simple habits that prevent them. databricks certification data-engineer-associate exam-tips https://bricksnotes.com/blog/databricks-customerlake-agentic-cdp-2026 https://bricksnotes.com/blog/databricks-customerlake-agentic-cdp-2026 Sat, 27 Jun 2026 16:20:50 GMT Team BricksNotes Databricks introduced CustomerLake, an agentic CDP built into the lakehouse. Here is what it changes, in plain language, for the people who build data systems. https://bricksnotes.com/blog/bricksnotes-story-helping-data-professionals-worldwide https://bricksnotes.com/blog/bricksnotes-story-helping-data-professionals-worldwide Thu, 25 Jun 2026 18:30:24 GMT Team BricksNotes Why BricksNotes exists, what we have built so far, and a genuine thank you to the learners who make this work worth doing. BricksNotes Data Engineering Community https://bricksnotes.com/blog/reyden-explained-clearly-lakehouse-rt-engine https://bricksnotes.com/blog/reyden-explained-clearly-lakehouse-rt-engine Wed, 24 Jun 2026 13:53:33 GMT Team BricksNotes A lot of people saw the word Reyden in the recent Databricks announcements and asked the same thing: what exactly is Reyden? Here is a calm explanation of the engine, the warehouse it powers, and why this matters for data engineers. databricks lakehouse-rt reyden real-time analytics 2026 https://bricksnotes.com/blog/five-fiefdoms-of-the-data-realm https://bricksnotes.com/blog/five-fiefdoms-of-the-data-realm Tue, 23 Jun 2026 02:30:14 GMT Team BricksNotes OLTP, data engineering, data science, the data warehouse, and real-time analytics each grew up as its own kingdom. Here is a calm map of the five fiefdoms, why their borders existed, and what changes for you when they begin to merge. data platform lakehouse data engineering 2026 https://bricksnotes.com/blog/mukesh-ambani-reliance-databricks-ai-for-everyone-2026 https://bricksnotes.com/blog/mukesh-ambani-reliance-databricks-ai-for-everyone-2026 Sun, 21 Jun 2026 15:42:29 GMT Team BricksNotes At the Databricks Data and AI Summit 2026, Mukesh Ambani called the Reliance-Databricks work one of the world's largest data and AI transformations. Behind the vision sits a quiet lesson every data engineer should hear. databricks reliance ai data engineering 2026 https://bricksnotes.com/blog/databricks-ltap-one-copy-of-data-2026 https://bricksnotes.com/blog/databricks-ltap-one-copy-of-data-2026 Thu, 18 Jun 2026 06:16:13 GMT Team BricksNotes For forty years every data stack split operations from analytics. LTAP and Lakebase quietly end that divide, and it reshapes what data engineers do next. databricks lakebase ltap ai agents lakehouse 2026 https://bricksnotes.com/blog/databricks-summit-2026-keynote-recap https://bricksnotes.com/blog/databricks-summit-2026-keynote-recap Wed, 17 Jun 2026 02:58:26 GMT Team BricksNotes A calm recap of the Databricks Data and AI Summit keynote: why the real story was context, not models, and what the new agents mean for data engineers. databricks summit 2026 ai agents lakehouse 2026 https://bricksnotes.com/blog/databricks-lakebase-postgres-lakehouse-2026 https://bricksnotes.com/blog/databricks-lakebase-postgres-lakehouse-2026 Tue, 16 Jun 2026 13:36:20 GMT Team BricksNotes Databricks took Lakebase to GA, a managed Postgres for real-time apps with branching, autoscaling, and Unity Catalog. A calm look at why it matters for data. databricks lakebase postgres lakehouse 2026 https://bricksnotes.com/blog/the-ai-flywheel-compute-data-loop-2026 https://bricksnotes.com/blog/the-ai-flywheel-compute-data-loop-2026 Mon, 15 Jun 2026 13:05:18 GMT Team BricksNotes More compute and data make AI better, better AI brings more usage, more usage makes more data. A calm look at the loop driving AI, and where data engineers fit. ai data engineering compute vision 2026 https://bricksnotes.com/blog/databricks-omnigent-meta-harness-agents-2026 https://bricksnotes.com/blog/databricks-omnigent-meta-harness-agents-2026 Sun, 14 Jun 2026 16:11:09 GMT Team BricksNotes Databricks Omnigent is a new open source meta-harness for AI agents. A calm look at why coordinating many agents, not building one more, is the real problem. databricks omnigent ai agents open source 2026 https://bricksnotes.com/blog/databricks-opensharing-delta-sharing-evolution-2026 https://bricksnotes.com/blog/databricks-opensharing-delta-sharing-evolution-2026 Thu, 11 Jun 2026 02:06:33 GMT BricksNotes Editorial Databricks open-sourced OpenSharing, the evolution of Delta Sharing. Here is what zero-copy data and AI sharing means for data engineers and enterprises. databricks opensharing delta sharing open standard ai 2026 https://bricksnotes.com/blog/databricks-genie-pay-as-you-go-pricing-budgets-2026 https://bricksnotes.com/blog/databricks-genie-pay-as-you-go-pricing-budgets-2026 Wed, 10 Jun 2026 03:03:14 GMT BrickNotes Editorial On July 6, 2026, Databricks Genie moves to pay-as-you-go pricing. Here is a calm look at the DBU costs, the compute underneath, and why usage budgets matter. databricks genie pricing cost budgets 2026