Compare Copy Activity, Copy Job, Notebook and Dataflow Gen2 for Bronze ingestion in Microsoft Fabric using the same large CSV source file.

A practical look at what belongs in Bronze, Silver and Gold, from preserving source data to building reusable datasets and shared business metrics.

Learn how caching materialises Spark DataFrames to avoid repeating source scans, joins and calculations, and when the performance trade-off is worthwhile.

Missing log backups, a full disk, and an unexpected opportunity. How fixing a SQL Server outage helped turn an app guy into a DBA.

I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark. At first glance the difference is stark, but it&…

I’ve had Fabric gnawing at me for too long. Whilst I’ve dabbled in a couple of areas, I haven’t taken the time to explore the breadth of what Microsoft’s flagship data platf…

Date handling in SQL Server tends to accumulate tried and trusted combinations of DATEPART(), DATEADD(), and DATEDIFF() – with nested variations. The challenge with these isn’t raw perf…

For August’s T-SQL Tuesday ask, Jeff has thrown temp tables to the wolves. Does the community like or loathe their usage? I’ll predict a few ‘it depends’ resp…

Computed columns allow us to bake logic into our schema. I find particular use for these when reviewing or optimising established solutions where you spot the quirky ways the data is being used. He…

Fragmented data estates usually share a trait – there’s no consistent representation of core business entities. It’s a common thread I’ve seen – system-centric entitie…

When developing Power BI models, we don’t always want to refresh the entire dataset before we start working with the data. We can speed up the refresh – and therefore our deve…

Schema drift is an inevitable part of environments where database changes are applied manually. Sometimes it’s dev-ing in production, other times it’s lax change control. Either way you…

For the bicentennial edition of T-SQL Tuesday, Brent has thrown down the topic of “When I’m looking at a query, I bet it’s bad if I see ____.”. I’m expecting the Brits out in forc…

From walking the dog before breakfast to getting off the bus a stop early, readers share their tips for building a daily walking habit.

As a database developer, when I started using Power BI, I was concerned about it retrieving reams of data only to perform transformations downstream. The Power Query editor misleads us into thinkin…

If you’re building a large INSERT statement using VALUES, you might find yourself hitting the limit of 1000 rows, for example: INSERT INTO #Numbers (Number, InWords) VALUES (1,…

Dealing with endless formats for SQL scripts is a special type of pain you get to enjoy as a data developer. With the arrival of SSMS v22.7 last week, we have a solution in the form of a T-SQL form…

This month the invitation from Koen asks what would happen if we came back from the cloud. Back to On-Prem. Cloud solutions have sliders which magically reduce downtime, performance issues, and acc…

Someone drops a KQL query in your lap and says “you know SQL right?”. Honestly, it’s different enough to trip you up, but similar enough that you’ll pick it up quickly. In t…

“AI will take our jobs” scares folks away. This past weekend I took a photo of a plant that’s in a bad way, and asked ChatGPT for support. I got a full rundown of positioning, car…

Large data solutions can span multiple databases. I’ve recently shared how amazing Database Projects can be but objects which reference across database boundaries can throw up warnings. Is th…

Updates, improvements, and bug fixes for the current version of SQL Server Management Studio (SSMS).

Being a Data Engineer, this month’s invitation from Meagan resonates. She’s asked us to share a tip, technique, or lesson about detecting changes – which can be crucial to reduce …

A regular Database Project deployment is static and delivers consistent results regardless of environment. When it comes to schema, that’s usually desired, but data is a different s…

Last month in SSMS v22.4, we had the Database DevOps (preview) workload introduced which brings with it Database Projects. Last week I shared my favourite features for them which make deployments a…

Buckinghamshire Council says it is a

If you’ve been using Database Projects for simply maintaining copies of your database objects – like I used to – then you’ve been missing out on the power of its deployments…

Failover Groups for Managed Instances are a great option to replicate data, but they don’t replicate key instance elements – one of which is logins that live in the master database. If …

Having TOP (1) return multiple rows feels wrong… but that’s what WITH TIES can do. For a long time I used patterns like this to get the first record in a group: WITH…

Most SQL Server environments have more jobs, schedules, and hidden complexities than you realise. It’s only when you arrive at a migration and peek under the hood that the scale is clear. Her…

Last week I demonstrated a fast binary search approach to quickly slice through large unindexed tables. I love the approach, but it fell short in two key areas – drop-in usage, and proper bou…

I know, clickbait right? Hear me out. Searching vast log or audit tables without indexes is painful. Narrowing down a specific time range often means scanning millions or billions of rows. But that…

Last week we looked at utilising Teams to make notifications more collaborative. This week I want to show how we can add more visual and interactive elements by adopting Adaptive Cards. What are Ad…

Whether running Data Factory, Synapse, or Fabric pipelines, things go wrong – and the de facto response is to send an email. We’ve looked at sending emails from pipelines before, but at…

SSIS packages provide great flexibility for integration between systems, but when they go wrong you can end up digging through logs or reports because every package logs differently. A standarised …

Last week, Database Instructions landed for GitHub Copilot as part of the SSMS v22.3 release. Database Instructions piggyback on extended properties to give Copilot specific context about database …

This month, Pat asks us to consider how our code has aged. Like fine wine, or old milk? A common time to revisit old code is when something breaks. I was contemplating Pat’s question this wee…

If you’ve tried to get header rows to repeat or scroll as you move through your paginated reports, you may have felt the frustration of getting them working correctly. Whether it’s Repo…

Part of the joy of new experiences is making mistakes and learning from them. For this month’s invitation, Louis Davidson asks us to share one of those mistakes to help others learn. The less…

When using Power BI or Fabric workspaces to browse reports, we’re greeted with a list of items and their attributes. While attention is given to report visuals such as bars, pies, candles, an…

It’s not the usual time of year to think about Halloween, but listening to Erik Darling and Kendra Little recently on the Dear SQL DBA podcast they mentioned a completely new topic to me: Hal…