iiinova — where your digital self keeps evolving

Bijil Subhash @bijilsubhash.bsky.social · 08/03/2025

Databricks vs Fabric feels a lot like Pied Piper vs Nucleus. Fans of the Silicon Valley show will get the reference :) #databs #databricks #fabric

060

Bijil Subhash @bijilsubhash.bsky.social · 18/02/2025

(3/3) As a data engineer, I could have spent my time learning one of the shiny tools out there or could have relied on co-pilot to write code. I chose to learn what was happening under the hood in Python, mainly in the interest of building a stronger foundation, which should be a priority.

050

Bijil Subhash @bijilsubhash.bsky.social · 18/02/2025

(2/3) I am no better and I am guilty of this myself, mainly from my own ignorance to writing Python idiomatically. This is why I spent the last 6 months re-learning Python, consuming multiple books, attending advanced lessons on specific topics, and wrote a #data ingestion package using Python.

120

Bijil Subhash @bijilsubhash.bsky.social · 18/02/2025

(1/3) Among programming languages, I consider #Python to be a relatively easy to learn language, opening doors for many to start coding without formal training. However this also results in some poorly written, unmaintainable, and non-extensible code; the infamous spaghetti code.

110

Bijil Subhash @bijilsubhash.bsky.social · 01/02/2025

Just finished watching the webinar on introducing SDF by dbt team. After seeing SDF in action, I have to admit that I am really looking forward to the future of dbt engine. I was wondering when dbt was going to bring in notable changes to the developer experience and this might be it. #databs

031

Bijil Subhash @bijilsubhash.bsky.social · 21/01/2025

I was speaking with someone who went all in on promoting duckdb to their clients. I did not get a chance to ask what exactly are they doing with duckdb. But I am curious to understand how duckdb is utilised in modern data pipelines.

010

Bijil Subhash @bijilsubhash.bsky.social · 18/01/2025

(4/4) It is crazy to think that this news came out just 2 weeks into the new year. I cannot wait to see what dbt and others have in store for the next 12 months. I am excited to see how the integration between sdf and dbt will be rolled out. Maybe we will have some updates at Coalesce 2025!!!

000

Bijil Subhash @bijilsubhash.bsky.social · 18/01/2025

(3/4) I have always found the incredibly slow compile times with dbt deeply frustrating, which is of no concern to business users. But considering the early adoption of dbt is strongly rooted in the technical community, the acquisition of SDF to improve the developer experience is well timed.

100

Bijil Subhash @bijilsubhash.bsky.social · 18/01/2025

(2/4) I have used dbt quite extensively, and do enjoy its utility when it comes to data transformations and acknowledge that it is not going anywhere. However, I do feel that dbt core have not had any significant upgrade in a while, especially when it comes to improving its developer experience.

100

Bijil Subhash @bijilsubhash.bsky.social · 18/01/2025

(1/4) SDF acquisition by dbt If you work in data, you probably would have come across a version of this headline this past week. A small disclaimer, I have not used SDF and neither do I have solid understanding of the tech that sits behind it, so take what I say with a grain of salt.

100

Bijil Subhash @bijilsubhash.bsky.social · 16/01/2025

(3/3) I am sure this will change as we iterate through the next generation of language models. However, if you are someone that is just starting out in the world of #data, I recommend you to reduce your reliance on code assist and instead spend some time understanding the basics of how SQL works.

100

Bijil Subhash @bijilsubhash.bsky.social · 16/01/2025

(2/2) But if you had to work with complex #SQL statements such as first touch attribution, finding the streak, calculating conversions etc with highly specific business logic and questionable quality of data, you probably will understand the sentiment that LLMs we have today are not powerful enough.

100

Bijil Subhash @bijilsubhash.bsky.social · 16/01/2025

(1/2) Maybe an unpopular opinion, SQL is a powerful language and despite what anyone says, it is unlikely to be replaced by an LLM, at least not with the models we have today. LLMs are powerful and can be leveraged to generate ideas or as a tool to unblock when you are stuck. #databs

120

Bijil Subhash @bijilsubhash.bsky.social · 16/01/2025

The thing that allows you to link data from the physical world in a format that a machine can understand coherently.

010

Bijil Subhash @bijilsubhash.bsky.social · 14/01/2025

SDF was on my list of things to try out. I'll just wait till they integrate it to dbt now I guess :)

110

Bijil Subhash @bijilsubhash.bsky.social · 14/01/2025

(3/3) Laktory also supports managing ETL pipelines, much like how you would do with dbt but with Spark and/or SQL. What I really like about Laktory is its ability to modularize the Databricks assets, which is big win when it comes to long term maintainability of your #data platform.

000

Bijil Subhash @bijilsubhash.bsky.social · 14/01/2025

(2/3) I am not affiliated with Laktory but if you are someone who works with Databricks and want to take a break from having to wrestle with Terraform/Pulumi, check out Laktory. It is an absolute game changer and I was able to go zero to managing multiple workspaces with a couple of yaml files.

100

Bijil Subhash @bijilsubhash.bsky.social · 14/01/2025

(1/3) Continuing from my previous thread on infrastructure as code for managing #Databricks. I have recently had the pleasure to work with an open source tool called Laktory, which is an abstraction that sits on top of Terraform/Pulumi to manage your Databricks workflow using YAML. #databs

121

Bijil Subhash @bijilsubhash.bsky.social · 11/01/2025

A default approach that I take when it comes to data modelling. It works because OBT is optimized for the modern vectorized data warehouses. At the same time, the underlying data is modelled using established best practices from Kimball.

010

Bijil Subhash @bijilsubhash.bsky.social · 11/01/2025

This is the approach I default to. Also, I always thought that this was the only way OBT was used. I guess not.

220

Bijil Subhash @bijilsubhash.bsky.social · 11/01/2025

To an extent, I agree. Though I still prefer to use documentation alongside AI assistance to verify some of the output.

010

Bijil Subhash @bijilsubhash.bsky.social · 11/01/2025

I doubt there is one course that covers them all. You can always pick what part of DE you want to learn and drill down on that first. So that would be for things like ingestion, then transformation,, orchestration and so on

000

Bijil Subhash @bijilsubhash.bsky.social · 10/01/2025

Senior data engineer!! That's tricky almost all of the content out there is generally catered towards early stage DEs. But I have heard Zach Wilson's boot camp is pretty good for seniors. Disclaimer - I have not attended the boot camp, so take my advice with a grain of salt.

100

Bijil Subhash @bijilsubhash.bsky.social · 10/01/2025

(2/2) I am no where near mastering it but I am happy that I was able to use it recently to build out a data platform on #Databricks and it was not as challenging as I imagined. Key takeaway, being comfortable with IaC can dramatically improve the efficiency and reliability of your #data pipelines.

020

Bijil Subhash @bijilsubhash.bsky.social · 10/01/2025

(1/2) Infrastructure as code (IaC) is ubiquitous in the data space. That being said, I have stayed away from doing any IaC work for as long as I can remember, mainly due to its aura of being difficult and also because I could pass the ball to the platform team. #datasky #databs

220

Bijil Subhash @bijilsubhash.bsky.social · 07/01/2025

(4/4) Some of the #saas products are amazing. In fact, I continue to use them where relevant. However, it is important that you do your due diligence when it comes to picking your tech stack so that you do not miss out on something better. Your future self will be grateful!

000

Bijil Subhash @bijilsubhash.bsky.social · 07/01/2025

(3/4) One of the key benefit of using dlthub is that you can run it anywhere with a #python runtime, i.e. airflow, dagster, serverless etc. There is also the added flexibility to bring unique functionality that are specific to your data into the ingestion framework.

110

Bijil Subhash @bijilsubhash.bsky.social · 07/01/2025

(2/4) This last year, I came across a python library called #dlthub, which I use regularly when it comes to data ingestion. What I like about dlthub is how it seamlessly integrate with widely used sources and destinations, and allows you to build production ready data pipelines in just a few lines.

100

Bijil Subhash @bijilsubhash.bsky.social · 07/01/2025

(1/4) What do you use for #data ingestion? Its true that there are no shortage of tools when it comes to data ingestion. But before you open the wallet to one of the many options out there, it might be worth doing a thorough due diligence based on your current and future needs. #databs

100

Bijil Subhash @bijilsubhash.bsky.social · 04/01/2025

I have been binging on the early chapters of the new book from @joereis.bsky.social on data modeling. I haven't consumed a lot of material on this topic besides Kimball but this one is a must read if you work with data in a modern context. Looking forward to the official release in 2025! #databs

170

Bijil Subhash @bijilsubhash.bsky.social · 02/01/2025

(5/5) Although I still use dbt core coupled with third party orchestrators for dbt projects. I would highly recommend dbt cloud for teams with a mix of technical and non-technicals. After all, the ethos behind dbt is to enhance collaboration between teams, and dbt cloud is essentially built for it.

000

Bijil Subhash @bijilsubhash.bsky.social · 02/01/2025

(4/5) That being said, there is a cost associated with running dbt cloud. But if goal is to put the dbt project to good use in a timely manner, dbt cloud is a viable option. Ultimately, the choice depends on data team's technical maturity, appetite, and business objectives.

100

Bijil Subhash @bijilsubhash.bsky.social · 02/01/2025

(3/5) However, I have recently put my biases aside and tried dbt cloud. I do not know how dbt cloud was like 2 years ago, but I am sold. I no longer need to wrestle with some third party orchestrator or K8s. There are also some nice to haves for governance, lineage, and data mesh implementations.

100

Bijil Subhash @bijilsubhash.bsky.social · 02/01/2025

(2/5) I have been an avid dbt core user for 2+ years now, and was fortunate to have worked with teams of varying size of dbt projects As someone who runs the other way at the thought of having to use click ops, I could not think of anything worse than having to use dbt cloud.

100

Bijil Subhash @bijilsubhash.bsky.social · 02/01/2025

(1/5) dbt core vs dbt cloud dbt has been a game changer to many #data teams, mainly for writing reusable and version controlled transformation logic. We are also witnessing an explosion of tools that wants to become the next #dbt. What is better, dbt core or dbt cloud? #databs

110

Bijil Subhash @bijilsubhash.bsky.social · 01/01/2025

That was a good read!! Learnt a thing or two, especially some of the concepts around linking best practices in data modelling to a mathematical construct.

110

Reposted by Bijil Subhash

Andy Pavlo @andypavlo.bsky.social · 01/01/2025

Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...

cs.cmu.edu

Databases in 2024: A Year in Review

Andy rises from the ashes of his dead startup and discusses what happened in 2024 in the database game.

1019963

Bijil Subhash @bijilsubhash.bsky.social · 01/01/2025

Depends, if it is something new, I like to go to the source. If it is something that has been around for a while, LLM seems to be a good option. This also depends on the context. I am referring to it in the context of learning a new tech.

010

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

(5/5) UC is enabled by default on all new Databricks workspaces, and most of the new features are almost exclusively designed with UC in mind. As such, teams working with Databricks should have a good handle on how to get the most out of UC while staying agile to adopt as new features are released.

000

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

(4/5) Considering how rapidly Databricks ecosystem is evolving, learning curve on UC implementation can be steep. Additionally, given the adoption of UC is still at its infancy, it is not uncommon to witness mistakes on how UC is enabled and/or see friction on its overall adoption rate.

100

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

(3/5) You can use UI, CLI, or API to manage permissions in UC. Though it is possible to manage the permissions with SQL, my experience with granting and revoking permissions have been through Terraform or similar, which has the added advantage of defining the permissions model declaratively.

100

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

(2/5) The most common use of UC is to apply universal permission model across all your data products. For instance, with UC, you can manage access to tables, views, notebooks, ML models, and more. The permission grouping also allows you to mask sensitive and PII data centrally.

100

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

(1/5) What is Unity Catalog (UC) in the context of #databricks? The standard definition that you get is that it is a unified governance solution built into Databricks. It is accurate but that was not intuitive to me when I started building on UC. See 🧵 for some additional context on UC. #databs

100

Bijil Subhash @bijilsubhash.bsky.social · 31/12/2024

Check out dlthub! It's essentially sits on top of requests amongst other things and supports duckdb as one of the destination

110

Bijil Subhash @bijilsubhash.bsky.social · 28/12/2024

(2/2) Under the hood, autoloader fully manages the file metadata in a scalable key-value store, providing both fault tolerance and exactly once writes out of the box. Furthermore, it also supports schema inference and schema evolution across a number of widely used file formats.

000

Bijil Subhash @bijilsubhash.bsky.social · 28/12/2024

(1/2) Autoloader is without doubt one of my favourite feature in Databricks! In a nutshell, it is an abstraction that simplifies the incremental ingestion of data by monitoring the files that arrive in the cloud storage, supporting reliable and resilient data pipelines cost efficiently. #databs

120

Bijil Subhash @bijilsubhash.bsky.social · 27/12/2024

Completely agree!! You can in fact run your dbt transformations on the fly as part of your dlt pipelines if you are using dlthub for ingestion. I would imagine it would only be a matter of time before we start to see these segregations disappear.

010

Bijil Subhash @bijilsubhash.bsky.social · 27/12/2024

As an alternative, I would also suggest looking into dlthub if you do not mind Python. It genuinely gives you more flexibility and control on your ingestion.

000