Blog

In the fall of 2023, Microsoft brought to general availability a new and exciting unified analytics solution called Fabric. The goal is to bring your data into one place with a suite of experiences that work seamlessly together. The introduction of Fabric brought about the first annual Fabric Conference (FABCON) 2024. To continue with the learning culture at Omnitech, we sent two Engineers to attend the conference and take their learnings back home. This allows us to stay ahead of technology and ensure we find the right tools and solutions for our clients.

FABCON 2024 was attended by over 4,000 people from around the world who work in various roles and industries. Microsoft pulled out all the stops by bringing in many of its Principal Program Managers, such as Bob Ward – a SQL guru, Adam Saxton – Guy in a Cube, and others, to speak about this new technology. The conference provided a wide range of sessions because of Fabric's broad toolset. With that, the people attending ranged from Data Scientists to Data Engineers, Program Managers, and many more. This shows the impact Fabric can have on businesses, and if it all works together as promised, it can help accelerate how we work with data.

Unlike other technologies in the past, Microsoft has promised very aggressive timeframes when it comes to releasing features within Fabric, and from what we saw at the conference, they are living up to their promise. Since the November 2023 release, Microsoft has introduced more features than we can cover in this blog, but you can check them all out here: https://learn.microsoft.com/en-us/fabric/get-started/whats-new.

With everything that Fabric has to offer, it now comes to us to figure out how to best support our clients in deciding if this is the right tool to manage their data. Microsoft's main message is that Fabric brings new and existing components into a single, integrated environment. They include these components in what they consider personas, meaning that you will choose the type of role you are portraying in the system, and the tools will support your purpose. An example would be choosing to work as a data scientist and giving you insights into features such as Power BI, AutoML, and blob storage, all of which are included as SaaS (Software as a Service).

All of this would not be possible without the foundation of Fabric which they are calling OneLake. This brings all of the services, storage, and analytics into a unified location, and it’s where all the experiences will operate. There is so much more to this OneLake than just the Data Lake we’ve known from before; it’s really a way to simplify how we share, manage, and consume data.

A key takeaway from FABCON is that they are trying to make it easier than ever to bring data into OneLake. One way of doing this is to create shortcuts to external data through tools such as a Gateway. This could be a Gateway sitting on a VM, or even on a VPC within Azure. These shortcuts allow the user to keep the data on their existing platforms and access it across all of Fabric. An even more exciting approach to getting data from other systems comes from what they call mirroring. Mirroring is basically having replication setup to sync up data from a source system to the OneLake. What is exciting about this feature is they are providing it at no cost for the storage and importing of the data, which really takes the “E” out of the ETL processing we’ve all done in the past.

As we mentioned above, there are a multitude of features either coming our way or that have already been released as part of Fabric, but the highlights include:

• Fast Copy – they have rethought how data is dealt with in the data lake environments and provided a way to move data at rates 30 times faster than before.
• T-SQL in Notebooks – Is the SQL world your home base? Now you can write T-SQL code in Notebooks like you would write on databases.
• Source control and sharing – workspaces can be linked to source control, including GIT.
• Schemas in Data Lakes – this allows developers to organize their data just like they would in a relational database.
• Semantic Link – Are you used to working in Power BI? Now, you can connect directly to the existing Power BI semantic models in tools like Python.
• Auto tuning within Spark – gives the ability to automate hyperparameter tuning in models, utilizing the Fast Library for Automated Machine Learning & Tuning (FLAML).

So, with all these new technologies we need to consider, we wouldn’t be doing our job if we didn’t talk about the massive elephant in the room: AI. Every conversation, whether it was in a session or with a vendor, revolved around AI and what it will mean for us. Every aspect of Fabric is built with AI in mind using CoPilot and AI Assistants. This could include helping with analysis, generating faster code, or establishing better Data Governance through Purview. We need to stop thinking of just putting AI on top of our existing data and start thinking about how we can structure our data to support the functionality AI can provide.

Data is the foundation of AI. This theme was echoed throughout much of FABCON. Without enough good data, AI cannot produce useful insights, predictions, or business value. Because of this, Fabric is well suited to be a solid foundation for AI. With data and analytics being stored in one unified platform, Fabric offers the capability to leverage AI on the data to provide advanced analytics solutions.

Fabric also includes the ability to create and experiment with machine learning (ML) models. Experiments allow the evaluation of different model configurations within Fabric notebooks and comparison of the results. Once a configuration that provides adequate results has been found, it can be saved as a model to be used in Fabric notebooks. Additionally, FLAML helps with the automation of this process within notebooks so engineers can spend less time tuning hyperparameters.

The ability of Fabric notebooks to access data across Fabric (e.g. from semantic models) allows engineers to use advanced analytics on a plethora of data more easily. Microsoft’s services, including Azure OpenAI Service and Azure AI Studio (both currently in public preview), offer the ability to deploy advanced large language models. Fabric notebooks can use those models through their API endpoints, in addition to any models developed within Fabric, to provide advanced analytics on data stored across Fabric and write the results back to OneLake.

It is both encouraging and exciting to see the investment Microsoft is making into the future of data. Data is driving more decisions than ever, and data analytics, as a field and industry, continues to change rapidly. Fabric is an analytics solution designed to support data-driven decisions and insights through a unified platform. With each new feature or set of features released, Fabric will continue to evolve. The first annual FABCON provided some insight into this evolution, and we look forward to continuing to learn more with each additional announcement.