Demo Fabric - Togaether

In this project, my fellow student Nathan Neve (website) and I collaborated to develop a demo for Togaether using the latest Microsoft Fabric tool. The goal was to create an advanced data pipeline that allows Togaether to demonstrate to their clients how data can be efficiently processed and visualized.

Functionalities

For this project, we worked with the Adventure Works dataset. The key functionalities and components of the pipeline include:

• A master pipeline with various subpipelines to structure the data processing.
• Use of Apache Spark notebooks to clean, enrich, and store data in delta tables.
• Implementation of slowly changing dimensions (SCD) type 1 to accurately track data changes.
• Preparation of the data for seamless integration with Power BI, enabling the creation of visualizations.

Project Execution

We started by taking a comprehensive course on Microsoft Fabric to gain a solid understanding of the new tool and learn how to set up an advanced pipeline. Next, we gathered the company’s requirements to ensure our solution fully aligned with their needs.

During development, we adopted an iterative approach. We began by ingesting the data, followed by step-by-step setup of the data processing. Processing the data according to SCD type 1 was a critical part of this process. Unfortunately, we didn’t have enough time to fully visualize the data, but the established structure makes this easily achievable in the future.

Conclusion

Nathan and I are very satisfied with the results of this project. Although we weren’t able to fully visualize the data within the given time, we built a robust and scalable pipeline. This project provided us with valuable experience with Microsoft Fabric and gave us insight into how advanced data processing can be applied in a business environment. The result is a demonstration that Togaether can use to showcase the power of this technology to their clients.