With the challenge presented, our team found the solution through the structuring of a Data Lakehouse.
The solution proposed by our team was the structuring of a Data Lakehouse, which combines concepts and techniques from both data warehouse and data lake.
The Data Lakehouse is a repository of company data accessible to users who need to query and analyze them. This type of project is based on three pillars: reliability, performance, and engineering.
For this project, governance control over data access was necessary, considering all guidelines of the General Data Protection Law (LGPD) regarding sensitive information of the company and its customers.
The data went through a cleansing and qualification process through batch (scheduled) and stream (real-time) workloads. This allows for consistent, optimized, and user-friendly visualizations.