An architecture of data lake Organizes how a company ingests, stores, transforms, governs, and consumes structured, semi-structured, and unstructured data. Its design should not start with choosing a cloud: it should start with the use cases, sources, quality levels, security, and SLA of the data. The goal is to turn scattered information into reusable assets for analytics and AI.
At MAIA by Scanda We approach this problem from Data Intelligence: integrating multiple sources, managing quality, MDM and governance, and enabling a modern architecture that enables converting existing data into reliable decisions. Our master document precisely states that Data Intelligence should be involved when there are inconsistent sources that hinder AI or BI initiatives.
Taxonomic definitions
1. Data lake
Data repository designed to store large volumes of information of different types and origins, including structured, semi-structured, and unstructured data. Unlike a traditional data warehouse, it can preserve information in a state close to its original form for later transformation according to each use case.
Microsoft identifies data lakes especially with scenarios of Exploratory analytics, data science, and machine learning, This approach, where raw information is stored and schemas are applied during processing, offers flexibility compared to exclusively structured models.
2. Data pipeline
Set of processes that transport information from the source systems to the storage and consumption layers. It can operate in batches (batch), events, CDC (Change Data Capture) or streaming.
An enterprise pipeline should not be limited to “moving files”. It should incorporate, as the case may be:
- traceability of origin;
- schema validation;
- quality controls;
- error handling;
- transformation;
- observability;
- safety rules;
- handling changes in the sources.
3. Data catalog
Layer that registers technical and business metadata to know what data exists, where it is located, where it comes from, who can use it and what it means.
AWS, for example, incorporates a centralized catalog into its reference architecture to decouple producers and consumers, share metadata, and facilitate governance and auditing functions.
Comparison table: layers of a data lake architecture
An implementation can use different names and is not required to follow exactly three layers. However, progressively separating the data according to its level of processing and trust is a widely used pattern. Microsoft describes raw, cleansed, and curated layers; Databricks uses the terms bronze, silver, and gold. Microsoft Learn
| Capa | State of the data | Main processing | Typical consumer | Risk if poorly designed |
| Raw / Bronze | Close to the origin | Ingestion and minimal metadata | Data engineering, auditing | Loss of traceability or preventing reprocessing |
| Cleansed / Silver | Validated and standardized | Quality, deduplication, typification, integration | Analysts, data scientists, engineering | Contradictory KPIs and models fed with inconsistent information |
| Curated / Gold | Ready for business | Rules, aggregations and consumption models | BI, AI, applications, business | Decisions based on ambiguous metrics |
| Catalogue and governance | Transversal | Metadata, lineage, permissions, and policies | IT, data, security, compliance | Exhibitionism, Duplicity and a Lack of Accountability |
Databricks documents that Bronze retains raw information for auditing and reprocessing; Silver performs cleaning, deduplication, validation, and normalization; and Gold prepares information and business-oriented aggregations.
The distinction is important: The layers represent levels of trust and purpose, not just folders with different names..
Technical team
H2: The components that an enterprise data lake architecture should have
A productive data lake needs more than just object storage. In MAIA by Scanda We recommend evaluating at least seven components.
1. Data sources
First, it must be inventoried what produces information and with what characteristics.
Sources can include:
- ERP and CRM;
- business applications;
- SQL and NoSQL databases;
- APIs;
- CSV, Excel, JSON and XML files;
- documents;
- IoT devices;
- telemetry;
- industrial systems;
- logs;
- images, audio and video;
- external sources.
The critical question is not simply “Can we connect?”, But rather:
What business decision will you use this data for and how often do you need to receive it?
A source that updates inventory every five minutes and a monthly financial history do not need the same input pattern.
H2: 2. Intake layer
The ingestion transports the data from the sources to the lake.
Three decisions must be made here.
Batch vs. streaming
Batch It is appropriate when the business tolerates latency of minutes or hours. Streaming It makes sense when the value of the information decreases rapidly over time.
Not everything needs real-time processing.
Converting each pipeline into a streaming one “because we can do it” can increase infrastructure, operation, and monitoring without generating additional return.
The correct decision is expressed as a Data SLA:
- How long can it take from the event happening to it being available?
- What does the business lose if it arrives 5 minutes, 1 hour, or 24 hours later?
- How much does it cost to reduce that latency?
That's where the business case comes in.
H2: 3. Storage and separation by layers
Storage must allow the original data to be preserved and transformed progressively.
Raw / Bronze: preserve before transforming
In this area, data must be kept with minimal alteration, accompanied by metadata such as origin, date of loading, or the identifier of the process.
There is an economic reason in addition to the technical one: if a transformation changes, the original data allows reconstructing the subsequent layers without having to extract all the information from the source systems again..
Databricks specifically points to the ability to reconstruct later layers from Bronze as an advantage of the pattern.
Cleansed / Silver: build reusable trust
Much of the work that determines whether the lake will end up being an asset or a swamp of data takes place here.
It usually includes:
- removal of duplicates;
- treatment of nullities;
- standardization of formats;
- validation of schemes;
- homologation of identifiers;
- quality rules;
- resolution of late registrations;
- integration between sources;
- enrichment;
- managing scheme changes.
Databricks explicitly places these activities in Silver and recommends not writing directly from the ingestion to this layer, precisely to isolate schema issues and corrupt records.
Curated / Gold: publish data products
Gold answers a different question:
What does the business need to consume?
It may contain:
- sales by region;
- margin per client;
- inventory rotation;
- store-by-store behavior × SKU;
- financial indicators;
- features for machine learning;
- datasets for AI agents;
- Dimensional models for BI.
In this layer, a critical metric must be defined and accountable. “Sales,” “active customer” or “margin” seem obvious concepts… until Finance, Commercial and Operations have three different figures in the same board.
H2: 4. Catalog, metadata and lineage
A data architecture is not governed because there is a central repository.
He/She must be able to answer:
- Who owns this dataset?
- From what system does it come?
- When was it updated?
- What transformations did he undergo?
- Which dashboards or models depend on it?
- Does it contain sensitive information?
- Who can consult it?
- What would happen if they changed their scheme?
AWS presents the centralized catalog precisely as a point where consumers can discover and access information from different producers, and where governance controls and auditing can also be applied.
Without a catalog, a data lake stores data. With a catalog, you start managing it as an asset.
H2: 5. Data Quality: Measure before you trust
Quality should not be reviewed solely when the project is being constructed.
It must become an ongoing process.
Recommended metrics
Depending on the dataset:
- completeness;
- uniqueness;
- accuracy;
- consistency;
- validity;
- opportunity or freshness;
- expected volume;
- percentage of records rejected.
For example, a pipeline can technically be “green” because it ended successfully, while at the same time it has received 40% fewer transactions than the historical average.
Available infrastructure does not equate to reliable information.
In the MAIA by Scanda model, AI Data is a transversal component of the CoE, It incorporates a platform governed by Data Quality, Data as a Service, and semantic layer.
H2: 6. Security and governance from design
A common mistake is to build the lake first and “add security” later.
It should happen the other way around.
The architecture must be defined from the beginning:
- identity;
- RBAC;
- minimum privilege;
- classification;
- encryption in transit and rest;
- segregation of environments;
- logs;
- auditing;
- retention;
- processing of sensitive information;
- Policies for data used by AI models.
This takes on even more significance when the data lake will feed generative agents or models.
Our principle at MAIA by Scanda is simple: AI does not automatically correct a poorly governed data; it can amplify its exposure. Our preparation framework includes inventorying and classifying sensitive information, correcting legacy permissions, mapping sources, and validating identities and privileges before enabling new AI cases.
H2: 7. Consumption layer: designing from the decision backwards
The last component determines much of the previous ones.
Consumers can be:
- dashboards;
- data warehouses;
- advanced analytics;
- applications;
- APIs;
- Data scientists;
- predictive models;
- RAG;
- AI agents;
- automations.
AWS includes analytics, query, consumption, and Data Science/AI/ML as distinct functions within a modern data architecture.
That's why we recommend designing from the result to the data, Not the other way around.
For example:
Decision: reduce the inventory of fixed assets
↓
KPI: Rotation and inventory days per SKU
↓
Product of data: inventory + sales + orders + catalog
↓
Transformations: SKU/store approval, quality, and business rules
↓
Sources: ERP + POS + e-commerce + WMS.
The data lake then ceases to be an “IT infrastructure” and acquires an explicit relationship with working capital and EBITDA.
H2: The 6 design decisions that must be taken before construction begins
1. Centralizing everything vs. prioritizing domains
It is not necessary to migrate every piece of data from the organization before producing value.
It is advisable to start with the sources necessary for a specific case and expand them later.
At MAIA by Scanda we work with a similar logic: First we demonstrate value and then justify a larger investment. The master document states that the business case must be built using real customer data and that the PoV acts as a gateway before scaling up.
2. ETL vs. ELT
ETL transforms it before uploading it to the destination repository.
ELT load first and take advantage of the platform's processing capacity to transform it later.
In modern lake/lakehouse architectures, ELT can offer greater flexibility because it preserves raw information and allows for the production of different subsequent representations.
However, the choice must consider privacy, volume, regulation, and processing cost.
3. Schema-on-read vs. schema-on-write
The data lake allows for delaying some of the modeling until consumption, but that doesn’t mean “saving anything without structure.”.
Architecture must decide At what point does flexibility become a data contract?.
A reasonable pattern is:
Flexibility in Raw → control in Silver → business semantics in Gold.
4. Cloud, hybrid or on-premise
The choice depends on:
- sovereignty;
- regulation;
- latency;
- connectivity;
- volume;
- transfer cost;
- security;
- existing infrastructure.
In our operating model we can start Cloud-first To accelerate the first outcome and maintain an on-premise route when sovereignty or security require it; the master document envisages keeping sensitive information local and exposing only aggregated versions when the case allows.
5. Proprietary vs. open architecture
A long-term decision is how much a platform depends on the provider's exclusive services.
Open formats, decoupling between storage and computing, and replaceable components can reduce lock-in, although they can also increase operational complexity.
There is no universal answer.
The executive question is:
How much would it cost to move this data or workload within three years?
6. Retaining everything vs. managing the life cycle
“Cheap storage” does not mean “free storage”.
Each dataset must have policies regarding:
- retention;
- archived;
- elimination;
- frequency of access;
- tiering;
- replication;
- support.
Indefinitely storing information without a consumer or regulatory requirement turns the growth of the lake into growth of OPEX.
H2: Reference architecture of a data lake
A conceptual architecture can be visualized as follows:
Source systems
ERP · CRM · POS · IoT · APIs · files · documents · logs
↓
Ingestion
Batch · CDC · Streaming · APIs
↓
Raw / Bronze
Original data + metadata + history
↓
Cleansed / Silver
Quality · deduplication · standardization · MDM · integration
↓
Curated / Gold
KPIs · business models · features · data products
↓
Consumption
BI · Analytics · ML · GenAI · Agents · Applications · Automation
And crossing all the architecture:
Catalog + lineage + identity + security + quality + observability + governance.
The reference architecture matters less because of the names of the boxes than because of the contracts between them: Owner, scheme, quality, latency, security, and consumer.
H2: How to know if the data lake is generating value
We do not recommend evaluating a data lake based on terabytes of stored data.
That measures occupancy, not results.
In MAIA by Scanda we use metrics closer to the actual decision-making capacity. Our master document proposes, among other things:, percentage of sources integrated into the master catalog, time from a data point to a decision, cost per use case in production, and business outcome attributed to AI.
An executive board could include:
| Metric | What it reveals | Business impact |
| % from integrated critical sources | Real coverage | Less manual reconciliation |
| % quality rules met | Trust | Lower risk of incorrect decisions |
| Freshness / latency | Speed | Given time → decision |
| Quality incidents | Stability | Less rework |
| Cost per pipeline/domain | Efficiency | OPEX control |
| Time to incorporate a new source | Scalability | Time-to-value |
| Data products with owner | Government | Accountability |
| AI cases in production on the platform | Utilization | Return on investment |
A particularly useful signal is the number of versions of the same figure. If each area continues to use its own Excel and the advice receives different numbers, the organization still does not have a truly operational source, even though it technically owns a data lake. This same signal is part of the MAIA diagnostic dashboard.
H2: What mistakes turn a data lake into a “data swamp”?
The main ones are usually not for storage.
They are of design and operation:
- uploading information without a use case;
- not maintain lineage;
- not assign owners;
- allow each area to re-clean the same data;
- creating pipelines without observability;
- duplicating datasets without contracts;
- mixing raw and certified information;
- apply permissions only at the infrastructure level;
- ignoring the cost of computing and transferring;
- not defining when a data can feed an AI model.
The layer pattern attempts to avoid exactly this: the quality and structure must increase progressively as the data progresses towards consumption.
In other words: The problem is not that the lake is growing; the problem is that it is growing faster than our ability to understand and govern it.
Predictive Conclusion
During the next modernization cycles, the difference will not simply be between companies that “have a data lake” and companies that don’t. It will be between organizations capable of converting raw data into governed and reusable products and those that continue to build integrations, definitions, and quality rules for each new project.
The pressure will increase with generative AI and agents: each new case will require identifiable, accessible, reliable, governed, and sufficiently up-to-date information.
Therefore, in MAIA by Scanda We do not view data architecture as a project that precedes AI and then ends and is archived. We view it as an operational capability.
Our approach Data Intelligence It integrates information management, MDM, Data Governance, quality, multi-source integration, analytics, Data as a Service, and hyperautomation. The goal is not to accumulate data: it is to enable reliable decisions and an architecture capable of supporting the following cases of analytics, automation, and AI.
CTA final of MAIA
Your data does not need another repository. They need an architecture that allows decision-making.
If your organization today manually integrates information from several systems, maintains different versions of the same number, or wants to implement AI without knowing whether its data is ready, the first step should not be buying another platform.
At MAIA by Scanda We can evaluate your sources, quality, integration, and governance to identify what architecture you need and what use case justifies building it. Our approach starts from the business outcome and builds the business case with Your own data, Before investing, you need to assess the investment.
Let's talk about your data architecture and define what you need to enable first to bring analytics and AI to production.
FAQ
1. Do I need to have all my data cleaned up before building a data lake?
No. Precisely, a layered architecture allows you to preserve raw information and apply quality progressively. The important thing is to identify which sources are needed first for the business case, define quality rules, and prevent unvalidated information from reaching consumers who expect certified data.
2. Does a data lake reduce costs compared to a data warehouse?
Not automatically. You can offer flexible storage and separate storage, processing, and consumption, but the TCO depends on ingestion, computation, transfer, retention, tools, governance, and operation. For a CFO, the right comparison is not just cost per TB: it must include Cost per data product and value generated by its use cases.
3. How do we know if our company needs a data lake?
This is a particularly relevant option when there are multiple data sources and types, you need to maintain detailed historical data, or you are planning advanced analytics workloads, machine learning, or AI. Microsoft recommends this pattern particularly for exploratory analytics, data science, and ML.
At MAIA by Scanda, a clear commercial sign is to find Multiple inconsistent sources that are slowing down BI or AI; That is precisely one of the defined entry scenarios for Data Intelligence.
Español 
English