Does the data platform implementation framework matter?
Data management technology has become one of the critical areas where tech companies invest in innovations to provide better ways to ingest and organise data to provide greater flexibility to consume data within the enterprise. The tech is superior and robustly built; however, it would become an enormous dormant technological piece with expensive overhead without proper implementation.
A well-sorted data analytics implementation framework would help modernise a data platform supporting the organisation’s analytical, operational, and research needs. A data analytics implementation framework typically includes a user-driven definition of the processes and artefacts like a data model, a data dictionary, a data architecture, and a data analytics process — a framework designed to define the artefacts dynamically based on the enterprise entities.
Purpose
- Faster time to market solution by building pre-defined Templates
- Automate build and deploy for common ELT patterns
- Improve the quality of data
- Helps to focus more on business solutions
- Implement process Standardisation and Governance
- Repeatable processes
Beyond Data Movement Guff
Data projects are not just about extraction, cleansing and moving data from point A to point B system, but we must also consider the other elementary building blocks for longer sustainability and interoperability. However, before determining the technology or architecture, one must define organisational data vision from the operational preview and the analytical plane.
Context
- Defining business vision and domain
- Reference data management
- Security policies
- Data volume sizing — Storage and Compute
- Data model and measure services
- Tools that can support interoperability
Interoperability
- ELT patterns
- Master data management
- Data aggregation
- Data consumption
- Data movement
Reliability
- Data integrity
- Data Governance
- Metadata management
- Data Quality
Operationalisation
- Job Scheduling
- Logging and monitoring framework
- Error handling and notification
- CI/CD process
Data Artefact — Key to building blocks for any data transformation
A data artefact plays a vital role in creating, maintaining or evolving the current ecosystem. Though generating artefacts look overhead from a project perspective, these tools are worthy of any data transformation with proper document and index management.
Some of the following data artefacts in my experience has been very useful.
- Functional outcome architecture
- Enterprise data architecture
- Data mapping processes
- Enterprise Data model
- Data Extraction processes
- Data quality processes
- Data transformation processes
- Post-load processing processes — including late change data capture or real-time ingestion
- Metadata management processes
- Logging and auditing processes
- Data Ops process
- Security patterns
Data artefacts play a significant role in structuring and governing the data definition and utilisation as the industry and technology evolve. They provide accelerators to grow the ongoing data needs or integration pattern and provide a single point of truth for an ever-changing paradigm.
