Data Discovery Process

At Apto we break down data using our data discovery process, this means using our data lifecycle to understand and develop organisational understanding of the given data landscape. For us the end product would be an operational specification bespoke to the customer but from this blog you can look to take away a top-level understanding of the facets of your data. By this we mean looking at data through: data sources, collection & ingestion, data processing, data storage, data consumption and data governance. 

Data Sources

When we look at data sources, we are aiming to understand at a source level the operational goals tied to using the given data set – understanding how this data can be valuable and impactful. From here we gauge the resilience of the data source, and the infrastructure needed to support it in a telemetry context, along with the infrastructure supporting the source itself. The next steps follow the development of key reporting, use case and service definitions – these artifacts are invaluable when data charts the rest of the lifecycle, they ensure its usable, meaningful and relevant. 

Data Collection

Data collection and ingestion is all about understanding the system architecture in place and the structure of the data – does it or should it conform to a pre-defined schema for example. Data processing ties neatly into this, consider the next steps from collection. How data within your organisation should be modeled, parsed, normalised and enriched regardless of source form. 

Data Storage

When we move though the lifecycle to data storage, we start to look at more tangible concepts that may be more familiar if you haven’t looked at MELT before. In this context we are looking at accessibility, durability and cost. Working though discovery from this point is looking at how the data is consumed by users: monitoring, analytics and visualisations. Effective data utilisation is not just a technical affair, to be valuable it must be consumed by the correct parties and represented accessibly. 

Data Lifecycle

The final stage of discovery for us ends with our data lifecycle, data governance – the frameworks and rulings that should be applied to any data sets within your business. In this section we look at retention, frameworks, taxonomies, KPIs and SLAs, and accessibility. Is data stored against its given parameters correctly, efficiently and serviceable when needed. 

To summarise, data discovery for us is a process of exploration – mapping your data from left to right. Visualising its transitions throughout its lifecycle in order to understand the global layout of data and the local state of data by source and phase – the spine of your organisation and the best ways to use it.