What is data grain?
The ‘grain’ of data refers to how detailed (high-grain) or aggregated (low-grain) any given data is in relation to the real-world phenomena that it describes. This is best explained by examples – let’s use supermarket store sales data to consider different levels of grains:
- Transaction x Item | 1 row of data represents 1 item code in 1 till transaction
Retailers collect tens or hundreds of millions of data rows like this every day; no UK supermarkets directly share data at this grain with their suppliers but ALL shared sales data is aggregated from this very lowest level of grain. - Day x Item x Store | 1 row represents all sales of 1 item code in 1 store on 1 day
This ‘daily item store grain’ is created by aggregating sales for every item, across all transactions, within each store every day and is the lowest level of grain that most UK supermarkets provide. It enables powerful insights into sales patterns, across different days of the week, in different store types and regions and supports many analytical use cases.
Collecting this grain of data from the ‘Big Four’ will result in thousands of rows of data for every item, every day, and can be overwhelming if you try to analyse several weeks of data for tens of products in a spreadsheet. - Week x Item x Store | 1 row represents all sales of 1 item code in 1 store in 1 week
This ‘weekly item store grain’ is an aggregate of [2] above and is (obviously!) seven times smaller. It is helpful for long-term sales analysis – for example, comparing performance over a two-year period (year-on-year).
It will be a little easier to analyse in a spreadsheet but is less useful for some analyses. For example, promotions usually start midweek and typically run for 3 weeks; a weekly dataset will obscure the start and end of the promotion and limit your understanding of demand, and the retailer’s ability to meet that demand. - Day x Item | 1 row represents all sales of 1 item code across all stores on 1 day
This ‘daily item grain’ is available from all UK supermarkets and is very accessible; an entire year of trading is represented in just 365 rows of data per item. This is a very popular dataset for suppliers to collect and analyse because it’s quick and easy to download from a retailer portal and load into Excel for reporting purposes.
It supports many general use cases, including overall promotional performance, long-term sales trends, price variation etc. and provides an excellent basis to understand WHAT is happening but can prove limiting when you want to understand WHY – the lack of breakdown by store means that you can’t identify variations by store type, region, promotional participation etc. - Week x Item | 1 row represents all sales of 1 item code across all stores in 1 week
When ‘daily item grain’ data is aggregated by week into ‘weekly item data’ it loses much of its value for a very limited reduction in size (sevenfold) so it’s not a popular dataset with any supplier.
It’s important to consider data grain because you face a trade-off between utility/insight and convenience/simplicity:
- Low-grain data [1, 2, 3 above] is much larger and more difficult to store and analyse in entry-level tools.
For example, a supplier with 100 items listed across most of the stores (supermarket and convenience format) of the ‘Big Four’ will be collecting around one million rows of data every day. Storing, organising, analysing and reporting on this scale of data is beyond the capability of an individual using Microsoft Excel (the world’s most popular reporting and analytics toolset!) and will require a database or data platform.
- High-grain data [4,5 above], however, is much easier to collect and manage but limits materially the nature and scope of the analysis that you can perform.
It’s easy to manage in Excel but if you have a suitable data platform (typically a data lake or a demand signal repository) then such high-grain data can be generated automatically from low-grain equivalents. For example, to get a weekly summary of sales by region, your reporting tool of choice will automatically aggregate ‘daily item store grain’ data by dates and region.
Data grain considerations also impact how you connect data from different grains. For example, to understand how the movement of items from depots into stores is impacting store sales, you will need to combine ‘daily item store’ and ‘daily item depot’ data, where each depot will be related to multiple stores.
Want to learn more? These articles can help:
