Spark data joy: Marie Kondō’s lessons for data overload and data engineering 

 

Large organisations, as our research shows, have vast oceans of data, but often lack the capabilities to extract enough value from them. How can they create a data landscape that is both more manageable and more valuable? After all, gaining an information advantage through real-time data delivers a direct benefit. We revisited tidying expert Marie Kondō’s six lessons and compared them with best practices in data engineering and data platforms. 

The conclusion of our recent research is clear: large organisations are hoarding data on a massive scale. They collect and retain large volumes of information without a clear purpose. As many as 63% of organisations deliberately collect more data than they currently use because it might prove valuable in the future.

The irony? Storing and keeping large amounts of data available consumes substantial storage capacity and distracts data analysts from their work. Ideally, analysts should be able to support better decisions at any time of day, using up-to-date data that gives them an information advantage. In reality, many spend their days navigating a jungle of irrelevant data. Ultimately, that data may not even be usable because its freshness, sampling frequency or level of detail does not suit the intended application.

Blocked data pipelines, an inability to extract value from data, high costs, frustrated data specialists and reduced agility and flexibility: the consequences of data overload are serious.

Fortunately, Marie Kondō offers a way forward. Although the Japanese tidying expert developed her KonMari method for homes, it applies equally well to your data landscape. Her six rules can help you create a tidy, manageable data environment. We see clear connections between her lessons and best practices in data engineering and data platforms.

“Commit yourself to tidying up,” says Kondō. Successfully bringing order to data chaos requires commitment from senior management, demonstrated through appropriate resources, methods, expertise and budget. If you are serious about extracting value from data, you must also take its availability seriously. Tidying up promptly saves hours of searching and prevents incorrect conclusions.

Management needs to commit to working with data. Alongside a clear data strategy, there must be sufficient budget for a properly functioning data platform. Data engineering should deliver a reliable, efficient platform that gives data analysts and machine learning specialists an information advantage. This provides the foundation for real-time insights that support better decisions throughout the day.

Commitment and a professional approach can prevent unnecessary data from accumulating. Usability should always be the starting point. Give your data team clear, measurable criteria, such as “90% of data is no more than four hours old”, “all data connections are available 95% of the time” and “fewer than 15% of pipelines have overlapping data sources”.

“Imagine your ideal lifestyle,” instructs Kondō. In other words, tidying up serves little purpose without clear goals. The same applies to data. Start with your business objectives, strategies and the questions you want to answer. Use these to determine what information you need and how current it must be. Then establish which data is required. What should you avoid doing first? Browsing through your data to see what stands out. Always begin with the end in mind.

What does your ideal data landscape look like? Your strategy for collecting and analysing data needs to align with your business objectives. One key objective is usually to improve decision-makers’ access to current information. Does your strategy involve making more data-driven decisions? Then both the underlying platform and the data engineering processes must focus fully on processing real-time data. That is very different from maintaining a data warehouse.

“Finish discarding first. Before getting rid of items, sincerely thank each item for serving its purpose,” says the third rule. KonMari places a strong emphasis on discarding things you do not need. The same applies to data: remove what is irrelevant, preferably before it reaches the cloud. Gartner estimates that around 80–90% of organisations’ data is either dark data, which is not used to generate insights, or ROT: redundant, outdated or trivial. Many businesses do not know which data falls into these categories, so they keep everything.

How do you clean up your data? By removing duplicates, outdated information and irrelevant data. Identifying and correcting all errors and inconsistencies often takes several iterations. Data engineers help identify and reduce this unnecessary data.

You also need a long-term plan to keep your data landscape manageable. Predefined, automated policies can ensure continuous data clean-up. When data is created, record its purpose and how long it will be retained.

Validate process data as it enters your platform by checking its accuracy, reliability and consistency, and classify it immediately. Validation should continue throughout operation, rather than happening only during development. Check that data is still arriving and that it remains meaningful. Modern data platforms offer these capabilities, although they are not always used. For example, if you measure outdoor temperatures and suddenly receive readings of 35°C in winter, that data should not get through the door.

Finally, regularly assess the progress of your data strategy and the volume and quality of your data. Use the findings to initiate further clean-up activities. In other words, carry out the full range of tasks expected of a modern data engineer.

“Tidy by category, not location,” is Kondō’s fourth rule. She recommends tidying by category rather than by cupboard or room. For organisations, this means avoiding a clean-up approach based on individual locations, regions or systems. Suppose the entire organisation uses a centralised IT platform. It is better to start by removing duplicate product records across the organisation than to remove all duplicate data separately for each location or region.

Tidying by category helps you analyse, categorise and prioritise data systematically. It also provides a different perspective, because information within a particular category, such as products, customers or orders, often exists in several systems. These are the categories you need to bring together in a well-designed data platform. Structuring data by category allows you to define clear pipelines that serve a specific purpose. Data analysts no longer need to search numerous sources for the right information. They can get straight to work, just as a tidy home lets you get on with what you intended to do.

“Follow the right order,” says lesson five. In a complex and diverse data landscape, start with data availability and freshness before considering complex algorithms. There is little debate about these requirements: they are essential foundations for creating value. Algorithms that support automated, real-time decision-making, however, tend to generate considerable discussion. Once the foundations are in place, that discussion becomes much more focused, without being distracted by questions about data accuracy.

“Ask yourself if it sparks joy,” is Kondō’s final lesson. Joy might not sound like the most suitable criterion for working with data. Instead, you could ask whether it has analytical value. Data engineering and the right platforms allow everyone in the organisation to focus on what they do best. Teams must be able to trust that others will deliver measurable data of the required quality. When your data specialists work with data they can extract more value from, their work becomes more rewarding. And that is how value sparks joy.

Retaining data for future use can be worthwhile. More historical data could help you improve business processes, gain deeper customer insights, develop new data products or build a stronger foundation for predictive analytics.

On paper, at least. In practice, we often find that data our customers have stored turns out to be unusable later. It may be too incomplete, insufficiently detailed or updated too infrequently.

Our advice? Follow best practices in data engineering and data platforms. Make sure your data processing and traceability work properly and result in usable data products. Once your data product is established, apply a strict deletion policy. If your service is still under development, retain a little more data to leave room for testing hypotheses and investigating causal relationships.

Above all, do not be afraid to discard data you cannot find an immediate purpose for. Or, figuratively speaking, put it in a box, write an expiry date on it and store it in the attic. If that date passes and you have never looked at it again, it can go in the bin.