How to Improve Data Integrity: 3 Questions To Ask
Every company talks about their data strategy and how important it is to their digital transformation but who really owns the data strategy and where does it live? When framed this way it almost feels like we are searching for a mythical creature—like chasing after Bigfoot. While everyone is trying to embrace digital transformations, which accelerated 5 years forward in a matter of 8 weeks (according to Mckinsey) during the pandemic, it feels like everyone is scrambling for that data strategy.
Most companies relate their data strategy to the data lake, data pipeline, and the efforts to get better analytics so they can make better business decisions. While all of those efforts are important, the time and effort involved in all of those systems creates an uphill battle that feels like you’ll never win. A data engineering team is always searching for why something is breaking, where data is coming from, and more context around the ever evolving and changing data.
The real truth is for most brands there is no single source of information, system, or person to clearly articulate the data strategy. There might be some PowerPoint slides or a long document on what should be done, but the minute these were finished they are already out of date.
Without a way for everyone to be able to understand the data strategy, trying to make sense of the data tends to fall on the data engineers or data analysts to infer context or tracking down answers. The cycle time to continue iterating on validating the data, understanding context, and fine tuning the ETL process creates data lag.
Everything in your data system may be working as expected and ending up in mostly the right places but the data is meaningless because it takes so long to reach a usable state. Data lag has a real impact on the business because your key asset and strategic advantage in the market sits idle for days or weeks.
To resolve this, you and your organization need to ask some key questions. They include:
1. How long does it take from the time data is created to support business decisions?
The best way to understand all of the systems and teams is to map out the full process from the time there is a marketing idea all the way through to the point where business decisions are being made.
For many companies, this time to arrive at final decisions can take weeks or longer before a report is able to be trusted. You can only assume everyone is making an effort to track their campaigns but by the time it arrives in your analytics platform, if the data is missing or not structured the same, the analyst is left to clean it up by reaching out to various teams to try and make sense of attribution.
2. Where are the key bottlenecks in your process and data flow?
After mapping out all the systems in place and the people involved, adding in a time component can be very helpful as you try and reduce any bottlenecks. While the time to launch and analyze campaigns can vary, even just using a few sample sizes can quickly identify where there may be inefficiencies.
For example, during the creative process to campaign launch, are changes being made through emailing a spreadsheet around or is there a workflow management tool? Not only are these back and forth steps adding to the time it takes to launch an experience, it also introduces opportunities for manual errors to occur along the way.
3. Are you getting all the data you are expecting?
The final area to focus on is auditing the systems along the way to validate you are getting all of the data you are expecting. While the numbers in systems are always going to vary slightly, it is fairly common to have experiences or campaigns not configured properly and data is lost along the way and unable to be accurately attributed.
The most common example is where clicks appear in the execution system like GCM or Facebook but don’t appear in your site analytics because the tracking was not set up correctly. Another example is a campaign that has tracking but it doesn’t align with the same granularity of other campaigns. When you are trying to assess attribution across channels you want to ensure you have consistency and alignment so you can trust you are making the right business decisions.
Once you have reviewed these areas you should have some areas of focus to improve your overall data integrity. While many people focus on cleaning up the data problem itself, there are usually many more problems identified at the point of origin or data source which, if resolved, can have a much bigger impact on creating better data. Ultimately, the goal is to trust the data in all of the systems and make sure it is complete, validated, and everyone in the process has the incentive to be part of the solution and not just part of the problem.
About the Author
Chris Comstock is Claravine’s Chief Product Officer and has more than 15 years in the digital marketing and data management fields – working as a consultant, brand and product leader for a variety of top companies. He now heads Claravine’s product vision.