
## Pivoting Towards Strategic Data Quality Management

In the rapidly shifting digital landscape, the importance of data quality for becoming a truly data-driven organization cannot be overstated. The quality of your data impacts every decision you make, and even the smallest error can have significant downstream repercussions. Recognizing this fact, many companies have implemented various measures to get in front of this by monitoring and improving data quality.

However, the journey towards high data quality is not a straightforward one. Companies tend to start with tactical, home grown solutions, utilizing tools and technologies already at their disposal. These solutions often include using Excel and SQL queries to sift through data, or incorporating data quality checks into Business Intelligence (BI) dashboards, allowing for some level of automation, monitoring and control over the data quality.

While these solutions add short term value they have difficulty evolving and scaling as an organization matures and its data needs change over time. In this article we discuss important limitations organizations face when relying on custom and home grown data quality solutions and why they might want to think more strategically.

## Limitations of tactical solutions

Tactical data quality solutions, as discussed above, have their place in an organization's toolkit: they provide immediate insights and help teams react quickly to data quality issues. But these solutions come with their own set of limitations around metrics, communication and collaboration, and scalability.

### Metrics

Tracking data quality and trends over time quickly gets increasingly complicated when just relying on queries and reports. SQL queries might pinpoint data quality issues, but they are not be able to identify patterns or trends over time. This makes it hard to predict and manage future issues.

While it's important to identify trends over time, it's just as important to ensure data quality issues that have been detected and resolved don’t reappear later on. Monitoring regressions like these is complex when you're just doing point in time analyses. Instead, to be able to alert teams of regressions, organizations must keep a full record of their data quality incidents.

Tracking data quality, trends and regressions often requires rethinking the framework and involves adding more complexity to an existing solution. Before you know it, your team is spending more time developing a system for data quality management rather than focusing on their core business operations.

### Communication and collaboration

Specialized scripts, SQL queries and BI dashboards are often managed by specific individuals or teams. When a data quality issue is detected, the process of a) notifying the relevant stakeholders, b) discussing the problem, and c) deciding on a solution is a largely manual one and often involves sending a string of emails back and forth. This often leads to delays in resolution, poor communication, and a lack of coordination among teams. Integrating these custom solutions with incident management processes and tools are also frequently complex and unwieldy undertakings.

In short, without a streamlined process and shared view on data quality, it becomes difficult to respond promptly and effectively to data quality issues when they arise.

### Scalability

Tactical solutions are effective for handling small to medium volumes of data. But as the number of data checks, frequency and scale increases, these solutions can easily become overwhelmed. They struggle to process large data sets efficiently, and their performance suffers as a result. What was initially a quick and easy solution becomes a bottleneck, limiting the organization's ability to handle and analyze data effectively.

The technical nature of custom solutions make them less accessible to non-technical users and make knowledge less transferable. This limits the number of people within the organization who can effectively maintain, update and utilize a data quality solution.

## Conclusion

A strategic approach to data quality management is about more than just fixing problems. It’s about anticipating issues, preventing or containing them, and continuously improving your data quality. It's laying the groundwork for your organization to become truly data-driven.

The challenges posed by custom solutions underscore the need for a shift in approach. Instead of relying on tactical short-term solutions, companies should aim for a strategic and long-term approach to data quality management. This shift is not just about better tools, but also about adopting a new mindset &embdash; one that values proactive measures, scalability, and collaboration.

_Mesoica’s data quality platform is designed to meet the evolving needs of today's organizations. By using our platform, you can continuously monitor data, identify trends, flag regressions, and foster communication and collaboration around data. Our platform is built to scale with your organization's growing data quality maturity needs. Start your journey towards becoming a truly data-driven organization today. Visit our [website](https://mesoica.com) or [contact](mailto:demo@mesoica.com) us to learn more about how Mesoica can empower your organization to anticipate, prevent, and continuously improve data quality._


Many organizations grapple with scaling homegrown data quality solutions that have limitations in metric tracking, communication, and scalability. This article proposes transitioning from these short-term tactics to a strategic, long-term approach using platforms like Mesoica's for improved data quality management.

From Quick Fixes to Long-Term Success


Companies waste a tremendous amount of time on data preparation. Studies Crowdflower estimate that this can sometimes consume up to 60% of people's time, disguising itself as a regular part of the job. This is a significant problem!

"Small data" can disrupt data-intensive processes and increase the amount of manual data preparation required. To us, small data refers to ad-hoc, manually exchanged, low-volume, and high-value data. Examples of small data include financial statements, capital calls, real estate deals, contracts, financial valuations, etc. all required as input into operational systems, data warehouses, or other downstream applications.

In other words, non-automated data processes can become very disorganized because of the data containers and structure that must be dealt with. These containers include files, documents, and images that contain data that must be extracted, structured, and processed.

## Impact of small data

In order to understand the impact of small data containers on data-intensive processes, it is important to consider the structure of the data and its readability by both machines and humans.

When considering data structure, it is important to determine whether it is tabular or organized differently. Tabular data is often easier to read and process, whereas differently organized data can create more complex processes and require more manual data preparation. Examples of differently structured data include financial statements, confirmations, emails, hierarchical and/or aggregated data.

In addition, data readability plays a crucial role in determining the efficiency of data-intensive processes. Data that is easily readable by both machines and humans can be processed and analyzed quickly, reducing the amount of time required for manual data preparation. A financial statement presented in XBRL format can be easily parsed by a machine, but not by a human. Conversely, the same statement presented in PDF is difficult for a machine to parse.

Data that is difficult for machines to read, and contains non-tabular data, often has the greatest impact on operational processes:

![Impact of manual data processing](danger-zone.png)

## Small data categories

The most common data containers used to exchange and process data are Excel sheets, PDFs, CSV documents and JSON/XML files

Excel sheets are readable by both humans and machines. However, a lot of manual intervention is often required due to the following issues:

- Consistency issues can easily creep in due to Excel sheets being the result of a manual reporting process. Column names can change and header rows can be moved around on a regular basis.
- Metadata sections and/or user-provided commentary can provide important clues about the data in the document. For example, a commentary field in a data collection Excel informing the recipient of data that all numbers are in USD can be important, even though numbers can be parsed automatically.
- Data may be located on different tabs and needs to be consolidated.
- Regional settings can be difficult to deal with when exchanging Excel data in an international context.
- Unpacking pivoted data or transposing requires manual steps.

PDFs are easy for humans to read, but difficult for machines to read. This is due to:

- Limited structure and visual-only cues, making it challenging to extract the right data.
- Data is presented in ways that prioritize human experience, such as summaries, hierarchical layout, headers, and roll-ups. These factors increase the amount of manual intervention necessary to extract data from these documents.

Structured tabular data in PDFs is hard to extract manually though, for example copy/paste, manual actions. These kind of containers suffer from the same problems mentioned under the previous bullet.

CSV-like data is well-structured and suitable for automated processes. Ironically, in manual data preparation processes (often Excel-based), CSV can actually cause more trouble:

- Quoting and separator characters usually require some Excel wrangling, such as converting to columns or removing characters, before getting the data into the desired shape.
- Regional settings can be painful when dealing with Excel or even CSV in an international context.

XML and JSON are primarily meant for use in automated processes and consumption by software systems. In our opinion, they have no place in an end-user environment.

## Conclusion

We find that small data, as defined by us, is a common phenomenon in many organizations and can have major impact on efficiency, scalability and quality of data operations. However, a structured approach to dealing with it is not widespread. Typical approaches involve using Excel macros or custom Python scripts. Often, there is no focussed approach to breaking out collection, validation, transformation, and delivery into discrete steps that can be optimized, improved, and automated.

How does your organisation deal with small data in a structured manner? We’d be very interested to learn, let us know in the comments.


This post explains the impact of small data on data-intensive processes, and the challenges associated with common data containers used to exchange and process data such as Excel, PDFs, CSVs, and JSON/XML files. The author concludes that small data is a common problem in many organizations and a structured approach to dealing with it is not widespread.

Small data, big problems


A couple of weeks ago I wrote about data management and exchange in the investment management industry (see previous article [here](bla)). I argued that it is necessary to bridge the gap between manual and hard to automate data management processes on the one hand, and formal data management or systems of record on the other. Standards and enterprise data management systems are not helping solve these real world data management problems.

To bridge that gap we need to look at the nature of these processes and what common issues need to be solved.

Prior to going into common issues and solutions, I want to set the stage by describing a generic view that we, as a company, have on data management and data exchange processes. I will discuss underlying key business requirements and challenges of each of the following steps:

![Key steps in data management](key-steps-in-data-management.png)

## Collection

Gathering data from different internal or external suppliers (e.g. back office teams, external asset managers, actuaries, accountants, service providers, etc.) involves receipt of the actual data sets, tracking delivery of data sets, sending reminders and escalating if data does not arrive on time. Data, on receipt, must be stored in a central location in order for the team or internal systems to locate it and be processed further. Tracking delivery ideally is related to an entity within the business context (e.g. external manager, fund, client, portfolio, data vendor) so that reporting on progress is easy and accessible.

Data can be received from internal as well as external sources. In an internal scenario a team might have more influence on how data is delivered to them. When dealing with external entities data delivery agreements might be more difficult to enforce. Finding the right level of leverage over data suppliers can be a balancing act between providing the best technical solutions (i.e. making it very easy) and a more contractual/transactional approach.

## Validation

Received data ideally passes through a series of technical and functional checks and controls to make sure that the it conforms to agreed quality standards. An exception driven process should enable specialist in the business teams to intervene in the data processing, make changes where applicable or escalate back to the data supplier with clear and structured feedback.

The challenge is to facilitate and structure the communication around expectations and actual data quality to improve that and create a virtuous cycle. More concretely, based on specific feedback data suppliers should be enabled to improve on an ongoing basis rather than recipients continue to fix data issues on behalf of.

## Preparation

Teams request data for one or more specific purposes. This step is about shaping data in such a way that it is fit for further consumption by other processes, applications and/or systems. For example, a spreadsheet tool requires input from various other departments, a data warehouse system relies on manual inputs for a number of it’s tables or an accounting system requires weekly or monthly valuations to be processed.

Preparation of data is not just simply renaming and moving columns around in the right order or restructuring data (e.g. denormalising, disaggregate, etc.). It is also about mapping data to internal structures and conventions as well: mapping data formats to something applicable (e.g. dates, numbers), cross referencing internal and external identifiers (e.g. ISIN, LEI), translating classification/rating schemes (e.g. GICS, Moody's, S&P), enrichment, etc. This requires loads of reference data, translation tables and an efficient method to maintain all of that.

## Delivery

Data delivery is the final piece in any process where validation and prepared data is moved on to a next step (e.g. processing by a different team, application, etc.). Here teams want to track delivery and understand that data was successfully consumed by the intended recipient. Delivery can happen in different shapes or forms: certain recipients might want to receive an email with attachment, others have structured systems in place that will allow delivery onto a Sharepoint or through SFTP, web services or API’s. The trick here is to minimize friction for the data recipients and set them up for an efficient experience.

As delivery of data can happen to both internal as well as external stakeholders, security is a big consideration when setting up these workflows. Striking a balance between usability and a secure delivery method is an interesting challenge for many organisations.

## Next up, practical solutions

Again, when dealing with highly structured, organised and predictable data flows the above issues might prove less urgent and traditional enterprise tools (EDM, ETL, etc.) could fit the bill. However this approach very quickly falls down if data frequently changes, data comes in at unpredictable intervals, progress needs to be tracked, visibility/audit of the process is necessary, structured communication/feedback on data is required. These problems are compounded by the typical tools that are used for this today: client portals, internal Sharepoints, Excel and email. These tools don’t integrate to provide a coherent, user friendly view on the process described above.

It is clear that each step in the process has different challenges and that point solutions have been created as coping mechanisms. Stay tuned as I will discuss each of the steps further and present practical and better solutions in the weeks to come.


Data management follows four steps: Collection, from diverse sources; Validation, for quality assurance; Preparation, for user-specific needs; and Delivery, ensuring secure and efficient data transfer. Read on to learn more and unlock the power of effective data management!

Key process steps in data management


Everyone seems to agree that data is a critical resource for serious financial institutions. Is it not curious then that in so many organisations data is not treated with the importance it deserves: quite often data is processed manually, is late, gets lost, is incomplete, or inconsistent or incorrect. These are particularly urgent problems when it comes to external data required for processes from portfolio management, compliance and risk management to accounting and reporting.

Regulatory pressure and client demand have greatly increased the need for more transparency into investment manager activities, which in turn has increased demand for more and increasingly detailed information on investments. As the picture below demonstrates this can give rise to quite some complexity. Especially as the different roles that each of the market participants can play in any given situation can change.

![Data management landscape and 3rd parties](data-management-landscape.png)

Obviously this is an oversimplified view of the market. But organisations do exchange massive amounts of data amongst each other sometimes in different capacities. A lot of the data that is changing hands is quite similar in nature, in the end everyone is interested in the same things: market data, risk/performance, portfolio holdings (in the broadest sense) and transactions. So we must have solved this, right?

Not quite: some standards exist and systems have been implemented to automate stable data exchange, but Email, Excel are still prevalent as a primary means of data exchange. This wreaks operational havoc and creates security nightmares. Let’s explore these themes a little bit further.

### Standards, maybe not so standard after all

Apart from well known standards like SWIFT and FIX, which cater to data exchange of highly stable and simple data, very little standardisation has been achieved on wider industry level.

There are many initiatives that aim to achieve this, for example: Solvency 2 TPT, Inrev, ILPA, FundsXML, etc. While these have found adoption in some products and services, they generally do not play a large role in manual data exchange processes. What makes these standards cumbersome in use is the technical handling; formats are often complex and unwieldy (think XML or very large Excel sheets). Also, these standards usually prescribe a layout in terms of columns and content but organisations generally have no way to actually enforce or validate any of the more in-depth data structures (like classifications, code schemes, formats, etc.).

In a lot of cases capabilities to effectively leverage standards are missing or standards are ‘abused’ and thus lose their power to truly harmonise data exchange.

### Systems? Not so fast

Many data management system vendors will promise to solve all your data management problems. Ranging from technical solutions to super smart AI, machine learning and robotics applications, systems like these are expensive and require large investments in license fees and implementation up front with little to no guarantee that they will actually make practical operational problems go away.

Data management systems are good at processing large amounts of structured and stable data: e.g. end-of-day market data feeds, accounting snapshots or transactions. They are less well suited to exception handling at every step in the process, dealing with volatile layouts, enabling manual intervention or providing a user friendly window into the data and the process.

Furthermore, traditional delivery models of these system require long implementation projects before actual business value is generated. Changing the system afterwards or getting new functionality rolled out is difficult.

Data management systems can play an effective role in automating simple, stable and structured data feeds. But they are ill-equipped to enable flexibility and agility on the end-user side.

### Excel, email and PDF, things of the future?

A short time-to-market, providing a good service to clients and to meet regulatory deadlines requires rapid onboarding of new or changed data sources to be integrated into operational and reporting systems. Unfortunately, data management systems, even when implemented properly, do not provide the basic flexibility or agility that end users often require.

Waiting for IT to get things implemented in a timely fashion is in a lot of cases not a realistic option. There’s also the issue of retrofitting all data sources into a data management system’s framework. This typically results in a lot of customisation and many one-off and unique processes which are costly to maintain and can prove a major headache during systems upgrades (think regression testing). All in all IT often has neither the time, appetite, or tools to optimally facilitate the business.

The result of this is that business teams will work around data management systems and revert to email/Excel solutions and simply work with the data they receive to get things done. This perpetuates the manual processing of data, introduces errors, lowers data quality and further complicates Excel solutions.

### Security

Last but not least, data management processes that exist in this parallel world, outside of the enterprise systems, create real security issues (such as compliance issues, GDPR violations, contract breaches, etc.).

Data that is required to fulfill client obligations is often confidential but is sent across in unsecured emails. Emails end up in group mailboxes causing organisations to lose track of who has access to what.

Other solutions are to deliver data through portal and data sharing solutions. These offer secure transport mechanism but often credentials (username/passwords) are shared among colleagues and also through email. Apart from the fact that it introduces more manual work and consequently errors, it does not fundamentally improve the security situation.

## What can you do?

There is a clear need for solutions that can bridge the gap between the manual and hard to automate data management processes on the one hand, and formal data management or operational systems on the other.

Ideally, organisations exchanging data would agree to execute on tight data delivery agreements and to implement highly automated exchange processes. Until we get there, a more pragmatic approach should probably be adopted, an approach where data as-is and a process of continuous improvement of data and relationship with supplier is accepted.

This means opening up processes and systems to deal with data in a structured manner. More concretely, this means organisations should follow clear workflow steps in collecting, validating, preparing and delivering data downstream. The end-result is flexible, agile and exception driven processes that provide visibility into each of the above steps and the ability to manually intervene only when required to do so.

In future blog posts we will describe how we look at each step of the above workflow, how this is operationalised and effectively supported by technology.


Data mismanagement in financial institutions leads to operational inefficiencies and security issues. Traditional systems lack the required agility, causing reliance on insecure methods like email and Excel. A structured approach to data handling and continuous improvement is needed to bridge this gap.

All things data quality, our company blog

From Quick Fixes to Long-Term Success

Small data, big problems

Key process steps in data management

Real world effective data management