Welcome Soda Cloud: End-to-End Data Observability and Collaboration

By Maarten Masschelein, CEO & Co-Founder

© Soda, created by Dahlia Ramirez

I got involved in the data space when I joined Collibra, a platform for data governance, cataloging, and discovery back in 2010. At that time, the role of the Chief Data Officer / Head of Data didn’t exist, and organizations were struggling to understand what was going on with their data, let alone manage data quality issues as the tooling that was available, was unwieldy and antiquated, and did not support the needs of the business (still does not in most cases). When I was introduced to Tom Baeyens, my fellow co-founder, it didn’t take long to connect the dots — and the data — and realize that there was a real world problem that we could solve combining Tom’s open source background and my data experience.

The real world problem? Silent data issues.

The issue continues to get bigger as systems keep going and processing bad data, with uncontrolled consequences including producing unexpected or erroneous results. This is why we call them silent data issues — data quality issues that are identified only once datasets are put to use in reports, campaigns, models, and used for decision making.

This can, and often does, result in data engineers spending too much time fire-fighting a data issue; data consumers having no confidence to trust the data; and the business spending too much time trying to resolve the far-reaching effects and consequences.

Let me be clear: data quality is an age-old problem. You’ll find it referenced in DAMA’s Data Management Body of Knowledge, whose first edition was published in 2009. The overall principles are solid; it’s the process, and technology that need to be hit with a refresh.

Our big idea was to empower data teams with tools to detect and resolve issues that matter earlier upstream to enable data confidence. And so Tom, the team and I, set to work to remove the fear of not knowing and the pain of finding out too late that a silent data issue has had a downstream impact.

Introducing Soda Cloud.

Early in our market research, it struck us that there was a general lack of observability into data systems (commonly called Data Observability). Many organizations had a way to index and discover data (catalog), but very few organizations could automatically discover and track the data problems across and within data sources that surface in the context of building data products. As a result, most data issues remained silent.

When teams were implementing a system to detect issues, it was mostly a rules-based testing service. To achieve this, they were relying on homegrown frameworks, most often using YAML files to configure the checks that needed to run each time new data arrived. At runtime, the DSL translates into a set of compute instructions to calculate metrics — typically in SQL or Spark — and evaluate checks.

The biggest problem data teams have with this set-up is that it doesn’t scale. It doesn’t scale because rules are hard to set-up and maintain, and it doesn’t scale because there are no automated issue discovery systems (e.g. tables are not refreshing based on the historical refresh schedule we’ve inferred from the query logs).

Another, even more important, scalability problem that we identified had nothing to do with technology, but with people and process. The majority of high-value checks were defined by subject matter experts (SMEs) of data, who are often not comfortable with Git + a YAML DSL. This limits adoption of both the new process and technology tremendously.

Data teams require a common framework to define and manage expectations for data behavior. Setting “Service Level Agreements” between data engineers and data consumers would bring clarity and consistency to teams when creating data products, eliminating (unwanted) assumptions and improving data quality on a continuous (automated) basis.

Soda Cloud is a new, prescriptive approach to get ahead of silent data issues and manage data quality. It combines predictive capabilities with a super simple, yet powerful rules-based system. This allows data teams to create coverage very quickly, and get end-to-end observability.

Soda Cloud is designed for a broad range of team members to get involved, from data platform engineers, to analytics engineers, product managers, and analysts. The goal is to help them to discover, prioritize, and resolve data issues collaboratively — and sooner. This way, the CDO / Head of Data can keep the oversight they need to help remove bottlenecks and ensure governance.

Discover issues with data products intelligently.

This Soda Cloud Dashboard shows the results of automated monitoring. Key aspects of your data, such as freshness, data volumes, schema changes, user defined tests, and column level metrics are reported.

Next to this automated anomaly detection for “known unknowns”, data issues can also be discovered by testing and validating data. For this, our team has developed a simple, yet very powerful low-code, Domain Specific Language (sometimes called DSL) that allows you to do a wide range of checks. These include, but are not limited to consistency-over-time checks, reconciliations, reference data checks, and virtually any business logic.

This Soda Cloud Data Validation user experience allows non-SQL savvy users to set-up complex data validations to catch issues with a known exception scenario.

Prioritize the data issues that matter most.

Data Freshness Dashboards — a mechanism to manage product performance
Soda Cloud gives teams a tangible framework to define, visualize and track data behavior through Data Freshness Dashboards. Whether it’s for a BI report for Finance, or machine-learning instructions set up by Operations, This approach enables the data owners to manage their data as a product, and understand what every team expects of the data downstream. These dashboards show the data owner the discovered data issues from across the organization to accelerate both discovery and resolution.

In this area of the Soda Cloud, scoring the health and fitness of data using data quality dimensions, can trigger an alert if for example a particular column is missing values, or a critical alert is missing more than 5%. This functionality defines “what good data looks like” based on the users’ expectations and helps eliminate alert fatigue.

Further analysis on the root cause of a data issue is often also needed. This can be achieved by, for example, exploring the data lineage in your data transformation, orchestration, and/or data cataloging tool.

Lineage graph in the DBT Data Transformation tool.

Resolve data issues collaboratively at the source.

At Soda, we believe that data quality is a team sport, however, you need to make sure that it’s the right people getting involved through role based alerting. The Soda platform facilitates collaboration by creating a shared context, and a clear resolution workflow for prioritizing and resolving issues and assigning tasks that matter most to the business.

Our approach and workflows take into consideration and recognize that it is the data owner that can ultimately make decisions on where to invest time and effort to improve the quality of data. Data owners should be able to easily ask both data engineers and data SMEs in the business to help analyze and fix issues. To streamline communications, integration options are available for the most-used channels such as e-mail, Chat, Slack or ServiceNow.

Resolution and root cause analysis begins with Annotation functionality. Here a data owner or business users can comment on an observation about the data and assign a Task for follow-up.

Data Quality is a Team Sport: Soda Cloud Brings Everyone Together

Soda’s approach is different. Whilst there are a number of offerings in the market (with more and more joining this growing data space every day), they tend to focus on only one part of the problem, or solve the challenges of only one role in the data team.

With Soda Cloud, the entire end-to-end data quality process is brought together in a single platform, integrated and centralized. Each user can work with the right tools and workflows, in an environment that best suits their needs and expertise. This is what we mean when we say that Soda brings everyone closer to the data.

We solve the problem with a combination of a cloud platform and a set of open source developer tools.

This is the Modern World of Data Quality

  • Data platform engineers have a central infrastructure to enable monitoring-as-a-service, so that others can build data products on it.
  • Data and analytics engineers are equipped with a simple way to test data each and every time it is transformed, to ensure data pipelines are more reliable. From the developer toolkit, they can use Soda SQL to easily stop data production and quarantine bad data. Soda Cloud will visualize the health of datasets and act as a communication hub for the data issues.
  • Analysts, stewards and subject matter experts can validate data in a self-service way.
  • Data owners and data product managers can get an overview of all data expectations so they can easily prioritize issues for resolution.
  • Heads of data and analytics, or Chief Data Officers, are able to implement a culture of data ownership and a community of good data practices.

Join the Community

Soda SQL is delivered in a white box approach that puts engineers in control. The Soda Developer Tools kit, available on GitHub, is built to fit naturally into the data engineer’s workflow.

Soda Cloud is available as a free trial version (now extended until June 30 2021). Users can get monitoring in just minutes and realize the power of Soda on their own data.

There is power in the community and there is power in good data.

I invite you to access our Developer Tools or Soda Cloud today. It’s time for every modern data team to create trust in the data.

If I can help, let me know.

Soda‘s data observability and collaboration platform keeps data fit for purpose, verifiable and trustworthy. We bring everyone closer to trusted data. soda.io

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store