On data trust, a silver data product and one field that fell through.

by Sem Lemmers

A good data product delivers not only data, but also trust. You know what the data means, what quality you can expect, and who is responsible if something is not right. Precisely when something does deviate, that approach proves its value.

That is what Iris discovered when she started with a simple assignment on Monday morning: a new dashboard about ride patterns in New York. She is an analytics developer, a combined role of analytics engineer and BI developer. That means she models the data from the quality layer to reporting-ready datasets, and she builds the dashboards on top of that.

She opens Qlik Cloud and finds the data product in the marketplace NYC Taxi & Limousine CommissionSilver status, according to the medallion architecture: the raw source data has been cleaned and transformed, but not yet enriched or aggregated. Exactly what Iris is looking for. Gold would be processed too far for her; in that case, someone else has already made choices that she wants to make herself. Raw is too early, as she would spend half the week cleaning. Silver is the transfer point: quality guaranteed, flexibility preserved.

Look first, then build

What matters to Iris before she writes even a single line of script: can I rely on this? She doesn't have to call anyone for that. The data product tells her itself.

The figures are right there: 99% validity and 99.5% completeness across the entire product, with a Trust Score of 3.8/5. She clicks through to the Trip fact table, the heart of the star schema, with one row per taxi ride featuring time, location, distance, and the complete fare breakdown. 100% valid, 100% complete. Every dataset has a description, and so does every field. There is a glossary attached in which terms such as Trip, Medallion and Silver layer are recorded and linked to the underlying datasets. The documentation describes the origin, the star schema, and even the ML applications for which the data is suitable. And it includes a key contact: the data engineer who manages the product.

This is the part of governance you normally don't see, because it simply works. Iris doesn't have to guess what a field means, email where the data came from, or hope that it is complete. She reads it straight off.

And then: the Date field

The date dimension is where it stalls. The validity of QLT_Date is at 93.6 — decent, but clearly lower than the rest. And the Trust Score recently dropped, from 76.2 to 75.0. Iris opens the quality details and sees the culprit: the Date field is being rejected by a custom validation rule.

Even context. Qlik handles out-of-the-box quite well whether something is a date. But "being a date" is not the same as "being our date". The agreed quality agreements state that date fields in the silver layer are a bare date, in a fixed format, without a time component. There is no standard check for that, so we defined our own semantic type with a validation rule for it. You create such a rule once and it then runs in every quality calculation.

And that rule is now doing exactly what it’s supposed to do. The “Date” field is displayed with twenty to twenty-two characters: a date with a time stamp. At first glance, it seems harmless—it’s just a date, right? But for Iris, it’s a problem: a date with a time stamp doesn’t display neatly in a dashboard, and any deviation from the agreed-upon format means that every developer will end up handling it slightly differently. Without her own rule, this would never have been noticed. The standard checks found it fine. The column was populated for 100%. Everything looked green, except for the one point where we said: this is our agreement.

Fix it yourself or return it?

Iris now has a choice, and it's a tempting one. Qlik has recently table recipea no-code, spreadsheet-like way to make a dataset ready for use. You open the table, see a quality bar for each column, and apply edits step by step. Converting, cleaning, formatting, from over sixty visual functions, with an immediate view of the result. Every step is recorded in the recipe, which is reusable and runs with every refresh. It was built for exactly this kind of job: turning a string with time into a neat date takes her just a few minutes, without a single line of script.

She even builds it temporarily, as a test. Select column, text to date, remove time component, format according to agreement. The quality bar turns green. Solved, for her.

And that is precisely where the doubt begins. Because the data product itself is still wrong. Every colleague who grabs it from the marketplace later on will run into the same field and fix it on their own accord. Three developers, three recipes, three slightly different dates. Then the agreement for which the validation rule existed has evaporated anyway; only now it is invisible, hidden in everyone's own final mile.

He who seeks, finds more

Now that Iris is already looking at the quality details, she investigates further. The profiling of QLT_Date reveals two more things. HolidayUSA is empty in 362 out of the 365 rows. Understandable (not every day is a holiday), but it depresses completeness and deserves a conscious choice: leave it empty or fill it with a default. More painful: a series of derived fields such as WeekOfYear and DayOfWeekInMonth contain the same value for all 365 days. Constant 1. No validation rule catches it, no empty cells, no error message. Just silently broken. Anyone building a weekly report on this will get one very long week one.

That clinches the choice. She could have fixed the Date format herself, but you don't repair broken derivations in the final stretch. Those belong where they are created. The lineage points her in two clicks to the quality app that produces the silver QVDs, and the key contact is right there with the product. One message to the data engineer, with the scores as proof and her test recipe as an attachment. Not a complaint, but a ready-made specification: this is what the field needs to be.

What this story actually tells

The temptation is to call this a story about a mistake. It is the reverse. The mistake already existed; the system merely made it visible, at a time when it didn't cost anything yet. Iris found it before the dashboard existed, not after it in a heated meeting about why the weekly figures don't add up.

That is what a dropping Trust Score is: no shame, but a signal that works. Quality scores that are always at 100 measure nothing. And generic checks only monitor generic agreements. The agreements that truly matter, those of your organization, must be defined by yourself. A single custom validation rule on a single field was the difference here between “looks good” and “is correct”.

And then Iris's choice. The table recipe gave her the power to solve it herself, and that is precisely why her decision matters. Being able to fix things yourself is worth its weight in gold; knowing when you shouldn't do it, that is governanceA recipe is perfect for the final mile: your dataset, your analysis, your pace. But a semantic difference between the silver layer and the data product is not a final-mile problem; it is a breach of contract at the source, and you solve that at the source. Otherwise, you are buying speed today with proliferation tomorrow.

Meanwhile, Iris just keeps pushing forward. The Trip table is spotless, the model will be up and running by the end of the day. She will connect the date dimension next sprint, as soon as the Date field is just a date again. She'll throw away her trial recipe then. The best recipe is the one you no longer need.

Iris is a fictional "persona" created to visualize the functionality and value of data products. Data products are a part of Qlik Cloud and serve as the gateway between data engineering and analytics. That the roles analytics engineer and BI developer merged is very common in the Qlik world, but not necessarily always the case. A data product with silver-status datasets can easily exist alongside data products with gold-status datasets. A data product is a curated, documented, and quality-assured set of data offered as a ready-to-use package to users, including an owner, agreements, and built-in trust.