Check the data behind a semantic layer
Data Quality profiles the warehouse data behind a semantic layer. It finds fields with many empty values, placeholder values and dates, identifiers that repeat, outliers and other problems, gives the layer a score out of 100, and suggests a fix for each finding.
Availability: Data Quality is turned on per tenant. It runs real queries in your warehouse, so it ships behind its own feature flag. If you do not see Data Quality under Evaluations, ask your Astrato contact to turn it on.
Run a profile
Section titled “Run a profile”Before you start: open a semantic layer that is connected to a warehouse. Profiling needs a connected layer.
- In the Semantic Layer Editor, open Evaluations > Data Quality.
- Select Run profile.
- Wait while the fields are profiled one at a time. Progress shows in the tab bar, so you can work on other tabs meanwhile. Select Stop to end the run early; results so far are kept.
A run profiles up to 60 fields, starting with the fields business questions use, then fields behind a dimension or measure. Each field is profiled with a few aggregate queries in your warehouse: counts, distinct values, the most common values and a histogram. No rows are copied out.
When the layer has unpublished changes that cannot be queried, the profile runs against the published layer and says so. If the layer or the checks change after a run, a notice says the results may be out of date and offers Run profile again.
The last run is stored with the layer, so the next person who opens Data Quality sees it without running it again. This holds only when every connection the profile uses queries the warehouse with shared credentials. If any of them queries with each user’s own credentials, or its settings cannot be read, the run stays visible only to the person who started it.
Read the results
Section titled “Read the results”- Score: each profiled field counts as healthy (100), warning (50) or error (0). Fields that business questions use count three times, fields behind a dimension or measure twice. Fields not profiled are left out.
- Issues: findings ranked worst first, each with the field, what was found, a suggested fix and the business questions and items that use the field.
- Fields: a table of every field with its status, empty share, distinct values, range and the top 10 values.
Select an issue or a field to open its profile: rows and empty values, distinct values, the range for numbers and dates, and either the top 10 values or a histogram of the distribution.
Mark a finding as expected
Section titled “Mark a finding as expected”Some findings are true but fine, such as an optional field that is often empty. You can mark them so they stop lowering the score.
Before you start: you need permission to edit the semantic layer, because the mark is saved in its draft.
- On the issue, select Mark as expected.
- Optionally, type why it is expected.
- Select Mark as expected again to confirm.
The finding moves to the Expected group, which is not counted in the score, and shows who marked it and when, with the note. The mark follows the field if it is renamed. If a later run finds the problem has got worse, the finding comes back into Issues with what it was when it was marked; select Still expected to keep it out of the score, or Stop expecting. To undo a mark, select Stop expecting on the finding.