September 11, 2026, by Kevin
I was recently talking with one of my colleagues who mentioned that our CEO used to ask him for ad-hoc data about our site’s usage. My friend would dutifully build and run a query to get the data, and when he’d share it, be met with the question: “…and this is correct?”
This question came up again in a more recent conversation, and it got me thinking: when you create a report that must be right, how do you guarantee it is in fact correct?
During 10 years at Bandcamp, I created tons of reports and dashboards that were used by customers, executives and other stakeholders. I also worked on our payments system and checkout, where mistakes in calculation would mean mistakes in the transfer of real money.
It turns out that ensuring correctness and building confidence in reports and critical transactional data requires an intersectional approach. If your users don’t have confidence in them, it doesn’t matter how right your numbers are. It is not enough to be a SQL wizard, you also have to account for UX and usability.
Here are seven ways to ensure that your reports are accurate and credible.
This is the “sniff test.” Once I became a manager, I started consuming more data that was produced by a data team or other engineers. Reports are always about something: imagine we are talking about monthly active users, and streams played per month. When I look at a new report, I do not immediately evaluate the direct metrics.
Instead, the first thing to think about is whether the report makes sense. You’ll be surprised how often a report does not make sense! Sense-check by seeing if it stays within your expectations. Think critically about what must have happened for this data to be true. Does the report reflect reasonable behavior?
For instance, do streams played indicate that every user on the site is streaming 100 albums per month? Is that a surprising thing for your average user to do? If—like me—you think that doesn’t sound right, dig deeper into your MAU and stream plays. You will discover that either your users are surprising you, or that there’s a mistake in one or both of your reports. Both are good things to discover!
If you are reporting in aggregate, does the data look correct at a scale that you can personally evaluate? Say you are reporting the total amount of VAT paid to the EU for the month.
How do you know this big aggregate number is correct? One way to do this is to check sales for just one seller. Maybe this seller had 50 sales in the last month. Manually evaluating this is an hour of worthwhile work.
At Bandcamp all of our reports were built with SQL, which makes it quite easy to add filtering criteria. Using SQL and other reporting tools, it is easy to filter the aggregate query down to just one seller, then take a look at that seller’s total. This is a slice of the aggregate report that exercises all of its joins and filters, but operates on a small enough number of records for you to hold in your head.
Do the line-by-line VAT amounts sum up to the aggregated amount in the filtered report? If not, you have a problem in your report and you need to dig deeper. If they do add up, your work isn’t done: add a second seller to the aggregated report’s filter and verify that both of them sum up, line by line, to the aggregate.
If you are creating a report with lots of records—perhaps a raw data report for user sales—grab 10 lines at random and manually check them against the live data in the database.
Are all the records exactly correct? Is there anything weird or unexpected? If so, you have some work to do to ensure correctness. If you just wrote the report, and you are running it for the first time, you will probably discover some things to fix with this technique. If you didn’t find any issues, sample another 10 random lines.
Consider creating a map for yourself and your data team.
Transactional databases can be very confusing, with records that look duplicative, but actually represent your application’s state. For instance, depending on how your system is implemented, you could end up with multiple “sale” records, each one representing a different phase of a sale. This is not ideal, but things like this can happen as a system evolves.
A naive data engineer may not know this, and could easily make the mistake of summing up all the sale records per user, without taking the state into account. This could lead to an inflated sales report.
Documenting your schema, including all tables and columns (and what they mean!) can go a long way toward preventing mistakes like this.
Alternatively, using a tool like dbt to create tightly controlled reporting views into your transactional data can all but eliminate errors that come from the conceptual mismatch between your transactional, stateful database, and a data warehouse used for reporting.
When we started to implement dbt at Bandcamp, we found that doing so was a shared responsibility between application developers who are experts at the transactional database, and data engineers who want to learn the schema that they are reporting from.
When writing SQL, it’s easy to end up with a 200-line behemoth that becomes tricky to reason about. It is important to make sure that the query you build to generate your report has a strong foundation.
Start with your primary record. Are you creating a sales report? If so, start by selecting your sale IDs, given your appropriate filtering criteria. If you get the set of IDs you expected, then start to join to other tables to get the data you need to report. Did the number of records returned change in an unexpected way? If so, you have a problem with your joins. Figure out whether you are joining tables that have missing data, look into whether there are more records in the joined table matching then you think there should be.
Building a little bit at a time is a great way to build a deep understanding about your data model, and it helps ensure that the reports you eventually create are correct.
If you are building a report or a dashboard, consider auditability and traceability.
When Bandcamp added VAT collection, it was obvious that the signals to determine the buyer’s country were opaque to users and administrators, so we had an internal page that explained all of the inputs to determine the buyer’s country, and we showed the calculated country for every input. Each individual calculation was trivially reproducible, so users would have confidence in the final computed country and VAT calculation.
In practice, this meant showing the country associated with the payment method, the geo-ip location, the buyer’s self-reported location, and other signals. This page also rendered the verdict: our most informed estimate of the country of purchase.
This page was used by our support team to build confidence in buyers that we were correctly calculating the VAT that we collected on the seller’s behalf.
When it comes to traceability, the tradeoffs come in terms of data storage and product effort. Because we anticipated users would have questions about VAT collection, we were able to build a system that stored each part of our VAT calculation, and exposed the audit trail to our support team. This requires more effort and storage than simply storing the final result, but when it comes to your users’ money, “trust me bro” is not an acceptable answer.
There’s one last way to build confidence: write your own SQL to validate your report’s correctness. Effectively, you are building a shadow report. If you can build it from a different system (say the report is generated from Snowflake, and your validation SQL is written against the OLTP system), you can have even greater confidence that if the final values match up, your report is correct and ready for consumption.
It is not possible to be 100% confident in the correctness of your reporting: you could have missing data, recording issues, incorrect queries, a bad data pipeline or any number of other confounding coincidences. But, if you ask these seven questions before handing your data off to your boss or your customers, you can be 99% sure you have it right.
This page is assembled with Pandoc, using the Computer Modern typeface. No AI was used to create the prose or ideas reflected here.
© 2026 Kevin Mooney