Synthetic Data Generation — Tabular Benchmark

A free, CPU-only, reproducible benchmark of open synthetic-data tools on three public enterprise-shaped datasets — measuring whether synthetic data is useful, safe, and rule-compliant enough to trust for analytics, ML augmentation, and safe data sharing.

Generated 2026-07-27T18:57:29+00:00 · pure function of results/registry.parquet · seeds vary per run, variance shown where ≥2 seeds

1The opportunity

Enterprise data is locked up by privacy, contractual, and regulatory walls. Synthetic data promises statistically faithful stand-ins that can move freely — for analytics, ML augmentation, and safe sharing. This benchmark tests that promise empirically, on our hardware, at zero licence cost.

2What we tested

Every tool fits the same train split, samples the same row count, and is scored by the same dual evaluator stack (SDMetrics + mostlyai-qa) across four metric families — fidelity, utility (TSTR), privacy, and business-rule compliance — with baselines that keep deep models honest.

3Headline results

Privacy–utility Pareto — the money chart

x = utility retained (TSTR AUC ÷ TRTR AUC, mean across datasets & seeds; 1.0 = synthetic trains as well as real). y = membership-inference attack AUC (0.5 = attacker guesses at chance; higher = privacy leak). The ideal corner is bottom-right. Points are labeled directly; color is a redundant cue.

Utility retained per dataset (TSTR gap)

TSTR = train on synthetic, test on real holdout; bars show retention vs the real-data ceiling (TRTR), mean ± std across seeds (tooltip). ≥0.95 excellent · ≥0.90 usable.

Tool scorecard (mean across datasets & seeds)

disc. AUC = can a classifier tell synthetic from real holdout (0.5 = indistinguishable). Exact match = synthetic rows identical to a training row. MIA = membership-inference attack AUC (0.5 = chance).

4Business vignette — fixing class imbalance (fraud)

Credit-card fraud is 0.17% of rows. Train a fraud model on real data alone and it misses most fraud. Augmenting the real training set with synthetic minority rows lifts recall — quantified below per tool.

LightGBM recall on the untouched real holdout (fraud class), before vs after augmenting real training data with synthetic fraud rows to class balance. Tooltip shows precision trade-off.

5The privacy dial — DP on the same model

The same MOSTLY AI ARGN model trained twice: once normally, once under differential privacy (DP-SGD). The delta is the visible price of a formal guarantee — the knob a regulator can be shown.

6Which tool for which job

7The accelerator we built

Everything above regenerates from one command. The harness (adapter-pattern synthesizers, config-driven experiments, append-only metric registry, this auto-report) is a reusable asset: adding a tool is one adapter file; adding a dataset is one catalog entry. Extending it to new modalities inherits all of it.

8Beyond tabular — future work

This benchmark covers single-table tabular data on CPU. The natural next steps reuse the same harness, registry, and reporting pipeline to go further:

  1. Larger models, full budgets: GPU-scale training (hosted safe-synthesizer services, full-epoch CTGAN/TVAE, diffusion) with no subsampling caps. This benchmark consumed of CPU for the grid — the same models are minutes on a GPU node.
  2. New modalities: relational / multi-table and time-series, then LLM-driven text and Q&A generation — the tracks this repository already scaffolds.
  3. Deeper privacy: a differential-privacy ε sweep to chart the full privacy–utility curve, plus stronger membership- and attribute-inference attacks.