This release brings user friendly improvements and bug fixes on the SDV constraints, to help
users generate their synthetic data easily.
Some predefined constraints have been renamed and redefined to be more user friendly & consistent.
The custom constraint API has also been updated for usability. The SDV now automatically determines
the best handling_strategy to use for each constraint, attempting transform by default and
falling back to reject_sampling otherwise. The handling_strategy parameters are no longer
included in the API.
Finally, this version of SDV also unifies the parameters for all sampling related methods for
all models (including TabularPreset).
-
GreatherThanconstraint is now separated in two new constraints:Inequality, which is intended to be used between two columns, andScalarInequality, which is intended to be used between a column and a scalar. -
Betweenconstraint is now separated in two new constraints:Range, which is intended to be used between three columns, andScalarRange, which is intended to be used between a column and low and high scalar values. -
FixedIncrementsa new constraint that makes the data increment by a certain value. -
New
create_custom_constraintfunction available to create custom constraints.
RoundingRounding is automatically being handled by therdt.HyperTransformer.ColumnFormulathecreate_custom_constrainttakes place over this one and allows more advanced usage for the end users.
- Improve error message for invalid constraints - Issue #801 by @fealho
- Numerical Instability in Constrained GaussianCopula - Issue #806 by @fealho
- Unify sampling params for reject sampling - Issue #809 by @amontanez24
- Split
GreaterThanconstraint intoInequalityandScalarInequality- Issue #814 by @fealho - Split
Betweenconstraint intoRangeandScalarRange- Issue #815 @pvk-developer - Change
columnstocolumn_namesinOneHotEncodingandUniqueconstraints - Issue #816 by @amontanez24 - Update columns parameter in
PositiveandNegativeconstraint - Issue #817 by @fealho - Create
FixedIncrementsconstraint - Issue #818 by @amontanez24 - Improve datetime handling in
ScalarInequalityandScalarRangeconstraints - Issue #819 by @pvk-developer - Support strict boundaries even when transform strategy is used - Issue #820 by @fealho
- Add
create_custom_constraintfactory method - Issue #836 by @fealho
- Remove
handling_strategyparameter - Issue #833 by @amontanez24 - Remove
fit_columns_modelparameter - Issue #834 by @pvk-developer - Remove the
ColumnFormulaconstraint - Issue #837 by @amontanez24 - Move
table_data.copyto base class of constraints - Issue #845 by @fealho
- Numerical Instability in Constrained GaussianCopula - Issue #801 by @tlranda and @fealho
- Fix error message for
FixedIncrements- Issue #865 by @pvk-developer - Fix constraints with conditional sampling - Issue #866 by @amontanez24
- Fix error message in
ScalarInequality- Issue #868 by @pvk-developer - Cannot use
max_tries_per_batchon sample:TypeError: sample() got an unexpected keyword argument 'max_tries_per_batch'- Issue #885 by @amontanez24 - Conditional sampling + batch size:
ValueError: Length of values (1) does not match length of index (5)- Issue #886 by @amontanez24 TabularPresetdoesn't support new sampling parameters - Issue #887 by @fealho- Conditional Sampling:
batch_sizeis being set toNoneby default? - Issue #889 by @amontanez24 - Conditional sampling using GaussianCopula inefficient when categories are noised - Issue #910 by @amontanez24
- Show the
APIforTabularPresetmodels - Issue #854 by @katxiao - Update handling constraints doc - Pull Request #856 by @amontanez24
- Update custom costraints documentation - Pull Request #857 by @pvk-developer
This release improves the speed of the GaussianCopula model by removing logic that previously searched for the appropriate distribution to
use. It also fixes a bug that was happening when conditional sampling was used with the TabularPreset.
The rest of the release focuses on making changes to improve constraints including changing the UniqueCombinations constraint to FixedCombinations,
making the Unique constraint work with missing values and erroring when null values are seen in the OneHotEncoding constraint.
- Silence warnings coming from univariate fit in copulas - Issue #769 by @pvk-developer
- Remove parameters related to distribution search and change default - Issue #767 by @fealho
- Update the UniqueCombinations constraint - Issue #793 by @fealho
- Make Unique constraint works with nans - Issue #797 by @fealho
- Error out if nans in OneHotEncoding - Issue #800 by @amontanez24
- Unable to sample conditionally in Tabular_Preset model - Issue #796 by @katxiao
- Support GPU computing and progress track? - Issue #478 by @fealho
This release adds a TabularPreset, available in the sdv.lite module, which allows users to easily optimize a tabular model for speed.
In this release, we also include bug fixes for sampling with conditions, an unresolved warning, and setting field distributions. Finally,
we include documentation updates for sampling and the new TabularPreset.
- Sampling with conditions={column: 0.0} for float columns doesn't work - Issue #525 by @shlomihod and @tssbas
- resolved FutureWarning with Pandas replaced append by concat - Issue #759 by @Deathn0t
- Field distributions bug in CopulaGAN - Issue #747 by @katxiao
- Field distributions bug in GaussianCopula - Issue #746 by @katxiao
- Set default transformer to categorical_fuzzy - Issue #768 by @amontanez24
- Model nulls normally when tabular preset has constraints - Issue #764 by @katxiao
- Don't modify my metadata object - Issue #754 by @amontanez24
- Presets should be able to handle constraints - Issue #753 by @katxiao
- Change preset optimize_for --> name - Issue #749 by @katxiao
- Create a speed optimized Preset - Issue #716 by @katxiao
- Add tabular preset docs - Issue #777 by @katxiao
- sdv.sampling module is missing from the API - Issue #740 by @katxiao
This release updates the sampling API and splits the existing functionality into three methods - sample, sample_conditions,
and sample_remaining_columns. We also add support for sampling in batches, displaying a progress bar when sampling with more than one batch,
sampling deterministically, and writing the sampled results to an output file. Finally, we include fixes for sampling with conditions
and updates to the documentation.
- Fix write to file in sampling - Issue #732 by @katxiao
- Conditional sampling doesn't work if the model has a CustomConstraint - Issue #696 by @katxiao
- Updates to GaussianCopula conditional sampling methods - Issue #729 by @katxiao
- Update conditional sampling errors - Issue #730 by @katxiao
- Enable Batch Sampling + Progress Bar - Issue #693 by @katxiao
- Create sample_remaining_columns() method - Issue #692 by @katxiao
- Create sample_conditions() method - Issue #691 by @katxiao
- Improve sample() method - Issue #690 by @katxiao
- Create Condition object - Issue #689 by @katxiao
- Is it possible to generate data with new set of primary keys? - Issue #686 by @katxiao
- No way to fix the random seed? - Issue #157 by @katxiao
- Can you set a random state for the sdv.tabular.ctgan.CTGAN.sample method? - Issue #515 by @katxiao
- generating different synthetic data while training the model multiple times. - Issue #299 by @katxiao
- Typo in the document documentation - Issue #680 by @katxiao
This release adds support for passing tabular constraints to the HMA1 model, and adds more explicit error handling for metric evaluation. It also includes a fix for using categorical columns in the PAR model and documentation updates for metadata and HMA1.
- Categorical column after sequence_index column - Issue #314 by @fealho
- Support passing tabular constraints to the HMA1 model - Issue #296 by @katxiao
- Metric evaluation error handling metrics - Issue #638 by @katxiao
- Make true/false values lowercase in Metadata Schema specification - Issue #664 by @katxiao
- Update docstrings for hma1 methods - Issue #642 by @katxiao
This release makes multiple improvements to different Constraint classes. The Unique constraint can now
handle columns with the name index and no longer crashes on subsets of the original data. The Between
constraint can now handle columns with nulls properly. The memory of all constraints was also improved.
Various other features and fixes were added. Conditional sampling no longer crashes when the num_rows argument
is not provided. Multiple localizations can now be used for PII fields. Scaffolding for integration tests was added
and the workflows now run pip check.
Additionally, this release adds support for Python 3.9!
- Gaussian Copula – Memory Issue in Release 0.10.0 - Issue #459 by @xamm
- Applying Unique Constraint errors when calling model.fit() on a subset of data - Issue #610 by @xamm
- Calling sampling with conditions and without num_rows crashes - Issue #614 by @xamm
- Metadata.visualize with path parameter throws AttributeError - Issue #634 by @xamm
- The Unique constraint crashes when the data contains a column called index - Issue #616 by @xamm
- The Unique constraint cannot handle non-default index - Issue #617 by @xamm
- ConstraintsNotMetError when applying Between constraint on datetime columns containing null values - Issue #632 by @katxiao
- Adds Multi localisations feature for PII fields defined in #308 - PR #609 by @xamm
- Support latest version of Faker - Issue #621 by @katxiao
- Add scaffolding for Metadata integration tests - Issue #624 by @katxiao
- Add support for Python 3.9 - Issue #631 by @amontanez24
- Add pip check to CI workflows - Issue #626 by @pvk-developer
- Anonymizing PII in single table tutorials states address field as e-mail type - Issue #604 by @xamm
Special thanks to @xamm, @katxiao, @pvk-developer and @amontanez24 for all the work that made this release possible!
This release fixes bugs in constraints, metadata behavior, and SDV documentation. Specifically, we added proper handling of data containing null values for constraints and timeseries data, and updated the default metadata detection behavior.
- ValueError: The parameter loc has invalid values - Issue #353 by @fealho
- Gaussian Copula is generating different data with metadata and without metadata - Issue #576 by @katxiao
- Make pomegranate an optional dependency - Issue #567 by @katxiao
- Small wording change for Question Issue Template - Issue #571 by @katxiao
- ConstraintsNotMetError when using GreaterThan constraint with datetime - Issue #590 by @katxiao
- GreaterThan constraint crashing with NaN values - Issue #592 by @katxiao
- Null values in GreaterThan constraint raises error - Issue #589 by @katxiao
- ColumnFormula raises ConstraintsNotMetError when checking NaN values - Issue #593 by @katxiao
- GreaterThan constraint raises TypeError when using datetime - Issue #596 by @katxiao
- Fix repository language - Issue #464 by @fealho
- Update init.py - Issue #578 by @dyuliu
- IndexingError: Unalignable boolean - Issue #446 by @fealho
This release focuses on improving and expanding upon the existing constraints. More specifically, the users can now
(1) specify multiple columns in Positive and Negative constraints, (2) use the new Uniqueconstraint and
(3) use datetime data with the Between constraint. Additionaly, error messages have been added and updated
to provide more useful feedback to the user.
Besides the added features, several bugs regarding the UniqueCombinations and ColumnFormula constraints have been fixed,
and an error in the metadata.json for the student_placements dataset was corrected. The release also added documentation
for the fit_columns_model which affects the majority of the available constraints.
- Change default fit_columns_model to False - Issue #550 by @katxiao
- Support multi-column specification for positive and negative constraint - Issue #545 by @sarahmish
- Raise error when multiple constraints can't be enforced - Issue #541 by @amontanez24
- Create Unique Constraint - Issue #532 by @amontanez24
- Passing invalid conditions when using constraints produces unreadable errors - Issue #511 by @katxiao
- Improve error message for ColumnFormula constraint when constraint column used in formula - Issue #508 by @katxiao
- Add datetime functionality to Between constraint - Issue #504 by @katxiao
- UniqueCombinations constraint with handling_strategy = 'transform' yields synthetic data with nan values - Issue #521 by @katxiao and @csala
- UniqueCombinations constraint outputting wrong data type - Issue #510 by @katxiao and @csala
- UniqueCombinations constraint on only one column gets stuck in an infinite loop - Issue #509 by @katxiao
- Conditioning on a non-constraint column using the ColumnFormula constraint - Issue #507 by @katxiao
- Conditioning on the constraint column of the ColumnFormula constraint - Issue #506 by @katxiao
- Update metadata.json for duration of student_placements dataset - Issue #503 by @amontanez24
- Unit test for HMA1 when working with a single child row per parent row - Issue #497 by @pvk-developer
- UniqueCombinations constraint for more than 2 columns - Issue #494 by @katxiao and @csala
- Add explanation of fit_columns_model to API docs - Issue #517 by @katxiao
This release primarily addresses bugs and feature requests related to using constraints for the single-table models.
Users can now enforce scalar comparison with the existing GreaterThan constraint and apply 5 new constraints: OneHotEncoding, Positive, Negative, Between and Rounding.
Additionally, the SDV will now auto-apply constraints for rounding numerical values, and for keeping the data within the observed bounds.
All related user guides are updated with the new functionality.
- Add OneHotEncoding Constraint - Issue #303 by @fealho
- GreaterThan Constraint should apply to scalars - Issue #410 by @amontanez24
- Improve GreaterThan constraint - Issue #368 by @amontanez24
- Add Non-negative and Positive constraints across multiple columns- Issue #409 by @amontanez24
- Add Between values constraint - Issue #367 by @fealho
- Ensure values fall within the specified range - Issue #423 by @amontanez24
- Add Rounding constraint - Issue #482 by @katxiao
- Add rounding and min/max arguments that are passed down to the NumericalTransformer - Issue #491 by @amontanez24
- GreaterThan constraint between Date columns rasises TypeError - Issue #421 by @amontanez24
- GreaterThan constraint's transform strategy fails on columns that are not float - Issue #448 by @amontanez24
- AttributeError on UniqueCombinations constraint with non-strings - Issue #196 by @katxiao
- Use reject sampling to sample missing columns for constraints - Issue #435 by @amontanez24
- Ensure privacy metrics are available in the API docs - Issue #458 by @fealho
- Ensure forumla constraint is called ColumnFormula everywhere in the docs - Issue #449 by @fealho
This release changes the way we sample conditions to not only group by the conditions passed by the user, but also by the transformed conditions that result from them.
- Conditionally sampling on variable in constraint should have variety for other variables - Issue #440 by @amontanez24
This release improves the constraint functionality by allowing constraints and conditions at the same time. Additional changes were made to update tutorials.
- Not able to use constraints and conditions in the same time - Issue #379 by @amontanez24
- Update benchmarking user guide for reading private datasets - Issue #427 by @katxiao
This release broadens the constraint functionality by allowing for the ColumnFormula
constraint to take lambda functions and returned functions as an input for its formula.
It also improves conditional sampling by ensuring that any id fields generated by the
model remain unique throughout the sampled data.
The CTGAN model was improved by adjusting a default parameter to be more mathematically
correct.
Additional changes were made to improve tutorials as well as fix fragile tests.
- Tutorials test sometimes fails - Issue #355 by @fealho
- Duplicate IDs when using reject-sampling - Issue #331 by @amontanez24 and @csala
- discriminator_decay should be initialized at 1e-6 but it's 0 - Issue #401 by @fealho and @YoucefZemmouri
- Tutorial typo - Issue #380 by @fealho
- Request for sdv.constraint.ColumnFormula for a wider range of function - Issue #373 by @amontanez24 and @JetfiRex
This release brings new privacy metrics to the evaluate framework which help to determine
if the real data could be obtained or deduced from the synthetic samples.
Additionally, now there is a normalized score for the metrics, which stays between 0 and 1.
There are improvements that reduce the usage of memory ram when sampling new data. Also there
is a new parameter to control the reject sampling crash, graceful_reject_sampling, which if
set to True and if it's not possible to generate all the requested rows, it will just issue a
warning and return whatever it was able to generate.
The Metadata object can now be visualized using different combinations of names and details,
which can be set to True or False in order to display only the table names with details or
without. There is also an improvement on the validation, which now will display all the errors
found at the end of the validation instead of only the first one.
This version also exposes all the hyperparameters of the models CTGAN and TVAE to allow a more
advanced usage. There is also a fix for the TVAE model on small datasets and it's performance
with NaN values has been improved. There is a fix for when using
UniqueCombinationConstraint with the transform strategy.
- Memory Usage Gaussian Copula Trained Model consuming high memory when generating synthetic data - Issue #304 by @pvk-developer and @AnupamaGangadhar
- Add option to visualize metadata with only table names - Issue #347 by @csala
- Add sample parameter to control reject sampling crash - Issue #343 by @fealho
- Verbose metadata validation - Issue #348 by @csala
- Missing the introduction of custom specification for hyperparameters in the TVAE model - Issue #344 by @imkhoa99 and @pvk-developer
This version adds conditional sampling for tabular models by combining a reject-sampling strategy with the native conditional sampling capabilities from the gaussian copulas.
It also introduces several upgrades on the HMA1 algorithm that improve data quality and robustness in the multi-table scenarios by making changes in how the parameters of the child tables are aggregated on the parent tables, including a complete rework of how the correlation matrices are modeled and rebuild after sampling.
- Fix probabilities contain NaN error - Issue #326 by @csala
- Conditional Sampling for tabular models - Issue #316 by @fealho and @csala
- HMA1: LinAlgError: SVD did not converge - Issue #240 by @csala
This release introduces a few changes in the HMA1 relational algorithm to decrease modeling and sampling times, while also ensuring that correlations are properly kept across tables and also adding support for some relational schemas that were not supported before.
A few changes in constraints and tabular models also ensure that situations that produced errors before now work without errors.
- Fix unique key generation - Issue #306 by @fealho
- Ensure tables that contain nothing but ids can be modeled - Issue #302 by @csala
- Metadata visualization improvements - Issue #301 by @csala
- Multi-parent re-model and re-sample issue - Issue #298 by @csala
- Support datetimes in GreaterThan constraint - Issue #266 by @rollervan
- Support for multiple foreign keys in one table - Issue #185 by @csala
SDMetrics version is updated to include the new Time Series metrics, which have also been added to the API Reference and User Guides documentation. Additionally, a few code has been refactored to reduce external dependencies and a few minor bugs related to single table constraints have been fixed
- Add timeseries metrics and user guides - Issue #289 by @csala
- Add functions to generate regex ids - Issue #288 by @csala
- Saving a fitted tabular model with UniqueCombinations constraint raises PicklingError - Issue #286 by @csala
- Constraints:
handling_strategy='reject_sampling'causes'ZeroDivisionError: division by zero'- Issue #285 by @csala
This release updates to the latest CTGAN, RDT and SDMetrics libraries to introduce a new TVAE model, multiple new metrics for single table and multi table, and fixes issues in the re-creation of tabular models from a metadata dict.
- Upgrade to SDMetrics v0.1.0 and add
sdv.metricsmodule - Issue #281 by @csala - Upgrade to CTGAN 0.3.0 and add TVAE model - Issue #278 by @fealho
- Add
dtype_transformerstoTable.from_dict- Issue #276 by @csala - Fix Metadata
from_dictbehavior - Issue #275 by @csala
This version updates the dependencies and makes a few internal changes in order to ensure that SDV works properly on Windows Systems, making this the first release to be officially supported on Windows.
Apart from this, some more internal changes have been made to solve a few minor issues from the older versions while also improving the processing speed when processing relational datasets with the default parameters.
- The
distributionargument of theGaussianCopulahas been renamed tofield_distributions. - The
HMA1andSDVclasses now use thecategorical_fuzzytransformer by default instead of theone_hot_encodingone.
- GaussianCopula: rename
distributionargument tofield_distributions- Issue #237 by @csala - GaussianCopula: Improve error message if an invalid distribution name is passed - Issue #220 by csala
- Import urllib.request explicitly - Issue #227 by @csala
- TypeError: cannot astype a datetimelike from [datetime64[ns]] to [int32] - Issue #218 by @csala
- Change default categorical transformer to
categorical_fuzzyin HMA1 - Issue #214 by @csala - Integer categoricals being sampled as strings instead of integer values - Issue #194 by @csala
In this version a new family of models for Synthetic Time Series Generation is introduced
under the sdv.timeseries sub-package. The new family of models now includes a new class
called PAR, which implements a Probabilistic AutoRegressive model.
This version also adds support for composite primary keys and regex based generation of id fields in tabular models and drops Python 3.5 support.
- Drop python 3.5 support - Issue #204 by @csala
- Support composite primary keys in tabular models - Issue #207 by @csala
- Add the option to generate string
idfields based on regex on tabular models - Issue #208 by @csala - Synthetic Time Series - Issue #142 by @csala
This version adds a new tabular model based on combining the CTGAN model with the reversible transformation applied in the GaussianCopula model that converts random variables with arbitrary distributions to new random variables with standard normal distribution.
The reversible transformation is handled by the GaussianCopulaTransformer recently added to RDT.
- Add CopulaGAN Model - Issue #202 by @csala
This release moves the models and algorithms related to generation of synthetic
relational data to a new sdv.relational subpackage (Issue #198)
As part of the change, also the old sdv.models have been removed and now
relational model is based on the recently introduced sdv.tabular models.
In this release the sdv.evaluation module has been reworked to include 4 different
metrics and in all cases return a normalized score between 0 and 1.
Included metrics are:
cstestkstestlogistic_detectionsvc_detection
This release fixes a couple of minor issues and introduces an important rework of the User Guides section of the documentation.
- Error Message: "make sure the Graphviz executables are on your systems' PATH" - Issue #182 by @csala
- Anonymization mappings leak - Issue #187 by @csala
In this release SDV gets new documentation, new tutorials, improvements to the Tabular API and broader python and dependency support.
Complete list of changes:
- New Documentation site based on the
pydata-sphinx-theme. - New User Guides and Notebook tutorials.
- New Developer Guides section within the docs with details about the SDV architecture, the ecosystem libraries and how to extend and contribute to the project.
- Improved API for the Tabular models with focus on ease of use.
- Support for Python 3.8 and the newest versions of pandas, scipy and scikit-learn.
- New Slack Workspace for development discussions and community support.
This release introduces a new concept of Constraints, which allow the user to define
special relationships between columns that will not be handled via modeling.
This is done via a new sdv.constraints subpackage which defines some well-known pre-defined
constraints, as well as a generic framework that allows the user to customize the constraints
to their needs as much as necessary.
- Support for Constraints - Issue #169 by @csala
This release introduces a new subpackage sdv.tabular with models designed specifically
for single table modeling, while still providing all the usual conveniences from SDV, such
as:
- Seamless multi-type support
- Missing data handling
- PII anonymization
Currently implemented models are:
- GaussianCopula: Multivariate distributions modeled using copula functions. This is stronger version, with more marginal distributions and options, than the one used to model multi-table datasets.
- CTGAN: GAN-based data synthesizer that can generate synthetic tabular data with high fidelity.
- Support for Multiple Parents - Issue #162 by @csala
- Sample by default the same number of rows as in the original table - Issue #163 by @csala
- Add benchmark - Issue #165 by @csala
- Use SDMetrics for evaluation - Issue #159 by @csala
- Improve metadata visualization - Issue #151 by @csala @JDTheRipperPC
-
Add Metadata Validation - Issue #134 by @csala @JDTheRipperPC
-
Add Metadata Visualization - Issue #135 by @JDTheRipperPC
-
Add path to metadata JSON - Issue #143 by @JDTheRipperPC
-
Use new Copulas and RDT versions - Issue #147 by @csala @JDTheRipperPC
- Create sdv.models subpackage - Issue #141 by @JDTheRipperPC
-
Adapt evaluation to the different data types - Issue #128 by @csala @JDTheRipperPC
-
Extend
load_demofunctionality to load other datasets - Issue #136 by @JDTheRipperPC
- Methods to generate Metadata from DataFrames - Issue #126 by @csala @JDTheRipperPC
- compatibility with rdt issue 72 - Issue #120 by @csala @JDTheRipperPC
- Error docstring sampler.__fill_text_columns - Issue #144 by @JDTheRipperPC
- Reach 90% coverage - Issue #112 by @JDTheRipperPC
- Review unittests - Issue #111 by @JDTheRipperPC
- Time required for sample_all function? - Issue #118 by @csala @JDTheRipperPC
- Add option to model the amount of child rows - Issue 93 by @ManuelAlvarezC
-
Add Evaluation Metrics - Issue 52 by @ManuelAlvarezC
-
Ensure unicity on primary keys on different calls - Issue 63 by @ManuelAlvarezC
- executing readme: 'not supported between instances of 'int' and 'NoneType' - Issue 104 by @csala
- Add warnings when trying to model an unsupported dataset structure. GH#73
- Add option to anonymize data. GH#51
- Add support for modeling data with different distributions, when using
GaussianMultivariatemodel. GH#68 - Add support for
VineCopulasas a model. GH#71 - Improve
GaussianMultivariateparameter sampling, avoiding warnings and unvalid parameters. GH#58 - Fix issue that caused that sampled categorical values sometimes got numerical values mixed. GH#81
- Improve the validation of extensions. GH#69
- Update examples. GH#61
- Replaced
Tableclass with aNamedTuple. GH#92 - Fix inconsistent dependencies and add upper bound to dependencies. GH#96
- Fix error when merging extension in
Modeler.CPAwhen running examples. GH#86
- First release on PyPI.