syntropy.neural package

syntropy.neural.differential_entropy(idxs, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the differential entropy of the data.

Parameters:

• idxs (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• data (torch.Tensor) – The training data, in samples x features format.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Returns:

• float

• nflows.flows.base.Flow

Return type:

tuple[Tensor, float, Flow]

syntropy.neural.mutual_information(idxs_x, idxs_y, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the mutual information between the elements given by idxs_x and idxs_y.

Parameters:

• idxs_x (tuple[int, ...]) – The tuple of indices the x variable.

• idxs_y (tuple[int, ...]) – The tuple of indices for the y variable.

• data (torch.Tensor) – The training data, in samples x features format.

• context (None | tuple[int]) – If not None, the indices of the conditioning variables.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Return type:

float

syntropy.neural.total_correlation(idxs, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the total correlation of the data using normalizing flow estimators.

Parameters:

• idxs (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• data (torch.Tensor) – The training data, in samples x features format.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Returns:

• torch.Tensor

• float

Return type:

tuple[Tensor, float]

Submodules

syntropy.neural.multivariate_mi module

syntropy.neural.multivariate_mi.total_correlation(idxs, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the total correlation of the data using normalizing flow estimators.

Parameters:

• idxs (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• data (torch.Tensor) – The training data, in samples x features format.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Returns:

• torch.Tensor

• float

Return type:

tuple[Tensor, float]

syntropy.neural.multivariate_mi.higher_order_information(idxs, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the O-information, S-information, total correlation, and dual total correlation for the data. Computing them all as a set is more efficient than computing each one independently.

Parameters:

• idxs (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• data (torch.Tensor) – The training data, in samples x features format.

• context (None | tuple[int]) – If not None, the indices of the conditioning variables.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Return type:

dict[str, dict[str, float | torch.Tensor]]

syntropy.neural.shannon module

syntropy.neural.shannon.differential_entropy(idxs, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the differential entropy of the data.

Parameters:

• idxs (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• data (torch.Tensor) – The training data, in samples x features format.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Returns:

• float

• nflows.flows.base.Flow

Return type:

tuple[Tensor, float, Flow]

syntropy.neural.shannon.conditional_entropy(idxs_x, idxs_y, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the differential entropy of the data.

Parameters:

• idxs_x (tuple[int, ...]) – The tuple of indices the differential entropy is computed for.

• idxs_y (tuple[int, ...]) – The tuple of indices to condition on.

• data (torch.Tensor) – The training data, in samples x features format.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Returns:

• torch.Tensor

• float

• nflows.flows.base.Flow

Return type:

tuple[Tensor, float, Flow]

syntropy.neural.shannon.mutual_information(idxs_x, idxs_y, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the mutual information between the elements given by idxs_x and idxs_y.

Parameters:

• idxs_x (tuple[int, ...]) – The tuple of indices the x variable.

• idxs_y (tuple[int, ...]) – The tuple of indices for the y variable.

• data (torch.Tensor) – The training data, in samples x features format.

• context (None | tuple[int]) – If not None, the indices of the conditioning variables.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

Return type:

float

syntropy.neural.shannon.conditional_mutual_information(idxs_x, idxs_y, idxs_z, data, data_test=None, flow_kwargs=None, train_kwargs=None, verbose=False)

Computes the mutual information between the elements given by idxs_x and idxs_y.

Parameters:

• idxs_x (tuple[int, ...]) – The tuple of indices the x variable.

• idxs_y (tuple[int, ...]) – The tuple of indices for the y variable.

• data (torch.Tensor) – The training data, in samples x features format.

• context (None | tuple[int]) – If not None, the indices of the conditioning variables.

• data_test (None | torch.Tensor) – If not None, the testing data in samples x features format.

• flow_kwargs (dict) – Arguments for the utils.initalize_flow function.

• train_kwargs (dict) – Arguments for the utils.train_flow function.

• verbose (bool) – Whether to print the training progress.

• idxs_z (tuple[int, ...])

Return type:

float

syntropy.neural.utils module

syntropy.neural.utils.initialize_flow(dim, dim_context=0, num_layers=5, hidden_features=64, dropout_probability=0.1)

Initializes a new normalizing flow network.

Parameters:

• dim (int) – The number of input dimensions.

• dim_context (int) – The number of conditioning dimension. The default is zero.

• num_layers (int) – The number of hidden layers. The default is 5 layers.

• hidden_features (int) – The number of neurons in each hidden layer. The default is 64 neurons.

• dropout_probability (int) – The probability of a neuron dropping out.

Return type:

nflows.flows.base.Flow

syntropy.neural.utils.train_flow(flow, data, context=None, batch_size=256, lr=0.0001, num_epochs=100, weight_decay=1e-05, convergence_threshold=0.0, alpha=0.1, verbose=False)

Trains a normalizing flow network to approximate the maximally likely distribution to have generated the given data.

Parameters:

• flow (nflows.flows.base.Flow) – An untrained normalizing flow network.

• data (torch.Tensor) – The training data, in samples x features format.

• context (None | torch.Tensor) – Conditioning random variables. Default is None.

• batch_size (int) – The size of each batch. The default value is 256.

• lr (float) – The learning rate. The default value is 1e-4

• num_epochs (int) – The number of training epochs. The default value is 100,

• weight_decay (float) – The rate at which parameter weights decay. A regularizer to reduce over-fitting. The default is 1e-5.

• convergence_threshold (float) – The value of the coefficient of variation below which the training terminates. The default value is 0.0, which means the training will not stop before num_epochs is hit.

• alpha (float) – How quickly the exponentially weighted moving standard deviation downweights older data. The default is 0.2.

• verbose (bool) – Whether to print the loss for each epoch throughout training. The default is False.

Return type:

nflows.flows.base.Flow

syntropy.neural.utils.evaluate_flow(flow, data, context=None)

Evaluates a trainied normalizing flow network on the given data.

Parameters:

• flow (nflows.flows.base.Flow) – A trained normalilzing flow network.

• data (torch.Tensor) – The testing data in samples x features format.

• context (None | torch.Tensor)

Returns:

The average entropy (in nat) The standard error of the estimate.

Return type:

tuple[float, float]