syntropy.discrete package

syntropy.discrete.shannon_entropy(joint_distribution)

Computes the Shannon entropy of the distribution \(P(x)\).

\[H(X) = -\sum_{x\in\mathcal{X}} P(x) \log P(x)\]

To compute the entropy of a subset of the variables in the joint distribution, use the syntropy.discrete.utils.get_marginals() function from the utils library.

Parameters:

joint_distribution (dict) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict) – The pointwise entropy for each state in the joint distribution.

• avg (float) – The average entropy

Return type:

tuple[dict, float]

syntropy.discrete.conditional_entropy(idxs_x, idxs_y, joint_distribution)

Computes the conditional entropy of X given Y.

\[H(X|Y) = H(X,Y) - H(Y)\]

Parameters:

• idxs_x (tuple) – The indices of the variables to compute the entropy on.

• idxs_y (tuple) – The indicies of the variables to contintue on.

• joint_distribution (dict) –

  DESCRIPTION.joint_distributiondict

  The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict) – The pointwise entropy for each state in the joint distribution.

• avg (float) – The average entropy

Return type:

tuple[dict, float]

syntropy.discrete.mutual_information(idxs_x, idxs_y, joint_distribution)

Computes the mutual information between X and Y.

\[\begin{split}I(X;Y) &= H(X) + H(Y) - H(X,Y) \\ &= H(X) - H(X|Y) \\ &= H(Y) - H(Y|X) \\ &= H(X,Y) - H(X|Y) - H(Y|X)\end{split}\]

Parameters:

• idxs_x (tuple) – The indices of the X variable(s).

• idxs_y (tuple) – The indices of the Y variable(s).

• joint_distribution (dict) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict) – The pointwise mutual information for each state in the joint distribution.

• avg (float) – The average mutual information

Return type:

tuple[dict, float]

syntropy.discrete.conditional_mutual_information(idxs_x, idxs_y, idxs_z, joint_distribution)

Computes the mutual information between X and Y condioned on Z.

\[\begin{split}I(X,Y|Z) &= H(X|Z) + H(Y|Z) - H(X,Y|Z) \\ &= I(X;Y,Z) - I(X;Z)\end{split}\]

Parameters:

• idxs_x (tuple) – The indices of the X variable(s).

• idxs_y (tuple) – The indices of the Y variable(s).

• idxs_z (tuple) – The indices of the variables to condition on.

• joint_distribution (dict) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict) – The pointwise mutual information for each state in the joint distribution.

• avg (float) – The average mutual information

Return type:

tuple[dict, float]

syntropy.discrete.kullback_leibler_divergence(posterior_distribution, prior_distribution)

Computes the Kullback-Leibler divergence from a prior distribution P(X) and and posterior distribution Q(X).

\[D_{KL}(P||Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}\]

Parameters:

• posterior_distribution (dict) – The joint distribution of the posterior distribution P(X).

• prior_distribution (dict) – The joint distribution of the prior distribution Q(X)

Returns:

• ptw (dict) – The pointwise Kullback-Leibler divergence for each state in the joint distribution.

• avg (float) – The average Kullback-Leibler divergence.

Return type:

tuple[dict, float]

syntropy.discrete.total_correlation(joint_distribution)

Computes the average and pointwise total correlations:

\[\begin{split}TC(X) &= D_{KL}(P(X) || \prod_{i=1}^{N}P(X_i) \\ &= \sum_{i=1}^{N}H(X_i) - H(X)\end{split}\]

Parameters:

joint_distribution (dict[tuple[int, ...]], float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise TC .

• avg (float) – The average TC.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Watanabe, S. (1960). Information Theoretical Analysis of Multivariate Correlation. IBM Journal of Research and Development, 4(1), Article 1. https://doi.org/10.1147/rd.41.0066

Tononi, G., Sporns, O., & Edelman, G. M. (1994). A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proceedings of the National Academy of Sciences, 91(11), Article 11. https://doi.org/10.1073/pnas.91.11.5033

syntropy.discrete.dual_total_correlation(joint_distribution)

Computes the local and expected dual total correlations for the joint distribution.

\[\begin{split}DTC(X) &= H(X) - \sum_{i=1}^{N}H(X_i|X^{-i}) \\ &= (N-1)\times TC(X) - \sum_{i=1}^{N}TC(X^{-i})\end{split}\]

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise DTC .

• avg (float) – The average DTC.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Abdallah, S. A., & Plumbley, M. D. (2012). A measure of statistical complexity based on predictive information with application to finite spin systems. Physics Letters A, 376(4), 275–281. https://doi.org/10.1016/j.physleta.2011.10.066

Rosas, F., Mediano, P. A. M., Gastpar, M., & Jensen, H. J. (2019). Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information. Physical Review E, 100(3), Article 3. https://doi.org/10.1103/PhysRevE.100.032305

syntropy.discrete.s_information(joint_distribution)

Computes the local and expected S-information for the joint distribution.

\[\begin{split}\Sigma(X) &= \sum_{i=1}^{N}I(X_i;X^{-i}) \\ &= N\times TC(X) - \sum_{i=1}^{N}TC(X^{-i}) \\ &= TC(X) + DTC(X)\end{split}\]

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise S-information .

• avg (float) – The average S-information.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Rosas, F., Mediano, P. A. M., Gastpar, M., & Jensen, H. J. (2019). Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information. Physical Review E, 100(3), Article 3. https://doi.org/10.1103/PhysRevE.100.032305

Varley, T. F., Pope, M., Faskowitz, J., & Sporns, O. (2023). Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex. Communications Biology, 6(1), Article 1. https://doi.org/10.1038/s42003-023-04843-w

syntropy.discrete.o_information(joint_distribution)

Computes the local and expected O-informations for the joint distribution. O-information quantifies the balance between redundancy (positive values) and synergy (negative values) in multivariate information.

\[\begin{split}\Omega(X) &= (2-N)TC(X) + \sum_{i=1}^{N}TC(X^{-i}) \\ &= TC(X) - DTC(X)\end{split}\]

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise O-information .

• avg (float) – The average O-information.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Rosas, F., Mediano, P. A. M., Gastpar, M., & Jensen, H. J. (2019). Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information. Physical Review E, 100(3), Article 3. https://doi.org/10.1103/PhysRevE.100.032305

Varley, T. F., Pope, M., Faskowitz, J., & Sporns, O. (2023). Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex. Communications Biology, 6(1), Article 1. https://doi.org/10.1038/s42003-023-04843-w

syntropy.discrete.tse_complexity(joint_distribution, num_samples)

The Tononi-Sporns-Edelman neural complexity measure, which provides a measure of the balance between integration and segregation across scales.

\[\begin{split}TSE(X) &= \sum_{k=1}^{\lfloor N/2\rfloor} \bigg\langle I(X^{k}_j;X^{-k}_j) \bigg\rangle_{j} \\ &= \sum_{k=2}^{N}\bigg[\bigg(\frac{k}{N}\bigg)TC(X) - \langle TC(X^{k}_{j}) \rangle_{j} \bigg]\end{split}\]

Runtimes scale very badly with system size (as it requires brute-forcing) all possible bipartitions of the system. If the system is too large, a sub-sampling approach is taken: at each scale, num_samples are drawn from the space of bipartitions.

Parameters:

• joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

• num_samples (int) – The number of samples to do for each subset size..

Returns:

The TSE complexity. No local complexity is computed. .

Return type:

float

References

Tononi, G., Sporns, O., & Edelman, G. M. (1994). A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proceedings of the National Academy of Sciences, 91(11), Article 11. https://doi.org/10.1073/pnas.91.11.5033

Varley, T. F., Pope, M., Faskowitz, J., & Sporns, O. (2023). Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex. Communications Biology, 6(1), Article 1. https://doi.org/10.1038/s42003-023-04843-w

syntropy.discrete.description_complexity(joint_distribution)

The description complexity was proposed by Tononi and Sporns as a heuristic, easy-to-compute approximation of the full TSE-Complexity. Later shown by Varley et al., to be directly proportional to the dual total correlation.

\[C(X) = \frac{DTC(X)}{N}\]

Where \(N\) is the number of elements in \(X\).

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise description complexity.

• avg (float) – The average description complexity.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Varley, T. F., Pope, M., Faskowitz, J., & Sporns, O. (2023). Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex. Communications Biology, 6(1), Article 1. https://doi.org/10.1038/s42003-023-04843-w

syntropy.discrete.co_information(joint_distribution)

Computes the Co-information, the third generalization of bivariate mutual information. Unlike total correlation and dual total correlation, the cO-information can be negative and is difficult to interpret.

\[Co(X) = \sum_{\xi\subseteq X}(-1)^{|\xi|}H(\xi)\]

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

• ptw (dict[tuple, float]) – The pointwise cO-information.

• avg (float) – The average cO-information.

Return type:

(tuple[dict, float], <class ‘float’>)

References

Bell, A. J. (2003, April). The Co-information lattice. 4th International Symposium on Independent Component Analysis and Blind Signal Separation, Nara, Japan. https://www.semanticscholar.org/paper/THE-CO-INFORMATION-LATTICE-Bell/25a0cd8d486d5ffd204485685226f189e6eadd4d

syntropy.discrete.connected_information(joint_distribution, maximum_order=-1)

Returns the connected information profile from Schneidman et al., which decomposes the total correlation into contributing parts of different orders:

\[TC(X) = \sum_{k=2}^{N}TC^{k}(X)\]

Where the k superscript refers to the maximum-entropy distribution that preserves all marginals of order k.

One of the few measures that can reliably distinguish between the JAMES_DYADIC and JAMES_TRIADIC distributions.

Parameters:

• joint_distribution (dict[tuple[int, ...], float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

• maximum_order (int, optional) – The highest order of marginals to sweep. The default sweeps all.

Returns:

The connected information profile.

Return type:

list[float]

References

Schneidman, E., Still, S., Berry, M. J., & Bialek, W. (2003). Network Information and Connected Correlations. Physical Review Letters, 91(23), 238701. https://doi.org/10.1103/PhysRevLett.91.238701

syntropy.discrete.partial_entropy_decomposition(joint_distribution, redundancy_function)

Computes the partial entropy decomposition of a joint distribution with up to four elements.

The available redundancy functions are h_min from Finn and Lizier and h_sx from Varley et al.,

\[\begin{split}h_{\min}(\alpha) &= \min(\alpha_i) \\ h_{sx}(\alpha) &= \log\frac{1}{P(\alpha_1\cup ... \cup\alpha_N)}\end{split}\]

Parameters:

• joint_distribution (DiscreteDist) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

• redundancy_function (str) – The localizable redundancy function. Options are: hmin and hsx.

Returns:

• ptw (dict[tuple, dict]) – A dictionary of dictionaries, The outer dictionary has one key for each joint state. Each inner dictionary is the lookup of partial entropy atoms.

• avg (dict[tuple, float]) – The expected value for each partial entropy atom.

Return type:

tuple[dict[tuple[int, …], dict[tuple[tuple[int, …], …], float]], dict[tuple[tuple[int, …], …], float]]

References

Finn, C., & Lizier, J. T. (2020). Generalised Measures of Multivariate Information Content. Entropy, 22(2), Article 2. https://doi.org/10.3390/e22020216

Varley, T. F., Pope, M., Maria Grazia, P., Joshua, F., & Sporns, O. (2023). Partial entropy decomposition reveals higher-order information structures in human brain activity. Proceedings of the National Academy of Sciences, 120(30), e2300888120. https://doi.org/10.1073/pnas.2300888120

syntropy.discrete.partial_information_decomposition(inputs, target, joint_distribution, redundancy_function)

Computes the partial information decomposition for up to four input variables onto one (potentially joint) target variable.

The available redundancy functions are \(MMI\), \(i_{pm}\) from Finn and Lizier and \(i_{sx}\) from Makkeh et al..

\[\begin{split}i_{pm}(\alpha;t) &= \min h(\alpha_i) - \min h(\alpha_i|t) \\ i_{sx}(\alpha;t) &= \log\frac{P(t)-P(t\cap(\alpha_1\cup ...\cup\alpha_k))}{1-P(\bar\alpha_1\cap ...\cap\alpha_N)} \\ i_{MMI}(\alpha;t) &= \min_i I(\alpha_i;T)\end{split}\]

Parameters:

• inputs (tuple[int, ...]) – The indices of the input elements.

• target (tuple[int, ...]) – The indices of the target element(s)

• joint_distribution (DiscreteDist) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

• redundancy_function (str) – The localizable redundancy function. Options are: hmin and hsx.

Return type:

Any

References

Williams, P. L., & Beer, R. D. (2010). Nonnegative Decomposition of Multivariate Information. arXiv:1004.2515 [Math-Ph, Physics:Physics, q-Bio]. http://arxiv.org/abs/1004.2515

Finn, C., & Lizier, J. T. (2018). Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices. Entropy, 20(4), Article 4. https://doi.org/10.3390/e20040297

Makkeh, A., Gutknecht, A. J., & Wibral, M. (2021). Introducing a differentiable measure of pointwise shared information. Physical Review E, 103(3), 032149. https://doi.org/10.1103/PhysRevE.103.032149

syntropy.discrete.generalized_information_decomposition(posterior_distribution, prior_distribution, redundancy_function)

Computes the generalized information decomposition from Varley et al. The GID is a decomposition of the Kullback-Leibler divergence of a posterior distribution from a prior distribution.

Available redundancy functions are “hmin” and “hsx”. See the documentation for the partial_entropy_decomposition() function for details.

Parameters:

• posterior_distribution (DiscreteDist) – The posterior distribution. The support set of this distribution must be a subset of the supppirt set of the prior distribution.

• prior_distribution (DicreteDist) – The prior distribution.

• redundancy_function (str) – The localizable redundancy function. Options are: hmin and hsx.

Returns:

• ptw (dict[tuple, dict]) – A dict of dicts. The local Kullback-Leibler divergence for each atom for each state.

• avg (dict[tuple, float]) – The average Kullback-Leibler divergence for each atom.

Return type:

(<class ‘dict’>, <class ‘dict’>)

References

Varley, T. F. (2024). Generalized decomposition of multivariate information. PLOS ONE, 19(2), e0297128. https://doi.org/10.1371/journal.pone.0297128

syntropy.discrete.integrated_information_decomposition(inputs, target, joint_distribution, redundancy_function)

Computes the integrated information decomposition introduced by Rosas, Mediano, et al. The PhiID relaxes the requirement of only having a single target, and instead allows for redundant-redundant, synergistic-synergistic, etc interactions.

Available redundancy functions are:

i_{pm}(alpha;beta) &= min_i h(alpha_i) + min_i h(beta_i) - min h(alpha_i, beta_i) \

i_{tsx}(alpha;beta) &= h_{sx}(alpha) + h_{sx}(beta) - h_{sx}(alphacapbeta) i_{MMI}(alpha;beta) &= min_{ij} I(alpha_i;beta_j)

Parameters:

• inputs (tuple[int, ...]) – The indices of the input elements.

• target (tuple[int, ...]) – The indices of the target element(s)

• joint_distribution (DiscreteDist) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

• redundancy_function (str) – The localizable redundancy function. Options are: hmin and hsx.

Return type:

Any

References

Mediano, P. A. M., Rosas, F. E., Luppi, A. I., Carhart-Harris, R. L., Bor, D., Seth, A. K., & Barrett, A. B. (2025). Toward a unified taxonomy of information dynamics via Integrated Information Decomposition. Proceedings of the National Academy of Sciences, 122(39), e2423297122. https://doi.org/10.1073/pnas.2423297122

Rosas, F. E., Mediano, P. A. M., Jensen, H. J., Seth, A. K., Barrett, A. B., Carhart-Harris, R. L., & Bor, D. (2020). Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data. PLOS Computational Biology, 16(12), Article 12. https://doi.org/10.1371/journal.pcbi.1008289

Varley, T. F. (2023). Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions. PLOS ONE, 18(3), e0282950. https://doi.org/10.1371/journal.pone.0282950

syntropy.discrete.representational_complexity(avg, comparator=<built-in function min>)

Computes the representational complexity of a given partial information or entropy lattice. The representational complexity is a measure of how much partial information atoms of a given degree of synergy contribute to the overall mutual information or entropy.

Parameters:

• avg (dict[tuple, float]) – The dictionary of partial information/entropy atoms. Returned from any of the above functions.

• comparator (function, optional) – Whether to consider the minimum complexity of an atom. or the maximum complexity of an atom. Options are: min, max, np.min, np.max. The default is min, following the original work by Ehrlich et al.,.

Returns:

The representational complexity.

Return type:

float

References

Ehrlich, D. A., Schneider, A. C., Priesemann, V., Wibral, M., & Makkeh, A. (2023). A Measure of the Complexity of Neural Representations based on Partial Information Decomposition. Transactions on Machine Learning Research. https://openreview.net/forum?id=R8TU3pfzFr

syntropy.discrete.lempel_ziv_complexity(X, return_dictionary=False)

Uses the classic Lempel-Ziv compression algorithm to estimate the entropy rate of a one-dimensional array with \(N\) samples. If each element in X is a multi-dimensional tuple, then the result is equivalent to the joint entropy rate.

The extension to multivariate Lempel-Ziv is straightforward and involves representing the joint state of each element at time t as a tuple (X(t), Y(t)) and treating the two sources as a single joint source.

Here, the dictionary length \(|D|\) is normalized:

\[\textnormal{Complexity}(X) = \frac{|D|\log|D|}{N}\]

Parameters:

• X (list) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

• return_substrings (bool, optional) – DESCRIPTION. The default is False.

• return_dictionary (bool)

Returns:

• float – The estimated Lempel-Ziv complexity.

• set – The dictionary (only returned if return_dictionary == True)

Return type:

float | tuple[float, set]

References

Schartner, M. M., Carhart-Harris, R. L., Barrett, A. B., Seth, A. K., & Muthukumaraswamy, S. D. (2017). Increased spontaneous MEG signal diversity for psychoactive doses of ketamine, LSD and psilocybin. Scientific Reports, 7, 46421. https://doi.org/10.1038/srep46421

Blanc, J.-L., Schmidt, N., Bonnier, L., Pezard, L., & Lesne, A. (2008). Quantifying Neural Correlations Using Lempel-Ziv Complexity. Deuxième conférence française de Neurosciences Computationnelles, Marseille, France. https://hal.science/hal-00331599/document

Zozor, S., Ravier, P., & Buttelli, O. (2005). On Lempel–Ziv complexity for multidimensional data analysis. Physica A: Statistical Mechanics and Its Applications, 345(1), 285–302. https://doi.org/10.1016/j.physa.2004.07.025

syntropy.discrete.lempel_ziv_mutual_information(X, Y)

Estimates the discrete mutual information rate for two channels X and Y with the Lempel-Ziv compression algorithm. This measure can be transiently negative, although in the limit it approximates the discrete information rate.

\[I_{LZ}(X;Y) = LZ(X) + LZ(Y) - LZ(X,Y)\]

Parameters:

• X (Iterable[Any]) – An iterable of discrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

• Y (Iterable[Any]) – An iterable of dicrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

Returns:

The estimated mutual information rate.

Return type:

float

References

Zozor, S., Ravier, P., & Buttelli, O. (2005). On Lempel–Ziv complexity for multidimensional data analysis. Physica A: Statistical Mechanics and Its Applications, 345(1), 285–302. https://doi.org/10.1016/j.physa.2004.07.025

Blanc, J.-L., Schmidt, N., Bonnier, L., Pezard, L., & Lesne, A. (2008). Quantifying Neural Correlations Using Lempel-Ziv Complexity. Deuxième conférence française de Neurosciences Computationnelles, Marseille, France. https://hal.science/hal-00331599/document

syntropy.discrete.lempel_ziv_total_correlation(data)

A straightforward generalization of the mutual information rate given by Zozor et al., and Blanc et al.,

\[TC_{LZ}(X) = \sum_{i=1}^{N} LZ(X_i) - LZ(X)\]

Parameters:

data (NDArray[Any]) – A multi-dimensional discrete array, assumed to be in channels x time format.

Returns:

The estimated total correlation rate.

Return type:

float

syntropy.discrete.cross_lempel_ziv_complexity(X, Y)

Computes the Lempel-Ziv complexity of a string X using a pre-constructed dictionary optimized on Y. The relative Lempel-Ziv complexity is the extra patterns that appear in X but not in the compressed Y.

This code uses the optimized dictionary for Y, rather than searching all substrings of Y. Using all possible substrings is very impractical for long time series. As a result, however, it is very sensitive to the particular temporal ordering of fluctuations in X and Y.

Should only be used when X and Y were recorded at the same time, such as channels in an EEG/fMRI/MEG recording.

MATHEMATICAL GARUNTEES GIVEN IN ZIV & MERHAV 1993 ARE NOT PRESERVED BY THIS METHOD. THIS FUNCTION REMAINS EXPERIMENTAL

Parameters:

• X (NDArray[Any]) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.).

• Y (NDArray[Any]) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.).

Returns:

complexity – The Lempel-ziv relative complexity.

Return type:

int

References

Ziv, J., & Merhav, N. (1993). A Measure of Relative Entropy between Individual Sequences with Application to Universal Classification. Proceedings. IEEE International Symposium on Information Theory, 352–352. https://doi.org/10.1109/ISIT.1993.748668

Submodules

syntropy.discrete.alpha_synergy module

syntropy.discrete.alpha_synergy.alpha_synergistic_entropy(joint_distribution, alpha, num_samples=-1, definition='min')

Computes the \(\alpha\)-synergistic entropy for a joint distribution for a given value of \(\alpha\).

\[h^{syn}_{\alpha}(x) = \min_{a\subseteq x, |a|=\alpha} h(x^{a}|x^{-a})\]

Parameters:

• joint_distribution (dict[tuple, float]) – The joint distribution dictionary object.

• alpha (int) – The scale to consider.

• num_samples (int, optional) – The number of samples to trial. The default is -1, in which case, all permutations are trialed.

• definition (str, optional) – How to define the loss of information. Can be “min”, “max”, or “avg”. The default is “min”.

Returns:

The local alpha-synergy for each state.

Return type:

dict[tuple, float]

References

Varley, T. F. (2024). A scalable synergy-first backbone decomposition of higher-order structures in complex systems. Npj Complexity, 1(1), 1–11. https://doi.org/10.1038/s44260-024-00011-1

syntropy.discrete.alpha_synergy.partial_entropy_spectra(joint_distribution, num_samples=-1, definition='min')

Computes the partial synergy for every value of \(alpha\) (the spectrum) for each local state.

Parameters:

• joint_distribution (dict[tuple,float]) – The joint probability dictionary object.

• num_samples (int, optional) – The number of samples to trial. The default is -1, in which case, all permutations are trialed.

• definition (str, optional) – How to define the loss of information. Can be “min”, “max”, or “avg”. The default is “min”.

Returns:

The alpha-synergistic entropy spectrum for each state.

Return type:

dict[tuple,list]

syntropy.discrete.alpha_synergy.partial_kullback_leibler_spectra(posterior, prior, num_samples=-1, definition='min')

Computes the local Kullback-Leibler spectrum for each state

Parameters:

• posterior (dict[tuple,float]) – The distribution that describes the posterior beliefs.

• prior (dict[tuple,float]) – The distribution that describes the prior beliefs.

• num_samples (int, optional) – The number of samples to trial. The default is -1, in which case, all permutations are trialed.

• definition (str, optional) – How to define the loss of information. Can be “min”, “max”, or “avg”. The default is “min”.

Returns:

The alpha-synergistic DKL spectrum for each state.

Return type:

dict[tuple,list]

syntropy.discrete.alpha_synergy.partial_total_correlation_spectra(joint_distribution, num_samples=-1, definition='min')

Computes the local total correlation spectrum for each state using the Kullback-Leibler divergence.

Parameters:

• joint_distribution (dict[tuple,float]) – The joint probability dictionary object.

• num_samples (int, optional) – The number of samples to trial. The default is -1, in which case, all permutations are trialed.

• definition (str, optional) – How to define the loss of information. Can be “min”, “max”, or “avg”. The default is “min”.

Returns:

The alpha-synergistic total correlation spectrum for each state.

Return type:

dict[tuple,list]

syntropy.discrete.alpha_synergy.partial_information_spectra(inputs, target, joint_distribution, num_samples=-1, definition='min')

Computes the local mutual information spectrum for each state.

Parameters:

• inputs (tuple) – The indices of the input variables.

• target (tuple) – The indices of the target variables.

• joint_distribution (dict[tuple,float]) – The joint probability dictionary object.

• num_samples (int, optional) – The number of samples to trial. The default is -1, in which case, all permutations are trialed.

• definition (str, optional) – How to define the loss of information. Can be “min”, “max”, or “avg”. The default is “min”.

Return type:

list[float]

syntropy.discrete.optimization module

syntropy.discrete.optimization.constrained_maximum_entropy_distributions(joint_distribution, marginal_constraints=[(None,)], order=-1, max_iters=10000, tol=1e-06)

Uses the iterated proportional fitting (IPF) algorithm to find the maximum-entropy distribution consistant with given constraints.

If the marginal constrains are given, uses those. If order is given, finds the maximum entropy distribution consistant with all marginals of the given order.

Parameters:

• joint_distribution (dict) – The joint distribution from which to compute the marginal constraints.

• marginal_constraints (list, optional) – A list of tuples: each tuple corresponds to a set of marginals to constrain. The default is (None,).

• order (int, optional) – The order of marginals to fix. The default is -1.

• max_iters (int, optional) – The maximum number of iterations the algorithm can run for. The default is 10_000.

• tol (float, optional) – The value below which further refinements of the maximum entropy distribution are stopped. The default is 1e-6.

Returns:

The optimized distribution consistent with the given marginal constraints.

Return type:

dict[tuple, float]

syntropy.discrete.temporal module

syntropy.discrete.temporal.lempel_ziv_complexity(X, return_dictionary=False)

Uses the classic Lempel-Ziv compression algorithm to estimate the entropy rate of a one-dimensional array with \(N\) samples. If each element in X is a multi-dimensional tuple, then the result is equivalent to the joint entropy rate.

The extension to multivariate Lempel-Ziv is straightforward and involves representing the joint state of each element at time t as a tuple (X(t), Y(t)) and treating the two sources as a single joint source.

Here, the dictionary length \(|D|\) is normalized:

\[\textnormal{Complexity}(X) = \frac{|D|\log|D|}{N}\]

Parameters:

• X (list) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

• return_substrings (bool, optional) – DESCRIPTION. The default is False.

• return_dictionary (bool)

Returns:

• float – The estimated Lempel-Ziv complexity.

• set – The dictionary (only returned if return_dictionary == True)

Return type:

float | tuple[float, set]

References

Schartner, M. M., Carhart-Harris, R. L., Barrett, A. B., Seth, A. K., & Muthukumaraswamy, S. D. (2017). Increased spontaneous MEG signal diversity for psychoactive doses of ketamine, LSD and psilocybin. Scientific Reports, 7, 46421. https://doi.org/10.1038/srep46421

Blanc, J.-L., Schmidt, N., Bonnier, L., Pezard, L., & Lesne, A. (2008). Quantifying Neural Correlations Using Lempel-Ziv Complexity. Deuxième conférence française de Neurosciences Computationnelles, Marseille, France. https://hal.science/hal-00331599/document

Zozor, S., Ravier, P., & Buttelli, O. (2005). On Lempel–Ziv complexity for multidimensional data analysis. Physica A: Statistical Mechanics and Its Applications, 345(1), 285–302. https://doi.org/10.1016/j.physa.2004.07.025

syntropy.discrete.temporal.lempel_ziv_mutual_information(X, Y)

Estimates the discrete mutual information rate for two channels X and Y with the Lempel-Ziv compression algorithm. This measure can be transiently negative, although in the limit it approximates the discrete information rate.

\[I_{LZ}(X;Y) = LZ(X) + LZ(Y) - LZ(X,Y)\]

Parameters:

• X (Iterable[Any]) – An iterable of discrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

• Y (Iterable[Any]) – An iterable of dicrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

Returns:

The estimated mutual information rate.

Return type:

float

References

Zozor, S., Ravier, P., & Buttelli, O. (2005). On Lempel–Ziv complexity for multidimensional data analysis. Physica A: Statistical Mechanics and Its Applications, 345(1), 285–302. https://doi.org/10.1016/j.physa.2004.07.025

Blanc, J.-L., Schmidt, N., Bonnier, L., Pezard, L., & Lesne, A. (2008). Quantifying Neural Correlations Using Lempel-Ziv Complexity. Deuxième conférence française de Neurosciences Computationnelles, Marseille, France. https://hal.science/hal-00331599/document

syntropy.discrete.temporal.conditional_lempel_ziv_complexity(X, Y)

The conditional entropy rate estimated with the Lempel-Ziv algorithm.

\[LZ(X|Y) = LZ(X,Y) - LZ(Y)\]

Parameters:

• X (Iterable[Any]) – An iterable of discrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

• Y (Iterable[Any]) – An iterable of dicrete RVs. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.)..

Returns:

The estimated conditional entropy rate.

Return type:

float

References

Zozor, S., Ravier, P., & Buttelli, O. (2005). On Lempel–Ziv complexity for multidimensional data analysis. Physica A: Statistical Mechanics and Its Applications, 345(1), 285–302. https://doi.org/10.1016/j.physa.2004.07.025

syntropy.discrete.temporal.lempel_ziv_total_correlation(data)

A straightforward generalization of the mutual information rate given by Zozor et al., and Blanc et al.,

\[TC_{LZ}(X) = \sum_{i=1}^{N} LZ(X_i) - LZ(X)\]

Parameters:

data (NDArray[Any]) – A multi-dimensional discrete array, assumed to be in channels x time format.

Returns:

The estimated total correlation rate.

Return type:

float

syntropy.discrete.temporal.cross_lempel_ziv_complexity(X, Y)

Computes the Lempel-Ziv complexity of a string X using a pre-constructed dictionary optimized on Y. The relative Lempel-Ziv complexity is the extra patterns that appear in X but not in the compressed Y.

This code uses the optimized dictionary for Y, rather than searching all substrings of Y. Using all possible substrings is very impractical for long time series. As a result, however, it is very sensitive to the particular temporal ordering of fluctuations in X and Y.

Should only be used when X and Y were recorded at the same time, such as channels in an EEG/fMRI/MEG recording.

MATHEMATICAL GARUNTEES GIVEN IN ZIV & MERHAV 1993 ARE NOT PRESERVED BY THIS METHOD. THIS FUNCTION REMAINS EXPERIMENTAL

Parameters:

• X (NDArray[Any]) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.).

• Y (NDArray[Any]) – A discrete array. Can contain digits 0-9 and/or single-character strings (“A”, “a”, “B”, “b”, etc.).

Returns:

complexity – The Lempel-ziv relative complexity.

Return type:

int

References

Ziv, J., & Merhav, N. (1993). A Measure of Relative Entropy between Individual Sequences with Application to Universal Classification. Proceedings. IEEE International Symposium on Information Theory, 352–352. https://doi.org/10.1109/ISIT.1993.748668

syntropy.discrete.utils module

syntropy.discrete.utils.make_powerset(iterable)

A utility function for quickly making powersets,

powerset([1,2,3]) –> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)

syntropy.discrete.utils.flatten_nested_tuple(x)

Parameters:

x (tuple[tuple[Any, ...], ...])

Return type:

tuple[Any, …]

syntropy.discrete.utils.clean_distribution(joint_distribution)

A utility function to remove states with 0 probability

Parameters:

joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

The joint probability distribution with zero-probability elements removed.

Return type:

dict

syntropy.discrete.utils.reduce_state(state, source)

A utility function for reducing tuples to just the elements in the source.

Parameters:

• state (tuple) – The particular state of each variable.

• source (tuple) – The indices of the variable to remove.

Returns:

The reduced state consisting only of those elements indexed in the source variable.

Return type:

tuple

syntropy.discrete.utils.construct_joint_distribution(data)

Given a channels x time, discrete Numpy array, computes the probability distribution that describes the data.

Parameters:

data (np.ndarray) – The data: assumed to be in elements x time format.

Returns:

The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Return type:

dict

syntropy.discrete.utils.get_marginal_distribution(idxs, joint_distribution)

Returns the marginal distribution of the variables indexed by the idxs tuple. The opposite of the marginalize_out() function.

Parameters:

• idxs (tuple) – The indices of the variable to retain.

• joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

The marginal joint probability distribution object.

Return type:

dict

syntropy.discrete.utils.marginalize_out(idxs, joint_distribution)

Returns a distribution with the variables indexed by idxs marginalized out.

Parameters:

• idxs (tuple) – The indices of the variables to be marginalized out.

• joint_distribution (dict[tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

A joint probability distribution dictionary.

Return type:

dict

syntropy.discrete.utils.get_all_marginal_distributions(joint_distribution)

Computes the set of all marginal probability distributions. If the original distribution has variables:

\(P(X_1, X_2, X_3)\)

Returns a dictionary of dictionaries for each:

\(P(X_1,), P(X_2,), P(X_3,), P(X_1, X_2), P(X_1, X_3), P(X_2, X_3), P(X_1,X_2,X_3)\)

Parameters:

joint_distribution (dict[tuple, float][tuple, float]) – The joint probability distribution. Keys are tuples corresponding to the state of each element. The valules are the probabilities.

Returns:

A dictionary of dictionaries: each key is a set of marginals, each value is the associated marginal distribution .

Return type:

dict[tuple, dict]

syntropy.discrete.utils.product_distribution(A, B)

Compute the product of two independent distributions.

Parameters:

• A (dict[tuple[int, ...], float]) – The first distribution

• B (dict[tuple[int, ...], float]) – The second distribution

Returns:

Joint distribution over concatenated states

Return type:

dict[tuple[int, …], float]

syntropy.discrete.utils.generate_closed_distribution(N, seed=None)

Generate a random closed discrete probability distribution on N binary elements.

A distribution is closed iff H(X_i | X^{-i}) = 0 for all i, meaning every variable is fully determined by the others. This requires the support to have minimum Hamming distance >= 2.

Parameters:

• N (int) – Number of binary elements.

• seed (int, optional) – Random seed for reproducibility.

Returns:

Probability distribution as {state: probability} mapping.

Return type:

dict[tuple[int, …], float]