diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 4b85d72..b351c7c 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.11.2","generation_timestamp":"2025-01-06T21:23:23","documenter_version":"1.7.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.11.2","generation_timestamp":"2025-01-06T21:29:09","documenter_version":"1.7.0"}} \ No newline at end of file diff --git a/dev/api/index.html b/dev/api/index.html index 10b52c0..f146dc0 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,5 +1,5 @@ -API Documentation · ExpFamilyPCA
GitHub

API Documentation

Contents

Index

Functions

The core of the ExpFamilyPCA.jl API is the EPCA abstract type. All supported and custom EPCA specifications are subtypes of EPCA and include three methods in their EPCA interface: fit!, compress and decompress.

ExpFamilyPCA.fit! — Function
fit!(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real

Fits the EPCA model on the dataset X. Call this function after creating an EPCA struct or to continue training on more data.

Should be called after creating an EPCA object or when you want to fit on new data.

Warning

The default fit! may have long run times depending on the dataset size and model complexity. Consider adjusting the verbosity (verbose) and number of iterations (maxiter) to better balance runtime and model performance.

Arguments

  • epca::EPCA: The EPCA model.
  • X::AbstractMatrix{T}: (n, indim) - The input training data matrix. Rows are observations. Columns are features or variables.

Keyword Arguments

  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.

Returns

  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.

Usage

Input:

using ExpFamilyPCA
+API Documentation · ExpFamilyPCA
GitHub
API Documentation
Contents
Index
Functions
The core of the ExpFamilyPCA.jl API is the EPCA abstract type. All supported and custom EPCA specifications are subtypes of EPCA and include three methods in their EPCA interface: fit!, compress and decompress.
ExpFamilyPCA.fit! — Function
fit!(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Fits the EPCA model on the dataset X. Call this function after creating an EPCA struct or to continue training on more data.
Should be called after creating an EPCA object or when you want to fit on new data.
Warning
The default fit! may have long run times depending on the dataset size and model complexity. Consider adjusting the verbosity (verbose) and number of iterations (maxiter) to better balance runtime and model performance.
Arguments
  • epca::EPCA: The EPCA model. 
  • X::AbstractMatrix{T}: (n, indim) - The input training data matrix. Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
Input:
using ExpFamilyPCA
 using Random; Random.seed!(1)
 
 # Create the model
@@ -16,9 +16,9 @@
 Iteration: 50/200 | Loss: 11.07389383509631
 Iteration: 100/200 | Loss: 10.971490262772905
 Iteration: 150/200 | Loss: 10.886018474442618
-Iteration: 200/200 | Loss: 10.718703556787007
source
ExpFamilyPCA.compress — Function
compress(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Compresses the input data X with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • X::AbstractMatrix{T}: (n, indim) - The input data matrix (can differ from the training data). Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
# Generate some random test data
+Iteration: 200/200 | Loss: 10.718703556787007
source
ExpFamilyPCA.compress — Function
compress(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Compresses the input data X with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • X::AbstractMatrix{T}: (n, indim) - The input data matrix (can differ from the training data). Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
# Generate some random test data
 m = 10
 Y = rand(0:1, m, indim)
 
 # Compress the test data using the fitted model from the previous example
-Y_compressed = compress(epca, Y)
source
ExpFamilyPCA.decompress — Function
decompress(epca::EPCA, A::AbstractMatrix{T}) where T <: Real
Decompress the compressed matrix A with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • A::AbstractMatrix{T}: (n, outdim) - A compressed data matrix.
Returns
  • X̂::AbstractMatrix{T}: (n, indim) - The reconstructed data matrix approximated using EPCA model parameters.
Usage
Y_reconstructed = decompress(epca, Y)
source
Options
ExpFamilyPCA.Options — Type
Options(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), A_init_value::Real = 1.0, A_lower::Union{Real, Nothing} = nothing, A_upper::Union{Real, Nothing} = nothing, A_use_sobol::Bool = false, V_init_value::Real = 1.0, V_lower::Union{Real, Nothing} = nothing, V_upper::Union{Real, Nothing} = nothing, V_use_sobol::Bool = false, low = -1e10, high = 1e10, tol = 1e-10, maxiter = 1e6)
Defines a struct Options for configuring various parameters used in optimization and calculus. It provides flexible defaults for metaprogramming, initialization values, optimization boundaries, and binary search controls.
Fields
  • metaprogramming::Bool: Enables metaprogramming for symbolic calculus conversions. Default is true.
  • μ::Real: A regularization hyperparameter. Default is 1.
  • ϵ::Real: A regularization hyperparameter. Default is eps().
  • A_init_value::Real: Initial value for parameter A. Default is 1.0.
  • A_lower::Union{Real, Nothing}: Lower bound for A, or nothing. Default is nothing.
  • A_upper::Union{Real, Nothing}: Upper bound for A, or nothing. Default is nothing.
  • A_use_sobol::Bool: Use Sobol sequences for initializing A. Default is false.
  • V_init_value::Real: Initial value for parameter V. Default is 1.0.
  • V_lower::Union{Real, Nothing}: Lower bound for V, or nothing. Default is nothing.
  • V_upper::Union{Real, Nothing}: Upper bound for V, or nothing. Default is nothing.
  • V_use_sobol::Bool: Use Sobol sequences for initializing V. Default is false.
  • low::Real: Lower bound for binary search. Default is -1e10.
  • high::Real: Upper bound for binary search. Default is 1e10.
  • tol::Real: Tolerance for stopping binary search. Default is 1e-10.
  • maxiter::Real: Maximum iterations for binary search. Default is 1e6.
Info
The metaprogramming flag controls whether metaprogramming is used during symbolic differentiation conversion. While conversion between Symbolics.jl atoms and base Julia can occur without it, this approach is slower and requires more calls. Nonetheless, the flag is provided for users who keenly want to avoid metaprogramming in their pipeline.
source
ExpFamilyPCA.NegativeDomain — Function
NegativeDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the negative domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = -1: Initializes A with a negative value.
  • A_upper = -1e-4: Upper bound for A is constrained to a small negative value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
ExpFamilyPCA.PositiveDomain — Function
PositiveDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the positive domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = 1: Initializes A with a positive value.
  • A_lower = 1e-4: Lower bound for A is constrained to a small positive value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
  • 1If compress is called before fit!, X will compressed using unfitted starting weights.

+Y_compressed = compress(epca, Y)
source
ExpFamilyPCA.decompress — Function
decompress(epca::EPCA, A::AbstractMatrix{T}) where T <: Real

Decompress the compressed matrix A with the EPCA model.

Arguments

  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • A::AbstractMatrix{T}: (n, outdim) - A compressed data matrix.

Returns

  • X̂::AbstractMatrix{T}: (n, indim) - The reconstructed data matrix approximated using EPCA model parameters.

Usage

Y_reconstructed = decompress(epca, Y)
source

Options

ExpFamilyPCA.Options — Type
Options(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), A_init_value::Real = 1.0, A_lower::Union{Real, Nothing} = nothing, A_upper::Union{Real, Nothing} = nothing, A_use_sobol::Bool = false, V_init_value::Real = 1.0, V_lower::Union{Real, Nothing} = nothing, V_upper::Union{Real, Nothing} = nothing, V_use_sobol::Bool = false, low = -1e10, high = 1e10, tol = 1e-10, maxiter = 1e6)

Defines a struct Options for configuring various parameters used in optimization and calculus. It provides flexible defaults for metaprogramming, initialization values, optimization boundaries, and binary search controls.

Fields

  • metaprogramming::Bool: Enables metaprogramming for symbolic calculus conversions. Default is true.
  • μ::Real: A regularization hyperparameter. Default is 1.
  • ϵ::Real: A regularization hyperparameter. Default is eps().
  • A_init_value::Real: Initial value for parameter A. Default is 1.0.
  • A_lower::Union{Real, Nothing}: Lower bound for A, or nothing. Default is nothing.
  • A_upper::Union{Real, Nothing}: Upper bound for A, or nothing. Default is nothing.
  • A_use_sobol::Bool: Use Sobol sequences for initializing A. Default is false.
  • V_init_value::Real: Initial value for parameter V. Default is 1.0.
  • V_lower::Union{Real, Nothing}: Lower bound for V, or nothing. Default is nothing.
  • V_upper::Union{Real, Nothing}: Upper bound for V, or nothing. Default is nothing.
  • V_use_sobol::Bool: Use Sobol sequences for initializing V. Default is false.
  • low::Real: Lower bound for binary search. Default is -1e10.
  • high::Real: Upper bound for binary search. Default is 1e10.
  • tol::Real: Tolerance for stopping binary search. Default is 1e-10.
  • maxiter::Real: Maximum iterations for binary search. Default is 1e6.
Info

The metaprogramming flag controls whether metaprogramming is used during symbolic differentiation conversion. While conversion between Symbolics.jl atoms and base Julia can occur without it, this approach is slower and requires more calls. Nonetheless, the flag is provided for users who keenly want to avoid metaprogramming in their pipeline.

source
ExpFamilyPCA.NegativeDomain — Function
NegativeDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)

Returns an instance of Options configured for optimization over the negative domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.

Specific Settings

  • A_init_value = -1: Initializes A with a negative value.
  • A_upper = -1e-4: Upper bound for A is constrained to a small negative value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.

Other fields inherit from the Options struct.

source
ExpFamilyPCA.PositiveDomain — Function
PositiveDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)

Returns an instance of Options configured for optimization over the positive domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.

Specific Settings

  • A_init_value = 1: Initializes A with a positive value.
  • A_lower = 1e-4: Lower bound for A is constrained to a small positive value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.

Other fields inherit from the Options struct.

source
  • 1If compress is called before fit!, X will compressed using unfitted starting weights.
diff --git a/dev/constructors/bernoulli/index.html b/dev/constructors/bernoulli/index.html index 3c83f57..b12573b 100644 --- a/dev/constructors/bernoulli/index.html +++ b/dev/constructors/bernoulli/index.html @@ -1,2 +1,2 @@ -Bernoulli · ExpFamilyPCA
GitHub

Bernoulli EPCA

Math

NameBernoulliEPCA
$G(\theta)$$\log(1 + e^\theta)$
$g(\theta)$$\frac{e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, 1)$
$\Theta$ Spacereal
Appropriate Databinary

$G$ is the softplus function and $g$ is the logistic function.

Documentation

ExpFamilyPCA.BernoulliEPCA — Function
BernoulliEPCA(indim::Integer, outdim::Integer; options = Options(μ = 0.5))

Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Bernoulli · ExpFamilyPCA
GitHub

Bernoulli EPCA

Math

NameBernoulliEPCA
$G(\theta)$$\log(1 + e^\theta)$
$g(\theta)$$\frac{e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, 1)$
$\Theta$ Spacereal
Appropriate Databinary

$G$ is the softplus function and $g$ is the logistic function.

Documentation

ExpFamilyPCA.BernoulliEPCA — Function
BernoulliEPCA(indim::Integer, outdim::Integer; options = Options(μ = 0.5))

Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/binomial/index.html b/dev/constructors/binomial/index.html index 2072f26..16f000f 100644 --- a/dev/constructors/binomial/index.html +++ b/dev/constructors/binomial/index.html @@ -1,2 +1,2 @@ -Binomial · ExpFamilyPCA
GitHub

Binomial EPCA

Math

NameBinomialEPCA
$G(\theta)$$n \log(1 + e^\theta)$
$g(\theta)$$\frac{n e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, n)$
$\Theta$ Spacereal
Appropriate Datacount
$n$$n \geq 0$ (number of trials)

$G$ is the scaled softplus function and $g$ is the scaled logistic function.

Documentation

ExpFamilyPCA.BinomialEPCA — Function
BinomialEPCA(indim::Integer, outdim::Integer, n::Integer; options::Options = Options(μ = 0.5))

Binomial EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • n::Integer: A known parameter representing the number of trials (nonnegative).
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the binomial distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Binomial · ExpFamilyPCA
GitHub

Binomial EPCA

Math

NameBinomialEPCA
$G(\theta)$$n \log(1 + e^\theta)$
$g(\theta)$$\frac{n e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, n)$
$\Theta$ Spacereal
Appropriate Datacount
$n$$n \geq 0$ (number of trials)

$G$ is the scaled softplus function and $g$ is the scaled logistic function.

Documentation

ExpFamilyPCA.BinomialEPCA — Function
BinomialEPCA(indim::Integer, outdim::Integer, n::Integer; options::Options = Options(μ = 0.5))

Binomial EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • n::Integer: A known parameter representing the number of trials (nonnegative).
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the binomial distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/continuous_bernoulli/index.html b/dev/constructors/continuous_bernoulli/index.html index 2cac702..c76ea56 100644 --- a/dev/constructors/continuous_bernoulli/index.html +++ b/dev/constructors/continuous_bernoulli/index.html @@ -1,2 +1,2 @@ -Continuous Bernoulli · ExpFamilyPCA
GitHub

Continuous Bernoulli EPCA

Math

NameContinuousBernoulliEPCA
$G(\theta)$$\log \frac{e^\theta - 1}{\theta}$
$g(\theta)$$\frac{\theta-1}{\theta} + \frac{1}{e^\theta - 1}$
$\mu$ Space[1]$(0, 1) \setminus \{\frac{1}{2}\}$
$\Theta$ Space$\mathbb{R} \setminus \{ 0 \}$
Appropriate Dataunit interval

Documentation

ExpFamilyPCA.ContinuousBernoulliEPCA — Function
ContinuousBernoulliEPCA(indim::Integer, outdim::Integer; options::Options = Options(μ = 0.5))

Continuous Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.25).

Returns

  • epca: An EPCA subtype for the continuous Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Continuous Bernoulli · ExpFamilyPCA
GitHub

Continuous Bernoulli EPCA

Math

NameContinuousBernoulliEPCA
$G(\theta)$$\log \frac{e^\theta - 1}{\theta}$
$g(\theta)$$\frac{\theta-1}{\theta} + \frac{1}{e^\theta - 1}$
$\mu$ Space[1]$(0, 1) \setminus \{\frac{1}{2}\}$
$\Theta$ Space$\mathbb{R} \setminus \{ 0 \}$
Appropriate Dataunit interval

Documentation

ExpFamilyPCA.ContinuousBernoulliEPCA — Function
ContinuousBernoulliEPCA(indim::Integer, outdim::Integer; options::Options = Options(μ = 0.5))

Continuous Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.25).

Returns

  • epca: An EPCA subtype for the continuous Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/gamma/index.html b/dev/constructors/gamma/index.html index 1b8504e..b7abef8 100644 --- a/dev/constructors/gamma/index.html +++ b/dev/constructors/gamma/index.html @@ -1,2 +1,2 @@ -Gamma · ExpFamilyPCA
GitHub

Gamma EPCA

Math

NameGammaEPCA or ItakuraSaitoEPCA
$G(\theta)$$-\log(-\theta)$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datapositive

Documentation

ExpFamilyPCA.GammaEPCA — Function
GammaEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -2, A_upper = -eps(), V_lower = eps()))

Gamma EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -2).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Gamma · ExpFamilyPCA
GitHub

Gamma EPCA

Math

NameGammaEPCA or ItakuraSaitoEPCA
$G(\theta)$$-\log(-\theta)$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datapositive

Documentation

ExpFamilyPCA.GammaEPCA — Function
GammaEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -2, A_upper = -eps(), V_lower = eps()))

Gamma EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -2).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/gaussian/index.html b/dev/constructors/gaussian/index.html index 5538863..553d03d 100644 --- a/dev/constructors/gaussian/index.html +++ b/dev/constructors/gaussian/index.html @@ -1,2 +1,2 @@ -Gaussian · ExpFamilyPCA
GitHub

Gaussian EPCA

NameGaussianEPCA or NormalEPCA
$G(\theta)$$\theta^2 / 2$
$g(\theta)$$\theta$
$\mu$ Space[1]real
$\Theta$ Spacereal
Appropriate Datacontinuous

The Gaussian EPCA objective is equivalent to regular PCA.

Documentation

ExpFamilyPCA.NormalEPCA — Function
GaussianEPCA(indim::Integer, outdim::Integer; options::Options = Options())

An EPCA model with Gaussian loss.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Gaussian distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Gaussian · ExpFamilyPCA
GitHub

Gaussian EPCA

NameGaussianEPCA or NormalEPCA
$G(\theta)$$\theta^2 / 2$
$g(\theta)$$\theta$
$\mu$ Space[1]real
$\Theta$ Spacereal
Appropriate Datacontinuous

The Gaussian EPCA objective is equivalent to regular PCA.

Documentation

ExpFamilyPCA.NormalEPCA — Function
GaussianEPCA(indim::Integer, outdim::Integer; options::Options = Options())

An EPCA model with Gaussian loss.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Gaussian distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/negative_binomial/index.html b/dev/constructors/negative_binomial/index.html index 8cd213d..eb9d7ca 100644 --- a/dev/constructors/negative_binomial/index.html +++ b/dev/constructors/negative_binomial/index.html @@ -1,2 +1,2 @@ -Negative Binomial · ExpFamilyPCA
GitHub

Negative Binomial EPCA

NameNegativeBinomialEPCA
$G(\theta)$$-r \log(1 - e^\theta)$
$g(\theta)$$- \frac{r e^\theta}{1 - e^\theta}$
$\mu$ Space[1]positive
$\Theta$ Spacenegative
Appropriate Datacount
$r$$r > 0$ (number of successes)

Documentation

ExpFamilyPCA.NegativeBinomialEPCA — Function
NegativeBinomialEPCA(indim::Integer, outdim::Integer, r::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Negative binomial EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • r::Integer: A known parameter of the negative binomial distribution representing the number of successes (positive).
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -1).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Negative Binomial · ExpFamilyPCA
GitHub

Negative Binomial EPCA

NameNegativeBinomialEPCA
$G(\theta)$$-r \log(1 - e^\theta)$
$g(\theta)$$- \frac{r e^\theta}{1 - e^\theta}$
$\mu$ Space[1]positive
$\Theta$ Spacenegative
Appropriate Datacount
$r$$r > 0$ (number of successes)

Documentation

ExpFamilyPCA.NegativeBinomialEPCA — Function
NegativeBinomialEPCA(indim::Integer, outdim::Integer, r::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Negative binomial EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • r::Integer: A known parameter of the negative binomial distribution representing the number of successes (positive).
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -1).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/pareto/index.html b/dev/constructors/pareto/index.html index f5d11fb..2682230 100644 --- a/dev/constructors/pareto/index.html +++ b/dev/constructors/pareto/index.html @@ -1,2 +1,2 @@ -Pareto · ExpFamilyPCA
GitHub

Pareto EPCA

Math

NameParetoEPCA
$G(\theta)$$-\log(-1 - \theta) + \theta \log m$
$g(\theta)$$\log m - \frac{1}{\theta + 1}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ \log{m} \}$
$\Theta$ Spacenegative
Appropriate Dataheavy-tail
$m$$m > 0$ (minimum value)

Documentation

ExpFamilyPCA.ParetoEPCA — Function
ParetoEPCA(indim::Integer, outdim::Integer, m::Real; options::Options = Options(μ = 2, A_init_value = 2, A_lower = 1 / indim, V_init_value = -2, V_upper = -1))

Pareto EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • m::Real: A known parameter of the Pareto distribution representing the minimum value in the support.
  • options::Options: Optional parameters for model initialization:
    • μ: Default value 2.
    • A_init_value: Initial value for matrix A (default: 2).
    • A_lower: Lower bound for matrix A (default: 1 / indim).
    • V_init_value: Initial value for matrix V (default: -2).
    • V_upper: Upper bound for matrix V (default: -1).

Returns

  • epca: An EPCA subtype for the Pareto distribution.
source
Tip

If your compression converges to a constant matrix, try processing your data to reduce the maximum (e.g., divide your data by a large constant, take the logarithm).

  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Pareto · ExpFamilyPCA
GitHub

Pareto EPCA

Math

NameParetoEPCA
$G(\theta)$$-\log(-1 - \theta) + \theta \log m$
$g(\theta)$$\log m - \frac{1}{\theta + 1}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ \log{m} \}$
$\Theta$ Spacenegative
Appropriate Dataheavy-tail
$m$$m > 0$ (minimum value)

Documentation

ExpFamilyPCA.ParetoEPCA — Function
ParetoEPCA(indim::Integer, outdim::Integer, m::Real; options::Options = Options(μ = 2, A_init_value = 2, A_lower = 1 / indim, V_init_value = -2, V_upper = -1))

Pareto EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • m::Real: A known parameter of the Pareto distribution representing the minimum value in the support.
  • options::Options: Optional parameters for model initialization:
    • μ: Default value 2.
    • A_init_value: Initial value for matrix A (default: 2).
    • A_lower: Lower bound for matrix A (default: 1 / indim).
    • V_init_value: Initial value for matrix V (default: -2).
    • V_upper: Upper bound for matrix V (default: -1).

Returns

  • epca: An EPCA subtype for the Pareto distribution.
source
Tip

If your compression converges to a constant matrix, try processing your data to reduce the maximum (e.g., divide your data by a large constant, take the logarithm).

  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/poisson/index.html b/dev/constructors/poisson/index.html index e9277fd..927269c 100644 --- a/dev/constructors/poisson/index.html +++ b/dev/constructors/poisson/index.html @@ -1,2 +1,2 @@ -Poisson · ExpFamilyPCA
GitHub

Poisson EPCA

NamePoissonEPCA
$G(\theta)$$e^\theta$
$g(\theta)$$e^\theta$
$\mu$ Space[1]positive
$\Theta$ Spacereal
Appropriate Datacount, probability

Poisson EPCA minimizes the generalized KL divergence making it well-suited for compressing probability profiles. Poisson EPCA has also been used in reinforcement learning to solve partially observed Markov decision processes (POMDPs) with belief compression (Roy et al., 2005).

Documentation

ExpFamilyPCA.PoissonEPCA — Function
PoissonEPCA(indim::Integer, outdim::Integer; options::Options = Options())

Poisson EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Poisson distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Poisson · ExpFamilyPCA
GitHub

Poisson EPCA

NamePoissonEPCA
$G(\theta)$$e^\theta$
$g(\theta)$$e^\theta$
$\mu$ Space[1]positive
$\Theta$ Spacereal
Appropriate Datacount, probability

Poisson EPCA minimizes the generalized KL divergence making it well-suited for compressing probability profiles. Poisson EPCA has also been used in reinforcement learning to solve partially observed Markov decision processes (POMDPs) with belief compression (Roy et al., 2005).

Documentation

ExpFamilyPCA.PoissonEPCA — Function
PoissonEPCA(indim::Integer, outdim::Integer; options::Options = Options())

Poisson EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Poisson distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/weibull/index.html b/dev/constructors/weibull/index.html index 6251bba..fd1f576 100644 --- a/dev/constructors/weibull/index.html +++ b/dev/constructors/weibull/index.html @@ -1,2 +1,2 @@ -Weibull · ExpFamilyPCA
GitHub

Weibull EPCA

NameWeibullEPCA
$G(\theta)$$-\log(-\theta) - \log k$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} / \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datanonnegative continuous

WeibullEPCA omits it the known shape parameter $k$ since it does not affect the Weibull EPCA objective.

Documentation

ExpFamilyPCA.WeibullEPCA — Function
WeibullEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Weibull EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters for model initialization. Default NegativeDomain().

Returns

  • epca: An EPCA subtype for the Weibull distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Weibull · ExpFamilyPCA
GitHub

Weibull EPCA

NameWeibullEPCA
$G(\theta)$$-\log(-\theta) - \log k$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} / \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datanonnegative continuous

WeibullEPCA omits it the known shape parameter $k$ since it does not affect the Weibull EPCA objective.

Documentation

ExpFamilyPCA.WeibullEPCA — Function
WeibullEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Weibull EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters for model initialization. Default NegativeDomain().

Returns

  • epca: An EPCA subtype for the Weibull distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/index.html b/dev/index.html index 86f4584..befef70 100644 --- a/dev/index.html +++ b/dev/index.html @@ -13,4 +13,4 @@ Y_compressed = compress(poisson_epca, Y; maxiter=200, verbose=true) X_reconstructed = decompress(poisson_epca, X_compressed) -Y_reconstructed = decompress(poisson_epca, Y_compressed)

Supported Distributions

DistributionExpFamilyPCA.jlObjectiveLink Function $g(\theta)$
BernoulliBernoulliEPCA$\log(1 + e^{\theta - 2x\theta})$$\frac{e^\theta}{1 + e^\theta}$
BinomialBinomialEPCA$n \log(1 + e^\theta) - x\theta$$\frac{ne^\theta}{1 + e^\theta}$
Continuous BernoulliContinuousBernoulliEPCA$\log\left(\frac{e^\theta - 1}{\theta}\right) - x\theta$$\frac{\theta - 1}{\theta} + \frac{1}{e^\theta - 1}$
Gamma¹GammaEPCA or ItakuraSaitoEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$
Gaussian²GaussianEPCA or NormalEPCA$\frac{1}{2}(x - \theta)^2$$\theta$
Negative BinomialNegativeBinomialEPCA$-r \log(1 - e^\theta) - x\theta$$\frac{-re^\theta}{e^\theta - 1}$
ParetoParetoEPCA$-\log(-1 - \theta) + \theta \log m - x\theta$$\log m - \frac{1}{\theta + 1}$
Poisson³PoissonEPCA$e^\theta - x\theta$$e^\theta$
WeibullWeibullEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$

¹: For the Gamma distribution, the link function is typically based on the inverse relationship.

²: For Gaussian, also known as Normal distribution, the link function is the identity.

³: The Poisson distribution link function is exponential.

Contributing

ExpFamilyPCA.jl was designed through the Stanford Intelligent Systems Laboratory (SISL) which helps maintain open-source projects posted on the SISL organization repository and the JuliaPOMDPs community. We subscribe to the community's contribution guidelines and encourage new users with all levels of software development experience to contribute and ask questions.

If you would like to create a new compressor or improve an existing model, please open a new issue that briefly describes the contribution you would like to make and the problem that it fixes, if there is one.

+Y_reconstructed = decompress(poisson_epca, Y_compressed)

Supported Distributions

DistributionExpFamilyPCA.jlObjectiveLink Function $g(\theta)$
BernoulliBernoulliEPCA$\log(1 + e^{\theta - 2x\theta})$$\frac{e^\theta}{1 + e^\theta}$
BinomialBinomialEPCA$n \log(1 + e^\theta) - x\theta$$\frac{ne^\theta}{1 + e^\theta}$
Continuous BernoulliContinuousBernoulliEPCA$\log\left(\frac{e^\theta - 1}{\theta}\right) - x\theta$$\frac{\theta - 1}{\theta} + \frac{1}{e^\theta - 1}$
Gamma¹GammaEPCA or ItakuraSaitoEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$
Gaussian²GaussianEPCA or NormalEPCA$\frac{1}{2}(x - \theta)^2$$\theta$
Negative BinomialNegativeBinomialEPCA$-r \log(1 - e^\theta) - x\theta$$\frac{-re^\theta}{e^\theta - 1}$
ParetoParetoEPCA$-\log(-1 - \theta) + \theta \log m - x\theta$$\log m - \frac{1}{\theta + 1}$
Poisson³PoissonEPCA$e^\theta - x\theta$$e^\theta$
WeibullWeibullEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$

¹: For the Gamma distribution, the link function is typically based on the inverse relationship.

²: For Gaussian, also known as Normal distribution, the link function is the identity.

³: The Poisson distribution link function is exponential.

Contributing

ExpFamilyPCA.jl was designed through the Stanford Intelligent Systems Laboratory (SISL) which helps maintain open-source projects posted on the SISL organization repository and the JuliaPOMDPs community. We subscribe to the community's contribution guidelines and encourage new users with all levels of software development experience to contribute and ask questions.

If you would like to create a new compressor or improve an existing model, please open a new issue that briefly describes the contribution you would like to make and the problem that it fixes, if there is one.

diff --git a/dev/math/appendix/expectation/index.html b/dev/math/appendix/expectation/index.html index fde928d..b2a5f7f 100644 --- a/dev/math/appendix/expectation/index.html +++ b/dev/math/appendix/expectation/index.html @@ -7,4 +7,4 @@ &= \int x \exp(x \theta - G(\theta)) h(x) \, dx \\ &= \int x p_\theta(x) \, dx \\ &= \mathbb{E}_\theta[X]. -\end{aligned}\]

+\end{aligned}\]

diff --git a/dev/math/appendix/gamma/index.html b/dev/math/appendix/gamma/index.html index 0027e56..569f918 100644 --- a/dev/math/appendix/gamma/index.html +++ b/dev/math/appendix/gamma/index.html @@ -11,4 +11,4 @@ &= \frac{p}{q} - \log \frac{p}{q} - 1, \end{aligned}\]

so $B_F$ is the Itakura-Saito (Itakura and Saito, 1968) distance as desired. Further, the EPCA objective is

\[\begin{aligned} B_F(x \| g(\theta)) = \frac{p}{g(\theta)} - \log \frac{p}{g(\theta)} - 1 = -p\theta - \log(-p\theta) - 1. -\end{aligned}\]

+\end{aligned}\]

diff --git a/dev/math/appendix/gaussian/index.html b/dev/math/appendix/gaussian/index.html index abd1bef..f39c171 100644 --- a/dev/math/appendix/gaussian/index.html +++ b/dev/math/appendix/gaussian/index.html @@ -6,4 +6,4 @@ &= \frac{1}{2} \big[ \langle A, A \rangle - 2\langle B, A \rangle + \langle B, B \rangle \big] \\ &= \frac{1}{2} \langle A - B, A - B \rangle \\ &= \frac{1}{2} \| A - B \|_F^2. -\end{aligned}\]

Similarly, the Bregman divergence induced from the log-partition of the Gaussian $G(\theta) = \theta^2/2$ is the squared Euclidean distance. Thus, EPCA generalizes PCA.

+\end{aligned}\]

Similarly, the Bregman divergence induced from the log-partition of the Gaussian $G(\theta) = \theta^2/2$ is the squared Euclidean distance. Thus, EPCA generalizes PCA.

diff --git a/dev/math/appendix/inverses/index.html b/dev/math/appendix/inverses/index.html index 4a14a9d..c6b8589 100644 --- a/dev/math/appendix/inverses/index.html +++ b/dev/math/appendix/inverses/index.html @@ -6,4 +6,4 @@ &= \nabla_\mu \Big[ \mu \theta - G(\theta)\Big] \\ &= \theta + \mu \nabla_\mu \theta - g(\theta) \nabla_\mu \theta \\ &= \theta. \\ -\end{aligned}\]

The converse is similar, so $f = g^{-1}$.

Remark

Since $f$ is the inverse link function $g^{-1}$ and $\mu = g(\theta)$, we also have $f(\mu) = \theta$.

+\end{aligned}\]

The converse is similar, so $f = g^{-1}$.

Remark

Since $f$ is the inverse link function $g^{-1}$ and $\mu = g(\theta)$, we also have $f(\mu) = \theta$.

diff --git a/dev/math/appendix/poisson/index.html b/dev/math/appendix/poisson/index.html index dcbebbe..edf0d64 100644 --- a/dev/math/appendix/poisson/index.html +++ b/dev/math/appendix/poisson/index.html @@ -9,4 +9,4 @@ &= F(p) - F(q) - \langle f(q), p - q \rangle \\ &= p \log p - p - q \log q + q - \langle \log q, p - q \rangle \\ &= p \log \frac{p}{q} - p + q. -\end{aligned}\]

so $B_F$ is generalized KL-divergence.

+\end{aligned}\]

so $B_F$ is generalized KL-divergence.

diff --git a/dev/math/bregman/index.html b/dev/math/bregman/index.html index f46ce3a..34dace9 100644 --- a/dev/math/bregman/index.html +++ b/dev/math/bregman/index.html @@ -2,4 +2,4 @@ Bregman Divergences · ExpFamilyPCA
GitHub

Bregman Divergences

The EPCA objective is formulated as a Bregman divergence (Bregman, 1967). Bregman divergences are a measure of difference between two points (often probability distributions); however, they are not proper metrics, because they do not always satisfy symmetry and the triangle inequality.

Definition

Formally, the Bregman divergence $B_F$ associated with a function $F(\theta)$ is defined as

\[B_F(p \| q) = F(p) - F(q) - \langle f(p), p - q \rangle\]

where

  • $F(\mu)$ is a strictly convex and continuously differentiable function,
  • $f(\mu) = \nabla_\mu F(\mu)$ is the gradient of $F$,
  • and $\langle \cdot, \cdot \rangle$ denotes an inner product.

Intuitively, the Bregman divergence expresses the difference at $p$ between $F$ and its first-order Taylor expansion about $q$.

Aside: Properties

Bregman divergences vanish $B_F(p \| q) = 0$ if and only if their inputs also vanish $p = q = 0$. They are also always non-negative $B_F(p \| q) \geq 0$ for all $p, q \in \mathrm{domain}(F)$.

Info

While the full EPCA objective is always non-negative, the EPCA loss may be negative because ExpFamilyPCA.jl uses transformed objectives that are equivalent but not equal to minimizing a sum of Bregman divergences.

The Exponential Family

The natural exponential family is a broad class of probability distributions that includes many common distributions such as the Gaussian, binomial, Poisson, and gamma distributions. A probability distribution belongs to the exponential family if its probability density function (or mass function for discrete variables) can be expressed in the following canonical form:

\[p_\theta(x) = h(x) \exp(\langle \theta, x \rangle - G(\theta) )\]

where

  • $\theta$ is the natural parameter (also called the canonical parameter) of the distribution,
  • $h(x)$ is the base measure, independent of $\theta$,
  • and $G(\theta)$ is the log-partition function (also called the cumulant function) defined as:

\[G(\theta) = \log \int h(x) \exp(\langle \theta, x \rangle) \, dx.\]

The log-partition function $G(\theta)$ ensures that the probability distribution integrates (or sums) to $1$.

Key Parameters

  1. The natural parameter $\theta$ controls the distribution’s shape in its canonical form. For example, the natural parameter for the Poisson distribution is $\log \lambda$.
  2. The expectation parameter $\mu$ is the expected value of the sufficient statistic,[1] computed as the mean of the data under the distribution. In exponential family distributions, it is related to the natural parameter through the link function $g$:

\[\mu = \mathbb{E}_{\theta}[X] = \nabla_\theta G(\theta) = g(\theta)\]

where $E_\theta$ is the expectation with respect to the distribution $p_\theta$ (see appendix). Similarly, we also have $\theta = f(\mu)$ (see [appendix])(./appendix/inverses.md).

Convex Conjugation

The fact that $f$ and $g$ are inverses follows from the stronger claim that $F$ and $G$ are convex conjugates. For a convex function $F$, its convex conjugate (or dual)[2] $F^*$ is

\[F^*(\theta) = \sup_{\mu} [\langle \theta, \mu \rangle - F(\mu)].\]

Convex conjugation is also an involution meaning it inverts itself, so $F^{**} = F$. Conjugation provides a rich structure for converting between natural and expectation parameters and, as we explain in the next section, helps induce multiple useful specifications of the EPCA objective.

Bregman Loss Functions

Bregman divergences are crucial to EPCA, because they are equivalent (up to a constant) to maximum likelihood estimation for the exponential family (Azoury and Warmuth, 2001; Forster and Warmuth, 2002). To see this, consider the negative log-likelihood of such a distribution:

\[-\ell(x; \theta) = G(\theta) - \langle x, \theta \rangle.\]

We want to show that this is equivalent to the Bregman divergence $B_F(x \| \mu)$. From the previous subsection, we know $G$ is the convex conjugate of $F$, so:

\[G(\theta) = \langle \theta, \mu \rangle - F(\mu).\]

Substituting $G$ back into the negative log-likelihood, then yields:

\[\begin{aligned} -\ell(x; \theta) &= (\langle \theta, \mu \rangle - F(\mu)) - \langle x, \theta \rangle \\ &= -F(\mu) - \langle \theta, x - \mu \rangle. -\end{aligned}\]

The last line is equivalent to $B_F(x \| \mu)$ up to a constant, meaning we can interpret the Bregman divergence as a loss function that generalizes the maximum likelihood of any exponential family distribution. In the next section, we expand this idea to derive specific constructors for the EPCA objective.

  • 1The sufficient statistic for the natural exponential family is simply the identity.
  • 2Duality also refers to the concept in convex analysis (Boyd and Vandenberghe, 2004).
+\end{aligned}\]

The last line is equivalent to $B_F(x \| \mu)$ up to a constant, meaning we can interpret the Bregman divergence as a loss function that generalizes the maximum likelihood of any exponential family distribution. In the next section, we expand this idea to derive specific constructors for the EPCA objective.

diff --git a/dev/math/intro/index.html b/dev/math/intro/index.html index 7cf006b..7355685 100644 --- a/dev/math/intro/index.html +++ b/dev/math/intro/index.html @@ -9,4 +9,4 @@ & & B_F(X \| g(\Theta)) + \epsilon B_F(\mu_0 \| g(\Theta)) \\ & \text{subject to} & & \mathrm{rank}\left(\Theta\right) = k. -\end{aligned}\]

In this formulation:

On the next page, we dive deeper into Bregman divergences. We’ll explore their properties and how they connect to the exponential family, providing a solid foundation for understanding the probabilistic framework of EPCA.

+\end{aligned}\]

In this formulation:

On the next page, we dive deeper into Bregman divergences. We’ll explore their properties and how they connect to the exponential family, providing a solid foundation for understanding the probabilistic framework of EPCA.

diff --git a/dev/math/objectives/index.html b/dev/math/objectives/index.html index abcf365..7ce33e4 100644 --- a/dev/math/objectives/index.html +++ b/dev/math/objectives/index.html @@ -19,4 +19,4 @@ g(θ) = exp(θ) poisson_epca = EPCA(indim, outdim, B, g, Val((:B, :g)))

7. Using $\tilde{B}$ and $g$

Similarly, using the transformed Bregman divergence $\tilde{B}(p, q) = B_F(p \| g(q))$ along with $g$ is straightforward. Given that $\tilde{B}$ is just $B_F$ evaluated at $g(q)$, and we already have the link function $g$, defining the EPCA objective is almost too obvious to mention. While mathematically plain, this specification is useful in practice to avoid domain errors that make optimization difficult with certain exponential family members (e.g., gamma, negative binomial, Pareto).

Example:

Bg(x, θ) = exp(θ) - x * θ + x * log(x) - x
 g(θ) = exp(θ)
-poisson_epca = EPCA(indim, outdim, Bg, g, Val((:Bg, :g)))

Conclusion

In summary, the EPCA objective function and the decompression function $g$ can be derived from various components. The flexibility afforded by Julia's multiple dispatch and symbolic differentiation capabilities allows for efficient computation of EPCA in different scenarios. Depending on the form of the data and the problem at hand, one can choose the most convenient set of components to define and compute the EPCA objective.

+poisson_epca = EPCA(indim, outdim, Bg, g, Val((:Bg, :g)))

Conclusion

In summary, the EPCA objective function and the decompression function $g$ can be derived from various components. The flexibility afforded by Julia's multiple dispatch and symbolic differentiation capabilities allows for efficient computation of EPCA in different scenarios. Depending on the form of the data and the problem at hand, one can choose the most convenient set of components to define and compute the EPCA objective.

diff --git a/dev/math/references/index.html b/dev/math/references/index.html index 2a23787..835fae4 100644 --- a/dev/math/references/index.html +++ b/dev/math/references/index.html @@ -1,2 +1,2 @@ -References · ExpFamilyPCA
GitHub

References

+References · ExpFamilyPCA
GitHub

References