Posted in 2023
Online learning for spiking neural networks with relative information rate
- 11 December 2023
Spiking neural networks (SNNs) mimic biological neural networks more closely than feedforward neural networks (FNNs), and are pursued because of the promise of low-energy training and inference. While learning methods such as Hebbian algorithms and STDP (spiking-timing-dependent plasticity) exist, the learning theory of SNNs is poorly understood. In particular, little is known about how SNNs with memory (i.e. latent or hidden variables) can be trained effectively, making it difficult to build large SNNs that rival the performance of large FNNs. In this talk, we attack this problem with the information theory of time series. Using relative information rate, Amari’s em algorithm and stochastic approximation theory, we derive online learning algorithms for SNNs with memory. It turns out that STDP is a consequence of this algorithm, rather than its basis. This is joint work with Tenzin Chan, Chris Hillar and Sarah Marzen.
Institute for Mathematical and Statistical Innovation; Bayesian Statistics and Statistical Learning
Relative Information and the Dual Numbers
- 25 October 2023
Relative information (Kullback-Leibler divergence) is a fundamental concept in statistics, machine learning and information theory.
In the first half of the talk, I will define conditional relative information, list its axiomatic properties, and describe how it is used in machine learning. For example, the generalization error of a learning algorithm depends on the structure of algebraic geometric singularities of relative information.
References on information cohomology
- 20 August 2023
Some references on the cohomological nature of various information theoretic concepts such as entropy and relative information.
Markov categories were defined by Fritz [Fri20].
All you need is relative information
- 26 June 2023
SLT has taught me that relative information (or Kullback-Leibler divergence) is all you need. For instance, the level sets of relative information give us state density functions, whose Fourier, Laplace and Mellin transforms reveal different facets of learning algorithms and their ability to generalize. In this talk, I will outline two ongoing projects involving relative information. The first project explores the information theory of time series, for the purpose of understanding language models and reinforcement learning. Using relative information rate, we derive stochastic learning algorithms for spiking neural networks with memory. The second project explores cohomological generalizations of relative information, building on recent work by Pierre Baudot and Daniel Bennequin and by Tai-Danae Bradley. The hope is to uncover new tools for studying non-statistical spaces, by analyzing the level sets of generalized relative information and their transforms.
Relative information is motivic
- 01 April 2023
A (Hochschild) cohomological view of relative information must be motivic!
Vigneaux gives some hints of this [Vig21].