Posted in 2022
Likelihood, greed and temperature in sequence learning
- 28 May 2022
Imagine we have a model \(D(w)\) of a dynamical system with states \(s \in S,\) that is parametrized by some weight \(w \in W\). Each state \(s\) comes with a set \(N(s) \subset S\) of neighbors and an associated energy function \(E(s'|s,w) \in \mathbb{R}\) that assigns an energy to each neighbor \(s' \in N(s)\).
For simplicity, we assume the following dynamics: when the system is in state \(s\), it picks the neighbor \(s'\) with the lowest energy \(E(s'|s,w)\) and jumps to state \(s'\) in the next time step (more to come later about what we mean by time step).
Parametric typeclasses aid generalization in program synthesis
- 22 January 2022
We envision programming being done in top-down fashion. The human describes the goal (e.g. sorting), and the machine reduces it to smaller subgoals based on well-known heuristics (e.g. divide and conquer). The easier subgoals could even be fulfilled automatically. This top-down heuristics approach will be more amenable to machine learning. See my Topos Institute talk for more info.
The problem with the current approach in type theory is as follows.
Information topos theory - motivation
- 22 January 2022
Relative information (also known as the Kullback-Leibler divergence) is an important fundamental concept in statistical learning and information theory.
The (conditional) relative information