Cross entropy

港控/mmm° 2022-09-22 01:13 167阅读 0赞

In [information theory][], the **cross entropy** between two [probability distributions][] \{\\displaystyle p\}![p][] and \{\\displaystyle q\}![q][] over the same underlying set of events measures the average number of [bits][] needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an "unnatural" probability distribution \{\\displaystyle q\}![q][], rather than the "true" distribution \{\\displaystyle p\}![p][].

The cross entropy for the distributions \{\\displaystyle p\}![p][] and \{\\displaystyle q\}![q][] over a given set is defined as follows:

\{\\displaystyle H(p,q)=\\operatorname \{E\} \_\{p\}\[-\\log q\]=H(p)+D\_\{\\mathrm \{KL\} \}(p\\|q),\\!\} ![H(p,q)=\\operatorname \{E\}\_\{p\}\[-\\log q\]=H(p)+D\_\{\{\{\\mathrm  \{KL\}\}\}\}(p\\|q),\\!][H_p_q_operatorname _E_p_-_log q_H_p_D_mathrm _KL_p_q]

where \{\\displaystyle H(p)\}![H(p)][H_p] is the [entropy][] of \{\\displaystyle p\}![p][], and \{\\displaystyle D\_\{\\mathrm \{KL\} \}(p||q)\}![D\_\{\{\{\\mathrm  \{KL\}\}\}\}(p||q)][D_mathrm _KL_p_q] is the [Kullback–Leibler divergence][Kullback_Leibler divergence] of \{\\displaystyle q\}![q][] from \{\\displaystyle p\}![p][] (also known as the *relative entropy* of *p* with respect to *q* — note the reversal of emphasis).

For [discrete][] \{\\displaystyle p\}![p][] and \{\\displaystyle q\}![q][] this means

\{\\displaystyle H(p,q)=-\\sum \_\{x\}p(x)\\,\\log q(x).\\!\} ![H(p,q)=-\\sum \_\{x\}p(x)\\,\\log q(x).\\!][H_p_q_-_sum _x_p_x_log q_x_.]

The situation for [continuous][] distributions is analogous:

\{\\displaystyle -\\int \_\{X\}p(x)\\,\\log q(x)\\,dx.\\!\} ![\-\\int \_\{X\}p(x)\\,\\log q(x)\\,dx.\\!][-_int _X_p_x_log q_x_dx.]

NB: The notation \{\\displaystyle H(p,q)\}![H(p,q)][H_p_q] is also used for a different concept, the [joint entropy][] of \{\\displaystyle p\}![p][] and \{\\displaystyle q\}![q][].

## Contents ##

\[[hide][]\]

*  [1Motivation][]
 *  [2Estimation][]
 *  [3Cross-entropy minimization][]
 *  [4Cross-entropy error function and logistic regression][]
 *  [5References][]
 *  [6See also][]
 *  [7External links][]

## Motivation\[[edit][]\] ##

In information theory, the [Kraft–McMillan theorem][Kraft_McMillan theorem] establishes that any directly decodable coding scheme for coding a message to identify one value \{\\displaystyle x\_\{i\}\}![x\_\{i\}][x_i] out of a set of possibilities \{\\displaystyle X\}![X][] can be seen as representing an implicit probability distribution \{\\displaystyle q(x\_\{i\})=2^\{-l\_\{i\}\}\}![q(x\_\{i\})=2^\{\{-l\_\{i\}\}\}][q_x_i_2_-l_i] over \{\\displaystyle X\}![X][], where \{\\displaystyle l\_\{i\}\}![l\_\{i\}][l_i] is the length of the code for \{\\displaystyle x\_\{i\}\}![x\_\{i\}][x_i] in bits. Therefore, cross entropy can be interpreted as the expected message-length per datum when a wrong distribution \{\\displaystyle Q\}![Q][] is assumed while the data actually follows a distribution \{\\displaystyle P\}![P][]. That is why the expectation is taken over the probability distribution \{\\displaystyle P\}![P][] and not \{\\displaystyle Q\}![Q][].

\{\\displaystyle H(p,q)=\\operatorname \{E\} \_\{p\}\[l\_\{i\}\]=\\operatorname \{E\} \_\{p\}\\left\[\\log \{\\frac \{1\}\{q(x\_\{i\})\}\}\\right\]\} ![H(p,q)=\\operatorname \{E\}\_\{p\}\[l\_\{i\}\]=\\operatorname \{E\}\_\{p\}\\left\[\\log \{\\frac  \{1\}\{q(x\_\{i\})\}\}\\right\]][H_p_q_operatorname _E_p_l_i_operatorname _E_p_left_log _frac _1_q_x_i_right]

\{\\displaystyle H(p,q)=\\sum \_\{x\_\{i\}\}p(x\_\{i\})\\,\\log \{\\frac \{1\}\{q(x\_\{i\})\}\}\\!\} ![H(p,q)=\\sum \_\{\{x\_\{i\}\}\}p(x\_\{i\})\\,\\log \{\\frac  \{1\}\{q(x\_\{i\})\}\}\\!][H_p_q_sum _x_i_p_x_i_log _frac _1_q_x_i]

\{\\displaystyle H(p,q)=-\\sum \_\{x\}p(x)\\,\\log q(x).\\!\} ![H(p,q)=-\\sum \_\{x\}p(x)\\,\\log q(x).\\!][H_p_q_-_sum _x_p_x_log q_x_.]

## Estimation\[[edit][edit 1]\] ##

There are many situations where cross-entropy needs to be measured but the distribution of \{\\displaystyle p\}![p][] is unknown. An example is [language modeling][], where a model is created based on a training set \{\\displaystyle T\}![T][], and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, \{\\displaystyle p\}![p][] is the true distribution of words in any corpus, and \{\\displaystyle q\}![q][] is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

\{\\displaystyle H(T,q)=-\\sum \_\{i=1\}^\{N\}\{\\frac \{1\}\{N\}\}\\log \_\{2\}q(x\_\{i\})\} ![H(T,q)=-\\sum \_\{\{i=1\}\}^\{N\}\{\\frac  \{1\}\{N\}\}\\log \_\{2\}q(x\_\{i\})][H_T_q_-_sum _i_1_N_frac _1_N_log _2_q_x_i]

where \{\\displaystyle N\}![N][] is the size of the test set, and \{\\displaystyle q(x)\}![q(x)][q_x] is the probability of event \{\\displaystyle x\}![x][] estimated from the training set. The sum is calculated over \{\\displaystyle N\}![N][]. This is a Monte Carlo estimate of the true cross entropy, where the training set is treated as samples from \{\\displaystyle p(x)\}![p(x)][p_x].

## Cross-entropy minimization\[[edit][edit 2]\] ##

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the [cross-entropy method][].

When comparing a distribution \{\\displaystyle q\}![q][] against a fixed reference distribution \{\\displaystyle p\}![p][], cross entropy and [KL divergence][Kullback_Leibler divergence] are identical up to an additive constant (since \{\\displaystyle p\}![p][] is fixed): both take on their minimal values when \{\\displaystyle p=q\}![p=q][p_q], which is \{\\displaystyle 0\}![\{\\displaystyle 0\}][displaystyle 0] for KL divergence, and \{\\displaystyle \\mathrm \{H\} (p)\}![\{\\mathrm  \{H\}\}(p)][mathrm _H_p] for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "[Principle of Minimum Discrimination Information][]") is often called the **Principle of Minimum Cross-Entropy** (MCE), or **Minxent**.

However, as discussed in the article *[Kullback–Leibler divergence][Kullback_Leibler divergence]*, sometimes the distribution \{\\displaystyle q\}![q][] is the fixed prior reference distribution, and the distribution \{\\displaystyle p\}![p][] is optimised to be as close to \{\\displaystyle q\}![q][] as possible, subject to some constraint. In this case the two minimisations are*not* equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be \{\\displaystyle D\_\{\\mathrm \{KL\} \}(p\\|q)\}![D\_\{\{\{\\mathrm  \{KL\}\}\}\}(p\\|q)][D_mathrm _KL_p_q 1], rather than \{\\displaystyle H(p,q)\}![H(p,q)][H_p_q].

## Cross-entropy error function and logistic regression\[[edit][edit 3]\] ##

Cross entropy can be used to define the loss function in machine learning and optimization. The true probability \{\\displaystyle p\_\{i\}\}![p\_\{i\}][p_i] is the true label, and the given distribution \{\\displaystyle q\_\{i\}\}![q\_\{i\}][q_i] is the predicted value of the current model.

More specifically, let us consider [logistic regression][], which (in its most basic form) deals with classifying a given set of data points into two possible classes generically labelled \{\\displaystyle 0\}![\{\\displaystyle 0\}][displaystyle 0] and \{\\displaystyle 1\}![1][]. The logistic regression model thus predicts an output \{\\displaystyle y\\in \\\{0,1\\\}\}![y\\in \\\{0,1\\\}][y_in _0_1], given an input vector \{\\displaystyle \\mathbf \{x\} \}![\\mathbf \{x\}][mathbf _x]. The probability is modeled using the [logistic function][] \{\\displaystyle g(z)=1/(1+e^\{-z\})\}![g(z)=1/(1+e^\{\{-z\}\})][g_z_1_1_e_-z]. Namely, the probability of finding the output \{\\displaystyle y=1\}![y=1][y_1] is given by

\{\\displaystyle q\_\{y=1\}\\ =\\ \{\\hat \{y\}\}\\ \\equiv \\ g(\\mathbf \{w\} \\cdot \\mathbf \{x\} )\\,,\} ![q\_\{\{y=1\}\}\\ =\\ \{\\hat  \{y\}\}\\ \\equiv \\ g(\{\\mathbf  \{w\}\}\\cdot \{\\mathbf  \{x\}\})\\,,][q_y_1_ _ _hat _y_ _equiv _ g_mathbf _w_cdot _mathbf _x]

where the vector of weights \{\\displaystyle \\mathbf \{w\} \}![\\mathbf \{w\}][mathbf _w] is optimized through some appropriate algorithm such as [gradient descent][]. Similarly, the complementary probability of finding the output \{\\displaystyle y=0\}![y=0][y_0] is simply given by

\{\\displaystyle q\_\{y=0\}\\ =\\ 1-\{\\hat \{y\}\}\} ![q\_\{\{y=0\}\}\\ =\\ 1-\{\\hat  \{y\}\}][q_y_0_ _ 1-_hat _y]

The true (observed) probabilities can be expressed similarly as \{\\displaystyle p\_\{y=1\}=y\}![p\_\{\{y=1\}\}=y][p_y_1_y] and \{\\displaystyle p\_\{y=0\}=1-y\}![p\_\{\{y=0\}\}=1-y][p_y_0_1-y].

Having set up our notation, \{\\displaystyle p\\in \\\{y,1-y\\\}\}![p\\in \\\{y,1-y\\\}][p_in _y_1-y] and \{\\displaystyle q\\in \\\{ \{\\hat \{y\}\},1-\{\\hat \{y\}\}\\\}\}![q\\in \\\{\{\\hat  \{y\}\},1-\{\\hat  \{y\}\}\\\}][q_in _hat _y_1-_hat _y], we can use cross entropy to get a measure for similarity between \{\\displaystyle p\}![p][] and \{\\displaystyle q\}![q][]:

\{\\displaystyle H(p,q)\\ =\\ -\\sum \_\{i\}p\_\{i\}\\log q\_\{i\}\\ =\\ -y\\log \{\\hat \{y\}\}-(1-y)\\log(1-\{\\hat \{y\}\})\} ![H(p,q)\\ =\\ -\\sum \_\{i\}p\_\{i\}\\log q\_\{i\}\\ =\\ -y\\log \{\\hat  \{y\}\}-(1-y)\\log(1-\{\\hat  \{y\}\})][H_p_q_ _ -_sum _i_p_i_log q_i_ _ -y_log _hat _y_-_1-y_log_1-_hat _y]

The typical loss function that one uses in logistic regression is computed by taking the average of all cross-entropies in the sample. For example, suppose we have \{\\displaystyle N\}![N][] samples with each sample labeled by \{\\displaystyle n=1,\\dots ,N\}![n=1,\\dots ,N][n_1_dots _N]. The loss function is then given by:

\{\\displaystyle \{\\begin\{aligned\}L(\\mathbf \{w\} )\\ &=\\ \{\\frac \{1\}\{N\}\}\\sum \_\{n=1\}^\{N\}H(p\_\{n\},q\_\{n\})\\ =\\ -\{\\frac \{1\}\{N\}\}\\sum \_\{n=1\}^\{N\}\\ \{\\bigg \[\}y\_\{n\}\\log \{\\hat \{y\}\}\_\{n\}+(1-y\_\{n\})\\log(1-\{\\hat \{y\}\}\_\{n\})\{\\bigg \]\}\\,,\\end\{aligned\}\}\} ![\{\\begin\{aligned\}L(\{\\mathbf  \{w\}\})\\ &=\\ \{\\frac  1N\}\\sum \_\{\{n=1\}\}^\{N\}H(p\_\{n\},q\_\{n\})\\ =\\ -\{\\frac  1N\}\\sum \_\{\{n=1\}\}^\{N\}\\ \{\\bigg \[\}y\_\{n\}\\log \{\\hat  y\}\_\{n\}+(1-y\_\{n\})\\log(1-\{\\hat  y\}\_\{n\})\{\\bigg \]\}\\,,\\end\{aligned\}\}][begin_aligned_L_mathbf _w_ _ _frac 1N_sum _n_1_N_H_p_n_q_n_ _ -_frac 1N_sum _n_1_N_ _bigg _y_n_log _hat y_n_1-y_n_log_1-_hat y_n_bigg _end_aligned]

where \{\\displaystyle \{\\hat \{y\}\}\_\{n\}\\equiv g(\\mathbf \{w\} \\cdot \\mathbf \{x\} \_\{n\})\}![\{\\hat  \{y\}\}\_\{n\}\\equiv g(\{\\mathbf  \{w\}\}\\cdot \{\\mathbf  \{x\}\}\_\{n\})][hat _y_n_equiv g_mathbf _w_cdot _mathbf _x_n], with \{\\displaystyle g(z)\}![g(z)][g_z] the logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It's also known as log loss (In this case, the binary label is often denoted by \{-1,+1\}).[\[1\]][1 1]

[information theory]: https://en.wikipedia.org/wiki/Information_theory
[probability distributions]: https://en.wikipedia.org/wiki/Probability_distribution
[p]: https://wikimedia.org/api/rest_v1/media/math/render/svg/81eac1e205430d1f40810df36a0edffdc367af36
[q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/06809d64fa7c817ffc7e323f85997f783dbdf71d
[bits]: https://en.wikipedia.org/wiki/Bit
[H_p_q_operatorname _E_p_-_log q_H_p_D_mathrm _KL_p_q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/80bd13c723dce5056a6f3aa1b29e279fb90d40bd
[H_p]: https://wikimedia.org/api/rest_v1/media/math/render/svg/d2ec3c386ddf7e6c880691d84669ad0f5d7d2596
[entropy]: https://en.wikipedia.org/wiki/Information_entropy
[D_mathrm _KL_p_q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/6e70eecd4bca2d0e9338aea6aebc9c29eb699e19
[Kullback_Leibler divergence]: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
[discrete]: https://en.wikipedia.org/wiki/Discrete_random_variable
[H_p_q_-_sum _x_p_x_log q_x_.]: https://wikimedia.org/api/rest_v1/media/math/render/svg/0cb6da032ab424eefdca0884cd4113fe578f4293
[continuous]: https://en.wikipedia.org/wiki/Continuous_random_variable
[-_int _X_p_x_log q_x_dx.]: https://wikimedia.org/api/rest_v1/media/math/render/svg/cc15728e952d3773449413bf6dd95895a1f7c54e
[H_p_q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/6e73dcf68e3f43f556220163269bb355b498936b
[joint entropy]: https://en.wikipedia.org/wiki/Joint_entropy
[hide]: https://en.wikipedia.org/wiki/Cross_entropy#
[1Motivation]: https://en.wikipedia.org/wiki/Cross_entropy#Motivation
[2Estimation]: https://en.wikipedia.org/wiki/Cross_entropy#Estimation
[3Cross-entropy minimization]: https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_minimization
[4Cross-entropy error function and logistic regression]: https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression
[5References]: https://en.wikipedia.org/wiki/Cross_entropy#References
[6See also]: https://en.wikipedia.org/wiki/Cross_entropy#See_also
[7External links]: https://en.wikipedia.org/wiki/Cross_entropy#External_links
[edit]: https://en.wikipedia.org/w/index.php?title=Cross_entropy&action=edit&section=1
[Kraft_McMillan theorem]: https://en.wikipedia.org/wiki/Kraft%27s_inequality
[x_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/e87000dd6142b81d041896a30fe58f0c3acb2158
[X]: https://wikimedia.org/api/rest_v1/media/math/render/svg/68baa052181f707c662844a465bfeeb135e82bab
[q_x_i_2_-l_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/eae4c79b205fcc4affe95605a5f4c6336bab777e
[l_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/48ba35055d00bcaa522bca9247c8857730998759
[Q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/8752c7023b4b3286800fe3238271bbca681219ed
[P]: https://wikimedia.org/api/rest_v1/media/math/render/svg/b4dc73bf40314945ff376bd363916a738548d40a
[H_p_q_operatorname _E_p_l_i_operatorname _E_p_left_log _frac _1_q_x_i_right]: https://wikimedia.org/api/rest_v1/media/math/render/svg/374e8c67971e6177b0123302c419a34cc5adc8c2
[H_p_q_sum _x_i_p_x_i_log _frac _1_q_x_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/4e41b8ce09c9e3f532c89013dca5396a5bf34484
[edit 1]: https://en.wikipedia.org/w/index.php?title=Cross_entropy&action=edit&section=2
[language modeling]: https://en.wikipedia.org/wiki/Language_model
[T]: https://wikimedia.org/api/rest_v1/media/math/render/svg/ec7200acd984a1d3a3d7dc455e262fbe54f7f6e0
[H_T_q_-_sum _i_1_N_frac _1_N_log _2_q_x_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/bb11eae1b2b1120c2bcccf741a51c2511c0cbffe
[N]: https://wikimedia.org/api/rest_v1/media/math/render/svg/f5e3890c981ae85503089652feb48b191b57aae3
[q_x]: https://wikimedia.org/api/rest_v1/media/math/render/svg/c38bbafe34a043d284f19231b946a76c0a4b16b4
[x]: https://wikimedia.org/api/rest_v1/media/math/render/svg/87f9e315fd7e2ba406057a97300593c4802b53e4
[p_x]: https://wikimedia.org/api/rest_v1/media/math/render/svg/8cb7afced134ef75572e5314a5d278c2d644f438
[edit 2]: https://en.wikipedia.org/w/index.php?title=Cross_entropy&action=edit&section=3
[cross-entropy method]: https://en.wikipedia.org/wiki/Cross-entropy_method
[p_q]: https://wikimedia.org/api/rest_v1/media/math/render/svg/4cf52da7809003406aee7e680eb5476a05a5c546
[displaystyle 0]: https://wikimedia.org/api/rest_v1/media/math/render/svg/2aae8864a3c1fec9585261791a809ddec1489950
[mathrm _H_p]: https://wikimedia.org/api/rest_v1/media/math/render/svg/76d5ade34024658505d6df9f6114079845d005b5
[Principle of Minimum Discrimination Information]: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Principle_of_minimum_discrimination_information
[D_mathrm _KL_p_q 1]: https://wikimedia.org/api/rest_v1/media/math/render/svg/01c66b767db801bc3b0b2910e69c1c3dca395861
[edit 3]: https://en.wikipedia.org/w/index.php?title=Cross_entropy&action=edit&section=4
[p_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/5bab39399bf5424f25d957cdc57c84a0622626d2
[q_i]: https://wikimedia.org/api/rest_v1/media/math/render/svg/2752dcbff884354069fe332b8e51eb0a70a531b6
[logistic regression]: https://en.wikipedia.org/wiki/Logistic_regression
[1]: https://wikimedia.org/api/rest_v1/media/math/render/svg/92d98b82a3778f043108d4e20960a9193df57cbf
[y_in _0_1]: https://wikimedia.org/api/rest_v1/media/math/render/svg/7c66170a51fd7b916d42c2cf8e8512c75c85a594
[mathbf _x]: https://wikimedia.org/api/rest_v1/media/math/render/svg/32adf004df5eb0a8c7fd8c0b6b7405183c5a5ef2
[logistic function]: https://en.wikipedia.org/wiki/Logistic_function
[g_z_1_1_e_-z]: https://wikimedia.org/api/rest_v1/media/math/render/svg/dd04cb873b2f781ab5193c1e900feaabc135d22f
[y_1]: https://wikimedia.org/api/rest_v1/media/math/render/svg/10f53b404b1fdd041a589f1f2425e45a2edba110
[q_y_1_ _ _hat _y_ _equiv _ g_mathbf _w_cdot _mathbf _x]: https://wikimedia.org/api/rest_v1/media/math/render/svg/04f709bcb8b37b2e3d3af449eb5cf92d5e5b577d
[mathbf _w]: https://wikimedia.org/api/rest_v1/media/math/render/svg/20795664b5b048744a2fd88977851104cc5816f8
[gradient descent]: https://en.wikipedia.org/wiki/Gradient_descent
[y_0]: https://wikimedia.org/api/rest_v1/media/math/render/svg/094f824655138f6b11d96a0da32e7f0716ba6959
[q_y_0_ _ 1-_hat _y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/2c30e8f9223b676407bc6b431d3ef55b46d2de4a
[p_y_1_y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/d0f948403fc13952e81827bfd89c4f84290ae387
[p_y_0_1-y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/ecacbd2ae760c4d4b057c940c6f5a24076df21d1
[p_in _y_1-y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/e3ab7d49ae71e4d41532ceabf91dfada3ed4774e
[q_in _hat _y_1-_hat _y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/625742439e4971c3234ca3099238b310ca5edeb7
[H_p_q_ _ -_sum _i_p_i_log q_i_ _ -y_log _hat _y_-_1-y_log_1-_hat _y]: https://wikimedia.org/api/rest_v1/media/math/render/svg/1f3f3acfb5549feb520216532a40082193c05ccc
[n_1_dots _N]: https://wikimedia.org/api/rest_v1/media/math/render/svg/ad5340f154b80ccd58b1343c863880a8155a1d06
[begin_aligned_L_mathbf _w_ _ _frac 1N_sum _n_1_N_H_p_n_q_n_ _ -_frac 1N_sum _n_1_N_ _bigg _y_n_log _hat y_n_1-y_n_log_1-_hat y_n_bigg _end_aligned]: https://wikimedia.org/api/rest_v1/media/math/render/svg/106c195cc961bd026ad949ad5ff89f3cde845e2c
[hat _y_n_equiv g_mathbf _w_cdot _mathbf _x_n]: https://wikimedia.org/api/rest_v1/media/math/render/svg/560a9b4b3a9c5377c693e5492aa5eab2828b148d
[g_z]: https://wikimedia.org/api/rest_v1/media/math/render/svg/8eb6fb9c0f0d402f5ebbebfd5e34bedd39a4a52b
[1 1]: https://en.wikipedia.org/wiki/Cross_entropy#cite_note-1