Hi Abbas,

You are right that the current loss function wouldn’t support a negative input from the previous layer. In practice this isn’t an issue since the output would typically be subject to a sigmoid or relu nonlinearity. In the case of this example, since the weights are initialized to 1, it is unlikely that they would drop below 0.

If you are working on a continuous control problem that requires negative values, then a different approach may be necessary, but otherwise this should work in the majority of cases.

PhD. Interests include Deep (Reinforcement) Learning, Computational Neuroscience, and Phenomenology.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store