Skip to content

LSTMNetwork

Unreliable

As far as I can tell, my LSTM implementation is a complete failure. It does not achieve any measurable success or advantage over feedforward networks and cannot predict even simple number sequences. The internet has little to no readable information on the inner mathematical workings of LSTMs since few have the interest of creating new libraries, so I have no way of knowing how a true LSTM behaves internally. Feel free to try it but I can't give any guarantees.

This class is responsible for constructing and managing LSTM (long short term memory) networks. This type of network is specialized for extrapolating a datapoint that would continue the series of given datapoints. A common application of this is text generation, though it can be applied to any scenario where there is a sequence that needs to be continued.

LSTMNetwork .new(array inputNamesArray, number numberOfLSTMLayers, number numberOfLSTMNodesPerLayer, array outputNamesArray, dictionary customSettings) Creates and returns the LSTMNetwork according to the given settings as !customSettings!. The settings determine how the network will perform and function and is open to customization. The available setting parameters and their default values are below. 
local customSettings = {
    Optimizer = StochasticGradientDescent.new();
    SoftMax = false;

    HiddenActivationName = "ReLU";
    OutputActivationName = "Sigmoid";
    Bias = 0;
    LearningRate = 0.3;
    NumOfInputsForLSTMUnits = {Default = 1};
    MakeDenseLayer = true;
    MakeDirectOutput = false;
    RandomizeWeights = true;
}

Inherited from NeuralNetwork:

Only unchanged and unmodified functions are listed below.

NeuralNetwork .newFromSave(string serial) Deserializes and returns the neural network from the !serial! string: the save string.
string :Save() Returns the serialized form of the neural network as a string, allowing you to save the network.
void :ConnectNodes(Node inNode, Node outNode, bool checkOveride=false) Creates a synapse for the 2 nodes with !inNode! as the input and !outNode! as the output. If !checkOveride! is true, this function will not check if the synapse between these 2 nodes already exists; this is purely for internal performance purposes where the check is unnecessary. Do not set this to true if you are experimenting unless you're confident.
dictionary :(dictionary inputValues, bool doNotClearOtherInputValues=false) Propagates the network with the given !inputValues! if provided. If not, the network will run with the input values already set previously. If !doNotClearOtherInputValues! is true, any inputs that are missing from the !inputValues! dictionary are set to 0.

This function is fired when the NeuralNetwork object is called, such as: 
neuralNetwork:({Input1 = 0.5, Input2 = -0.2})
dictionary :GetOutputValues() Returns the current output values.
void :SetInputValues(dictionary inputValues, bool doNotClearOtherInputValues=false) Sets the given input values as !inputValues! into the input nodes. If !doNotClearOtherInputValues! is true, any inputs that are missing from the !inputValues! dictionary are set to 0.
void :ClearInputValues() Sets all input values of the input nodes to 0.
void :ClearValues() Sets all the output values of every functional node to 0.
array :GetNodes() Returns every node in the network in an unreliable order.
array :GetInputNodes() Returns the input nodes in the network in the order they were assigned by the user upon network creation.
void :AddInputNode(InputNode inputNode) Adds input node !inputNode! to the network.
array :GetOutputNodes() Returns the output nodes in the network in the order they were assigned by the user upon network creation.
void :AddOutputNode(OutputNode outputNode) Adds output node !outputNode! to the network.
array :GetHiddenNodes() Returns the hidden nodes in the network in the order they were created (first node to last node of first layer, first node to last node of second layer, etc).
void :AddHiddenNode(Node node) Adds the hidden node !node! to the network.
array :GetFunctionNodes() Returns the functional nodes in the network. Functional nodes, in my definition, are nodes that have a working bias and weight set. Thus, hidden nodes and output nodes are all functional nodes.
void :AddNode(Node node) Adds the node !node! to the network.
BackPropagator :GetBackPropagator() Returns the BackPropagator object for this network.
void :RandomizeWeights(number min=-0.5, number max=0.5) Randomizes all weights in the network with the given minimum !min! and maximum !max! values.
Optimizer :GetOptimizer() Returns the Optimizer object the network is using.
void :AddRandomNoise(number min, number max) Adds a random noise to every parameter in the network with the given minimum !min! and maximum !max! values.