Dynamic neural network architectures can deal naturally with sequential data through recursive processing enabled by feedback connections. We show how such architectures are predisposed for suffix-based Markovian input sequence representations in both supervised and unsupervised learning scenarios. In particular, in the context of such architectural predispositions, we study computational and learning capabilities of typical dynamic neural network architectures. We also show how efficient finite memory models can be readily extracted from untrained networks and argue that such models should be used as baselines when comparing dynamic network performances in a supervised learning task. Finally, potential applications of the Markovian architectural predisposition of dynamic neural networks in bioinformatics are presented.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Christiansen, M., Chater, N.: Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23 (1999) 417-437
Kolen, J.: Recurrent networks: state machines or iterated function systems? In Mozer, M., Smolensky, P., Touretzky, D., Elman, J., Weigend, A., eds.: Pro-ceedings of the 1993 Connectionist Models Summer School. Erlbaum Associates, Hillsdale, NJ (1994) 203-210
Kolen, J.: The origin of clusters in recurrent neural network state space. In: Proceedings from the Sixteenth Annual Conference of the Cognitive Science Society, Hillsdale, NJ: Lawrence Erlbaum Associates (1994) 508-513
Tiňo, P., Čerňanský, M., Beňušková, L.: Markovian architectural bias of recurrent neural networks. IEEE Transactions on Neural Networks 15 (2004) 6-15
Hammer, B., Tino, P.: Recurrent neural networks with small weights implement definite memory machines. Neural Computation 15 (2003) 1897-1929
Ron, D., Singer, Y., Tishby, N.: The power of amnesia. Machine Learning 25 (1996)
Tiňo, P., Hammer, B.: Architectural bias in recurrent neural networks: Fractal analysis. Neural Computation 15 (2004) 1931-1957
Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78 (1990) 1464-1479
Chappell, G., Taylor, J.: The temporal kohonen map. Neural Networks 6 (1993) 441-445
Koskela, T., znd J. Heikkonen, M.V., Kaski, K.: Recurrent SOM with local linear models in time series prediction. In: 6th European Symposium on Artificial Neural Networks. (1998) 167-172
Horio, K., Yamakawa, T.: Feedback self-organizing map and its application to spatio-temporal pattern classification. International Journal of Computational Intelligence and Applications 1 (2001) 1-18
Voegtlin, T.: Recursive self-organizing maps. Neural Networks 15 (2002) 979-992
Strickert, M., Hammer, B.: Merge SOM for temporal data. Neurocomputing 64 (2005)39-72
Hagenbuchner, M., Sperduti, A., Tsoi, A.: Self-organizing map for adaptive processing of structured data. IEEE Transactions on Neural Networks 14 (2003) 491-505
Hammer, B., Micheli, A., Strickert, M., Sperduti, A.: A general framework for unsupervised processing of structured data. Neurocomputing 57 (2004) 3-35
Tiňo, P., Farkaš, I., van Mourik, J.: Dynamics and topographic organization of recursive self-organizing maps. Neural Computation 18 (2006) 2529-2567
Falconer, K.: Fractal Geometry: Mathematical Foundations and Applications. John Wiley and Sons, New York (1990)
Barnsley, M.: Fractals everywhere. Academic Press, New York (1988)
Ron, D., Singer, Y., Tishby, N.: The power of amnesia. In: Advances in Neural Information Processing Systems 6, Morgan Kaufmann (1994) 176-183
Guyon, I., Pereira, F.: Design of a linguistic postprocessor using variable memory length markov models. In: International Conference on Document Analysis and Recognition, Monreal, Canada, IEEE Computer Society Press (1995) 454-457
Weinberger, M., Rissanen, J., Feder, M.: A universal finite memory source. IEEE Transactions on Information Theory 41 (1995) 643-652
Buhlmann, P., Wyner, A.: Variable length markov chains. Annals of Statistics 27 (1999) 480-513
Rissanen, J.: A universal data compression system. IEEE Trans. Inform. Theory 29 (1983) 656-664
Giles, C., Omlin, C.: Insertion and refinement of production rules in recurrent neural networks. Connection Science 5 (1993)
Doya, K.: Bifurcations in the learning of recurrent neural networks. In: Proc. of 1992 IEEE Int. Symposium on Circuits and Systems. (1992) 2777-2780
Tiňo, P., Dorffner, G.: Predicting the future of discrete sequences from fractal representations of the past. Machine Learning 45 (2001) 187-218
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. Technical Report GMD Report 148, German National Research Center for Information Technology (2001)
Jaeger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304 (2004) 78-80
Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14 (2002) 2531-2560
Maass, W., Legenstein, R.A., Bertschinger, N.: Methods for estimating the computational power and generalization capability of neural microcircuits. In: Advances in Neural Information Processing Systems. Volume 17., MIT Press (2005)865—872
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks 2 (1989) 359-366
Hammer, B.: Generalization ability of folding networks. IEEE Transactions on Knowledge and Data Engineering 13 (2001) 196-206
Koiran, P., Sontag, E.: Vapnik-chervonenkis dimension of recurrent neural net-works. In: European Conference on Computational Learning Theory. (1997) 223-237
Vapnik, V.: Statistical Learning Theory. Wiley-Interscience (1998)
Shawe-Taylor, J., Bartlett, P.L.: Structural risk minimization over data-dependent hierarchies. IEEE Trans. on Information Theory 44 (1998) 1926-1940
Williams, R., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1 (1989) 270-280
Williams, R.: Training recurrent networks using the extended kalman filter. In: Proc. 1992 Int. Joint Conf. Neural Networks. Volume 4. (1992) 241-246
Patel, G., Becker, S., Racine, R.: 2d image modelling as a time-series prediction problem. In Haykin, S., ed.: Kalman filtering applied to neural networks. Wiley (2001)
Manolios, P., Fanelli, R.: First order recurrent neural networks and deterministic finite state automata. Neural Computation 6 (1994) 1155-1173
Tiňo, P., Hammer, B.: Architectural bias in recurrent neural networks - fractal analysis. In Dorronsoro, J., ed.: Artificial Neural Networks - ICANN 2002. Lecture Notes in Computer Science, Springer-Verlag (2002) 1359-1364
de A. Barreto, G., Araújo, A., Kremer, S.: A taxanomy of spatiotemporal connectionist networks revisited: The unsupervised case. Neural Computation 15 (2003) 1255-1320
Kilian, J., Siegelmann, H.T.: On the power of sigmoid neural networks. Infor-mation and Computation 128 (1996) 48-56
Siegelmann, H.T., Sontag, E.D.: Analog computation via neural networks. The-oretical Computer Science 131 (1994) 331-360
Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Networks 17 (2004) 1061-1086
Hammer, B., Neubauer, N.: On the capacity of unsupervised recursive neural networks for symbol processing. In d’Avila Garcez, A., Hitzler, P., Tamburrini, G., eds.: Workshop proceedings of NeSy’06. (2006)
Sonnenburg, S.: New methods for splice site recognition. Master’s thesis, Diplom thesis, Institut für Informatik, Humboldt-Universität Berlin (2002)
Martinetz, T., Berkovich, S., Schulten, K.: ‘neural-gas’ networks for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Networks 4 (1993) 558-569
Baldi, P., Brunak, S.: Bioinformatics: The machine learning approach. MIT Press, Cambridge, Mass (2001)
Bodén, M., Hawkins, J.: Improved access to sequential motifs: A note on the architectural bias of recurrent networks. IEEE Transactions on Neural Networks 16 (2005) 491-494
Bailey, T.L., Elkan, C.: Fitting a mixture model by expectation maximization to discover motifs in biopolymers. In: Proceedings of ISMB, Stanford, CA (1994) 28-36
Stormo, G.D.: Dna binding sites: representation and discovery. Bioinformatics 16 (2000) 16-23
Sorek, R., Shemesh, R., Cohen, Y., Basechess, O., Ast, G., Shamir, R.: A non-EST-based method for exon-skipping prediction. Genome Research 14 (2004) 1617-1623
Hawkins, J., Bodén, M.: The applicability of recurrent neural networks for bio-logical sequence analysis. IEEE/ACM Transactions on Computational Biology and Bioinformatics 2 (2005) 243-253
Dyrlöv Bendtsen, J., Nielsen, H., von Heijne, G., Brunak, S.: Improved predic-tion of signal peptides: SignalP 3.0. Journal of Molecular Biology 340 (2004) 783-795
Hawkins, J., Bodén, M.: Detecting and sorting targeting papetides with neural networks and support vector machines. Journal of Bioinformatics and Compu-tational Biology 4 (2006) 1-18
Baldi, P., Brunak, S., Frasconi, P., Soda, G., Pollastri, G.: Exploiting the past and the future in protein secondary structure prediction. Bioinformatics 15 (1999) 937-946
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Tiňo, P., Hammer, B., Bodén, M. (2007). Markovian Bias of Neural-based Architectures With Feedback Connections. In: Hammer, B., Hitzler, P. (eds) Perspectives of Neural-Symbolic Integration. Studies in Computational Intelligence, vol 77. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73954-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-73954-8_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-73953-1
Online ISBN: 978-3-540-73954-8
eBook Packages: EngineeringEngineering (R0)