Emergence of modularity within one sheet of intrinsically active stochastic neurons

We investigate how modular structure within a neural net of stochastic neurons can emerge from intrinsic dynamics and from unsupervised learning of data. The model consists of an input layer and two hierarchically organized hidden layers and differs from the Helmholtz machine only in that hidden neurons are not assigned to a single one of the layers prior to learning. On the basis of a maximum likelihood framework the task is (i) to infer a hidden code from data using the recognition weights and (ii) to generate given input data from the hidden code using the generative weights. Hidden neurons are fully connected via both weight types allowing to code on different hierarchical levels. The hidden neurons are separated into two groups by their intrinsic parameters. One group, designed to be on the highest hierarchical level, is highly spontaneously active and is thus responsible for the initiation of the hidden code. The other group, designed to be a lower level, is highly responsive to stimulation by input only and thus likely transmits information between the input and the highest level in both directions. Besides demonstrating the emergence of this hierarchical structure, we show using a different data set that a parallel structure emerges which matches the data. Finally, if the net is trained without any input, then a weak hierachical structure emerges by the differential intrinsic activity.

@InProceedings{WO00, 
 	 author =  {Weber, Cornelius and Obermayer, Klaus},  
 	 title = {Emergence of modularity within one sheet of intrinsically active stochastic neurons}, 
 	 booktitle = {Proc. 7th International Conference on Neural Information Processing (ICONIP)},
 	 editors = {},
 	 number = {},
 	 volume = {},
 	 pages = {},
 	 year = {2000},
 	 month = {},
 	 publisher = {},
 	 doi = {}, 
 }