Search

A scheme to provide various mean-field-type approximation algorithms is presented by employing the Bethe free energy formalism to a family of replicated systems in conjunction with analytical continuation with respect to the number of replicas. In the scheme, survey propagation (SP), which is an efficient algorithm developed recently for analyzing the microscopic properties of glassy states for a fixed sample of disordered systems, can be reproduced by assuming the simplest replica symmetry on stationary points of the replicated Bethe free energy. Belief propagation and generalized SP can also be offered in the identical framework under assumptions of the highest and broken replica symmetries, respectively., Comment: appeared in Journal of the Physical Society of Japan 74, 2133-2136 (2005)

The demand for extracting rules from high dimensional real world data is increasing in various fields. However, the possible redundancy of such data sometimes makes it difficult to obtain a good generalization ability for novel samples. To resolve this problem, we provide a scheme that reduces the effective dimensions of data by pruning redundant components for bicategorical classification based on the Bayesian framework. First, the potential of the proposed method is confirmed in ideal situations using the replica method. Unfortunately, performing the scheme exactly is computationally difficult. So, we next develop a tractable approximation algorithm, which turns out to offer nearly optimal performance in ideal cases when the system size is large. Finally, the efficacy of the developed classifier is experimentally examined for a real world problem of colon cancer classification, which shows that the developed method can be practically useful., Comment: 13 pages, 6 figures

Belief propagation (BP) and the concave convex procedure (CCCP) are both methods that utilize the Bethe free energy as a cost function and solve information processing tasks. We have developed a new algorithm that also uses the Bethe free energy, but changes the roles of the master variables and the slave variables. This is called the Bowman-Levin (BL) approximation in the domain of statistical physics. When we applied the BL algorithm to decode the Gallager ensemble of short-length regular low-density parity check codes (LDPCC) over an additive white Gaussian noise (AWGN) channel, its average performance was somewhat better than that of either BP or CCCP. This implies that the BL algorithm can also be successfully applied to other problems to which BP or CCCP has already been applied., Comment: 2005 IEEE International Symposium on Information Theory

We present herein a scheme by which to accurately evaluate the error exponents of a lossy data compression problem, which characterize average probabilities over a code ensemble of compression failure and success above or below a critical compression rate, respectively, utilizing the replica method (RM). Although the existing method used in information theory (IT) is, in practice, limited to ensembles of randomly constructed codes, the proposed RM-based approach can be applied to a wider class of ensembles. This approach reproduces the optimal expressions of the error exponents achieved by the random code ensembles, which are known in IT. In addition, the proposed framework is used to show that codes composed of non-monotonic perceptrons of a specific type can provide the optimal exponents in most cases, which is supported by numerical experiments., Comment: 20 pages, 4 figures

An expression for the moment of partition function valid for any finite system size $N$ and complex power $n (\Re(n)>0)$ is obtained for a simple spin glass model termed the {\em discrete random energy model} (DREM). We investigate the behavior of the moment in the thermodynamic limit $N \to \infty$ using this expression, and find that a phase transition occurs at a certain real replica number when the temperature is sufficiently low, directly clarifying the scenario of replica symmetry breaking of DREM in the replica number space {\em without using the replica trick}. The validity of the expression is numerically confirmed., Comment: 31 pages, 8 eps figures

We investigate dynamics of an inference algorithm termed the belief propagation (BP) when employed in spin glass (SG) models and show that its macroscopic behaviors can be traced by recursive updates of certain auxiliary field distributions whose stationary state reproduces the replica symmetric solution offered by the equilibrium analysis. We further provide a compact expression for the instability condition of the BP's fixed point which turns out to be identical to that of instability for breaking the replica symmetry in equilibrium when the number of couplings per spin is infinite. This correspondence is extended to a SG model of finite connectivity to determine the phase diagram, which is numerically supported.

The task of CDMA multiuser detection is to simultaneously estimate binary symbols of $K$ synchronous users from the received $N$ base-band CDMA signals. Mathematically, this can be formulated as an inference problem on a complete bipartite graph. In the research on graphically represented statistical models, it is known that the belief propagation (BP) can exactly perform the inference in a polynomial time scale of the system size when the graph is free from cycles in spite that the necessary computation for general graphs exponentially explodes in the worst case. In addition, recent several researches revealed that the BP can also serve as an excellent approximation algorithm even if the graph has cycles as far as they are relatively long. However, as there exit many short cycles in a complete bipartite graph, one might suspect that the BP would not provide a good performance when employed for the multiuser detection. The purpose of this paper is to make an objection to such suspicion. More specifically, we will show that appropriate employment of the central limit theorem and the law of large numbers to BP, which is one of the standard techniques in statistical mechanics, makes it possible to develop a novel multiuser detection algorithm the convergence property of which is considerably better than that of the conventional multistage detection without increasing the computational cost significantly. Furthermore, we will also provide a scheme to analyse the dynamics of the proposed algorithm, which can be naturally linked to the equilibrium analysis recently presented by Tanaka., Comment: Submitted to 2003 IEEE International Symposium on Information Theory

We investigate the use of Gallager's low-density parity-check (LDPC) codes in a broadcast channel, one of the fundamental models in network information theory. Combining linear codes is a standard technique in practical network communication schemes and is known to provide better performance than simple timesharing methods when algebraic codes are used. The statistical physics based analysis shows that the practical performance of the suggested method, achieved by employing the belief propagation algorithm, is superior to that of LDPC based timesharing codes while the best performance, when received transmissions are optimally decoded, is bounded by the timesharing limit., Comment: 14 pages, 4 figures

The performance of ``typical set (pairs) decoding'' for ensembles of Gallager's linear code is investigated using statistical physics. In this decoding, error happens when the information transmission is corrupted by an untypical noise or two or more typical sequences satisfy the parity check equation provided by the received codeword for which a typical noise is added. We show that the average error rate for the latter case over a given code ensemble can be tightly evaluated using the replica method, including the sensitivity to the message length. Our approach generally improves the existing analysis known in information theory community, which was reintroduced by MacKay (1999) and believed as most accurate to date., Comment: 7 pages

Statistical physics is employed to evaluate the performance of error-correcting codes in the case of finite message length for an ensemble of Gallager's error correcting codes. We follow Gallager's approach of upper-bounding the average decoding error rate, but invoke the replica method to reproduce the tightest general bound to date, and to improve on the most accurate zero-error noise level threshold reported in the literature. The relation between the methods used and those presented in the information theory literature are explored., Comment: 11 pages, 1 figure

A variation of low density parity check (LDPC) error correcting codes defined over Galois fields ($GF(q)$) is investigated using statistical physics. A code of this type is characterised by a sparse random parity check matrix composed of $C$ nonzero elements per column. We examine the dependence of the code performance on the value of $q$, for finite and infinite $C$ values, both in terms of the thermodynamical transition point and the practical decoding phase characterised by the existence of a unique (ferromagnetic) solution. We find different $q$-dependencies in the cases of C=2 and $C \ge 3$; the analytical solutions are in agreement with simulation results, providing a quantitative measure to the improvement in performance obtained using non-binary alphabets., Comment: 7 pages, 1 figure

We employ the methods of statistical physics to study the performance of Gallager type error-correcting codes. In this approach, the transmitted codeword comprises Boolean sums of the original message bits selected by two randomly-constructed sparse matrices. We show that a broad range of these codes potentially saturate Shannon's bound but are limited due to the decoding dynamics used. Other codes show sub-optimal performance but are not restricted by the decoding dynamics. We show how these codes may also be employed as a practical public-key cryptosystem and are of competitive performance to modern cyptographical methods., Comment: 6 pages

An exact solution to a family of parity check error-correcting codes is provided by mapping the problem onto a Husimi cactus. The solution obtained in the thermodynamic limit recovers the replica symmetric theory results and provides a very good approximation to finite systems of moderate size. The probability propagation decoding algorithm emerges naturally from the analysis. A phase transition between decoding success and failure phases is found to coincide with an information-theoretic upper bound. The method is employed to compare Gallager and MN codes., Comment: 7 pages, 3 figures, with minor corrections

A variation of Gallager error-correcting codes is investigated using statistical mechanics. In codes of this type, a given message is encoded into a codeword which comprises Boolean sums of message bits selected by two randomly constructed sparse matrices. The similarity of these codes to Ising spin systems with random interaction makes it possible to assess their typical performance by analytical methods developed in the study of disordered systems. The typical case solutions obtained via the replica method are consistent with those obtained in simulations using belief propagation (BP) decoding. We discuss the practical implications of the results obtained and suggest a computationally efficient construction for one of the more practical configurations., Comment: 35 pages, 4 figures

The relation between Ising spin systems and public-key cryptography is investigated using methods of statistical physics. The insight gained from the analysis is used for devising a matrix-based cryptosystem whereby the ciphertext comprises products of the original message bits; these are selected by employing two predetermined randomly-constructed sparse matrices. The ciphertext is decrypted using methods of belief-propagation. The analyzed properties of the suggested cryptosystem show robustness against various attacks and competitive performance to modern cyptographical methods., Comment: 4 pages, 2 figures

We discuss the properties of equilibrium states in an autoassociative memory model storing hierarchically correlated patterns (hereafter, hierarchical patterns). We will show that symmetric mixed states (hereafter, mixed states) are bi-stable on the associative memory model storing the hierarchical patterns in a region of the ferromagnetic phase. This means that the first-order transition occurs in this ferromagnetic phase. We treat these contents with a statistical mechanical method (SCSNA) and by computer simulation. Finally, we discuss a physiological implication of this model. Sugase et al. analyzed the time-course of the information carried by the firing of face-responsive neurons in the inferior temporal cortex. We also discuss the relation between the theoretical results and the physiological experiments of Sugase et al., Comment: 18 pages, 6 figures

Low-density parity-check codes with irregular constructions have been recently shown to outperform the most advanced error-correcting codes to date. In this paper we apply methods of statistical physics to study the typical properties of simple irregular codes. We use the replica method to find a phase transition which coincides with Shannon's coding bound when appropriate parameters are chosen. The decoding by belief propagation is also studied using statistical physics arguments; the theoretical solutions obtained are in good agreement with simulations. We compare the performance of irregular with that of regular codes and discuss the factors that contribute to the improvement in performance., Comment: 20 pages, 9 figures, revised version submitted to JPA

The performance of Gallager's error-correcting code is investigated via methods of statistical physics. In this approach, the transmitted codeword comprises products of the original message bits selected by two randomly-constructed sparse matrices; the number of non-zero row/column elements in these matrices constitutes a family of codes. We show that Shannon's channel capacity is saturated for many of the codes while slightly lower performance is obtained for others which may be of higher practical relevance. Decoding aspects are considered by employing the TAP approach which is identical to the commonly used belief-propagation-based decoding., Comment: 6 pages, latex, 1 figure

We investigate the performance of parity check codes using the mapping onto Ising spin systems proposed by Sourlas. We study codes where each parity check comprises products of K bits selected from the original digital message with exactly C checks per message bit. We show, using the replica method, that these codes saturate Shannon's coding bound for $K\to\infty$ when the code rate K/C is finite. We then examine the finite temperature case to asses the use of simulated annealing methods for decoding, study the performance of the finite K case and extend the analysis to accommodate different types of noisy channels. The connection between statistical physics and belief propagation decoders is discussed and the dynamics of the decoding itself is analyzed. Further insight into new approaches for improving the code performance is given., Comment: 32 pages, 12 figures, to appear in PRE

We investigate the generalization ability of a simple perceptron trained in the off-line and on-line supervised modes. Examples are extracted from the teacher who is a non-monotonic perceptron. For this system, difficulties of training can be controlled continuously by changing a parameter of the teacher. We train the student by several learning strategies in order to obtain the theoretical lower bounds of generalization errors under various conditions. Asymptotic behavior of the learning curve has been derived, which enables us to determine the most suitable learning algorithm for a given value of the parameter controlling difficulties of training., Comment: LaTeX 10 pages including 6 ps figures, using llncs.sty, Proc. of Theoretical Aspects of Neural Computation 97, to be published from Springer-Verlag

We investigate the generalization ability of a perceptron with non-monotonic transfer function of a reversed-wedge type in on-line mode. This network is identical to a parity machine, a multilayer network. We consider several learning algorithms. By the perceptron algorithm the generalization error is shown to decrease by the ${\alpha}^{-1/3}$-law similarly to the case of a simple perceptron in a restricted range of the parameter $a$ characterizing the non-monotonic transfer function. For other values of $a$, the perceptron algorithm leads to the state where the weight vector of the student is just opposite to that of the teacher. The Hebbian learning algorithm has a similar property; it works only in a limited range of the parameter. The conventional AdaTron algorithm does not give a vanishing generalization error for any values of $a$.We thus introduce a modified AdaTron algorithm which yields a good performance for all values of $a$. We also investigate the effects of optimization of the learning rate as well as of the learning algorithm. Both methods give excellent learning curves proportional to ${\alpha}^{-1}$. The latter optimization is related to the Bayes statistics and is shown to yield useful hints to extract maximum amount of information necessary to accelerate learning processes., Comment: Latex 20 pages with 10 figures

We study the generalization ability of a simple perceptron which learns unlearnable rules. The rules are presented by a teacher perceptron with a non-monotonic transfer function. The student is trained in the on-line mode. The asymptotic behaviour of the generalization error is estimated under various conditions. Several learning strategies are proposed and improved to obtain the theoretical lower bound of the generalization error., Comment: LaTeX 20 pages using IOP LaTeX preprint style file, 14 figures