Search

The first part of the thesis focuses on asymptotic energy-distortion performance of zero- and low-delay transmission of Gaussian sources over energy-limited Gaussian channels. A lower bound for the leading term in the negative logarithm of the distortion, termed the energy-distortion exponent, is derived through an achievable scheme based on high-resolution quantization coupled with orthogonal signaling. The higher-order term in the negative logarithm of the distortion, termed the energy-distortion dispersion, is optimized while keeping the leading term, the energy-distortion exponent, at its optimal (respectively, the best known) value for the zero-delay (respectively, low-delay) regime. In contrast with the decaying dispersion previously reported in the literature, the proposed coding scheme achieves a constant dispersion. When the scheme is optimized, this constant can be improved with respect to its naive value, i.e., that achieved by optimizing purely the source coding performance instead of the end-to-end distortion. Lastly, a tradeoff of achievable energy-distortion exponents is derived for broadcast scenarios by extending the point-to-point scheme to include a successive refinement source coder coupled with two rounds of orthogonal signaling. The second part examines the idea of randomized response together with the method of types and large deviations techniques to analyze the accuracy of potential electronic privacy-preserving voting and survey schemes. Previous work by Tuncel [1] proposed a voting scheme in which votes are randomly changed by the system before being transmitted with a chosen flipping probability to preserve the privacy of the voters. The vulnerable interval in [1], where the voting results are very close to 50%-50%, is tackled by introducing a third possible outcome referred to as "too close to call". This third outcome is used as a feedback to adjust the flipping probability so that small upper bounds on the probability of wrongly callin

The first part of the thesis focuses on asymptotic energy-distortion performance of zero- and low-delay transmission of Gaussian sources over energy-limited Gaussian channels. A lower bound for the leading term in the negative logarithm of the distortion, termed the energy-distortion exponent, is derived through an achievable scheme based on high-resolution quantization coupled with orthogonal signaling. The higher-order term in the negative logarithm of the distortion, termed the energy-distortion dispersion, is optimized while keeping the leading term, the energy-distortion exponent, at its optimal (respectively, the best known) value for the zero-delay (respectively, low-delay) regime. In contrast with the decaying dispersion previously reported in the literature, the proposed coding scheme achieves a constant dispersion. When the scheme is optimized, this constant can be improved with respect to its naive value, i.e., that achieved by optimizing purely the source coding performance instead of the end-to-end distortion. Lastly, a tradeoff of achievable energy-distortion exponents is derived for broadcast scenarios by extending the point-to-point scheme to include a successive refinement source coder coupled with two rounds of orthogonal signaling. The second part examines the idea of randomized response together with the method of types and large deviations techniques to analyze the accuracy of potential electronic privacy-preserving voting and survey schemes. Previous work by Tuncel [1] proposed a voting scheme in which votes are randomly changed by the system before being transmitted with a chosen flipping probability to preserve the privacy of the voters. The vulnerable interval in [1], where the voting results are very close to 50%-50%, is tackled by introducing a third possible outcome referred to as "too close to call". This third outcome is used as a feedback to adjust the flipping probability so that small upper bounds on the probability of wrongly callin

In this thesis, we incorporate information theory into statistical signal processing and machine learning in order to achieve robust learning in presence of outliers (or generally in environments with non-Gaussian structure). In other words, by incorporating information theory especially when structure of environment is non-Gaussian, the efficiency of information extraction from data and consequently precision of the learning are increased. Note that, although the well-known central limit theorem creates the expectations that we should see the Gaussian distribution everywhere in the real world, this is not necessarily true. Indeed central limit theorem is concluded under some assumptions that do not hold in many real scenarios. For instance, in many real scenarios heavy-tailed distributions arise which result in outliers. The problem of learning a system affected by these types of distributions is of great importance inasmuch as conventional learning approaches that are mostly based on Gaussian assumption are not effective anymore. In this thesis, we address this problem by means of information theory. Our work finds broad applications from communication channel estimation, adaptive equalization, adaptive echo noise cancellation, system identification, underwater and satellite communications, blind source separation to physics, biology, computer science, the social sciences, and beyond where real-world environments may change in time and are conducive to non-Gaussian probability density functions. In such environments higher order statistics of data is needed, therefore conventional methodologies are not efficient anymore. Specifically, the problem of online linear regression (or interchangeably linear adaptive filtering) in environments corrupted by non-Gaussian noise is addressed in this thesis. In such environments, the error between system outputs and labels (or desired responses) does not follow a Gaussian distribution and there might exist abnormally

In this thesis, we incorporate information theory into statistical signal processing and machine learning in order to achieve robust learning in presence of outliers (or generally in environments with non-Gaussian structure). In other words, by incorporating information theory especially when structure of environment is non-Gaussian, the efficiency of information extraction from data and consequently precision of the learning are increased. Note that, although the well-known central limit theorem creates the expectations that we should see the Gaussian distribution everywhere in the real world, this is not necessarily true. Indeed central limit theorem is concluded under some assumptions that do not hold in many real scenarios. For instance, in many real scenarios heavy-tailed distributions arise which result in outliers. The problem of learning a system affected by these types of distributions is of great importance inasmuch as conventional learning approaches that are mostly based on Gaussian assumption are not effective anymore. In this thesis, we address this problem by means of information theory. Our work finds broad applications from communication channel estimation, adaptive equalization, adaptive echo noise cancellation, system identification, underwater and satellite communications, blind source separation to physics, biology, computer science, the social sciences, and beyond where real-world environments may change in time and are conducive to non-Gaussian probability density functions. In such environments higher order statistics of data is needed, therefore conventional methodologies are not efficient anymore. Specifically, the problem of online linear regression (or interchangeably linear adaptive filtering) in environments corrupted by non-Gaussian noise is addressed in this thesis. In such environments, the error between system outputs and labels (or desired responses) does not follow a Gaussian distribution and there might exist abnormally

In this thesis we investigate the lossy transmission of single and bivariate Gaussian sources over bandwidth-mismatched additive Gaussian white noise and broadcast channels. For these scenarios we proposed novel hybrid digital/analog based joint source-channel coding schemes which generalize or outperform existing schemes. In the first scenario we assume that side information is available at the receiver, channel state information of additive interference is available at the transmitter, and power is limited. For this scenario we proposed hybrid digital/analog schemes, for both bandwidth expansion and bandwidth compression cases, which can attain the optimum reconstruction levels. For bandwidth expansion we showed that the scheme can attain optimum distortion levels for a set of receivers with different side information and channel qualities simultaneously with a single set of scheme parameters. In the second scenario, where no side information or interference are present, we consider the robustness of scheme where it must attain the optimal distortion at a target signal-to-noise-ratio and we would like to attain the best distortion pair for two possible receivers one with better and the other with worse channel quality. We extended Tian et al.’s result to a set of non-integer bandwidth expansion ratios. Then we investigate the transmission of bivariate sources over broadcast channels. For this scenario we proposed a scheme which outperforms the known schemes which are either purely digital or hybrid schemes. Finally we analyzed energy-distortion tradeoff for lossy transmission of a Gaussian source over bandwidth-unlimited channel. We performed asymptotical analyses as signal- to-noise-ratio goes to infinity. We also considered zero-delay transmission of the source.

In this thesis we investigate the lossy transmission of single and bivariate Gaussian sources over bandwidth-mismatched additive Gaussian white noise and broadcast channels. For these scenarios we proposed novel hybrid digital/analog based joint source-channel coding schemes which generalize or outperform existing schemes. In the first scenario we assume that side information is available at the receiver, channel state information of additive interference is available at the transmitter, and power is limited. For this scenario we proposed hybrid digital/analog schemes, for both bandwidth expansion and bandwidth compression cases, which can attain the optimum reconstruction levels. For bandwidth expansion we showed that the scheme can attain optimum distortion levels for a set of receivers with different side information and channel qualities simultaneously with a single set of scheme parameters. In the second scenario, where no side information or interference are present, we consider the robustness of scheme where it must attain the optimal distortion at a target signal-to-noise-ratio and we would like to attain the best distortion pair for two possible receivers one with better and the other with worse channel quality. We extended Tian et al.’s result to a set of non-integer bandwidth expansion ratios. Then we investigate the transmission of bivariate sources over broadcast channels. For this scenario we proposed a scheme which outperforms the known schemes which are either purely digital or hybrid schemes. Finally we analyzed energy-distortion tradeoff for lossy transmission of a Gaussian source over bandwidth-unlimited channel. We performed asymptotical analyses as signal- to-noise-ratio goes to infinity. We also considered zero-delay transmission of the source.

This thesis focuses on the communication of a low bandwidth Gaussian source with specific energy restrictions through an AWGN channel. Since the source has low bandwidth, it is assumed that the transmitter is capable of delivering many dimensions per symbol without any loss of information. The offered schemes use high resolution quantization which requires an optimization of the MSE distortion to reach optimality. Several practical schemes with ML estimator and orthogonal modulation are used with different types of (high resolution) quantizers so that the distortion expression can be derived analytically. Due to the cumbersome in analytically expressing the distortion for MMSE estimation, MMSE estimator is offered under suboptimal conditions. To increase the performance of the MMSE estimator, an iterative design algorithm that transitions between receiver and transmitter is introduced. The algorithm assumes that the decoder is applying ML rule and it optimizes the quantizer's point density function accordingly. Then it fixes the transmitter side and runs Monte Carlo Simulations using MMSE estimator in a bounded set of points in the variables' space in search of the minimum distortion.

This thesis focuses on the communication of a low bandwidth Gaussian source with specific energy restrictions through an AWGN channel. Since the source has low bandwidth, it is assumed that the transmitter is capable of delivering many dimensions per symbol without any loss of information. The offered schemes use high resolution quantization which requires an optimization of the MSE distortion to reach optimality. Several practical schemes with ML estimator and orthogonal modulation are used with different types of (high resolution) quantizers so that the distortion expression can be derived analytically. Due to the cumbersome in analytically expressing the distortion for MMSE estimation, MMSE estimator is offered under suboptimal conditions. To increase the performance of the MMSE estimator, an iterative design algorithm that transitions between receiver and transmitter is introduced. The algorithm assumes that the decoder is applying ML rule and it optimizes the quantizer's point density function accordingly. Then it fixes the transmitter side and runs Monte Carlo Simulations using MMSE estimator in a bounded set of points in the variables' space in search of the minimum distortion.

A novel video coding scheme based on leaky prediction is proposedand its superiority against conventional predictive video coding isdemonstrated in environments with moderate to heavy packet loss. Thestrength of the scheme stems from the fact that it explicitly takesinto account the two modes of operation (packet loss or no packetloss) at the decoder and optimizes the corresponding reconstructionfilters together with the prediction filter at the encodersimultaneously. The prediction coefficient tends to be significantlysmaller than the correlation coefficient between two correspondingframes.

A novel video coding scheme based on leaky prediction is proposedand its superiority against conventional predictive video coding isdemonstrated in environments with moderate to heavy packet loss. Thestrength of the scheme stems from the fact that it explicitly takesinto account the two modes of operation (packet loss or no packetloss) at the decoder and optimizes the corresponding reconstructionfilters together with the prediction filter at the encodersimultaneously. The prediction coefficient tends to be significantlysmaller than the correlation coefficient between two correspondingframes.

One of the innovations brought about by the emerging and thriving of wireless sensor networks is Wyner-Ziv (WZ) coding, in other words, lossy source coding with side information at the decoder. While previous work mostly focuses on using capacity-achieving codes to approach the Wyner-Ziv bound, which naturally introduces huge block lengths and huge delay, we study extremely low-delay Wyner-Ziv coding of Gaussian sources. Three related but distinct problems are considered. The first involves scalar quantization and scalar noiseless coding of the quantization indices when only decoder has access to side information. Under high-resolution assumptions and appropriately defined decodability constraints, the optimal quantization level density is conjectured to be periodic. The performance of variable-length coding with uniform quantization is also characterized. The results are then incorporated in predictive Wyner-Ziv coding for Gaussian sources with memory, and optimal prediction filters are numerically designed so as to strike a balance between maximally exploiting both temporal and spatial correlation and limiting the propagation of distortion due to occasional decoding errors. Finally, zero-delay schemes are also employed in transform coding with small block lengths, where the Gaussian source and side information are transformed separately with the premise that corresponding transform coefficient pairs exhibit good spatial correlation and minimal temporal correlation. For the specific source-side information pairs studied, it is shown that transform coding, even with a small block-length, outperforms predictive coding. In the second part, we study the zero-delay joint source-channel coding (JSCC) problem of transmitting a Gaussian source over a Gaussian channel in Wyner-Ziv scenario. To achieve zero-delay, after applying scalar quantization to the source, the properly scaled analog information, namely the quantization error, is superimposed on the scaled digital info

This dissertation studies several problems regarding the joint/separate source-channel coding in data transmission over networks. It has been known that separation based transmission schemes achieves optimality when there is only one sender and one receiver. However, other than this point-to-point case, separation is often suboptimal and it is generally unclear how to achieve optimality.In Chapter 2, several new hybrid digital/analog schemes are proposed for the problem of transmitting a Gaussian source over a Gaussian channel under different scenarios. Previously existing optimal schemes are combined in a way that the output of one encoder is taken as effective CSI at the other encoder. In each case, the optimum distortion-power tradeoff can be achieved by a continuum of auxiliary random variables for any power allocation between the two encoders.In Chapter 3, a new scheme is proposed for lossy transmission of a Gaussian source over a Gaussian broadcast channel with source side information available at each re- ceiver. The proposed scheme combines two schemes that were previously shown to achieve optimal point-to-point distortion/power tradeoff simultaneously at all receivers under two distinct conditions stated in terms of channel and side information quality parameters. For the two-receiver case, the combined scheme is shown to achieve the same kind of optimality for the entire region in the parameter space sandwiched between those two conditions. Crucial to this result is the new degree of freedom discovered in Chapter 2.In Chapter 4, the problem of broadcasting a pair of correlated Gaussian sources using optimal separate source and channel codes is studied. A universal upper bound of rate (bandwidth compression/expansion ratio) penalty is given and in particular, for a low-distortion scenario, separate coding can be shown to achieve optimum. For the case with a Gaussian broadcast channel, the power loss of separate coding is also dis- cussed. Although source-ch

One of the innovations brought about by the emerging and thriving of wireless sensor networks is Wyner-Ziv (WZ) coding, in other words, lossy source coding with side information at the decoder. While previous work mostly focuses on using capacity-achieving codes to approach the Wyner-Ziv bound, which naturally introduces huge block lengths and huge delay, we study extremely low-delay Wyner-Ziv coding of Gaussian sources. Three related but distinct problems are considered. The first involves scalar quantization and scalar noiseless coding of the quantization indices when only decoder has access to side information. Under high-resolution assumptions and appropriately defined decodability constraints, the optimal quantization level density is conjectured to be periodic. The performance of variable-length coding with uniform quantization is also characterized. The results are then incorporated in predictive Wyner-Ziv coding for Gaussian sources with memory, and optimal prediction filters are numerically designed so as to strike a balance between maximally exploiting both temporal and spatial correlation and limiting the propagation of distortion due to occasional decoding errors. Finally, zero-delay schemes are also employed in transform coding with small block lengths, where the Gaussian source and side information are transformed separately with the premise that corresponding transform coefficient pairs exhibit good spatial correlation and minimal temporal correlation. For the specific source-side information pairs studied, it is shown that transform coding, even with a small block-length, outperforms predictive coding. In the second part, we study the zero-delay joint source-channel coding (JSCC) problem of transmitting a Gaussian source over a Gaussian channel in Wyner-Ziv scenario. To achieve zero-delay, after applying scalar quantization to the source, the properly scaled analog information, namely the quantization error, is superimposed on the scaled digital info

This dissertation studies several problems regarding the joint/separate source-channel coding in data transmission over networks. It has been known that separation based transmission schemes achieves optimality when there is only one sender and one receiver. However, other than this point-to-point case, separation is often suboptimal and it is generally unclear how to achieve optimality.In Chapter 2, several new hybrid digital/analog schemes are proposed for the problem of transmitting a Gaussian source over a Gaussian channel under different scenarios. Previously existing optimal schemes are combined in a way that the output of one encoder is taken as effective CSI at the other encoder. In each case, the optimum distortion-power tradeoff can be achieved by a continuum of auxiliary random variables for any power allocation between the two encoders.In Chapter 3, a new scheme is proposed for lossy transmission of a Gaussian source over a Gaussian broadcast channel with source side information available at each re- ceiver. The proposed scheme combines two schemes that were previously shown to achieve optimal point-to-point distortion/power tradeoff simultaneously at all receivers under two distinct conditions stated in terms of channel and side information quality parameters. For the two-receiver case, the combined scheme is shown to achieve the same kind of optimality for the entire region in the parameter space sandwiched between those two conditions. Crucial to this result is the new degree of freedom discovered in Chapter 2.In Chapter 4, the problem of broadcasting a pair of correlated Gaussian sources using optimal separate source and channel codes is studied. A universal upper bound of rate (bandwidth compression/expansion ratio) penalty is given and in particular, for a low-distortion scenario, separate coding can be shown to achieve optimum. For the case with a Gaussian broadcast channel, the power loss of separate coding is also dis- cussed. Although source-ch