Showing 1–36 of 36 results for author: Stavrou, P A

Copula-based Estimation of Continuous Sources for a Class of Constrained Rate-Distortion-Functions

Authors: Giuseppe Serra, Photios A. Stavrou, Marios Kountouris

Abstract: We present a new method to estimate the rate-distortion-perception function in the perfect realism regime (PR-RDPF), for multivariate continuous sources subject to a single-letter average distortion constraint. The proposed approach is not only able to solve the specific problem but also two related problems: the entropic optimal transport (EOT) and the output-constrained rate-distortion function… ▽ More We present a new method to estimate the rate-distortion-perception function in the perfect realism regime (PR-RDPF), for multivariate continuous sources subject to a single-letter average distortion constraint. The proposed approach is not only able to solve the specific problem but also two related problems: the entropic optimal transport (EOT) and the output-constrained rate-distortion function (OC-RDF), of which the PR-RDPF represents a special case. Using copula distributions, we show that the OC-RDF can be cast as an I-projection problem on a convex set, based on which we develop a parametric solution of the optimal projection proving that its parameters can be estimated, up to an arbitrary precision, via the solution of a convex program. Subsequently, we propose an iterative scheme via gradient methods to estimate the convex program. Lastly, we characterize a Shannon lower bound (SLB) for the PR-RDPF under a mean squared error (MSE) distortion constraint. We support our theoretical findings with numerical examples by assessing the estimation performance of our iterative scheme using the PR-RDPF with the obtained SLB for various sources. △ Less

Submitted 30 January, 2024; originally announced January 2024.

On the Computation of the Gaussian Rate-Distortion-Perception Function

Authors: Giuseppe Serra, Photios A. Stavrou, Marios Kountouris

Abstract: In this paper, we study the computation of the rate-distortion-perception function (RDPF) for a multivariate Gaussian source under mean squared error (MSE) distortion and, respectively, Kullback-Leibler divergence, geometric Jensen-Shannon divergence, squared Hellinger distance, and squared Wasserstein-2 distance perception metrics. To this end, we first characterize the analytical bounds of the s… ▽ More In this paper, we study the computation of the rate-distortion-perception function (RDPF) for a multivariate Gaussian source under mean squared error (MSE) distortion and, respectively, Kullback-Leibler divergence, geometric Jensen-Shannon divergence, squared Hellinger distance, and squared Wasserstein-2 distance perception metrics. To this end, we first characterize the analytical bounds of the scalar Gaussian RDPF for the aforementioned divergence functions, also providing the RDPF-achieving forward "test-channel" realization. Focusing on the multivariate case, we establish that, for tensorizable distortion and perception metrics, the optimal solution resides on the vector space spanned by the eigenvector of the source covariance matrix. Consequently, the multivariate optimization problem can be expressed as a function of the scalar Gaussian RDPFs of the source marginals, constrained by global distortion and perception levels. Leveraging this characterization, we design an alternating minimization scheme based on the block nonlinear Gauss-Seidel method, which optimally solves the problem while identifying the Gaussian RDPF-achieving realization. Furthermore, the associated algorithmic embodiment is provided, as well as the convergence and the rate of convergence characterization. Lastly, for the "perfect realism" regime, the analytical solution for the multivariate Gaussian RDPF is obtained. We corroborate our results with numerical simulations and draw connections to existing results. △ Less

Submitted 15 November, 2023; originally announced November 2023.

Comments: This paper has been submitted for journal publication

Indirect Rate Distortion Functions with $f$-Separable Distortion Criterion

Authors: Photios A. Stavrou, Yanina Shkel, Marios Kountouris

Abstract: We consider a remote source coding problem subject to a {distortion function}. Contrary to the use of the classical separable distortion criterion, herein we consider the more general, $f$-separable distortion measure and study its implications on the characterization of the minimum achievable rates (also called $f$-separable indirect rate distortion function (iRDF)) under both excess and average… ▽ More We consider a remote source coding problem subject to a {distortion function}. Contrary to the use of the classical separable distortion criterion, herein we consider the more general, $f$-separable distortion measure and study its implications on the characterization of the minimum achievable rates (also called $f$-separable indirect rate distortion function (iRDF)) under both excess and average distortion constraints. First, we provide a single-letter characterization of the optimal rates subject to an excess distortion using properties of the $f$-separable distortion. Our main result is a single-letter characterization of the $f$-separable iRDF subject to an average distortion constraint. As a consequence of the previous results, we also show a series of equalities that hold using either indirect or classical RDF under $f$-separable excess or average distortions. We corroborate our results with two application examples in which new closed-form solutions are derived, and based on these, we also recover known special cases. △ Less

Submitted 17 May, 2023; originally announced May 2023.

Comments: 6 pages, 2 figures, accepted for publication to the proceedings of ISIT 2023

MSC Class: 94A17; 94A34; 68P30; 94A15 ACM Class: H.1.1; G.3

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

Authors: Giuseppe Serra, Photios A. Stavrou, Marios Kountouris

Abstract: In this paper, we study the computation of the rate-distortion-perception function (RDPF) for discrete memoryless sources subject to a single-letter average distortion constraint and a perception constraint that belongs to the family of f-divergences. For that, we leverage the fact that RDPF, assuming mild regularity conditions on the perception constraint, forms a convex programming problem. We f… ▽ More In this paper, we study the computation of the rate-distortion-perception function (RDPF) for discrete memoryless sources subject to a single-letter average distortion constraint and a perception constraint that belongs to the family of f-divergences. For that, we leverage the fact that RDPF, assuming mild regularity conditions on the perception constraint, forms a convex programming problem. We first develop parametric characterizations of the optimal solution and utilize them in an alternating minimization approach for which we prove convergence guarantees. The resulting structure of the iterations of the alternating minimization approach renders the implementation of a generalized Blahut-Arimoto (BA) type of algorithm infeasible. To overcome this difficulty, we propose a relaxed formulation of the structure of the iterations in the alternating minimization approach, which allows for the implementation of an approximate iterative scheme. This approximation is shown, via the derivation of necessary and sufficient conditions, to guarantee convergence to a globally optimal solution. We also provide sufficient conditions on the distortion and the perception constraints which guarantee that our algorithm converges exponentially fast. We corroborate our theoretical results with numerical simulations, and we draw connections with existing results. △ Less

Submitted 8 May, 2023; originally announced May 2023.

Comments: Accepted paper to ISIT 2023 without proofs

Privacy signaling games with binary alphabets

Authors: Photios A. Stavrou, Serkan Sarıtaş, Mikael Skoglund

Abstract: In this paper, we consider a privacy signaling game problem for binary alphabets and single-bit transmission where a transmitter has a pair of messages, one of which is a casual message that needs to be conveyed, whereas the other message contains sensitive data and needs to be protected. The receiver wishes to estimate both messages to acquire as much information as possible. For this setup, we s… ▽ More In this paper, we consider a privacy signaling game problem for binary alphabets and single-bit transmission where a transmitter has a pair of messages, one of which is a casual message that needs to be conveyed, whereas the other message contains sensitive data and needs to be protected. The receiver wishes to estimate both messages to acquire as much information as possible. For this setup, we study the interactions between the transmitter and the receiver with non-aligned information-theoretic objectives (modeled by mutual information and hamming distance) due to the privacy concerns of the transmitter. We derive conditions under which Nash and/or Stackelberg equilibria exist and identify the optimal responses of the encoder and decoders strategies for each type of game. One particularly surprising result is that when both types of equilibria exist, they admit the same encoding and decoding strategies. We corroborate our analysis with simulation studies. △ Less

Submitted 30 March, 2022; v1 submitted 10 November, 2021; originally announced November 2021.

Comments: 11 pages, 9 figures, European Control Conference 2022 (accepted)

MSC Class: 91A05; 91A10; 91A12; 91A28

Generalized Talagrand Inequality for Sinkhorn Distance using Entropy Power Inequality

Authors: Shuchan Wang, Photios A. Stavrou, Mikael Skoglund

Abstract: In this paper, we study the connection between entropic optimal transport and entropy power inequality (EPI). First, we prove an HWI-type inequality making use of the infinitesimal displacement convexity of optimal transport map. Second, we derive two Talagrand-type inequalities using the saturation of EPI that corresponds to a numerical term in our expression. We evaluate for a wide variety of di… ▽ More In this paper, we study the connection between entropic optimal transport and entropy power inequality (EPI). First, we prove an HWI-type inequality making use of the infinitesimal displacement convexity of optimal transport map. Second, we derive two Talagrand-type inequalities using the saturation of EPI that corresponds to a numerical term in our expression. We evaluate for a wide variety of distributions this term whereas for Gaussian and i.i.d. Cauchy distributions this term is found in explicit form. We show that our results extend previous results of Gaussian Talagrand inequality for Sinkhorn distance to the strongly log-concave case. △ Less

Submitted 17 September, 2021; originally announced September 2021.

Comments: The paper has been accepted to Information Theory Workshop 2021

Secure Source Coding with Side-information at Decoder and Shared Key at Encoder and Decoder

Authors: Hamid Ghourchian, Photios A. Stavrou, Tobias J. Oechtering, Mikael Skoglund

Abstract: We study the problem of rate-distortion-equivocation with side-information only available at the decoder when an independent private random key is shared between the sender and the receiver. The sender compresses the sequence, and the receiver reconstructs it such that the average distortion between the source and the output is limited. The equivocation is measured at an eavesdropper that intercep… ▽ More We study the problem of rate-distortion-equivocation with side-information only available at the decoder when an independent private random key is shared between the sender and the receiver. The sender compresses the sequence, and the receiver reconstructs it such that the average distortion between the source and the output is limited. The equivocation is measured at an eavesdropper that intercepts the source encoded message, utilizing side-information correlated with the source and the side-information at the decoder. We have derived the entire achievable rate-distortion-equivocation region for this problem. △ Less

Submitted 23 January, 2022; v1 submitted 3 September, 2021; originally announced September 2021.

Comments: Long version of the accepted paper in the Information Theory Workshop (ITW 2021). 23 pages, 4 figures

Secure Block Source Coding with Sequential Encoding

Authors: Hamid Ghourchian, Photios A. Stavrou, Tobias J. Oechtering, Mikael Skoglund

Abstract: We introduce fundamental bounds on achievable cumulative rate distribution functions (CRDF) to characterize a sequential encoding process that ensures lossless or lossy reconstruction subject to an average distortion criterion using a non-causal decoder. The CRDF describes the rate resources spent sequentially to compress the sequence. We also include a security constraint that affects the set of… ▽ More We introduce fundamental bounds on achievable cumulative rate distribution functions (CRDF) to characterize a sequential encoding process that ensures lossless or lossy reconstruction subject to an average distortion criterion using a non-causal decoder. The CRDF describes the rate resources spent sequentially to compress the sequence. We also include a security constraint that affects the set of achievable CRDF. The information leakage is defined sequentially based on the mutual information between the source and its compressed representation, as it evolves. To characterize the security constraints, we introduce the concept of cumulative leakage distribution functions (CLF), which determines the allowed information leakage as distributed over encoded sub-blocks. Utilizing tools from majorization theory, we derive necessary and sufficient conditions on the achievable CRDF for a given independent and identically distributed (IID) source and CLF. One primary result of this paper is that the concave-hull of the CRDF characterizes the optimal achievable rate distribution. △ Less

Submitted 30 August, 2021; originally announced August 2021.

Comments: 38 pages, 5 figures

Adaptive Interference Coordination over Channels with Unknown State at the Encoder and the Decoder

Authors: Michail Mylonakis, Photios A. Stavrou, Mikael Skoglund

Abstract: We generalize the problem of controlling the interference created to an external observer while communicating over a discrete memoryless channel (DMC) which was studied in \cite{serrano:2014}. In particular, we consider the scenario where the transmission is established over a compound DMC channel with unknown state at both the encoder and the decoder. Depending on the exact state $s$ of the chann… ▽ More We generalize the problem of controlling the interference created to an external observer while communicating over a discrete memoryless channel (DMC) which was studied in \cite{serrano:2014}. In particular, we consider the scenario where the transmission is established over a compound DMC channel with unknown state at both the encoder and the decoder. Depending on the exact state $s$ of the channel, we ask for a different level of average precision $Δ_s$ on the establishment of the interference coordination with the external observer. For this set-up, we fully characterize the capacity region. △ Less

Submitted 3 June, 2021; v1 submitted 16 May, 2021; originally announced May 2021.

Quadratic Signaling Games with Channel Combining Ratio

Authors: Serkan Sarıtaş, Photios A. Stavrou, Ragnar Thobaben, Mikael Skoglund

Abstract: In this study, Nash and Stackelberg equilibria of single-stage and multi-stage quadratic signaling games between an encoder and a decoder are investigated. In the considered setup, the objective functions of the encoder and the decoder are misaligned, there is a noisy channel between the encoder and the decoder, the encoder has a soft power constraint, and the decoder has also noisy observation of… ▽ More In this study, Nash and Stackelberg equilibria of single-stage and multi-stage quadratic signaling games between an encoder and a decoder are investigated. In the considered setup, the objective functions of the encoder and the decoder are misaligned, there is a noisy channel between the encoder and the decoder, the encoder has a soft power constraint, and the decoder has also noisy observation of the source to be estimated. We show that there exist only linear encoding and decoding strategies at the Stackelberg equilibrium, and derive the equilibrium strategies and costs. Regarding the Nash equilibrium, we explicitly characterize affine equilibria for the single-stage setup and show that the optimal encoder (resp. decoder) is affine for an affine decoder (resp. encoder) for the multi-stage setup. For the decoder side, between the information coming from the encoder and noisy observation of the source, our results describe what should be the combining ratio of these two channels. Regarding the encoder, we derive the conditions under which it is meaningful to transmit a message. △ Less

Submitted 3 February, 2021; originally announced February 2021.

Comments: 19 pages, 2 figures

Remote Empirical Coordination

Authors: Michail Mylonakis, Photios A. Stavrou, Mikael Skoglund

Abstract: We apply the framework of imperfect empirical coordination to a two-node setup where the action $X$ of the first node is not observed directly but via $L$ agents who observe independently impaired measurements $\hat X$ of the action. These $L$ agents, using a rate-limited communication that is available to all of them, help the second node to generate the action $Y$ in order to establish the desir… ▽ More We apply the framework of imperfect empirical coordination to a two-node setup where the action $X$ of the first node is not observed directly but via $L$ agents who observe independently impaired measurements $\hat X$ of the action. These $L$ agents, using a rate-limited communication that is available to all of them, help the second node to generate the action $Y$ in order to establish the desired coordinated behaviour. When $L<\infty$, we prove that it suffices $R_i\geq I\left(\hat X;\hat{Y}\right)$ for at least one agent whereas for $L\longrightarrow\infty$, we show that it suffices $R_i\geq I\left(\hat X;\hat Y|X\right)$ for all agents where $\hat Y$ is a random variable such that $X-\hat X-\hat Y$ and $\|p_{X,\hat Y}\left(x,y\right)-p_{X,Y}\left(x,y\right)\|_{TV}\leq Δ$ ( $Δ$ is the pre-specified fidelity). △ Less

Submitted 22 September, 2020; v1 submitted 18 April, 2020; originally announced April 2020.

Indirect NRDF for Partially Observable Gauss-Markov Processes with MSE Distortion: Complete Characterizations and Optimal Solutions

Authors: Photios A. Stavrou, Mikael Skoglund

Abstract: In this paper we study the problem of characterizing and computing the nonanticipative rate distortion function (NRDF) for partially observable multivariate Gauss-Markov processes with hard mean squared error (MSE) distortion constraints. For the finite time horizon case, we first derive the complete characterization of this problem and its corresponding optimal realization which is shown to be a… ▽ More In this paper we study the problem of characterizing and computing the nonanticipative rate distortion function (NRDF) for partially observable multivariate Gauss-Markov processes with hard mean squared error (MSE) distortion constraints. For the finite time horizon case, we first derive the complete characterization of this problem and its corresponding optimal realization which is shown to be a linear functional of the current time sufficient statistic of the past and current observations signals. We show that when the problem is strictly feasible, it can be computed via semidefinite programming (SDP) algorithm. For time-varying scalar processes with average total MSE distortion we derive an optimal closed form expression by means of a dynamic reverse-waterfilling solution that we also implement via an iterative scheme that convergences linearly in finite time, and a closed-form solution under pointwise MSE distortion constraint. For the infinite time horizon, we give necessary and sufficient conditions to ensure that asymptotically the sufficient statistic process of the observation signals achieves a steady-state solution for the corresponding covariance matrices and impose conditions that allow existence of a time-invariant solution. Then, we show that when a finite solution exists in the asymptotic limit, it can be computed via SDP algorithm. We also give strong structural properties on the characterization of the problem in the asymptotic limit that allow for an optimal solution via a reverse-waterfilling algorithm that we implement via an iterative scheme that converges linearly under a finite number of spatial components. Our results are corroborated with various simulation studies and are also compared with existing results in the literature. △ Less

Submitted 20 October, 2021; v1 submitted 16 December, 2019; originally announced December 2019.

Comments: 17 double column pages, 4 figures, 1 Table

MSC Class: 15-00; 94D99; 60J05; 93E11; 93E20; 35Q93

Empirical Coordination with Multiple Descriptions

Authors: Michail Mylonakis, Photios A. Stavrou, Mikael Skoglund

Abstract: We extend the framework of empirical coordination to a distributed setup where for a given action by nature, multiple descriptions of the action of the decoder are available. We adopt the coding strategy applied by El Gamal and Cover in \cite{gamal:1982} to get a lower bound of the coordination region. Then, we improve this region by applying the coding scheme applied by Zhang and Berger in \cite{… ▽ More We extend the framework of empirical coordination to a distributed setup where for a given action by nature, multiple descriptions of the action of the decoder are available. We adopt the coding strategy applied by El Gamal and Cover in \cite{gamal:1982} to get a lower bound of the coordination region. Then, we improve this region by applying the coding scheme applied by Zhang and Berger in \cite{zhang:1987}. △ Less

Submitted 20 April, 2020; v1 submitted 5 October, 2019; originally announced October 2019.

Empirical Coordination Subject to a Fidelity Criterion

Authors: Michail Mylonakis, Photios A. Stavrou, Mikael Skoglund

Abstract: We study the problem of empirical coordination subject to a fidelity criterion for a general set-up. We prove a result which indicates a strong connection between our framework and the framework of empirical coordination developed in [1]. It turns out that when we design codes that achieve empirical coordination according to a given distribution and subject to the fidelity criterion, it is suffici… ▽ More We study the problem of empirical coordination subject to a fidelity criterion for a general set-up. We prove a result which indicates a strong connection between our framework and the framework of empirical coordination developed in [1]. It turns out that when we design codes that achieve empirical coordination according to a given distribution and subject to the fidelity criterion, it is sufficient to consider codes that produce actions of the same joint type for a class of types which is close enough to our desired distribution is some sense. △ Less

Submitted 15 July, 2019; originally announced July 2019.

Sequential Source Coding for Stochastic Systems Subject to Finite Rate Constraints

Authors: Photios A. Stavrou, Mikael Skoglund, Takashi Tanaka

Abstract: In this paper, we revisit the sequential source coding framework to analyze fundamental performance limitations of discrete-time stochastic control systems subject to feedback data-rate constraints in finite-time horizon. The basis of our results is a new characterization of the lower bound on the minimum total-rate achieved by sequential codes subject to a total (across time) distortion constrain… ▽ More In this paper, we revisit the sequential source coding framework to analyze fundamental performance limitations of discrete-time stochastic control systems subject to feedback data-rate constraints in finite-time horizon. The basis of our results is a new characterization of the lower bound on the minimum total-rate achieved by sequential codes subject to a total (across time) distortion constraint and a computational algorithm that allocates optimally the rate-distortion for any fixed finite-time horizon. This characterization facilitates the derivation of analytical, non-asymptotic, and finite-dimensional lower and upper bounds in two control-related scenarios. (a) A parallel time-varying Gauss-Markov process with identically distributed spatial components that is quantized and transmitted through a noiseless channel to a minimum mean-squared error (MMSE) decoder. (b) A time-varying quantized LQG closed-loop control system, with identically distributed spatial components and with a random data-rate allocation. Our non-asymptotic lower bound on the quantized LQG control problem, reveals the absolute minimum data-rates for (mean square) stability of our time-varying plant for any fixed finite time horizon. We supplement our framework with illustrative simulation experiments. △ Less

Submitted 18 May, 2020; v1 submitted 10 June, 2019; originally announced June 2019.

Comments: 40 pages, 6 figures

Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources

Authors: Photios A. Stavrou, Jan Ostergaard, Charalambos D. Charalambous

Abstract: We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function… ▽ More We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function and operational zero-delay RDF. We recall the realization that corresponds to the optimal "test-channel" of the Gaussian NRDF, when considering a vector Gauss-Markov source subject to a MSE distortion in the finite time horizon. Then, we introduce sufficient conditions to show existence of solution for this problem in the infinite time horizon. For the asymptotic regime, we use the asymptotic characterization of the Gaussian NRDF to provide a new equivalent realization scheme with feedback which is characterized by a resource allocation (reverse-waterfilling) problem across the dimension of the vector source. We leverage the new realization to derive a predictive coding scheme via lattice quantization with subtractive dither and joint memoryless entropy coding. This coding scheme offers an upper bound to the operational zero-delay vector-valued Gaussian RDF. When we use scalar quantization, then for "r" active dimensions of the vector Gauss-Markov source the gap between the obtained lower and theoretical upper bounds is less than or equal to 0.254r + 1 bits/vector. We further show that it is possible when we use vector quantization, and assume infinite dimensional Gauss-Markov sources to make the previous gap to be negligible, i.e., Gaussian NRDF approximates the operational zero-delay Gaussian RDF. We also extend our results to vector-valued Gaussian sources of any finite memory under mild conditions. Our theoretical framework is demonstrated with illustrative numerical experiments. △ Less

Submitted 29 September, 2018; originally announced October 2018.

Comments: 32 pages, 9 figures, published in IEEE Journal of Selected Topics in Signal Processing

The Time-Invariant Multidimensional Gaussian Sequential Rate-Distortion Problem Revisited

Authors: Photios A. Stavrou, Takashi Tanaka, Sekhar Tatikonda

Abstract: We revisit the sequential rate-distortion (SRD) trade-off problem for vector-valued Gauss-Markov sources with mean-squared error distortion constraints. We show via a counterexample that the dynamic reverse water-filling algorithm suggested by [1, eq. (15)] is not applicable to this problem, and consequently the closed form expression of the asymptotic SRD function derived in [1, eq. (17)] is not… ▽ More We revisit the sequential rate-distortion (SRD) trade-off problem for vector-valued Gauss-Markov sources with mean-squared error distortion constraints. We show via a counterexample that the dynamic reverse water-filling algorithm suggested by [1, eq. (15)] is not applicable to this problem, and consequently the closed form expression of the asymptotic SRD function derived in [1, eq. (17)] is not correct in general. Nevertheless, we show that the multidimensional Gaussian SRD function is semidefinite representable and thus it is readily computable. △ Less

Submitted 27 November, 2017; originally announced November 2017.

Comments: 7 pages, 2 figures

MSC Class: 90C22; 94A15

An Upper Bound to Zero-Delay Rate Distortion via Kalman Filtering for Vector Gaussian Sources

Authors: Photios A. Stavrou, Jan Ostergaard, Charalambos D. Charalambous, Milan Derpich

Abstract: We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the correspon… ▽ More We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the corresponding optimal "test-channel" of the NRDF, when considering vector Gaussian AR(1) sources subject to an average MSE distortion. We give conditions on the vector Gaussian AR(1) source to ensure asymptotic stationarity of the realization scheme (bounded performance). Then, we encode the vector innovations due to Kalman filtering via lattice quantization with subtractive dither and memoryless entropy coding. This coding scheme provides a tight upper bound to the zero-delay Gaussian RDF. We extend this result to vector Gaussian AR sources of any finite order. Further, we show that for infinite dimensional vector Gaussian AR sources of any finite order, the NRDF coincides with the zero-delay RDF. Our theoretical framework is corroborated with a simulation example. △ Less

Submitted 21 August, 2017; v1 submitted 23 January, 2017; originally announced January 2017.

Comments: 7 pages, 6 figures, accepted for publication in IEEE Information Theory Workshop (ITW)

Sequential Necessary and Sufficient Conditions for Capacity Achieving Distributions of Channels with Memory and Feedback

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: We derive sequential necessary and sufficient conditions for any channel input conditional distribution ${\cal P}_{0,n}\triangleq\{P_{X_t|X^{t-1},Y^{t-1}}:~t=0,\ldots,n\}$ to maximize the finite-time horizon directed information defined by $$C^{FB}_{X^n \rightarrow Y^n} \triangleq \sup_{{\cal P}_{0,n}} I(X^n\rightarrow{Y^n}),~~~ I(X^n \rightarrow Y^n) =\sum_{t=0}^n{I}(X^t;Y_t|Y^{t-1})$$ for channe… ▽ More We derive sequential necessary and sufficient conditions for any channel input conditional distribution ${\cal P}_{0,n}\triangleq\{P_{X_t|X^{t-1},Y^{t-1}}:~t=0,\ldots,n\}$ to maximize the finite-time horizon directed information defined by $$C^{FB}_{X^n \rightarrow Y^n} \triangleq \sup_{{\cal P}_{0,n}} I(X^n\rightarrow{Y^n}),~~~ I(X^n \rightarrow Y^n) =\sum_{t=0}^n{I}(X^t;Y_t|Y^{t-1})$$ for channel distributions $\{P_{Y_t|Y^{t-1},X_t}:~t=0,\ldots,n\}$ and $\{P_{Y_t|Y_{t-M}^{t-1},X_t}:~t=0,\ldots,n\}$, where $Y^t\triangleq\{Y_0,\ldots,Y_t\}$ and $X^t\triangleq\{X_0,\ldots,X_t\}$ are the channel input and output random processes, and $M$ is a finite nonnegative integer. \noi We apply the necessary and sufficient conditions to application examples of time-varying channels with memory and we derive recursive closed form expressions of the optimal distributions, which maximize the finite-time horizon directed information. Further, we derive the feedback capacity from the asymptotic properties of the optimal distributions by investigating the limit $$C_{X^\infty \rightarrow Y^\infty}^{FB} \triangleq \lim_{n \longrightarrow \infty} \frac{1}{n+1} C_{X^n \rightarrow Y^n}^{FB}$$ without any á priori assumptions, such as, stationarity, ergodicity or irreducibility of the channel distribution. The necessary and sufficient conditions can be easily extended to a variety of channels with memory, beyond the ones considered in this paper. △ Less

Submitted 18 April, 2016; v1 submitted 10 April, 2016; originally announced April 2016.

Comments: 57 pages, 9 figures, part of the paper was accepted for publication in the proceedings of the IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain 10-15 July, 2016 (Date of submission of the conference paper: 25/1/2016)

Optimal Estimation via Nonanticipative Rate Distortion Function and Applications to Time-Varying Gauss-Markov Processes

Authors: Photios A. Stavrou, Themistoklis Charalambous, Charalambos D. Charalambous, Sergey Loyka

Abstract: In this paper, we develop {finite-time horizon} causal filters using the nonanticipative rate distortion theory. We apply the {developed} theory to {design optimal filters for} time-varying multidimensional Gauss-Markov processes, subject to a mean square error fidelity constraint. We show that such filters are equivalent to the design of an optimal \texttt{\{encoder, channel, decoder\}}, which en… ▽ More In this paper, we develop {finite-time horizon} causal filters using the nonanticipative rate distortion theory. We apply the {developed} theory to {design optimal filters for} time-varying multidimensional Gauss-Markov processes, subject to a mean square error fidelity constraint. We show that such filters are equivalent to the design of an optimal \texttt{\{encoder, channel, decoder\}}, which ensures that the error satisfies {a} fidelity constraint. Moreover, we derive a universal lower bound on the mean square error of any estimator of time-varying multidimensional Gauss-Markov processes in terms of conditional mutual information. Unlike classical Kalman filters, the filter developed is characterized by a reverse-waterfilling algorithm, which ensures {that} the fidelity constraint is satisfied. The theoretical results are demonstrated via illustrative examples. △ Less

Submitted 10 February, 2017; v1 submitted 14 March, 2016; originally announced March 2016.

Comments: 35 pages, 6 figures, submitted for publication in SIAM Journal on Control and Optimization (SICON)

Information Nonanticipative Rate Distortion Function and Its Applications

Authors: Photios A. Stavrou, Christos K. Kourtellaris, C. D. Charalambous

Abstract: This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. The… ▽ More This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. These applications are described using two running examples, the Binary Symmetric Markov Source with parameter p, (BSMS(p)) and the multidimensional partially observed Gaussian-Markov source. For the multidimensional Gaussian-Markov source with square error distortion, the solution of the nonanticipative RDF is derived, its operational meaning using JSCC design via a noisy coding theorem is shown by providing the optimal encoding-decoding scheme over a vector Gaussian channel, and the RL of causal and zero-delay codes with respect to noncausal codes is computed. For the BSMS(p) with Hamming distortion, the solution of the nonanticipative RDF is derived, the RL of causal codes with respect to noncausal codes is computed, and an uncoded noisy coding theorem based on excess distortion probability is shown. The information nonanticipative RDF is shown to be equivalent to the nonanticipatory epsilon-entropy, which corresponds to the classical RDF with an additional causality or nonanticipative condition imposed on the optimal reproduction conditional distribution. △ Less

Submitted 18 November, 2015; v1 submitted 7 May, 2014; originally announced May 2014.

Comments: 34 pages, 12 figures, part of this paper was accepted for publication in IEEE International Symposium on Information Theory (ISIT), 2014 and in book Coordination Control of Distributed Systems of series Lecture Notes in Control and Information Sciences, 2015

Applications of Information Nonanticipative Rate Distortion Function

Authors: Photios A. Stavrou, Christos K. Kourtellaris, Charalambos D. Charalambous

Abstract: The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter $p$ (BSMS($p$)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS($p$), we give the solution to the NRDF, and w… ▽ More The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter $p$ (BSMS($p$)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS($p$), we give the solution to the NRDF, and we use it to compute the Rate Loss (RL) of causal codes with respect to noncausal codes. For the multidimensional Gaussian-Markov source, we give the solution to the NRDF, we show its operational meaning via joint source-channel matching over a vector of parallel Gaussian channels, and we compute the RL of causal and zero-delay codes with respect to noncausal codes. △ Less

Submitted 29 April, 2014; v1 submitted 22 January, 2014; originally announced January 2014.

Comments: 5 pages, 3 figures, accepted for publication in IEEE International Symposium on Information Theory (ISIT) proceedings, 2014

Source-Channel Matching for Sources with Memory

Authors: Christos Kourtellaris, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper we analyze the probabilistic matching of sources with memory to channels with memory so that symbol-by-symbol code with memory without anticipation are optimal, with respect to an average distortion and excess distortion probability. We show achievability of such a symbolby- symbol code with memory without anticipation, and we show matching for the Binary Symmetric Markov source (BSM… ▽ More In this paper we analyze the probabilistic matching of sources with memory to channels with memory so that symbol-by-symbol code with memory without anticipation are optimal, with respect to an average distortion and excess distortion probability. We show achievability of such a symbolby- symbol code with memory without anticipation, and we show matching for the Binary Symmetric Markov source (BSMS(p)) over a first-order symmetric channel with a cost constraint. △ Less

Submitted 25 March, 2014; v1 submitted 11 October, 2013; originally announced October 2013.

Comments: 6 pages. arXiv admin note: substantial text overlap with arXiv:1304.6528

Rate Distortion Function for a Class of Relative Entropy Sources

Authors: Farzad Rezaei, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding… ▽ More This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding with fidelity criterion. It is shown that minimax and maxmin strategies can be computed explicitly, and they are generalizations of the classical solution. Finally, for discrete memoryless uncertain sources, the rate distortion theorem is stated for the class omitting the derivations while the converse is derived. △ Less

Submitted 6 May, 2013; originally announced May 2013.

Comments: 6 pages, final version presented at Internation Symposium on Mathematical Theory of Networks and Systems (MTNS), Budapest, Hungary, 2010, pp. 1853-1858

Nonanticipative Rate Distortion Function for General Source-Channel Matching

Authors: Christos Kourtellaris, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper we invoke a nonanticipative information Rate Distortion Function (RDF) for sources with memory, and we analyze its importance in probabilistic matching of the source to the channel so that transmission of a symbol-by-symbol code with memory without anticipation is optimal, with respect to an average distortion and excess distortion probability. We show achievability of the symbol-by-… ▽ More In this paper we invoke a nonanticipative information Rate Distortion Function (RDF) for sources with memory, and we analyze its importance in probabilistic matching of the source to the channel so that transmission of a symbol-by-symbol code with memory without anticipation is optimal, with respect to an average distortion and excess distortion probability. We show achievability of the symbol-by-symbol code with memory without anticipation, and we evaluate the probabilistic performance of the code for a Markov source. △ Less

Submitted 25 April, 2013; v1 submitted 24 April, 2013; originally announced April 2013.

Comments: 5 pages, 3 figures

Directed Information on Abstract Spaces: Properties and Variational Equalities

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology o… ▽ More Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information. △ Less

Submitted 22 December, 2015; v1 submitted 16 February, 2013; originally announced February 2013.

Comments: 49 pages, 2 figures, Submitted for publication to IEEE Transactions on Information Theory

Optimal Nonstationary Reproduction Distribution for Nonanticipative RDF on Abstract Alphabets

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: In this paper we introduce a definition for nonanticipative Rate Distortion Function (RDF) on abstract alphabets, and we invoke weak convergence of probability measures to show various of its properties, such as, existence of the optimal reproduction conditional distribution, compactness of the fidelity set, lower semicontinuity of the RDF functional, etc. Further, we derive the closed form expres… ▽ More In this paper we introduce a definition for nonanticipative Rate Distortion Function (RDF) on abstract alphabets, and we invoke weak convergence of probability measures to show various of its properties, such as, existence of the optimal reproduction conditional distribution, compactness of the fidelity set, lower semicontinuity of the RDF functional, etc. Further, we derive the closed form expression of the optimal nonstationary reproduction distribution. This expression is computed recursively backward in time. Throughout the paper we point out an operational meaning of the nonanticipative RDF by recalling the coding theorem derive in \cite{tatikonda2000}, and we state relations to Gorbunov-Pinsker's nonanticipatory $ε-$entropy \cite{gorbunov-pinsker}. △ Less

Submitted 28 January, 2013; originally announced January 2013.

Comments: 5 pages, submitted to ISIT 2013

Variational Equalities of Directed Information and Applications

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $ε$-e… ▽ More In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $ε$-entropy $R^{\varepsilon}_{0,n}(D)$. By invoking certain results we first establish existence of the infimizing reproduction distribution for ${R}^{na}_{0,n}(D)$, and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of ${R}^{na}_{0,n}(D)$. △ Less

Submitted 16 May, 2013; v1 submitted 28 January, 2013; originally announced January 2013.

Comments: 5 pages, to appear in proceedings of International Symposium on Information Theory (ISIT), 2013

Nonanticipative Rate Distortion Function and Filtering Theory: A weak Convergence Approach

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF… ▽ More In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown using the topology of weak convergence of probability measures. Subsequently, we use the solution of the nonanticipative RDF to present the realization of a multidimensional partially observable source over a scalar Gaussian channel. We show that linear encoders are optimal, establishing joint source-channel coding in real-time. △ Less

Submitted 18 January, 2014; v1 submitted 29 December, 2012; originally announced December 2012.

Comments: 26 pages and 4 figures, part of this work is published in European Control Conference (ECC' 13)

On the relation of nonanticipative rate distortion function and filtering theory

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is investigated using the topology of weak convergence of probability measures on Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal repro… ▽ More In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is investigated using the topology of weak convergence of probability measures on Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown, while the optimal nonanticipative reproduction conditional distribution for stationary processes is derived in closed form. The realization procedure of nonanticipative RDF which is equivalent to joint-source channel matching for symbol-by-symbol transmission is described, while an example is introduced to illustrate the concepts. △ Less

Submitted 25 April, 2013; v1 submitted 7 October, 2012; originally announced October 2012.

Comments: 6 pages, 4 figures, final version submitted for publication at 12th Biannual European Control Conference (ECC), 2013

MSC Class: 60B11; 37A50

Nonanticipative Rate Distortion Function and Relations to Filtering Theory

Authors: Charalambos D. Charalambous, Photios A. Stavrou, Nasir U. Ahmed

Abstract: The relation between nonanticipative Rate Distortion Function (RDF) and filtering theory is discussed on abstract spaces. The relation is established by imposing a realizability constraint on the reconstruction conditional distribution of the classical RDF. Existence of the extremum solution of the nonanticipative RDF is shown using weak$^*$-convergence on appropriate topology. The extremum recons… ▽ More The relation between nonanticipative Rate Distortion Function (RDF) and filtering theory is discussed on abstract spaces. The relation is established by imposing a realizability constraint on the reconstruction conditional distribution of the classical RDF. Existence of the extremum solution of the nonanticipative RDF is shown using weak$^*$-convergence on appropriate topology. The extremum reconstruction conditional distribution is derived in closed form, for the case of stationary processes. The realization of the reconstruction conditional distribution which achieves the infimum of the nonanticipative RDF is described. Finally, an example is presented to illustrate the concepts. △ Less

Submitted 18 September, 2013; v1 submitted 3 October, 2012; originally announced October 2012.

Comments: 41 pages,5 figures, Provisionally Accepted Version of the manuscript submitted to IEEE Transactions on Automatic Control; Part of this work was presented in 20th International Symposium on Mathematical Theory of Networks and Systems (MTNS '12)

MSC Class: 28A33; 28A35; 60B10

Realizable Rate Distortion Function and Bayesian FIltering Theory

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: The relation between rate distortion function (RDF) and Bayesian filtering theory is discussed. The relation is established by imposing a causal or realizability constraint on the reconstruction conditional distribution of the RDF, leading to the definition of a causal RDF. Existence of the optimal reconstruction distribution of the causal RDF is shown using the topology of weak convergence of pro… ▽ More The relation between rate distortion function (RDF) and Bayesian filtering theory is discussed. The relation is established by imposing a causal or realizability constraint on the reconstruction conditional distribution of the RDF, leading to the definition of a causal RDF. Existence of the optimal reconstruction distribution of the causal RDF is shown using the topology of weak convergence of probability measures. The optimal non-stationary causal reproduction conditional distribution of the causal RDF is derived in closed form; it is given by a set of recursive equations which are computed backward in time. The realization of causal RDF is described via the source-channel matching approach, while an example is briefly discussed to illustrate the concepts. △ Less

Submitted 13 April, 2012; originally announced April 2012.

Comments: 5 pages, 3 figures, 1 table, 1 graph, submitted to Information Theory Workshop 2012

MSC Class: 28A33

Directed Information on Abstract spaces: Properties and Extremum Problems

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedb… ▽ More This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedback, and non-anticipative and sequential rate distortion are analyzed showing existence of maximizing and minimizing distributions, respectively. △ Less

Submitted 19 May, 2012; v1 submitted 5 February, 2012; originally announced February 2012.

Comments: 5 pages, to appear in proceedings of International Symposium on Information Theory (ISIT), 2012

Causal Rate Distortion Function on Abstract Alphabets: Optimal Reconstruction and Properties

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: A causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and a coding theorem is derived. Existence of the minimizing kernel is shown using the topology of weak convergence of probability measures. The optimal reconstruction kernel is derived, which is causal, and certain properties of the causal rate distortion function are presented. A causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and a coding theorem is derived. Existence of the minimizing kernel is shown using the topology of weak convergence of probability measures. The optimal reconstruction kernel is derived, which is causal, and certain properties of the causal rate distortion function are presented. △ Less

Submitted 4 February, 2012; originally announced February 2012.

Comments: 5 pages, Submitted to Internation Symposium on Information Theory(ISIT) 2012

Causal Rate Distortion Function and Relations to Filtering Theory

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: A causal rate distortion function is defined, its solution is described, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable. A causal rate distortion function is defined, its solution is described, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable. △ Less

Submitted 6 June, 2012; v1 submitted 5 April, 2011; originally announced April 2011.

Comments: 8 pages; 3 figures; Presented in 20th International Symposium on Mathematical Theory of Networks and Systems (MTNS 2012)

Causal Rate Distortion Function on Abstract Alphabets and Optimal Reconstruction Kernel

Authors: Charalambos D. Charalambous, Photios A. Stavrou, Christos K. Kourtellaris

Abstract: A Causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and the optimal reconstruction kernel is derived, which consists of a product of causal kernels. In the process, general abstract spaces are introduced to show existence of the minimizing kernel using weak*-convergence. Certain properties of the causal rate distortion function are presented. A Causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and the optimal reconstruction kernel is derived, which consists of a product of causal kernels. In the process, general abstract spaces are introduced to show existence of the minimizing kernel using weak*-convergence. Certain properties of the causal rate distortion function are presented. △ Less

Submitted 16 February, 2011; originally announced February 2011.