Electrical Engineering

This paper presents a predictive controller for handling plug-and-play (P&P) charging requests of electric vehicles (EVs) in a distribution system. The proposed method uses a two-stage hierarchical control scheme based on a model predictive control (MPC) formulation for tracking periodic references. The first stage computes a reachable periodic reference that trades off deviation from the nominal voltage with the required generation control. The second stage computes a controller that tracks this reference and charges the EVs, while satisfying system constraints at all times. Under the assumption of a time-periodic load, it is shown that the proposed controller is recursively feasible and exponentially stable i.e. the EVs' state of charge (SOC) and bus voltages converge to the desired SOC and to the optimal periodic reference respectively. Finally, the proposed scheme is illustrated in a set of examples.

In recently proposed multiple access techniques such as IDMA and OFDM-IDMA, the user separation is done by user specific interleavers in contrast to conventional CDMA scheme where user separation is assured with user-specific signature sequences. The user specific interleavers must demonstrate minimum probability of collision amongst each other in addition to other merits including minimal consumption of bandwidth, least hardware for their generation, and least memory requirement. Previously, random interleaver and other interleavers proposed by researchers is still leaving some space for further research leading to optimality of interleavers. In this paper, we propose a novel interleaver based on prime numbers for generation of user specific interleavers to remove the problem of high consumption of bandwidth. The simulation results demonstrate the optimal performance of prime interleaver (PI) apart from other merits in comparison to random and other interleavers.

We consider the problem where M agents interact with M identical and independent environments with S states and A actions using reinforcement learning for T rounds. The agents share their data with a central server to minimize their regret. We aim to find an algorithm that allows the agents to minimize the regret with infrequent communication rounds. We provide DIST-UCRL which runs at each agent and prove that the total cumulative regret of M agents is upper bounded as Õ(DS √ MAT ) for a Markov Decision Process with diameter D, number of states S, and number of actions A. The agents synchronize after their visitations to any state-action pair exceeds a certain threshold. Using this, we obtain a bound of O (MSA log(MT )) on the total number of communications rounds. Finally, we evaluate the algorithm against multiple environments and demonstrate that the proposed algorithm performs at par with an always communication version of the UCRL2 algorithm, while with significantly lower comm...

Teleoperated surgical robots can provide immediate medical assistance in austere and hostile environments. However, such scenarios are time-sensitive and thus, require highbandwidth and low-latency communication links which might be unavailable. Systems with a higher degree of autonomy can address these issues as they can operate even with intermittent feedback from the surgeon. The system presented in this paper has a standard surgical teleoperation interface, which provides surgeons with an environment on which they are trained. In our semi-autonomous robotic framework, high level instructions are inferred from the surgeon's actions and then executed semi-autonomously on the robot. The framework consists of two main modules: (i) Recognition Module-which recognizes atomic sub-tasks (i.e., surgemes) performed at the operator end, and (ii) Execution Module-which executes the identified surgemes at the robot end using task contextual information. The peg transfer task was selected for this paper due to its importance in laparoscopic surgical training. The experiments were performed on the DESK surgical dataset to show the effectiveness of our framework using two metrics: user intervention, measured in terms of degree of autonomy, and success rate of surgeme execution. We achieved an average accuracy of 91.5% for surgeme recognition and a success rate of 86% during surgeme execution. Furthermore, we obtained an average success rate of 53.9% for the overall task, using a model-based approach with a degree of autonomy of 99.33%.

In the future, deployable, teleoperated surgical robots can save the lives of critically injured patients in battlefield environments. These robotic systems will need to have autonomous capabilities to take over during communication delays and unexpected environmental conditions during critical phases of the procedure. Understanding and predicting the next surgical actions (referred as "surgemes") is essential for autonomous surgery. Most approaches for surgeme recognition cannot cope with the high variability associated with austere environments and thereby cannot "transfer" well to field robotics. We propose a methodology that uses compact image representations with kinematic features for surgeme recognition in the DESK dataset. This dataset offers samples for surgical procedures over different robotic platforms with a high variability in the setup. We performed surgeme classification in two setups: 1) No transfer, 2) Transfer from a simulated scenario to two real deployable robots. Then, the results were compared with recognition accuracies using only kinematic data with the same experimental setup. The results show that our approach improves the recognition performance over kinematic data across different domains. The proposed approach produced a transfer accuracy gain up to 20% between the simulated and the real robot, and up to 31% between the simulated robot and a different robot. A transfer accuracy gain was observed for all cases, even those already above 90%.

Many engineering problems have multiple objectives, and the overall aim is to optimize a non-linear function of these objectives. In this paper, we formulate the problem of maximizing a non-linear concave function of multiple long-term objectives. A policy-gradient based model-free algorithm is proposed for the problem. To compute an estimate of the gradient, a biased estimator is proposed. The proposed algorithm is shown to achieve convergence to within an of the global optima after sampling O( M σ (1−γ)8 4 ) trajectories where γ is the discount factor and M is the number of the agents, thus achieving the same dependence on as the policy gradient algorithm for the standard reinforcement learning.

We consider the problem of constrained Markov Decision Process (CMDP) where an agent interacts with a unichain Markov Decision Process. At every interaction, the agent obtains a reward. Further, there are K cost functions. The agent aims to maximize the long-term average reward while simultaneously keeping the K long-term average costs lower than a certain threshold. In this paper, we propose CMDP-PSRL, a posterior sampling based algorithm using which the agent can learn optimal policies to interact with the CMDP. Further, for MDP with S states, A actions, and diameter D, we prove that following CMDP-PSRL algorithm, the agent can bound the regret of not accumulating rewards from optimal policy by Õ(poly(DSA) √ T ). Further, we show that the violations for any of the K constraints is also bounded by Õ(poly(DSA) √ T ). To the best of our knowledge, this is the first work which obtains a Õ( √ T ) regret bounds for ergodic MDPs with long-term average constraints.

Many real-world problems like Social Influence Maximization face the dilemma of choosing the best K out of N options at a given time instant. This setup can be modeled as a combinatorial bandit which chooses K out of N arms at each time, with an aim to achieve an efficient trade-off between exploration and exploitation. This is the first work for combinatorial bandits where the feedback received can be a non-linear function of the chosen K arms. The direct use of multi-armed bandit requires choosing among N -choose-K options making the state space large. In this paper, we present a novel algorithm which is computationally efficient and the storage is linear in N . The proposed algorithm is a divide-and-conquer based strategy, that we call CMAB-SM. Further, the proposed algorithm achieves a regret bound of Õ(K 1 2N 1 3T 2 3 ) for a time horizon T , which is sub-linear in all parameters T , N , and K.

Quantum key distribution (QKD) allows two distant parties to share encryption keys with security based on laws of quantum mechanics. In order to share the keys, the quantum bits have to be transmitted from the sender to the receiver over a noisy quantum channel. In order to transmit this information, efficient encoders and decoders need to be designed. However, large-scale design of quantum encoders and decoders have to depend on the channel characteristics and require look-up tables which require memory that is exponential in the number of qubits. In order to alleviate that, this paper aims to design the quantum encoders and decoders for expander codes by adapting techniques from machine learning including reinforcement learning and neural networks to the quantum domain. The proposed quantum decoder trains a neural network which is trained using the maximum aposteriori error for the syndromes, eliminating the use of large lookup tables. The quantum encoder uses deep Q-learning base...

Mean field control (MFC) is an effective way to mitigate the curse of dimensionality of cooperative multi-agent reinforcement learning (MARL) problems. This work considers a collection of Npop heterogeneous agents that can be segregated into K classes such that the k-th class contains Nk homogeneous agents. We aim to prove approximation guarantees of the MARL problem for this heterogeneous system by its corresponding MFC problem. We consider three scenarios where the reward and transition dynamics of all agents are respectively taken to be functions of (1) joint state and action distributions across all classes, (2) individual distributions of each class, and (3) marginal distributions of the entire population. We show that, in these cases, the K-class MARL problem can be approximated by MFC with errors given as e1 = O( √ |X ||U| Npop ∑

We consider the bandit problem of selecting $K$ out of $N$ arms at each time step. The reward can be a non-linear function of the rewards of the selected individual arms. The direct use of a multi-armed bandit algorithm requires choosing among $\\binom{N}{K}$ options, making the action space large. To simplify the problem, existing works on combinatorial bandits {typically} assume feedback as a linear function of individual rewards. In this paper, we prove the lower bound for top-$K$ subset selection with bandit feedback with possibly correlated rewards. We present a novel algorithm for the combinatorial setting without using individual arm feedback or requiring linearity of the reward function. Additionally, our algorithm works on correlated rewards of individual arms. Our algorithm, aDaptive Accept RejecT (DART), sequentially finds good arms and eliminates bad arms based on confidence bounds. DART is computationally efficient and uses storage linear in $N$. Further, DART achieves a...

Many real-world problems face the dilemma of choosing best $K$ out of $N$ options at a given time instant. This setup can be modelled as combinatorial bandit which chooses $K$ out of $N$ arms at each time, with an aim to achieve an efficient tradeoff between exploration and exploitation. This is the first work for combinatorial bandit where the reward received can be a non-linear function of the chosen $K$ arms. The direct use of multi-armed bandit requires choosing among $N$-choose-$K$ options making the state space large. In this paper, we present a novel algorithm which is computationally efficient and the storage is linear in $N$. The proposed algorithm is a divide-and-conquer based strategy, that we call CMAB-SM. Further, the proposed algorithm achieves a regret bound of $\\tilde O(K^\\frac{1}{2}N^\\frac{1}{3}T^\\frac{2}{3})$ for a time horizon $T$, which is sub-linear in all parameters $T$, $N$, and $K$. The evaluation results on different reward functions and arm distribution fun...

We consider the problem where N agents collaboratively interact with an instance of a stochastic K arm bandit problem for K ≫ N . The agents aim to simultaneously minimize the cumulative regret over all the agents for a total of T time steps, the number of communication rounds, and the number of bits in each communication round. We present Limited Communication Collaboration Upper Confidence Bound (LCC-UCB), a doubling-epoch based algorithm where each agent communicates only after the end of the epoch and shares the index of the best arm it knows. With our algorithm, LCC-UCB, each agent enjoys a regret of Õ ( √

Gradient descent and its variants are widely used in machine learning. However, oracle access of gradient may not be available in many applications, limiting the direct use of gradient descent. This paper proposes a method of estimating gradient to perform gradient descent, that converges to a second-order stationary point for general non-convex optimization problems. Beyond the first-order stationary properties, the second-order stationary properties are important in machine learning applications to achieve better performance. We show that the proposed model-free non-convex optimization algorithm returns an ε-second-order stationary point with $\\tilde O\\left( {\\frac{{{d^{2 + \\frac{\\theta }{2}}}}}{{{\\varepsilon ^{8 + \\theta }}}}} \\right)$ queries of the function for any arbitrary θ > 0.

We consider the problem of tabular infinite horizon concave utility reinforcement learning (CURL) with convex constraints. Various learning applications with constraints, such as robotics, do not allow for policies that can violate constraints. To this end, we propose a model-based learning algorithm that achieves zero constraint violations. To obtain this result, we assume that the concave objective and the convex constraints have a solution interior to the set of feasible occupation measures. We then solve a tighter optimization problem to ensure that the constraints are never violated despite the imprecise model knowledge and model stochasticity. We also propose a novel Bellman error based analysis for tabular infinite-horizon setups which allows to analyse stochatic policies. Combining the Bellman error based analysis and tighter optimization equation, for T interactions with the environment, we obtain a regret guarantee for objective which grows as Õ(1/ √ T ), excluding other f...

Reinforcement Learning (RL) is being increasingly applied to optimize complex functions that may have a stochastic component. RL is extended to multi-agent systems to find policies to optimize systems that require agents to coordinate or to compete under the umbrella of Multi-Agent RL (MARL). A crucial factor in the success of RL is that the optimization problem is represented as the expected sum of rewards, which allows the use of backward induction for the solution. However, many real-world problems require a joint objective that is non-linear and dynamic programming cannot be applied directly. For example, in a resource allocation problem, one of the objective is to maximize long-term fairness among the users. This paper addresses and formalizes the problem of joint objective optimization, where not only the sum of rewards of each agent but a function of the sum of rewards of each agent needs to be optimized. The proposed algorithms at the centralized controller aims to learn the...

Reinforcement learning is widely used in applications where one needs to perform sequential decisions while interacting with the environment. The problem becomes more challenging when the decision requirement includes satisfying some safety constraints. The problem is mathematically formulated as constrained Markov decision process (CMDP). In the literature, various algorithms are available to solve CMDP problems in a model-free manner to achieve -optimal cumulative reward with feasible policies. An -feasible policy implies that it suffers from constraint violation. An important question here is whether we can achieve -optimal cumulative reward with zero constraint violations or not. To achieve that, we advocate the use of randomized primal-dual approach to solve the CMDP problems and propose a conservative stochastic primal-dual algorithm (CSPDA) which is shown to exhibit Õ ( 1/ 2 ) sample complexity to achieve -optimal cumulative reward with zero constraint violations. In the prio...

Gradient descent and its variants are widely used in machine learning. However, oracle access of gradient may not be available in many applications, limiting the direct use of gradient descent. This paper proposes a method of estimating gradient to perform gradient descent, that converges to a stationary point for general nonconvex optimization problems. Beyond the first-order stationary properties, the second-order stationary properties are important in machine learning applications to achieve better performance. We show that the proposed modelfree non-convex optimization algorithm returns an ǫ-second-order stationary point with O(d 2+ θ 2 ǫ 8+θ) queries of the function for any arbitrary θ > 0.