Performing dynamic double lane-change maneuvers can be a challenge for highly automated vehicles.... more Performing dynamic double lane-change maneuvers can be a challenge for highly automated vehicles. The algorithm must meet safety requirements while keeping the vehicle stable and controllable. The problem of path planning is numerically complex and must be run at a high refresh rate. The article presents a new approach to avoiding obstacles for autonomous vehicles. To solve this problem, a geometric path generation is provided by a single-step continuous Reinforcement Learning (RL) agent. At the same time, a model-predictive controller (MPC) handles the lateral control to perform the dual lane-change maneuver. The task of the learning agent in this architecture is optimization. It is trained for different scenarios to provide geometric route planning parameters at the output of a neural network. During training, the goodness of the generated track is evaluated using an MPC controller. A hardware architecture was developed to test the local planner on a test track. The real-time operation of the planner has been proven. Its performance has also been compared to human drivers.
The gradually evolving automated driving and ADAS functions require more enhanced environment per... more The gradually evolving automated driving and ADAS functions require more enhanced environment perception. The key to reliable environmental perception is large amounts of data that are hard to collect. Several simulators provide realistic, raw sensor data based on physical sensor models. However, besides their high price, they also require very high computation capacity. Furthermore, most sensor suppliers provide high-level data, such as object detections, that is complicated to reproduce from simulated raw sensor data. This paper proposes a method that directly simulates the detections or object tracks provided by smart sensors. The model involves several uncertainties of the sensors, such as missed-, false detections, and measurement noise. In contrast to the conventional sensor models, this method tackles with state-dependent clutter model and considers the field of view in the detections model. The parameters of the proposed model are identified for an automotive smart radar and camera based on pre-evaluated real-world measurements. The resulting model provides synthetic object-level data with higher fidelity than the conventional probabilistic models, differing less than 2% from the precision and recall metrics of the actual sensors.
The rapid growth of urbanization and the constant demand for mobility have put a great strain on ... more The rapid growth of urbanization and the constant demand for mobility have put a great strain on transportation systems in cities. One of the major challenges in these areas is traffic congestion, particularly at signalized intersections. This problem not only leads to longer travel times for commuters, but also results in a significant increase in local and global emissions. The fixed cycle of traffic lights at these intersections is one of the primary reasons for this issue. To address these challenges, applying reinforcement learning to coordinating traffic light controllers has become a highly researched topic in the field of transportation engineering. This paper focuses on the traffic signal control problem, proposing a solution using a multi-agent deep Q-learning algorithm. This study introduces a novel rewarding concept in the multi-agent environment, as the reward schemes have yet to evolve in the following years with the advancement of techniques. The goal of this study is...
Proceedings of The First Conference on ZalaZONE Related R&I Activities of Budapest University of Technology and Economics 2022
The article presents the development of a low-budget positioning device that aims to provide an a... more The article presents the development of a low-budget positioning device that aims to provide an alternative in selfdriving vehicle development research that could replace costly, commercially available devices. In addition to being financially advantageous, it has the added benefit of allowing students to be involved in development. The primary function of the device is the sensor fusion, which outputs position, velocity, and orientation estimation based on data provided by Real-Time Kinematic (RTK) Global Navigation Satellite System (GNSS) technology and an Inertial Measurement Unit (IMU). High-frequency estimates are generated by running an Extended Kalman Filter (EKF) on a microcontroller in an embedded environment. During the work, new challenges arose several times that required solutions. For example, delays due to the operation of GNSS receivers, which the estimation algorithm must compensate, and proper calibration of the sensors for the measurement vehicle. In addition to the software, the development of the tool includes the complete design, manufacture, and testing of the hardware, which allows testing the completed software units not only in a simulation but also in a real environment. During testing, the output of the developed device was compared several times with commercially available hardware for similar purposes.
In this paper, we study the problem of traffic signal control in general intersections by applyin... more In this paper, we study the problem of traffic signal control in general intersections by applying a recent reinforcement learning technique. Nowadays, traffic congestion and road usage are increasing significantly as more and more vehicles enter the same infrastructures. New solutions are needed to minimize travel times or maximize the network capacity (throughput). Recent studies embrace machine learning approaches that have the power to aid and optimize the increasing demands. However, most reinforcement learning algorithms fail to be adaptive regarding goal functions. To this end, we provide a novel successor feature-based solution to control a single intersection to optimize the traffic flow, reduce the environmental impact, and promote sustainability. Our method allows for flexibility and adaptability to changing circumstances and goals. It supports changes in preferences during inference, so the behavior of the trained agent (traffic signal controller) can be changed rapidly ...
The railway timetables are designed in an optimal manner to maximize the capacity usage of the in... more The railway timetables are designed in an optimal manner to maximize the capacity usage of the infrastructure concerning different objectives besides avoiding conflicts. The realtime railway traffic management problem occurs when the preplanned timetable cannot be fulfilled due to various disturbances; therefore, the trains must be rerouted, reordered, and rescheduled. Optimizing the real-time railway traffic management aims to resolve the conflicts minimizing the delay propagation or even the energy consumption. In this paper, the existing mixedinteger linear programming optimization models are extended considering a safety-relevant issue of railway traffic management, the overlaps. However, solving the resulting model can be time-consuming in complex control areas and traffic situations involving many trains. Therefore, we propose different runtime efficient multi-stage heuristic models by decomposing the original problem. The impact of the model decomposition is investigated mathematically and experimentally in different rail networks and various simulated traffic scenarios concerning the objective value and the computational demand of the optimization. Besides providing a more realistic solution for the traffic management problem, the proposed multi-stage models significantly decrease the optimization runtime.
The paper presents real-world test cases of an optimal trajectory design solution that combines m... more The paper presents real-world test cases of an optimal trajectory design solution that combines modern control techniques with machine learning. The first step of the current research is to train a reinforcement learning agent in a simulated environment, where the conditions and the applied vehicle are modeled. System dynamics is described by a nonlinear single-track vehicle with dynamic wheel model. The designed trajectory is evaluated by driving the vehicle using a control loop. The reward of the method is based on the sum of different measures considering safety and passenger comfort. The proposed method forms a special one-step reinforcement learning task handled by Deep Deterministic Policy Gradient (DDPG) learning agent. As a result, the learning process provides a real-time neural-network-based motion planner and a tracking algorithm. The evaluation of the algorithm under real conditions is made by using an experimental test vehicle. The test setup contains a high precision GPS module, an automotive inertial sensor, an industrial PC, and communication interface devices. The test cases were performed on the ZalaZone automotive proving ground.
2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), 2018
The paper presents multimodel state estimation with constrained filtering applicable in a road tr... more The paper presents multimodel state estimation with constrained filtering applicable in a road traffic situation. The state to be estimated describes the motion of a car observed from the ego-vehicle. Multiple motion modes of the target vehicle are predefined and each is associated with a suitable constraining method. Both Kalman and particle filters are used to perform the estimations. The constrained filters are implemented in the Interacting Multiple Model structure. The outputs are the actual state of the observed vehicle and the motion mode that is in effect at every moment with an associated likelihood value. The performance of the proposed method is evaluated in a simulated environment.
2020 IEEE 24th International Conference on Intelligent Engineering Systems (INES), 2020
This work presents a powerful and intelligent driver agent, designed to operate in a preset highw... more This work presents a powerful and intelligent driver agent, designed to operate in a preset highway situation using Policy Gradient Reinforcement Learning (RL) agent. Our goal is to create an agent that is capable of navigating safely in changing highway traffic and successfully accomplish to get through the defined section keeping the reference speed. Meanwhile, creating a state representation that is capable of extracting information from images based on the actual highway situation. The algorithm uses Convolutional Neural Network (CNN) with Long-Short Term Memory (LSTM) layers as a function approximator for the agent with discrete action space on the control level, e.g., acceleration and lane change. Simulation of Urban MObility (SUMO), an open-source microscopic traffic simulator is chosen as our simulation environment. It is integrated with an open interface to interact with the agent in real-time. The agent can learn from numerous driving and highway situations that are created and fed to it. The representation becomes more general by randomizing and customizing the behavior of the other road users in the simulation, thus the experience of the agent can be much more diverse. The article briefly describes the modeling environment, the details on the learning agent, and the rewarding scheme. After evaluating the experiences gained from the training, some further plans and optimization ideas are briefed.
Proceedings of the 16th International Conference on Informatics in Control, Automation and Robotics, 2019
The paper presents a motion planning solution which combines classic control techniques with mach... more The paper presents a motion planning solution which combines classic control techniques with machine learning. For this task, a reinforcement learning environment has been created, where the quality of the fulfilment of the designed path by a classic control loop provides the reward function. System dynamics is described by a nonlinear planar single track vehicle model with dynamic wheel mode model. The goodness of the planned trajectory is evaluated by driving the vehicle along the track. The paper shows that this encapsulated problem and environment provides a one-step reinforcement learning task with continuous actions that can be handled with Deep Deterministic Policy Gradient learning agent. The solution of the problem provides a real-time neural network-based motion planner along with a tracking algorithm, and since the trained network provides a preliminary estimate on the expected reward of the current state-action pair, the system acts as a trajectory feasibility estimator as well.
Performing dynamic double lane-change maneuvers can be a challenge for highly automated vehicles.... more Performing dynamic double lane-change maneuvers can be a challenge for highly automated vehicles. The algorithm must meet safety requirements while keeping the vehicle stable and controllable. The problem of path planning is numerically complex and must be run at a high refresh rate. The article presents a new approach to avoiding obstacles for autonomous vehicles. To solve this problem, a geometric path generation is provided by a single-step continuous Reinforcement Learning (RL) agent. At the same time, a model-predictive controller (MPC) handles the lateral control to perform the dual lane-change maneuver. The task of the learning agent in this architecture is optimization. It is trained for different scenarios to provide geometric route planning parameters at the output of a neural network. During training, the goodness of the generated track is evaluated using an MPC controller. A hardware architecture was developed to test the local planner on a test track. The real-time operation of the planner has been proven. Its performance has also been compared to human drivers.
The gradually evolving automated driving and ADAS functions require more enhanced environment per... more The gradually evolving automated driving and ADAS functions require more enhanced environment perception. The key to reliable environmental perception is large amounts of data that are hard to collect. Several simulators provide realistic, raw sensor data based on physical sensor models. However, besides their high price, they also require very high computation capacity. Furthermore, most sensor suppliers provide high-level data, such as object detections, that is complicated to reproduce from simulated raw sensor data. This paper proposes a method that directly simulates the detections or object tracks provided by smart sensors. The model involves several uncertainties of the sensors, such as missed-, false detections, and measurement noise. In contrast to the conventional sensor models, this method tackles with state-dependent clutter model and considers the field of view in the detections model. The parameters of the proposed model are identified for an automotive smart radar and camera based on pre-evaluated real-world measurements. The resulting model provides synthetic object-level data with higher fidelity than the conventional probabilistic models, differing less than 2% from the precision and recall metrics of the actual sensors.
The rapid growth of urbanization and the constant demand for mobility have put a great strain on ... more The rapid growth of urbanization and the constant demand for mobility have put a great strain on transportation systems in cities. One of the major challenges in these areas is traffic congestion, particularly at signalized intersections. This problem not only leads to longer travel times for commuters, but also results in a significant increase in local and global emissions. The fixed cycle of traffic lights at these intersections is one of the primary reasons for this issue. To address these challenges, applying reinforcement learning to coordinating traffic light controllers has become a highly researched topic in the field of transportation engineering. This paper focuses on the traffic signal control problem, proposing a solution using a multi-agent deep Q-learning algorithm. This study introduces a novel rewarding concept in the multi-agent environment, as the reward schemes have yet to evolve in the following years with the advancement of techniques. The goal of this study is...
Proceedings of The First Conference on ZalaZONE Related R&I Activities of Budapest University of Technology and Economics 2022
The article presents the development of a low-budget positioning device that aims to provide an a... more The article presents the development of a low-budget positioning device that aims to provide an alternative in selfdriving vehicle development research that could replace costly, commercially available devices. In addition to being financially advantageous, it has the added benefit of allowing students to be involved in development. The primary function of the device is the sensor fusion, which outputs position, velocity, and orientation estimation based on data provided by Real-Time Kinematic (RTK) Global Navigation Satellite System (GNSS) technology and an Inertial Measurement Unit (IMU). High-frequency estimates are generated by running an Extended Kalman Filter (EKF) on a microcontroller in an embedded environment. During the work, new challenges arose several times that required solutions. For example, delays due to the operation of GNSS receivers, which the estimation algorithm must compensate, and proper calibration of the sensors for the measurement vehicle. In addition to the software, the development of the tool includes the complete design, manufacture, and testing of the hardware, which allows testing the completed software units not only in a simulation but also in a real environment. During testing, the output of the developed device was compared several times with commercially available hardware for similar purposes.
In this paper, we study the problem of traffic signal control in general intersections by applyin... more In this paper, we study the problem of traffic signal control in general intersections by applying a recent reinforcement learning technique. Nowadays, traffic congestion and road usage are increasing significantly as more and more vehicles enter the same infrastructures. New solutions are needed to minimize travel times or maximize the network capacity (throughput). Recent studies embrace machine learning approaches that have the power to aid and optimize the increasing demands. However, most reinforcement learning algorithms fail to be adaptive regarding goal functions. To this end, we provide a novel successor feature-based solution to control a single intersection to optimize the traffic flow, reduce the environmental impact, and promote sustainability. Our method allows for flexibility and adaptability to changing circumstances and goals. It supports changes in preferences during inference, so the behavior of the trained agent (traffic signal controller) can be changed rapidly ...
The railway timetables are designed in an optimal manner to maximize the capacity usage of the in... more The railway timetables are designed in an optimal manner to maximize the capacity usage of the infrastructure concerning different objectives besides avoiding conflicts. The realtime railway traffic management problem occurs when the preplanned timetable cannot be fulfilled due to various disturbances; therefore, the trains must be rerouted, reordered, and rescheduled. Optimizing the real-time railway traffic management aims to resolve the conflicts minimizing the delay propagation or even the energy consumption. In this paper, the existing mixedinteger linear programming optimization models are extended considering a safety-relevant issue of railway traffic management, the overlaps. However, solving the resulting model can be time-consuming in complex control areas and traffic situations involving many trains. Therefore, we propose different runtime efficient multi-stage heuristic models by decomposing the original problem. The impact of the model decomposition is investigated mathematically and experimentally in different rail networks and various simulated traffic scenarios concerning the objective value and the computational demand of the optimization. Besides providing a more realistic solution for the traffic management problem, the proposed multi-stage models significantly decrease the optimization runtime.
The paper presents real-world test cases of an optimal trajectory design solution that combines m... more The paper presents real-world test cases of an optimal trajectory design solution that combines modern control techniques with machine learning. The first step of the current research is to train a reinforcement learning agent in a simulated environment, where the conditions and the applied vehicle are modeled. System dynamics is described by a nonlinear single-track vehicle with dynamic wheel model. The designed trajectory is evaluated by driving the vehicle using a control loop. The reward of the method is based on the sum of different measures considering safety and passenger comfort. The proposed method forms a special one-step reinforcement learning task handled by Deep Deterministic Policy Gradient (DDPG) learning agent. As a result, the learning process provides a real-time neural-network-based motion planner and a tracking algorithm. The evaluation of the algorithm under real conditions is made by using an experimental test vehicle. The test setup contains a high precision GPS module, an automotive inertial sensor, an industrial PC, and communication interface devices. The test cases were performed on the ZalaZone automotive proving ground.
2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), 2018
The paper presents multimodel state estimation with constrained filtering applicable in a road tr... more The paper presents multimodel state estimation with constrained filtering applicable in a road traffic situation. The state to be estimated describes the motion of a car observed from the ego-vehicle. Multiple motion modes of the target vehicle are predefined and each is associated with a suitable constraining method. Both Kalman and particle filters are used to perform the estimations. The constrained filters are implemented in the Interacting Multiple Model structure. The outputs are the actual state of the observed vehicle and the motion mode that is in effect at every moment with an associated likelihood value. The performance of the proposed method is evaluated in a simulated environment.
2020 IEEE 24th International Conference on Intelligent Engineering Systems (INES), 2020
This work presents a powerful and intelligent driver agent, designed to operate in a preset highw... more This work presents a powerful and intelligent driver agent, designed to operate in a preset highway situation using Policy Gradient Reinforcement Learning (RL) agent. Our goal is to create an agent that is capable of navigating safely in changing highway traffic and successfully accomplish to get through the defined section keeping the reference speed. Meanwhile, creating a state representation that is capable of extracting information from images based on the actual highway situation. The algorithm uses Convolutional Neural Network (CNN) with Long-Short Term Memory (LSTM) layers as a function approximator for the agent with discrete action space on the control level, e.g., acceleration and lane change. Simulation of Urban MObility (SUMO), an open-source microscopic traffic simulator is chosen as our simulation environment. It is integrated with an open interface to interact with the agent in real-time. The agent can learn from numerous driving and highway situations that are created and fed to it. The representation becomes more general by randomizing and customizing the behavior of the other road users in the simulation, thus the experience of the agent can be much more diverse. The article briefly describes the modeling environment, the details on the learning agent, and the rewarding scheme. After evaluating the experiences gained from the training, some further plans and optimization ideas are briefed.
Proceedings of the 16th International Conference on Informatics in Control, Automation and Robotics, 2019
The paper presents a motion planning solution which combines classic control techniques with mach... more The paper presents a motion planning solution which combines classic control techniques with machine learning. For this task, a reinforcement learning environment has been created, where the quality of the fulfilment of the designed path by a classic control loop provides the reward function. System dynamics is described by a nonlinear planar single track vehicle model with dynamic wheel mode model. The goodness of the planned trajectory is evaluated by driving the vehicle along the track. The paper shows that this encapsulated problem and environment provides a one-step reinforcement learning task with continuous actions that can be handled with Deep Deterministic Policy Gradient learning agent. The solution of the problem provides a real-time neural network-based motion planner along with a tracking algorithm, and since the trained network provides a preliminary estimate on the expected reward of the current state-action pair, the system acts as a trajectory feasibility estimator as well.
Uploads
Papers by Szilárd Aradi