1

EECS 4314 Assignment 1:

Openpilot’s Conceptual Architecture Report
Group: Beyond

Authors:
Student No. Name of the author Author’s email

217537739 Nishaant Dhingra [email protected]

218550111 Abdirazak Yusuf [email protected]

218305847 Hashim Khan [email protected]

218452805 Kawshar Patel [email protected]

217692690 ​Mahdiar Shoraka [email protected]

217847328 Raj Panjabi [email protected]

218908368 ​Ryan Lyn [email protected]

217554668 ​Sanchit Duggal [email protected]

 2

Abstract:
This conceptual architecture report provides a thorough overview of openpilot.
Openpilot is a powerful open-source driver assistance system. The report gives a
high-level overview of the system’s structure, functionality, and interactions among its
parts. The report examines openpilot’s essential components and the overall
functionality of the system. We look into how the different subsystems of the system
including hardware and software interact and communicate with each other to provide
the functionality offered by the system. To facilitate a newcomer’s understanding of
openpilot, the report includes a sequence diagram to show the interaction between
processes during one of its main use cases. The sequence diagrams give a visual
picture of the system’s interactions and processes, which makes it simpler to
understand its functionality. The report also includes a state diagram to provide an
insight into one of the salient features of openpilot that ensure the driver is attentive at
all times to take over the vehicle if need be. We finish off by analyzing how the
architecture of the system affects how work is distributed among developers which is of
utmost importance here since this is an open source project and anyone can contribute
to making it better.

Introduction & Overview:

openpilot is one of the cutting-edge technologies in autonomous driving. A rapidly
evolving project due to the continuous development of the open-source community
consisting of users like you and Comma. The project alone represents a collaborative
effort to create an accessible and innovative platform for an advanced driver assistance
system; available on a tremendous number of car models.

Furthermore, openpilot offers various features from adaptive cruise control to automated
lane centering. The massive software is built on various technologies such as machine
learning, computer vision, and automotive engineering. Additionally, the massive
open-source project codebase is built on the foundation of various design patterns and
a well-defined architectural style. At the core, openpilot embraces effective means to
enhance scalability, maintainability, and modularity.

Moreover, openpilot consists of various subsystems working together and

communicating efficiently. Comma.ai the creators of openpilot include other various
open-source projects to add great functionality to their data and control flow as well as
safety and other unique features. Openpilot stands as one of the top advanced driver
assistance systems.

openpilot requires either a comma device or a PC running Linux or Windows to run the
software, it also requires a car harness to allow the software to interact with the car's
CAN buses [1] [2] [3] [4]. All the hardware required is sold separately by comma and
supported for 1 year after the next generation is sold [5]. The software is marketed as a
driver assistant rather than fully automatic driving, requiring driver attention at all times
[1]. A camera is located within the comma device and uses machine learning to monitor
the driver and ensure their attention is maintained at all times [1].
 3

Comma stores a list of cars with known functionality with openpilot as a tab on their
main website and under a repository named OpenDBC and tools to add cars not on the
list [6] [7]. Comma generally recommends using newer generations of Toyota and
Hyundai cars as the majority of cars they have confirmed connectivity with belong to
those brands [6]. The Hyundai brand encompasses Hyundai, Kia, and Genesis totaling
89 cars with confirmed support while the Toyota brand encompasses Toyota and Lexus
totaling 66 more cars [6] [8].

Comma also markets openpilot as an implementation of modern machine learning.

Openpilot makes use of a framework called Tinygrad which is another open-source and
community-developed framework. Tinygrad is built on tensor libraries and was built to
be easily implemented into other projects and expandable with accelerators for specific
components. The project was founded by George Hotz who is also the founder of
comma.ai meaning the connection between the 2 projects predetermined upon startup
[9].

openpilots' main functionality stems from the communication between their hardware or
a client's personal computer if they choose, the open-source software, and a car's
built-in hardware. Openpilot offers features such as adaptive cruise control and
automated lane centering with the communication between the software and a car's
CAN buses. The CAN buses on modern cars allow for commands to control electronic
steering and the pedals openpilot takes full advantage of to offer its assistance [1].

This paper will delve deeper into the conceptual design of openpilot’s development.
Focusing on key elements which is the high-level architecture, each subsystem, the
derivation of its architectural style, integration of design patterns, and other various
topics. It will explore the abstract foundation that empowers openpilot’s advanced driver
assistance system; providing insights into the collaborative efforts that developed the
system.

High-Level Architecture:
Openpilot, developed by comma.ai, represents a forefront integration of hardware and
software components designed to augment the driving experience through advanced
automation features. This system, conceptualized to enhance vehicle safety and
operational efficiency, embodies a complex interplay of services that seamlessly interact
to execute its functionalities.

Before diving into the intricacies of how all the different components interact with each
other to provide safe autonomous driving assistance we can understand how software
for autonomous driving can work. We should be able to read data from sensors, make a
decision about what to do given the data collected, and then act upon the decision using
actuators like accelerators or steering.
 4

Figure 1: Core System Functionality Abstraction

This simple abstraction in Figure 1 of such a system provides us with an understanding

of how it will work and what components can be involved in the system. Looking at this
we can see that the architecture exhibits Process Control architecture.

Description of subsystems -
The system has four main components/subsystems: Sensors and Actuators,
Localization and Calibration, Neural network runner, and Controls. Here is an overview
of each of these:

1. Sensors and Actuators: The sensors and actuators component sets up the first
link with the surroundings and gathers important information for later use. At the
heart of openpilot's ecosystem is the boardd service, which acts as the primary
liaison between the main device and the panda, an interface device facilitating
communication with the car's peripherals. The peripherals, including the GNSS
module for global positioning, infrared LEDs for driver monitoring under low light
conditions, and the vehicle's electronic systems, are crucial for the holistic
operation of openpilot. Through boardd, data from these peripherals is funneled
into the system, serving as a foundational input for subsequent processing
and decision-making. The camerad service is pivotal in managing the visual
aspects of openpilot's operation. It handles the acquisition of visual data from
multiple cameras aimed at the road ahead and the driver, ensuring optimal image
quality through autofocus and auto-exposure techniques. This visual data is
essential for both navigation and monitoring driver attentiveness. The integration
with visionipc allows for efficient distribution of image data across the system,
minimizing latency and overhead in data handling. Sensory input extends
beyond visual cues, with the sensord service dedicated to configuring and
reading data from gyroscopes, accelerometers, magnetometers, and light
sensors. This multi-dimensional sensory input provides a comprehensive
understanding of the vehicle's dynamics and environmental conditions,
enhancing the system's decision-making capabilities.
 5

2. Localization and Calibration: Localization and calibration play critical roles in

accurately positioning the vehicle within its environment. The ubloxd and
locationd services parse GNSS data and apply sophisticated filtering techniques
to determine the vehicle's precise location, orientation, and dynamics.
Calibrationd adjusts the input to the neural network, ensuring that data
interpretation aligns with the physical orientation of the vehicle. Vehicle-specific
parameters are managed by paramsd, employing a Kalman Filter to estimate
and adjust for variations in vehicle dynamics, enhancing the accuracy and
reliability of control signals sent to the car.

3. Neural network runners: Central to openpilot's functionality are the Neural

Network Runners, particularly modeld, which ingests the stream of image data
to predict the optimal driving path and maneuvers, such as lane changes. The
supercombo model within the modeld processes this data, alongside other
metadata like lane lines and vehicle positions, to generate real-time driving
instructions. The neural network operates in real-time, continuously analyzing the
incoming sensor data to provide timely feedback to the openpilot system. Not
only are the neural network models used in driving assistance, but they are also
used for driving monitoring feature dmonitoringmodeId predicts if the driver is
attentive or not based on the driver’s head position, eye movements, and facial
features.

4. Controls: The Control components include radard, plannerd, and controls

which convert the prediction and plans from the neural networks into signals that
guide the car along the intended path while meeting the safety parameters.
Radard processes radar data for obstacle detection, plannerd devises strategic
plans for maneuvering, and controlsd translates these plans into actionable
commands for the vehicle's steering, acceleration, and braking systems.

The flowchart below visualizes the openpilot system's architecture, illustrating how
data flows from hardware inputs through various components, leading to the
execution of control commands in the car. Starting with boardd, which interfaces with
the car and its peripherals, data flows through camera and sensor management
modules (camerad and sensord), into the neural network processing for driving
decisions and driver monitoring (modeld and dmonitoringmodeld). Localization and
calibration services (ubloxd/locationd and calibrationd/paramsd) refine the system's
understanding of the vehicle's position and orientation, allowing plannerd to plan the
path. Finally, controlsd translates these plans into specific control signals,
executing commands in the car. This end-to-end process encapsulates the intricate
orchestration of data and decisions within openpilot, enabling advanced driver
assistance functionalities. ​
 6

Figure 2: Flowchart of Data Flow in openpilot

The architecture of openpilot employs a variety of architecture making it heterogeneous.

The several architectural styles that can be seen in the system are discussed in the
following text. As shown in figure 1, The core, Process Control System, operates in
real-time with a feedback loop and continuously works on making driving better and
safer. The system employs a closed loop where sensors feed data to a
machine-learning model, and the actuators respond accordingly. Process Control
Architecture allows for real-time responsiveness by enabling quick and accurate
responses to changes in the environment and ensures safety by continuously
monitoring the environment. This iterative process aligns with the system's goal of
 7

continual improvement. Openpilot employs a Layered architecture, featuring the lowest

layer that deals with the hardware interactions, a middle layer being the
decision-making, and the top layer. The layers only interact with the layer adjacent to it
hence providing low coupling.

Figure 3: Interaction between different components [12]

Referring to the figure above, the bottom layer consists of sensors that are installed in
the car that send messages to CAN that are relayed over the comma3 device by a
hardware component called Panda. The openpilot software runs multiple software
processes that communicate with others and exhibit different architectural styles and
design patterns that are discussed later. These software modules also work
independently and interact with others using specified interfaces and protocols. All
messages to and from the panda are managed by the boardd background process.
boardd publishes these messages for other processes to use. The processes also
include radard, plannerd, and controlsd which play an important role in monitoring radar
messages, planning the car’s path, and converting plans to CAN messages. These
processes use our Neural Network Runners that provide real-time feedback to our
system. These runners are critical in terms of performance since they need to respond
instantaneously to navigate the car depending on the situation. The neural models also
need to be accurate and consistent and they are optimized and trained in parallel using
the driving data from drivers using openpilot’s system. The layered architectural
approach makes sure that each component has its own domain and can be developed
and enhanced independently of each other as long the interfaces remain the same. This
promotes high cohesion with clearly defined roles for each module, ensuring modularity
and efficiency in enhancing the overall driving experience.

Architectural Style Derivation -

To facilitate efficient coordination and communication between software components,
openpilot adopts an Implicit Invocation (specifically Pub-Sub) architecture, forming
the basis of asynchronous communication. This mechanism ensures a clear flow of
information for efficient coordination, even if the components are written in two different
languages. This architecture integrates with the event-driven interactions in openpilots’
systems. For instance, dmonitoringmodeId is a publisher that announces events related
to the driver’s attentiveness. Simultaneously, the user interface (UI) functions as one of
 8

the subscribers that receive notifications about the event and updates the display to
reflect the changes

In parallel to these architectural principles, openpilot prioritizes a Data Abstraction and

Object-Oriented Organization, using a shared car interface for various car types to
showcase an adaptable and encapsulated design. The primary focus is on achieving
low coupling, allowing modules to function independently, and providing flexibility for
changes. To do this, openpilot uses a common CarState object as an abstraction layer
to simplify the adjustments in the machine learning model and other code. It acts as a
shared repository for important information which establishes a standardized interface
for communication between different components. It serves as a key input to the
machine learning model influencing the adjustment in the control loop and helps to
enforce safety mechanisms by preventing and monitoring changes in car behavior. This
adaptability and standardization allow openpilot to be compatible with different car types
and make it extensible to support new car models in the future.

Alternative Architectural Style -

The architectural style seen to be used in different tasks has been given to different
software modules that execute independently and coordinate with other modules using
defined interfaces communicating using pub sub-architecture. One way this could have
been interpreted is instead of having several different processes having one monolith
application that takes in the input from sensors, localizes and calibrates it, uses a
decision-making process to decide how to navigate the car given the current situation,
and then give the results back to the the car actuators after converting them in the CAN
format. This would have led to a lot of problems as changes to each section in this
monolith module could have broken other parts and there would be very low cohesion.
Moreover with such an architecture adding more features or making enhancements to
the code would be very difficult.

Design Pattern -
One of the main design patterns that can be observed in the system is Facade design
in the software processes. The comma3 device acts as a client in this example and
does not interact with any of the sub-modules running or is aware of how they are
implemented. The device only sends data to the boardd module and that publishes
them to all the sub-modules that interact with each other and return CAN messages that
can be sent to the car by boardd via Panda.

Concurrency -
openpilot uses continuous integration to automatically test code changes, ensuring
safety, stability, and reliability as it evolves with new features and fixes. For instance,
before implementing parallel CI tests, openpilot required 35 minutes to complete its test
suite, significantly slowing development. By adopting parallelization, testing time was
reduced to just 3 minutes on CI servers and 10 minutes on local machines. This change
has dramatically increased the speed of the development cycle, providing quicker
feedback to developers and enhancing the project's agility and efficiency. Another
instance is openpilot's machine learning models, crucial for interpreting sensor data and
 9

making driving decisions, being trained and executed in parallel to respond in real-time
when sensory inputs come in for a decision.

Use Cases:
UC 1 - Adaptive Cruise Control (ACC) -
Scenario: A user traveling at a high speed on the highway wants AI to handle the car
and maintain the speed accordingly. The user is willing to trust the openpilot system to
handle the speed and steering while they sit back and enjoy the ride.

Solution: The user can engage openpilot's Adaptive Cruise Control (ACC) feature,
which allows them to relax while maintaining a safe distance from the vehicle in front
(lead car distance) and adjusts the vehicle’s speed accordingly.

Using a combination of sensors, including stock sensors and comma sensors, along
with sophisticated software components like boardd, radard, plannerd, and controlsd,
ACC processes real-time data to maintain a safe distance from the vehicle ahead,
adjusts speed accordingly, and navigates traffic flow with precision. The communication
between openpilot and your car is facilitated by the CAN bus and Panda interface,
ensuring accurate translation of commands for optimal performance.

Let us consider Figure 4 and understand in detail how ACC works:

Figure 4: Sequence diagram for ACC

Detailed explanation of how ACC works:

- In both scenarios, visual input is processed by the boardd software component.
- In the first scenario, visual data is transmitted via the car's CAN bus, facilitated
by the Panda interface, which serves as a bridge between the vehicle's hardware
systems and the openpilot software.
- In the second scenario, data comes directly from comma sensors to the boardd.
 10

- From there, radard processes radar data, generating a radarState message

having information about the lead car distance (it uses modeld for machine
learning).
- This message is then used by plannerd to plan the car's path.
- plannerd then publishes the planned path to controlsd, which converts it into
actionable commands (CAN messages) for the car's actuators [10] [11].
- These commands are sent back to the car's CAN bus via boardd, ultimately
controlling acceleration, braking, and steering. The entire process
demonstrates the seamless integration of hardware and software in the openpilot
system [10] [11].

UC 2 - Driver Monitoring -
Scenario: A user wants to go on a long road trip from one city to another. The road trip
comprises multiple stops and long hours of driving on the highway. The user is excited
to embark on this journey but also a little worried as it is their first time driving this far
and of all the challenges that the road trip poses, like fatigue and distractions.

Solution: The user can activate the openpilot system in their car, which includes a
robust feature of driver monitoring. The system utilizes sensors and cameras to monitor
the behavior and attentiveness of the driver while the driver is on the road.

Through facial recognition and tracking algorithms, the system can detect the driver’s
head positions, eye movements, head positions, etc. These measures are fed into a
neural network model that has been trained over a large dataset of drive behavior. The
model then classifies the user into different categories such as attentive or distracted
and issues alerts to prompt the driver to regain attentiveness. Since the model is
designed to adapt to various driving conditions, the user can adjust the sensitivity of the
system based on road complexity, traffic density, etc.

Using the driver monitoring feature, the user can travel smoothly without any worries.
 11

Figure 5: State diagram for Use Case 2

A detailed explanation of how the Driver Monitoring system works:

- The system starts in the Initial State, where it awaits messages to determine
driver status.
- Once the system has detected a user, it begins Driver State Monitoring, the
system determines whether the model is certain about the driver's state.
- If the state is detected to be a Driver Distracted State, the system sends that
message to the neural network that processes the information and triggers an
Alert.
- During the initial detection, the system also detects the Driver’s Interaction with
the Car, and if the driver is disengaged or distracted, the system stays in the
driver's distracted state and continues to alert the driver.
- Once the driver gains attention and engages with the car, the Alert is Cleared.
- After clearing the alert, the system returns to the Normal State to continue
monitoring the driver.

Implications of the division of responsibility among participating

developers:
Since the project is under the MIT license (meaning that it can be used for free, and
without any specific copyright rules) and it is open source (code is accessible to the
public), all developers can contribute, and give feedback to the other developers on the
project by adding Issues (can be a bug report, feature request, etc.) under the Issues
tab on GitHub.
 12

Figure 6: openpilot GitHub repository tabs

As seen in Figure 6, the project currently has 198 issues, and 92 Pull requests
(explained in the next paragraph). This is one of the implications of division of
responsibilities and collaboration.

In addition, under the Pull requests tab, we can see other developers' branches that
contain extra code and functionalities. If the branch passes the code review check, it
can be merged into the main (or an upper-level) branch. This also shows developers
can collaborate and add new specifications to the main project.

Furthermore, a set of rules for written communication among the developers can be
found in the repository within a file named CONTRIBUTING.md. [13] These rules and
definitions make collaborations, feature requests, bug report readings, and code
reviews easier. In addition, you can fork the project on Github from the original
repository to your copy on your profile. When you fork a project, you make a duplicate of
that project in your GitHub profile, which always references the main project. [14]

Figure 7: GitHub Fork menu

As seen in Figure 7, this project is forked ~84000 times, meaning that many developers
made duplicates of the project to make different versions and specifications of their
own.

In addition, GitHub has a feature that shows a list of all developers (contributors) on
the main page of the project repository. This is the most explicit sign of collaboration
and division of responsibilities on a Version Control system like GitHub.
 13

Conclusions:
In conclusion, openpilot’s conceptual architecture offers a strong framework for
understanding the idea that drives the success of this automated driving assistance
system. As we look to the future, openpilot will continue to improve due to its ongoing
development in developer resources and community collaboration. Moreover, with
continuous improvements in neural network models that are being optimized as more
and more data comes from drivers using the system, openpilot’s popularity will keep
growing. This report acts as a guide for both novices and experts looking to understand
the architecture of an automated driving assistance system. The fundamental building
blocks and procedures that support openpilot have been demonstrated through this
report. Moreover, this report has provided new information on how openpilot works,
develops, subsystems, deals with concurrency, and the division of responsibilities
among developers. The following is a summary of our key findings:
1. Compatibility: The architecture of openpilot is built to accommodate more and
more cars and be compatible with different models of different cars. It is
openpilot’s compatibility that demonstrates the relevance it will continue to have
in the future as new upcoming models can be supported with minor changes.
2. Utilizing sequence diagrams to visualize interactions among systems: Sequence
diagrams have been crucial in giving a visual picture of the interactions between
openpilot's components during particular processes. They provide useful
perceptions of the operation of the system.

The following are our proposals for future directions:

1. Promoting community collaboration: As openpilot is open-source, the community
plays a vital role in its development. Community collaboration can foster new use
cases hence enhancing its functionality.
2. Providing more essential tools and resources for developers: One of openpilot’s
future goals could be to provide even more tools, documentation, and other
resources to speed up process development and promote the education of all of
its users since the documentation to understand the system remains minimal.

Lessons Learned:
The choice of framework to power openpilot’s machine learning is not based on
completeness of features or being the most well-known. The founder of comma.ai
George Hotz is also the founder of tinygrad, the framework used in openpilot, being the
founder of both it could be possible to create a framework that is specifically designed to
work between his projects. Tinygrad on its own has the intention of being simple to
implement and highly expandable with accelerators which allows the machine learning
of the vehicles running openpilot to be tuned to run faster for specific tasks. The
emphasis on using a data flow pattern is interesting as while the architecture of the
modules is mostly following layered there is a sequence upon startup that requires CAN
information to be passed through multiple components. The process flowing from
OpenDBC, through Cabana, to Panda, and finally to Cereal is lengthy but is most likely
important for code rigor and safety as Panda upholds the safety standards while Cereal
controls the vehicle's functions. The use of implicit invocation makes the most sense for
 14

the vehicle controls as there are many sensors with multiple different functions, which
allows for straightforward packets from the controller to be sent to the sensors that
subscribe to each specific type of packet.

Serial Abbreviation Table

No.
Abbreviation Actual Words

1. PC Personal Computer

2. CAN Controller Area Network

3. GNSS Global Navigation Satellite System

4. UI User Interface

5. CI Continuous Integration

6. ACC Adaptive Cruise Control

7. MIT Massachusetts Institute of Technology

 15

References:

[1] Openpilot - Open source advanced driver assistance system. comma.ai - make
driving chill. (n.d.). https://www.comma.ai/openpilot

[2] Getting started with openpilot. comma.ai blog. (2023, August 2).
https://blog.comma.ai/gettingstarted/

[3] Commaai. (n.d.). Commaai/openpilot: Openpilot is an open-source driver assistance

system. openpilot performs the functions of Automated Lane centering and adaptive
cruise control for 250+ supported car makes and models. GitHub.
https://github.com/commaai/openpilot#running-on-a-dedicated-device-in-a-car

[4] Openpilot tools. GitHub. (n.d.).

https://github.com/commaai/openpilot/tree/master/tools

[5] Comma 3x - make driving chill. - make driving chill. (n.d.).

https://www.comma.ai/shop/comma-3x

[6] Vehicle compatibility. comma. (n.d.). https://www.comma.ai/vehicles

[7] Commaai. (n.d.-a). Commaai/opendbc at

3d1be8427a7e801e7da4e8506e5d5c2605de9176. GitHub.
https://github.com/commaai/opendbc/tree/3d1be8427a7e801e7da4e8506e5d5c2605de
9176

[8] Bartlett, J. S. (n.d.). Who owns which car brands? Consumer Reports.
https://www.consumerreports.org/cars/who-owns-which-car-brands-a5925868661/

[9] Hotz, G. (n.d.). Geohot. geohot. https://geohot.com/

[10] How does openpilot work?. comma.ai blog. (2017, June 26).
https://blog.comma.ai/how-does-openpilot-work/

[11] From vision to architecture: How to use openpilot and live. From Vision To
Architecture: How to use openpilot and live - DESOSA 2020. (2020, March 11).
https://desosa.nl/projects/openpilot/2020/03/11/from-vision-to-architecture

[12] How openpilot works in 2021. comma.ai blog. (2021, October 12).
https://blog.comma.ai/openpilot-in-2021/

[13] How to contribute. GitHub. (n.d.-a).

https://github.com/commaai/openpilot/blob/master/docs/CONTRIBUTING.md

[14] Science, T. C. for O. (n.d.). Fork a project (OSF projects). OSF Support.
https://bit.ly/3SOG198