Visual Programming for Interactive Robotic Fabrication

Processes
Process flow definition in robotic fabrication

Johannes Braumann1, Emanuel Gollob2, Karl Singline3

1,2,3Creative Robotics, UfG Linz, Austria
[email protected] [email protected]
[email protected]

Visual, flow-based programming environments in architecture and design are built to

control data flow but not process flow. However, controlling the process flow is essential
for interacting with robotic fabrication processes, so that they can react to input such as
user interaction or sensor data. In this research, we combine two visual programming
environments, utilizing Grasshopper for defining complex, robotic toolpaths, and Unity
Visual Scripting for controlling the overall process flow and process interaction. Through
that, we want to enable architects and designers to define more complex, interactive
production processes, with accessible, bespoke user-interfaces allowing non-experts to
operate these processes - a crucial step for the commercialization of innovations. This
approach is evaluated in a case study that creates a mobile, urban microfactory that
prototypically fabricates location-specific objects through additive manufacturing.

Keywords: Visual Programming; State Machines; Industrial Robotics; Unity Visual

Scripting.

INTRODUCTION Especially when robotic processes move from

A priority of our research into parametric robot academic research to commercial application, there
control (Braumann and Brell-Cokcan, 2011) has is a need to create maintainable systems with easy
always been to increase the accessibility for new user user-interaction that allow non-experts to manage a
groups, targeting primarily the creative industries production process.
with their existing expertise in visual programming. An area where accessibility, low-code, and
Visual programming environments such as decision-making intersect is game development.
McNeel Grasshopper and Autodesk Dynamo, Popular game engines like Unity and Unreal have
supported by specialized plugins, today allow the created rich ecosystems that support developers
definition of geometrically complex, individualizable with their visual programming environments.
fabrication processes that go far beyond the scope This paper presents a prototypical system that
of industry-standard CAM (Computer-Aided allows accessible visual programming of interactive
Manufacturing) software, but due to their inherently robotic fabrication processes within Unity. By
acyclic design are significantly limited when it comes building upon established user interaction concepts
to the handling of user interaction, real-time data and programming environments, we aim to
and decision-making. democratize the definition of complex, interactive

Volume 2 – Co-creating the Future – eCAADe 40 | 427

robotic fabrication sequences. As such, the research several architectural applications, for example by
targets designers who are already familiar with Amtsberg et al (2015) and Johns and Anderson
parametric visual programming environments and (2018). Frameworks for real-time robotic control
get empowered to create flexible, branching have been presented by Braumann and Brell-Cokcan
fabrication logic informed by user-input or sensor (2015) and Garcia del Castillo y López (2019).
data. The underlying programming paradigm of flow-
In a commercial context, this logic can then be based programming is relatively old, having been
“packaged” so that non-experts only need to interact developed by Morrison (2010) in the late 60s for IBM,
with a focused, custom-tailored UI that guides them building upon the concept of co-routines by Conway
through a fabrication process, without requiring (1963). Today, flow-based visual programming
expertise in the underlying visual programming environments like McNeel Grasshopper can be
code. considered part of the trend towards low-code
This approach is verified through the rapid (Fryling, 2019), allowing non-experts to create
prototyping of a mobile, robotic 3D printing geometric algorithms intuitively.
platform that acts as an urban microfactory, Of particular benefit to non-programmers is
enabling local and customized fabrication while Grasshopper’s immediate reaction to user
operated by a non-coder. interaction, coupled with a well-thought-out
approach to user interaction through colouring and
FLOW-BASED PROGRAMMING FOR visual cues, and its thorough catching of most
ARCHITECTURAL ROBOTICS exceptions.
Industrial robots are today commonly encountered As Grasshopper enforces an acyclic dataflow,
in architectural research, education, and increasingly current workflows for robot programming in
also in practice. Grasshopper work particularly well for processes
that are self-contained and do not depend on
previous iterations, like for example typical Figure 1
production processes where some parameters are Real-time robotic
set, and the resulting robot trajectory is then either process defined
written into a file or directly streamed to the robot. through flow-based
However, within the scope of interactive robotic programming,
fabrication, we see a major challenge in the fact that using Grasshopper
Grasshopper’s data flows cannot be natively
directed and looped.
As such, users must either use third-party
plugins (Zheng, Zhe, and Yang, 2019) to implement
ways of feeding data back to the start of a
Grasshopper definition (Celani and Vaz, 2012) or
switch to systems like Processing for generative
An enabling factor in that regard has been the processes (Pantazis, 2017; Elashry and Glynn, 2014).
development of accessible robot simulation and In reference to human’s tendency to adjust ideas
programming tools like HAL (Schwartz, 2013), in the process of making, Kim et al. (2017) outlined a
KUKA|prc (Braumann and Brell-Cokcan, 2011) (see speculative system for interactive human-machine
figure 1) and the open-source Robots (Soler, 2021). fabrication that would allow a bidirectional
Due to these and other interfaces, sending and fabrication process, seeing machines as
receiving data from a robot has been realized in

428 | eCAADe 40 – Volume 2 – Co-creating the Future

 collaborators instead of output devices; such could translate the default right-handed coordinate
currently not be realized natively on Grasshopper. system (Z+ up) to Unity’s left-handed (Z+ forward)
We therefore see a need for systems that allow a coordinate system and to express distances in
more accessible and visual definition of complex meters instead of millimetres.
data flows, to enable interactive fabrication, where A significant difference compared to
the data may originate from a user through a Grasshopper is that UVS also provides a graphical
graphical user interface, a machine learning model, way to define the data flow (see figure 2), so that e.g.,
an external sensor system, or another robot. after an if-else node, only the nodes connected to
the chosen option are executed. Furthermore,
METHODS: TOWARDS INTERACTING multiple so-called flow-graphs (in Grasshopper:
WITH FABRICATION definitions) can be executed in parallel.
In the wider field of robotics, “pre-defined” processes
Figure 2 - as in the fabrication of customized, architectural
UVS allowing components - are comparably rare. In contrast,
process flow dynamic processes that adapt to navigate in new
control in addition environments (Kruse et al, 2013), grasp unknown
to data flow objects (Zeng et al, 2019) and in general react to real-
time sensor data are the norm. Popular software
environments like ROS (Quigley et al, 2009) provide
a middleware but generally depend on text-based
programming languages like Python and C++. This is unlike Grasshopper, where only one
Reacting to user input, dealing with multiple definition can be active at a time. In addition, UVS
object states, and making decisions based on various also introduces state-graphs, which are state
input data are core aspects of game development machines that can be used to define the transition of
that are therefore well represented in these one flow-graph to another one.
environments. Within the context of this research, The concept of a state machine dates back to an
Unity Visual Scripting (UVS) was chosen as both even earlier date than flow-based programming
Unity, and Grasshopper support plugins written in (McCulloch and Pitts 1943). UVS uses a Mealy-style
the .NET framework, allowing us to make the existing finite state (Mealy, 1955) consisting of a finite
capabilities of KUKA|prc available in both number of states that are connected through
environments, greatly facilitating the exchange of transitions, which in turn are tied to certain
data, as well as troubleshooting. conditions.
There are already several projects within the It can therefore be easily represented by a state
field of architecture and robotics that were built on transition diagram, which in UVS’s state-graph not
Unity, e.g., for creating a graphical user interface for only visualizes the data flow but becomes a visual
complex robotic 3D printing (Soler, 2017) and acting way of defining and connecting program states. A
as a machine learning environment (Hosmer, 2019). designer can therefore sketch out the process flow
as a state graph and then define the individual flow
Bringing KUKA|prc to UVS graphs within the state machine.
While initially a Grasshopper plugin, KUKA|prc is Within the context of architecture, Jensen and
written as a .NET Standard library that can be used Das (2020) propose the use of state machines for
on all platforms that support the .NET framework. human-robot collaboration, utilizing the addon
Integrating the CAD-originating KUKA|prc into Metahopper with custom Python scripts to realize a
the computer-graphics-based Unity requires us to state machine within Grasshopper. While not

Volume 2 – Co-creating the Future – eCAADe 40 | 429

referred to as a state machine, the “feedback loop” CASE STUDY: WANDERING FACTORY
by Moorman et al (2016) fulfils a similar purpose. In order to verify the capabilities of the developed
Garcial (2019) utilizes a state-based “enactive robot system, we have applied it towards creating an
system” that can communicate with external visual interactive, mobile, robotic fabrication process
programming environments. that…

CAD in Unity • …allows UI-based operation of a production

In UVS, the user creates a state machine (“state process by an unskilled user.
graph”) and then adds various flow states, each of • …may be programmed by a non-coder solely
which contains a script-graph, with the transitions through visual, flow-based programming in
also represented by a flow-graph. Once a data flow Grasshopper and UVS.
reaches a transition element, the state machines
change to the according state and executes the The Wandering Factory is intended as a
associated flow, until the next transition is triggered. demonstrator within the context of the FabCity
This allows the user to define complex interactive Global Initiative, towards enabling the return of
fabrication processes through a flow-based manufacturing to cities. The demonstrator (see
programming tool. figure 3) consists of four dedicated parts that are
However, the generation of robot trajectories is linked through the software architecture that was
challenging, as Unity is missing many of Rhinoceros developed as part of our research: A KUKA Agilus
3D’s and Grasshopper’s geometric features. KR10-R1100-2 robotic arm, a Mattro ROVO-2 tracked,
We therefore use Rhino Compute to allow access electric vehicle, an Intel RealSense 3D camera and a
to Rhinoceros 3D functions within Unity through a Noztek pellet extruder. Communication is
representational state transfer (REST) application established through a variety of interfaces: KUKA
programming interface. mxAutomation controls the robot in real-time, the
Doing so allows us to split the relatively light vehicle uses CAN bus for communication, the 3D
user-interaction and data-flow tasks from the more camera is connected via USB and the extruder via the
computation heavy path planning and toolpath EtherCAT fieldbus with a Beckhoff-PLC
verification. As such, even a relatively low-powered (programmable logic controller) in between. These
device like a tablet or mixed reality headset can data sources are managed on a compact PC running
orchestrate a complex robotic production process. Unity which at the same time provides a 3D
visualization and a user interface.
The idea of the demonstrator is to showcase Figure 3
approaches towards local, customized, and Wandering Factory
sustainable fabrication: An environment is scanned, demonstrator (left),
and the resulting point cloud used to customize an individualized,
object so that it precisely fits at the captured locally adapted 3D
location. Toolpaths are automatically generated so print representing
that the object may be immediately 3D-printed by urban furniture
the industrial robot using recycled materials, (right).
without requiring an expert-user to be on-site

Implementation
The Wandering Factory is implemented through two
UVS graphs: A script graph is constantly running in

430 | eCAADe 40 – Volume 2 – Co-creating the Future

Figure 4 the background and manages the communication
State-graph with the KUKA robot via mxAutomation, while a
representing the state graph largely follows the user’s interaction with
Wandering Factory the user interface, processing and visualizing the
project in UVS data coming from the robot as well as the various
sensors and connected systems.
The unskilled user is guided through the
fabrication process in five steps that correspond to
Figure 5 the interface sections (see figure 4 and 5).
State-graph Initially, the user moves the mobile platform to
representing the the desired location. In the “Scanning” state, the
Wandering Factory point-cloud generated by an Intel Realsense 3D
project camera is displayed in real-time within the robot’s
conceptually workspace (see figure 6). Once on-site, the “Preview”
state saves the point cloud to a local file as
embedding it into the Grasshopper file would lead
to a decrease in performance. The provided
Grasshopper script then turns the point cloud into a
NURBS surfaces, fits the targeted print object onto
the surface and generates the necessary non-planar
toolpaths to fabricate it with the robot’s pellet
extruder. Finally, the script returns both the list of
serialized robot commands as well as the serialized
Figure 6 preview mesh geometry, which is then displayed in
Point cloud the Unity viewport (see figure 7). The user can then
visualization in the either change some core parameters of the targeted
user interface print object, or proceed to the “Fabricate” state,
which deserializes the KUKA|prc robot movement
commands and streams them to the robot via the
mxAutomation interface, while visualizing the
robot’s current position in the user interface. Once
completed, the user can immediately start a new
process.

Figure 7 Evaluation
3D print preview One of the most visible projects towards mobile
based on scan data, fabrication in an architectural context is ETH’s “In-
generated via Situ Fabricator” (Buchli et al, 2018). While its
Rhino Compute capabilities are clearly far beyond the presented case
study, doing so required a team of many researchers
and a large budget over multiple years, creating a
multi-layered, custom software architecture
building upon software such as OpenCV, ROS, and
Eigen as well many custom-developed tools

Volume 2 – Co-creating the Future – eCAADe 40 | 431

 In contrast, the Wandering Factory was put The presented case study uses the transition of
together within 6 weeks, with a minimal budget and states mostly in synchronization with the user
using a control system that is built exclusively on interface, following the user input. When working
flow-based programming, showcasing the with real-time sensor data, these cycle times may be
increasing accessibility of technologies that were reduced towards enabling fully interactive
previously limited to high-end research institutions. fabrication processes that immediately react to user
input, changes in the robot’s environment or the
RESULTS AND DISCUSSION behaviour of complex materials. Peng et al. (2018)
We consider the combination of using UVS for the presented approaches towards interactive
global process planning and user interaction with fabrication for their Robotic Modelling Assistant.
Grasshopper through Rhino Compute to be a Relating to mixed reality, Braumann and Singline
powerful way for defining complex robotic (2021) presented a system for linking a WiFi-
fabrication processes in a visual, low-code way. That connected HoloLens 2 mixed reality headset
approach builds upon the designer’s existing through a robust, latency-tolerant web technology
knowledge with Grasshopper but expands its scope to mxAutomation. This approach would fit the
significantly towards enabling processes that presented technology, keeping the process control
require decision making based on human or in the headset but outsourcing low-latency
machinic input - from simply waiting for user communication and intensive computing tasks to an
interaction to more complex robot behaviour external PC.
defined through the state machine. Through the Unity Machine Learning Agents
While the underlying technologies like state Toolkit (Juliani et al, 2018), Unity has exposed
machines and flow-based programming belong to machine learning processes in a very accessible way,
the foundations of computation, the accessibility of with a specific focus on applying it in a 3D
both Rhinoceros 3D and Unity sets the present environment. Besides being used to train agents for
approach apart from other solutions. gaming environments, recent research has also
Both software environments benefit from their shown its potential for training robotic agents in the
diverse, user-focused ecosystems that greatly context of architecture (Hosmer, 2019).
expand their core-functionality beyond their initially Compared to Grasshopper, where machine
intended purpose. learning plugins utilize Q-learning through the
We see this prototypical case study as a proof of Accord.Net library (Tamke, Nicholas and Zwierzycki,
concept and a first step towards making visual 2018), ML Agents provides a wider variety of
programming of complex interactive robotic reinforcement learning techniques, model-based
fabrication more accessible to experts and designers training as well as with its grid sensors ready to use
with a background in parametric design. tools for complex robotic path training in
unstructured environments.
Outlook Looking ahead, we expect the Unity ML Agents
For the future, we see two connected areas where Toolkit to be programmable from within UVS in the
further development of the presented technologies near term. In combination with the technology
will potentially lead to innovation: The immediate presented in this paper, this integration will make
interaction with machinic processes, as well as the deep reinforcement learning with industrial robots
easier integration of advanced computational accessible to non-coders.
methods and technologies, like mixed reality and We hope that the research presented in this
machine learning. paper will contribute towards enabling new,
interactive robotic processes that so far were limited

432 | eCAADe 40 – Volume 2 – Co-creating the Future

by the constraints of popular visual programming https://doi.org/10.1016/j.cemconres.2018.05.01
environments, while also making it possible to 3.
integrate innovative tools and interfaces like Celani, G., & Vaz, C. E. V. (2012). “CAD Scripting and
machine learning and mixed reality in the near Visual Programming Languages for
future. Implementing Computational Design Concepts:
A Comparison from a Pedagogical Point of
REFERENCES View.” International Journal of Architectural
Amtsberg, F., Raspall F., and Trummer A. (2015). Computing 10 (1): 121–37.
“Digital-Material Feedback in Architectural https://doi.org/10.1260/1478-0771.10.1.121
Design.” In Proceedings of the 20th Conway, M. E. (1963). “Design of a Separable
International Conference of the Association for Transition-Diagram Compiler.” Communications
Computer-Aided Architectural Design Research of the ACM 6 (7): 396–408. Available at:
in Asia (CAADRIA 2015). https://doi.org/10.1145/366663.366704
Brandt, E., & Dannenberg, R. B. (1998). “Low-Latency Davis, D., Burry, J., & Burry, M. (2011).
Music Software Using off-the-Shelf Operating “Understanding Visual Scripts: Improving
Systems.” In Proceedings of the International Collaboration through Modular Programming.”
Computer Music Conference. San Francisco. International Journal of Architectural
Braumann, J., & Brell-Cokcan, S. (2011). “Parametric Computing 9 (4). https://doi.org/10.1260/1478-
Robot Control: Integrated CAD/CAM for 0771.9.4.361
Architectural Design.” In Proceedings of the Elashry, K., & Glynn, R. (2014). “An Approach to
31st Annual Conference of the Association for Automated Construction Using Adaptive
Computer Aided Design in Architecture, 242– Programing.” In Robotic Fabrication in
51. Banff. Architecture, Art and Design 2014, edited by
Braumann, J., & Brell-Cokcan, S. (2015). “Adaptive Wes McGee and Monica Ponce de Leon, 51–66.
Robot Control - New Parametric Workflows Cham: Springer International Publishing.
Directly from Design to KUKA Robots.” In Real Available at: https://doi.org/10.1007/978-3-319-
Time - Proceedings of the 33rd ECAADe 04663-1_4 (Accessed: 27 March 2022)
Conference, 243–50. Fryling, M. (2019). “Low Code App Development.”
Braumann, J., & Singline, K. (2021). “Towards Real- Journal of Computing Sciences in Colleges 34
Time Interaction with Industrial Robots in the (6): 119.
Creative Industrie.” In 2021 IEEE International Garcia del Castillo y López, J. L. (2019). “Robot Ex
Conference on Robotics and Automation - ICRA. Machina.” In Proceedings of the 39th Annual
Brell-Cokcan, S., & Braumann J. (2015). “Toward Conference of the Association for Computer
Adaptive Robot Control Strategies.” In ACADIA Aided Design in Architecture (ACADIA).
2105: Computational Ecologies: Design in the Hosmer, T., & Tigas, P. (2019). “Deep Reinforcement
Anthropocene, 223–31. Cincinnatti. Learning for Autonomous Robotic Tensegrity
Buchli, J., M. Giftthaler, N. Kumar, M. Lussi, T. Sandy, (ART).” In Proceedings of the 39th Annual
K. Dörfler, and N. Hack. 2018. “Digital in Situ Conference of the Association for Computer
Fabrication - Challenges and Opportunities for Aided Design in Architecture (ACADIA).
Robotic in Situ Fabrication in Architecture, Jensen, M. B., & Das, A. (2020). “Technologies and
Construction, and Beyond.” Cement and Techniques for Collaborative Robotics in
Concrete Research, SI : Digital concrete 2018, Architecture - - Establishing a Framework for
112 (October): 66–75. Human-Robotic Design Exploration.” In
Proceedings of the 25th CAADRIA Conference.

Volume 2 – Co-creating the Future – eCAADe 40 | 433

Johns, R. L., & Anderson, J. (2018). “Interfaces for “RoMA: Interactive Fabrication with Augmented
Adaptive Assembly.” In Proceedings of the 38th Reality and a Robotic 3D Printer.” In
Annual Conference of the Association for Proceedings of the 2018 CHI Conference on
Computer Aided Design in Architecture Human Factors in Computing Systems, 1–12.
(ACADIA). CHI ’18. New York, NY, USA: Association for
Juliani, A., Berges, V. P., Vckay, E., Gao, Y., Henry, H., Computing Machinery.
Mattar, M., & Lange, D. (2018). “Unity: A General https://doi.org/10.1145/3173574.3174153.
Platform for Intelligent Agents.” Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T.,
ArXiv:1809.02627 [Cs, Stat]. Leibs, J., ... & Ng, A. Y. (2009). “ROS: An Open-
Kim, J., H. Takahashi, H. Miyashita, M. Annett, and T. Source Robot Operating System.” In Proc. of the
Yeh. 2017. “Machines as Co-Designers: A Fiction IEEE Intl. Conf. on Robotics and Automation
on the Future of Human-Fabrication Machine (ICRA) Workshop on Open Source Robotics.
Interaction.” In Proceedings of the 2017 CHI Kobe, Japan.
Conference Extended Abstracts on Human Schwartz, T. (2013). “HAL.” In ROB|ARCH, edited by
Factors in Computing Systems, 790–805. CHI EA Sigrid Brell-Çokcan and Johannes Braumann,
’17. New York, NY, USA: Association for 92–101. Vienna: Springer.
Computing Machinery. https://doi.org/10.1007/978-3-7091-1465-0_8
https://doi.org/10.1145/3027063.3052763. Soler. (2021). “Visose/Robots.” [Online]. Available at:
Kruse, T., Pandey, A. K., Alami, R., & Kirsch, A. (2013). https://github.com/visose/Robots (Accessed: 27
“Human-Aware Robot Navigation: A Survey.” March 2022)
Robotics and Autonomous Systems 61 (12): Soler, V., Retsin, G., & Jimenez Garcia, M. (2017). “A
1726–43. Generalized Approach to Non-Layered Fused
https://doi.org/10.1016/j.robot.2013.05.007 Filament Fabrication.” In Proceedings of the
Mealy, G. H. (1955). “A Method for Synthesizing 37th Annual Conference of the Association for
Sequential Circuits.” The Bell System Technical Computer Aided Design in Architecture
Journal 34 (5): 1045–79. (ACADIA).
https://doi.org/10.1002/j.1538- Tamke, M., Nicholas, P., & Zwierzycki, M. (2018).
7305.1955.tb03788.x “Machine Learning for Architectural Design:
Moorman, A., Liu, J., & Sabin, J. E. (2016). Practices and Infrastructure.” International
“RoboSense: Context-Dependent Robotic Journal of Architectural Computing 16 (2): 123–
Design Protocols and Tools.” In Proceedings of 43. https://doi.org/10.1177/1478077118778580
the 36th Annual Conference of the Association Zeng, A., Song, S., Yu, K. T., Donlon, E., Hogan, F. R.,
for Computer Aided Design in Architecture Bauza, M., ... & Rodriguez, A. (2018) “Robotic
(ACADIA). Pick-and-Place of Novel Objects in Clutter with
Morrison, J. P. (2010). Flow-Based Programming: A Multi-Affordance Grasping and Cross-Domain
New Approach to Application Development. Image Matching.” The International Journal of
J.P. Morrison Enterprises. Robotics Research,
Pantazis, E., & Gerber, D. (2017). “Emergent Order Zheng, H., Guo, Z., & Liang, Y. (2019). “Iterative
through Swarm Fluctuations - A Framework for Pattern Design via Decodes Python Scripts in
Exploring Self-Organizing Structures Using Grasshopper.” In Proceedings of the 18th CAAD
Swarm Robotics.” In Proceedings of the 35th Futures Conference.
ECAADe Conference.
Peng, H., J. Briggs, C.Y. Wang, K. Guo, J. Kider, S.
Mueller, P. Baudisch, and F. Guimbretière. 2018.

434 | eCAADe 40 – Volume 2 – Co-creating the Future