Intro to Zedboard System Architecture

Intro to OS (FreeRTOS, Linux)

Joint ICTP-IAEA School on Hybrid Reconfigurable Devices
for Scientific Instrumentation
Trieste, 1-12 June 2015

Fernando Rincón
[email protected]

Zedboard / OS © Grupo ARCO-UCLM, 2015 1

 Contents

● Zedboard
– I/O
– Boot Configuration
● Xilinx layered drivers architecture
● GNU/Linux for the Zynq
● FreeRTOS
● Asymmetric vs Symmetric Processing

Zedboard / OS © Grupo ARCO-UCLM, 2015 2

 Zedboard

● Stands for Zynq Evaluation and Development Board

● Features
– Zynq™-7000 SoC XC7Z020-CLG484-1
– 512 MB DDR3
– 256 Mb Quad-SPI Flash
– 4 GB SD card
– Onboard USB-JTAG Programming
– 10/100/1000 Ethernet
– USB OTG 2.0 and USB-UART
– PS & PL I/O expansion (FMC, Pmod™ Compatible, XADC)
– Multiple displays (1080p HDMI, 8-bit VGA, 128 x 32 OLED)
– I2S Audio CODEC

Zedboard / OS © Grupo ARCO-UCLM, 2015 3

 Zedboard I/O

Zedboard / OS © Grupo ARCO-UCLM, 2015 4

 Board setup

USB-JTAG
(programming, host PC)

USB-UART
(host PC comms)

USB-OTG
(peripherals)

Zedboard / OS © Grupo ARCO-UCLM, 2015 5

 Board configuration

● 4 options
– Wired
● Appropriate during development cycle
● USB-JTAG:
– default method
– Only requires micro-usb cable
● Traditional JTAG
– Trough the JTAG connector
– Requires a Xilinx Platform USB cable or Digitlent USB-TAG prog. Cable
– Autonomous
● Secure booting
● Quad-SPI flash memory
● SD card

Zedboard / OS © Grupo ARCO-UCLM, 2015 6

 Boot mode

JTAG Mode
Cascaded JTAG (default)

Independent JTAG

Boot Device
JTAG (default)
PLL Mode
PLL used (default)
Quad-SPI Flash

PLL Bypassed
SD Card

Zedboard / OS © Grupo ARCO-UCLM, 2015 7

Zynq All Programmable Platform

Zedboard / OS © Grupo ARCO-UCLM, 2015 8

 Programmers View

● Xilinx Device Drivers

– Provide maximum portability
● The device drivers are provided as ANSI C source code
– Support FPGA configurability
● Supports multiple instances of the device without code duplication for each
instance, while at the same time managing unique characteristics on a per-
instance basis
– Support simple and complex use cases
● A layered device driver architecture provides both
– Simple device drivers with minimal memory footprints
– Full-featured device drivers with larger memory footprints
– Ease of use and maintenance
● Xilinx uses coding standards and provides well-documented source code for
developers

Zedboard / OS © Grupo ARCO-UCLM, 2015 9

 Device Drivers: Layered Architecture

● The layered architecture provides seamless integration with…

– (Level 2) RTOS application layer
– (Level 1) High-level device drivers that are full-featured and portable across
operating systems and processors
– (Level 0) Low-level drivers for simple use cases

Level 2, RTOS Adaptation

Level 1, High-level Drivers

Level 0, Low-level Drivers

Zedboard / OS © Grupo ARCO-UCLM, 2015 10

 Device Drivers: Levels 0 & 1

● Low-level device drivers ● high-level device drivers

● Macros & functions to create a ● Macros & functions to utilize all
small system of the features of a device
● Characteristics: ● Characteristics:
– Small memory footprint – Abstract API
– Little to no error checking – All device features & configs.
– Primary device features only – Multiple instances of a device
– No support of device – Polled and interrupt driven I/O
configuration parameters – Non-blocking function calls to aid
– Supports multiple instances of a complex applications
device with base address input to – May have a large memory
the API
footprint
– Polled I/O only – Buffer interfaces as opposed to
– Blocking function calls byte interfaces
Zedboard / OS © Grupo ARCO-UCLM, 2015 11
 Device Drivers: Levels 0 & 1: Example

● UARTPS Level 1 ● UARTPS Level 0

– XUartPs_CfgInitialize() - – XUartPs_SendByte()- Sends one
Initializes a specific XUartPs byte using the device.
instance such that it is ready to be – XUartPs_RecvByte()- Receives a
used
byte from the device.
– XUartPs_Send() - Sends the
specified buffer using the device
in either polled or interrupt driven
mode.
– XUartPs_Recv() - Receive a
specified number of bytes of data
from the device and store it into
the specified buffer.
– XUartPs_SetBaudRate() - Sets
the baud rate for the device.

Zedboard / OS © Grupo ARCO-UCLM, 2015 12

 Operating Systems Support
● Open Source Linux
– Provided by Xilinx
– Git Repository (2014.4 update)
– Based on kernel 3.0
– Xilinx acquired Petalogix
– Xillybus – Xillinux
● Ubuntu-based with desktop
● Open Source Android
– 2.3 version
● Open Source FreeRTOS
– Light-weight real-time OS
● Some others Licensed provided by third
parties
– Nucleus RTOS (Mentor Graphics)
– uC/OS (Micrium)
Zedboard / OS © Grupo ARCO-UCLM, 2015 13
 OS Request for Zynq

Source: avnet

Zedboard / OS © Grupo ARCO-UCLM, 2015 14

 Choosing the optimal OS

● It depends on the application requirements

Zedboard / OS © Grupo ARCO-UCLM, 2015 15

 GNU/Linux

● Why GNU/Linux?
– Advantages:
● Broad use as an open source desktop, server, and embedded OS
● Feature-rich
– Symmetric multiprocessing
– Preemptive multitasking
– Shared libraries
– Device drivers
– Memory management
– IP networking
– Support for multiple file systems
– Free & Open Source

Zedboard / OS © Grupo ARCO-UCLM, 2015 16

 GNU/Linux Components
Tools

Application Utilities User Applications Root File sys-

Linux Build & Development
GNU Cross-Compiler tool

tem (rootfs)
chain & Debugger

Libraries – Graphics, Video, Networking, etc.

Tool

Kernel
Linux Kernel Device Drivers Image

U-boot
Boot
Loader
First Stage Boot Loader

Hardware QEMU
Linux Build tools: PetaLinux,
Yocto, OpenEmbedded, buildroot
Other tool chains: Linaro, GCC
mainline Board Support Package (BSP): Collection of SoC/board specific Linux
components that include Linux kernel & device drivers, boot loader and tools

Zedboard / OS © Grupo ARCO-UCLM, 2015 17

 Anatomy of a GNU/Linux System

● User applications
– Task specific
● Libraries
– C run time, networking,
graphics, for example
● Kernel
● Hardware
– CPU, memory
– Timer, interrupt controller,
I/O

Zedboard / OS © Grupo ARCO-UCLM, 2015 18

 Linux Kernel
● System call interface (SCI)
– Function calls from user space to
kernel
– Standard C library as a wrapper
● Ex: malloc()
● Device Drivers:
– 3 types
● Char:
– stream of bytes
– open(), close(), read(), write()
● Block-oriented
● Network: Packet-oriented
– char & block exported as device
nodes in the filesystem:
● /dev/ttyUSB0
Zedboard / OS © Grupo ARCO-UCLM, 2015 19
 Linux-based Design Flow

Zedboard / OS © Grupo ARCO-UCLM, 2015 20

 Linux Boot Sequence (2)

Zedboard / OS © Grupo ARCO-UCLM, 2015 21

 First Stage Boot Loader (FSBL)

● Platform/board specific initializations

● Generated by SDK
● Uses initialization settings specified in Vivado Software Export
● Packaged into boot image file BOOT.BIN
● Optionally performs PL bitstream configuration

Zedboard / OS © Grupo ARCO-UCLM, 2015 22

 U-Boot

● Open-source, second stage boot loader that boots the Linux system
– Platform/Board specific configuration required
– e.g. custom clock settings
● Powerful features:
– Load Linux kernel, applications, data blobs, root file system, device trees,
PL bitstream
– Boot from Flash, SPI Flash, SD/MMC, USB, Ethernet, I2C, etc.
– Decompression (including bitstream) and unpacking of combined images of
various types
– Has a simple shell/script environment
● Packaged into boot image file BOOT.BIN

Zedboard / OS © Grupo ARCO-UCLM, 2015 23

 U-Boot example

● Booting from the SD card:

fatload
fatload mmc
mmc 00 0x8000
0x8000 zImage;
zImage;
fatload
fatload mmc
mmc 00 0x1000000
0x1000000 devicetree.dtb;
devicetree.dtb;
fatload mmc
fatload mmc 00 0x2000000 ramdisk32M.image.gz;
0x2000000 ramdisk32M.image.gz;
go
go 0x8000
0x8000

Linux kernel begins executing from memory at address 0x8000

where it is expected to find the device tree at address 0x1000000
and the root file system at address 0x2000000

Zedboard / OS © Grupo ARCO-UCLM, 2015 24

 Boot Image file

● The boot image can contain FSBL, bitstream and application code or a
second stage bootloader
● Stored in persistent memory and loaded at boot time

Zedboard / OS © Grupo ARCO-UCLM, 2015 25

 Linux Kernel

● Booting a kernel
– Boot arguments and pass-over
– Kernel will setup the platform/architecture based on the device tree
– Setup, Initialize and Allocate memory, setup early devices (e.g. UART)
– Drivers are probed and initialized (based on device tree and scanning)
– Root file system is mounted
– Start user space ‘init’ program (PID = 1)
● Numerous kernel subsystems varying levels of hardware abstraction
– E.g. Disk Controller Drivers, Block Devices, File Systems, Virtual File
Systems

Zedboard / OS © Grupo ARCO-UCLM, 2015 26

 The Device Tree

● Data structure that describes the system hardware

● Passed to the kernel at boot time
● Created a a text file (DTS) and later compiled into a binary form (DTB)

Zedboard / OS © Grupo ARCO-UCLM, 2015 27

 The Root Filesystem

● Contains all user space components

– Required to boot Linux and run applications
– A minimal rootfs will include
● libc shared libraries
● 'init' and various boot up sequence scripts and tools
● Shell (/bash/etc) and a set of coreutils (in an embedded system busybox
provides coreutils)
● Packaging and booting rootfs
– Common methods for the Zynq All Programmable SoC: RAM disk
(initrd/initramfs), Flash/SD, NFS
– Linux can mount and use physical media as well as archives (cpio)

Zedboard / OS © Grupo ARCO-UCLM, 2015 28

 Kernel Space vs User Space

● Kernel Space
– Virtual and Physical memory
– CPU ‘Kernel/Supervisor Mode’ Application

(ARM Privileged) Application Libraries

Kernel Space User Space

– Context switching & interrupts libc Application

● User Space
Virtual memory only (kernel Kernel syscall API
–
handles the mapping and page Linux Kernel

faults)
– CPU ‘User Mode’ (ARM Hardware

Unprivileged)
– All hardware access via kernel
syscall interface

Zedboard / OS © Grupo ARCO-UCLM, 2015 29

 Writing Drivers

● Developing a kernel driver is a

challenging task
– Use of internal kernel functions
and macros
– Complex to debug
– Kernel failures lock the device
● Simpler alternatives
– Direct acces to memory registers
● /dev/mem device
– UserSpace IO (IUO)
● Generic framework for user space
drivers

Zedboard / OS © Grupo ARCO-UCLM, 2015 30

 /dev/mem Drivers

● Based on mapping physical device memory into the user space

– Access through a pointer returned from mmap() system call
● The simplest alternative
● But no IRQ handling
● Requires root or appropriate privileges
● Be careful
– Can bypass protections provided by the MMU
– Can corrupt kernel, memory or other processes

Zedboard / OS © Grupo ARCO-UCLM, 2015 31

 /dev/mem Driver Example
/*
/* poke
poke utility
utility -- for
for those
those who
who remember
remember the
the good
good old
old days!*/
days!*/
#include <stdio.h>
#include <stdio.h>
#include
#include <stdlib.h>
<stdlib.h>
#include
#include <unistd.h>
<unistd.h>
#include <sys/mman.h>
#include <sys/mman.h>
#include
#include <fcntl.h>
<fcntl.h>
int
int main(int
main(int argc,
argc, char
char *argv[])
*argv[])
Open {{
int
int fd;
fd;
/dev/mem void
void *ptr;
*ptr;
unsigned
unsigned val,
val, addr,
addr, page_addr,
page_addr, page_offset;
page_offset;
unsigned page_size=sysconf(_SC_PAGESIZE);
unsigned page_size=sysconf(_SC_PAGESIZE);
fd=open("/dev/mem",O_RDWR);
fd=open("/dev/mem",O_RDWR);
if(fd<1)
if(fd<1) {{ perror(argv[0]);
perror(argv[0]); exit(-1);}
exit(-1);}
if(argc!=3)
if(argc!=3) {{ printf(“Usage:
printf(“Usage: poke
poke <addr>
<addr> <data>”\n”);
<data>”\n”); exit(-1);}
exit(-1);}
Memory map addr=strtoul(argv[1],NULL,0);
addr=strtoul(argv[1],NULL,0);
val=strtoul(argv[2],NULL,0);
val=strtoul(argv[2],NULL,0);
page_addr=(addr
page_addr=(addr && ~(page_size-1));
~(page_size-1));
page_offset=addr-page_addr;
page_offset=addr-page_addr;
Access via
ptr=mmap(NULL,page_size,PROT_READ|PROT_WRITE,MAP_SHARED,fd,page_addr);
ptr=mmap(NULL,page_size,PROT_READ|PROT_WRITE,MAP_SHARED,fd,page_addr);
pointer if((int)ptr==-1)
if((int)ptr==-1) {{ perror(argv[0]);
perror(argv[0]); exit(-1);}
exit(-1);}
*((unsigned
*((unsigned *)(ptr+page_offset))=val;
*)(ptr+page_offset))=val;
return
return 0;
0;
}}

Zedboard / OS © Grupo ARCO-UCLM, 2015 32

 /dev/mem Pros & Cons

● Pros
– Very simple – no kernel module or code
– Good for quick prototyping / IP verification
– peek/poke utilities
– Portable (in a very basic sense)
● Cons
– No interrupt handling possible
– No protection against simultaneous access
– Need to know physical address of IP
● Hard-code?
● OK for prototyping – not recommended for production

Zedboard / OS © Grupo ARCO-UCLM, 2015 33

 UIO Drivers

● Introduced in Linux 2.6.22

– Basic interrupt handling
capabilities
● Very thin kernel-level driver
– Register UIO device
– Trivial interrupt handler
● All of the real work happens in
user space
● generates a set of directories
and attribute files in sysfs Linux
kernel memory management

Zedboard / OS © Grupo ARCO-UCLM, 2015 34

 UIO Drivers - SysFS

● Dynamic “virtual” file sytem

● Mounted on /sys
● Representation of the device model in the user space
– Device attributes represented as files
– Can read/write over the files to set parameters from the user space

/sys/class/gpio/gpio7
/sys/class/gpio/gpio7 ## echo
echo 11 >> value
value Useful for controlling
/sys/class/gpio/gpio7
/sys/class/gpio/gpio7 ## cat value
cat value Devices from shell scripts

fprintf(file_led7,
fprintf(file_led7, “%d”,
“%d”, 1);
1); The programmatic way
fscanf(file_led7, “%d”, &n_ch);
fscanf(file_led7, “%d”, &n_ch);

Zedboard / OS © Grupo ARCO-UCLM, 2015 35

 UIO Drivers – Application Level

● Opening the device

– Walk through sysfs mounted /sys/class/uio/uioX
– Check virtual file 'name'
– If it matches

fd=open("/dev/uioX",O_RDWR);
fd=open("/dev/uioX",O_RDWR);

● Memory mapping the resources

void
void *ptr=mmap(NULL,
*ptr=mmap(NULL, size,
size, PROT_READ|PROT_WRITE,
PROT_READ|PROT_WRITE,
MAP_SHARED,
MAP_SHARED, fd,
fd, nn ** PAGE_SIZE);
PAGE_SIZE);

– n is the mapping number (device specific)

● ptr may now be safely used for direct access to the hardware

Zedboard / OS © Grupo ARCO-UCLM, 2015 36

 UIO Drivers – Interrupt handling

● Several options
– Issuing a read() on the device returns number of interrupts since last read
call

read(fd,
read(fd, &num_irqs,
&num_irqs, sizeof(num_irqs));
sizeof(num_irqs));

– Can be blocking or non blocking

● O_NONBLOCK flag in open() call
– select() system call on the file descriptor
● optionally block until an IRQ occurs
– Actual handling of the interrupt is device dependent

Zedboard / OS © Grupo ARCO-UCLM, 2015 37

 UIO Drivers – Kernel Interface
● By default, even UIO requires a thin kernel-space driver
– Register and remap device address map
– Specify IRQ handler function
– Register driver with UIO subsystem
● The PetaLinux tools includes an extension to UIO
– Tag device entry in DTS file with compatible=”generic-uio”
– Enable "Userspace I/O OF driver with generic IRQ handling" kernel option
● Kconfig > Drivers > UserspaceIO
● No custom kernel code required at all!
● Userspace interface is the same
– Write 0/1 to the UIO device to disable/enable IRQ
– Read from the device to wait for IRQ
– mmap() for device register access

Zedboard / OS © Grupo ARCO-UCLM, 2015 38

 UIO Drivers - Example

/*
/* open
open the
the device
device */
*/
int fd=open(“/dev/uioX”,O_RDWR);
int fd=open(“/dev/uioX”,O_RDWR);
/*
/* memory
memory map
map it
it */
*/
volatile
volatile unsigned int
unsigned int *ptr
*ptr == \\
mmap(NULL,
mmap(NULL, 4096,
4096, PROT_READ|PROT_WRITE,
PROT_READ|PROT_WRITE, MAP_SHARED,
MAP_SHARED, fd,
fd, 0);
0);
/*
/* setup
setup the
the device
device registers
registers */
*/
*(ptr+REG_OFFSET)
*(ptr+REG_OFFSET) == ...
...
/*
/* main
main IRQ/processing
IRQ/processing loop
loop */*/
while(1)
while(1) {{
unsigned
unsigned tmp
tmp == 0x1;
0x1;
/*
/* Enable the interrupt */
Enable the interrupt */
write(fd, &tmp,sizeof(tmp));
write(fd, &tmp,sizeof(tmp));
/*
/* Wait
Wait for
for an
an IRQ
IRQ
read(fd, &tmp,
read(fd, &tmp, sizeof(tmp);
sizeof(tmp);
/*
/* Handle
Handle the
the IRQ
IRQ */
*/
tmp = *(ptr+REG_OFFSET);
tmp = *(ptr+REG_OFFSET);
...
...
*(ptr+REG_OFFSET)
*(ptr+REG_OFFSET) == ...
...
}}

Zedboard / OS © Grupo ARCO-UCLM, 2015 39

 UIO Drivers – Kernel IFace

● Things to watch for

– UIO IRQ enable/disable is at kernel IRQ line level
● Userspace code must
– Setup device for IRQ generation
– ACK the IRQ
● Cannot use shared IRQs in this scheme
– Only usable for address-mapped devices
● Cannot support FSL, APU etc

Zedboard / OS © Grupo ARCO-UCLM, 2015 40

 UIO Pros & Cons

● Pros
– Benefits of /dev/mem and mmap()
– Plus IRQ handling
– No kernel code at all
● If using OF_GENIRQ extensions
– No need to recompile and reboot kernel
● Kernel drivers can easily break the kernel and force a reboot
– UIO driver errors not usually fatal
● Open driver development to non-kernel developers
● Cons
– Interrupt model is simple but adequate
– Subject to variable or high latency
– No support for DMA to/from user space
Zedboard / OS © Grupo ARCO-UCLM, 2015 41
 Application Development

● Select the BSP from Linux

● Requires de configuration of the
appropriate toolchain
● The eclipse environment
provides also integrated
debugging

Zedboard / OS © Grupo ARCO-UCLM, 2015 42

 FreeRTOS

● Real-Time Operating System

● Widely used and known
● Open source
● Free use and distribute
● Simple and easy to port
● Feature rich but small size
● Path to commercial versions
● OpenRTOS: Full support, additional components
● SafeRTOS: Functional safety, IEC 61508 compliant
● More on: http://www.freertos.org/

Zedboard / OS © Grupo ARCO-UCLM, 2015 43

 FreeRTOS

● Pre-emptive scheduling
– Highest priority ready task
– Time slice same priority task
● Idle task
– For low priority or background
tasks
– System in low-power
● Supports
– Tasks
– Queues
– Semaphores & Mutex
– Dynamic memory allocation

Zedboard / OS © Grupo ARCO-UCLM, 2015 44

 Asymmetric Multiprocessing

● Independent Sw stacks on each processor core

● Unsupervised AMP
– One of the OS responsible for management & coordination
● Supervised
– Using ARM Trustzone architecture: some kind of light virtual machine
– OS execute independently:
● Ex: Linux + RTOS
● Following some Open Source Solutions
● Some other commercial
– Refer to
http://www.wiki.xilinx.com/Multi-OS+Support+%28AMP+%26+Hypervisor%
29

Zedboard / OS © Grupo ARCO-UCLM, 2015 45

Bare-metal / Bare-metal AMP

XAPP1079

Zedboard / OS © Grupo ARCO-UCLM, 2015 46

 Linux / Bare-metal AMP

XAPP1078 Wiki

Zedboard / OS © Grupo ARCO-UCLM, 2015 47

Bare-Metal (Cortex A9) / Bare-metal
(MicroBlaze) AMP

XAPP1093

Zedboard / OS © Grupo ARCO-UCLM, 2015 48

 Petalinux AMP

Zedboard / OS © Grupo ARCO-UCLM, 2015 49

 References

● Linux Device Drivers, 3rd ed., Corbet, Rubini and Kroah-Hartman,

O'Reilly
● Using the FreeRTOS Real time Kernel, a Practical Guide - Richard
Barry
● http://www.zedboard.org
● http://www.wiki.xilinx.com/Zynq-7000+AP+SoC+Operating+Systems
● Avnet speedway courses: www.em.avnet.com/zynqspeedways

Zedboard / OS © Grupo ARCO-UCLM, 2015 50