TC35874 to UVC gadget application README 
by Kin Wei Lee September 2020


Orignal Work
=============
UVC original work by
Laurent Pinchart ideasonboard.com  
and patches by Bhupesh Sharma 
but conveniently collated by jdonald (github.com/jdonald/uvc-gadget)
V4L2 MMAL original work by
6by9 github.com/6by9/v4l2_mmal

Overview
=========
This software uses a Raspberry Pi Zero and a TC35874 bridge to convert an HDMI
source signal and output the video over USB so that it can be used as a USB 
webcam.


Introduction
=============

This README file documents the v4l2-mmal-uvc application available
here git://github.com/kinweilee/v4l2-mmal-uvc, which can be used to
create a UVC webcam gadget using a HDMI input as the source video.
HDMI conversion utilises a TC35874 HDMI to CSI-2 converter.

The TC35874 outputs UYVY or RGB24 data to the CSI-2 camera port and is 
incompatible with UVC gadget, which requires either  YUYV or MJPEG formatted data.

By using the MMAL framework, supported by the Raspberry Pi Zero, hardware 
conversion of UYVY/RG24 data to YUYV or MJPEG can be performed. 

Prequisites
===========
Raspberry Pi Zero or Raspberry Pi Zero 2 W (other boards should also work)
TC358743 based converter board
Raspberry Pi OS Lite (Legacy) or Raspberry Pi OS (Legacy) with desktop. 
Unfortunately this software is currently incompatible with Bullseye based 
images from Raspberry Pi.

Setup
======
1. Use raspi-config to enable 'CAMERA' and 'I2C'

2. Add to /boot/config.txt the following two lines:
dtoverlay=tc358743,i2c_pins_28_29=1
dtoverlay=dwc2

3. Add to /boot/cmdline.txt after rootwait the following commands:
modules-load=dwc2,libcomposite cmd=32M

4. TC358743 must be setup up with the following commnads
v4l2-ctl --set-edid=file=./1080P50EDID.txt --fix-edid-checksums 
v4l2-ctl --query-dv-timings
v4l2-ctl --set-dv-bt-timings query

5. Setup the ConfigFS using the uvc-setup.sh script
The UVC gadget can be used in either RAW YUYV or MJPEG mode. MJPEG will give
better frame rates (30fps) but will use compressed images.

6.1 Setup using YUYV
sudo ./uvc-setup.sh start yuyv
./v4l2-mmal-uvc -v /dev/video0 -u /dev/video1 -n3 -f0  # -v is camera device -u is g_webcam device

6.2 Setup using MJPEG
sudo ./uvc-setup.sh start mjpeg
./v4l2-mmal-uvc -v /dev/video0 -u /dev/video1 -n3 -f1

ConfigFS can be unloaded using 
sudo ./uvc-setup.sh stop

Resolution
==========
The output resolution is fixed at 1280x720, this will give about 30fps when
using MJPEG or about 8.8fps when using YUYV RAW.

NOTES
======

Audio is NOT handled.

This software basically splices (hacked and diced with a very blunt blade) the
V4L2 mmal part of v4l2-mmal by 6by9 and with the UVC part of uvc-gadget by 
Laurent Pinchart.
v4l2-mmal has video output and H264 encoding but this has been removed, as we
only need the ISP to do the RGB24/UYVY to YUYV format conversion.

V4L2 capture uses V4L2_MEMORY_MMAP, DMA is used to transfer this to VC
(Raspberry Pi video core) and UVC output uses V4L2_MEMORY_USERPTR.

Internally the code will select UYVY output from TC35874 (RGB24 can be used
too).

Resolution:
Resolution presented to UVC is fixed at 1280x720.
Only raw YUYV data is presented to UVC, MJPEG is not used.
ISOCH USB transfers work.
BULK mode transfer does not work.

Framerate:
The max framerate is about 8.8fps, (I think) it's limited to UBR microframe 
period and streaming_maxpacket size for g_webcam.
microframe period = 125us (ISOCH transfer)
image size = 1280*720*2
streaming_maxpacket = 2048
framerate = 1/(1280*720*2/2048*125e-6) = 8.8fps

ConfigFS is used to setup the UVC USB gadget. g_webcam can also work.

Pipe Setup:
Two different pipe setups are used for RAW YUYV, and compressed MJPEG formats.
RAW YUYV
V4L2 -> ISP -> UVC 
MJPEG
V4L2 -> ISP -> Encoder -> UVC

The software may or may not work for you and it may kill your hardware. My first
PiZero's USB OTG functionality died during development. Essentially it no longer
works in gadget mode. The cause could have come from power coming from two 
different sources into the Pi Zero.

Issues
=======

Stopping and restarting a stream doesn't work. It may be necessary to quit
v4l2-mmal-uvc and the webcam viewer (e.g. QuickTime player) before a new stream
can be viewed.

Compiling reveals alot of warnings.

Test Setup
===========
Hardware
-H2C-RPI0-B01 HDMI to CSI-2 Bridge using TC358743 (same foot print as Pi Zero)
-Pi Zero W (4.19.122+ recompiled from source using rpi-4.19.y branch)
-Pi Zero 2 W using stock Raspberry Pi OS Lite (Legacy)

Input HDMI sources:
-PC 1280x720 and 1920x1080 ok
-EOS M - 720x480 interlaced signal needs code modification to use interlace mode
(V4L2_FIELD_INTERLACED) and the source must be RGB24. (use Magic Lantern for clean HDMI!)
-EOS M5 - 1920x1080 signal ok 

Software on MacBook Pro (Retina, 13-inch, Late 2013)
-Facetime (Cataline 10.15.5) working (!)
-QuickTime Player working
-VLC (macOS 3.0.11.1) - selecting webcam gadget fails on first try and reverts
to the internal webcam, on second select webcam gadget -> facetime camera -> 
webcam gadget to get it working. VLC however has a large latency.

Windows 10
- Skype working use UVC Camera as input and MJPEG mode, YUYV didn't work
- VLC doesn't work error 'Your input couldn't be opened'

On Pi Zero (first version) CPU usage is typically 45-50% using htop.
On Pi Zero 2 W CPU usage is less than 15% on any core.

Things to work on
==================
Frame rate could be improved but streaming_maxpacket seems to be limited to 
2048.  USB_Video_Class 1.1 states that 3072 bytes should work however setting 
3072 causes a USB disconnect error. It could be a dwc2 issue or a webcam.c 
issue.
Bulk mode cannot be used since f_uvc.c does not implement any bulk modes.
uvc-setup.sh needs a clean up