Domotics using a Raspberry Pi 3B for our own-built campervan.

At the moment it is just a simple prototype aiming to become a complete domotic voice-controled system.

Commands can be executed either by voice or by sending HTTP requests to a server.

• RaspVan (codename: Fiona)
  • Table of Contents
  • Requirements
  • Structure
    • Hotword
    • ASR
    • NLU
    • Respeaker
    • Raspvan
      • Relays
      • Bluetoth
  • How to
    • Installation
    • Finding the sound input device ID
    • WiFi and automatic hotspot
    • Wiring and Connections
  • Misc

Apart from any other requirement defined in the root or any of the sub-components we need the follwing:

• Raspbian Buster (installation guide)
• python >= 3.7
• Docker & Docker-compose

This repo is organized in a series of sub-components plus the main solution code under raspvan.

To understand how to train, configure, test and run each sub-component please refer to the individual readme files.

.
├── asr                     # ASR component (uses vosk-kaldi)
├── assets
├── common
├── config
├── data
├── docker-compose.yml
├── external
├── hotword                 # HotWord detection (uses Mycroft/Precise)
├── Makefile
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── respeaker
├── scripts
├── setup.cfg
└── raspvan                 # client and server systems

Most of the following components communicate through AMQP using rabbitMQ.

To run the broker backbone glueing all together:

docker-compose up -d rabbit

⚠️ TBD

To run:

source .env
source .venv/bin/activate
python -m raspvan.workers.hotword

We use the dockerized vosk-server from the jmrf/pyvosk-rpi repo.

This server listens via websocket to a sounddevice stream and performs STT on the fly.

💡 For a complete list of compatible models check: vosk/models

# Run the dockerized server
docker-compose up asr-server

Then, run one of the clients:

source .env
source .venv/bin/activate
# ASR from a audio wav file
python -m  asr.client -v 2 -f <name-of-the-16kHz-wav-file>
# Or ASR listening from the microphone
python -m  asr.client -v 2 -d <microphone-ID>

Or run the rabbitMQ-triggered raspvan ASR worker:

python -m raspvan.workers.asr

⚠️ While the rest of the components use numpy~=1.16 the NLU components requires a newer version in order to work with scikit.

The best thing if running locally is to create a separate virtual environment

See nlu/README.md

The NLU engine has two parts:

• A Spacy vectorizer + SVM classifier for intent classification
• A Conditional Random Field (CRF) for entity extraction

💡 Check the details in this Colab notebook: simple-NLU.ipynb

💡 It is advices to collect some voice samples and run them through ASR to use these as training samples for the NLU component to train it on real data.

To collect voice samples and apply ASR for the NLU, run:

# discover the audio input device to use and how many input channel are available
python -m scripts.mic_vad_record -l
# Run voice recording
python -m scripts.mic_vad_record sample.wav -d 5 -c 4

We use respeaker 4mic hat as microphone and visual feedbac with its LED array.

To run the LED pixel demo:

python -m respeaker.pixels

This is the main module which coordinates all the different components.

i2c relay demo: python -m raspvan.workers.relay

To run the bluetooth server make run-ble-server

[Unit]
Description=RaspVan BLE Server + Redis container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/bin/bash /home/pi/start_ble.sh
ExecStop=

[Install]
WantedBy=default.target

Create a virtual environment

python3.7 -m venv .venv
source .venv/bin/activate

And install all the python dependencies

pip install -r requirements.txt

First list all audio devices:

python -m respeaker.get_audio_device_index

You should get a table simlar to this:

┏━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ Index ┃ Name     ┃ Max Input Channels ┃ Max Output Channels ┃
┡━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│     0 │ upmix    │                  0 │                   8 │
│     1 │ vdownmix │                  0 │                   6 │
│     2 │ dmix     │                  0 │                   2 │
│     3 │ default  │                128 │                 128 │
└───────┴──────────┴────────────────────┴─────────────────────┘

Device with index 3, which can handle several input and output channels, is the one to pass to the hotword and ASR workers.

⚠️ ALSA won't allow for audio devices to be shared, i.e.: accessed simultaneously by more than one application when using the sound card directly. ⚠️

Solution: Use the pcm devices, i.e.: plugins. Specifically the dsnoop (to have shared input between processes) and dmix (to have several audio outputs on one card).

Copy config/.asoundrc to ~./asoundrc