09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

How It All Works

Understanding TRMM
Anything you configure: scripts, tasks, patching, etc is queued and scheduled on the
server to do something. Everything that is queued, happens immediately when agents
are online. The agent gets a NATS command, the server tells it to do xyz and it does it.

When agents are not connected to the server nothing happens. The Windows Task
Scheduler says do x at some time, what it's asked to do is get x command from the
server. If the server is offline, nothing happens. If an agent comes online, every x
interval (Windows Update, pending tasks etc) check and see if there is something for
me to do that I missed while I was offline. When that time occurs (eg agent sees if it
needs to update itself at 35 minutes past every hour Update Agents ) it'll get requested
on the online agent.

That's the simplified general rule for everything TRMM.

Image Source

Still need graphics for:

1. Agent installer steps

2. Agent checks / tasks and how they work on the workstation/interact with
server

Server

https://docs.tacticalrmm.com/howitallworks/ 1/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Has a Postgres database located here:

Django Admin

Description

A web interface for the Postgres database

All Tactical RMM dependencies are listed here.

A complete list of all packages used by Tactical RMM are listed here.

Outbound Firewall Rules

If you have strict outbound firewall rules these are the outbound rules needed for all
functionality:

Regular Use

1. Outbound traffic to all agent IP scopes for reflect traffic from agents

During installation/updating

1. Access to Github for downloading and installing TRMM

2. Access to nginx.org to install

3. Access to mongodb.org to install

4. Access to python.org to install

5. Access to postgresql.org to install

6. Whatever servers Let's Encrypt uses for DNS-01 challenges

Server Without Code Signing Key

No additional rules needed.

Server With Code Signing Key

No additional rules needed.

System Services

This lists the system services used by the server.

Nginx Web Server

https://docs.tacticalrmm.com/howitallworks/ 2/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Nginx is the web server for the rmm , api , and mesh domains. All sites redirect port 80
(HTTP) to port 443 (HTTPS).

https://docs.tacticalrmm.com/howitallworks/ 3/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

nginx configuration (a.k.a. sites available)

nginx configuration docs

rmm.example.com

This serves the frontend website that you interact with.

Config: /etc/nginx/sites-enabled/frontend.conf

root: /var/www/rmm/dist

Access log: /var/log/nginx/frontend-access.log

Error log: /var/log/nginx/frontend-error.log

TLS certificate: /etc/letsencrypt/live/example.com/fullchain.pem

api.example.com

This serves the TRMM API for the frontend and agents.

Config: /etc/nginx/sites-enabled/rmm.conf

roots:

/rmm/api/tacticalrmm/static/

/rmm/api/tacticalrmm/tacticalrmm/private/

Upstreams:

unix://rmm/api/tacticalrmm/tacticalrmm.sock
unix://rmm/daphne.sock

Access log: /rmm/api/tacticalrmm/tacticalrmm/private/log/access.log

Error log: /rmm/api/tacticalrmm/tacticalrmm/private/log/error.log

TLS certificate: /etc/letsencrypt/live/example.com/fullchain.pem

mesh.example.com

This serves MeshCentral for remote access.

Config: /etc/nginx/sites-enabled/meshcentral.conf

Upstream: http://127.0.0.1:4430/

Access log: /var/log/nginx/access.log (uses default)

Error log: /var/log/nginx/error.log (uses default)

TLS certificate: /etc/letsencrypt/live/example.com/fullchain.pem

https://docs.tacticalrmm.com/howitallworks/ 4/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

default

This is the default site installed with nginx. This listens on port 80 only.

Config: /etc/nginx/sites-enabled/default

root: /var/www/rmm/dist

Access log: /var/log/nginx/access.log (uses default)

Error log: /var/log/nginx/error.log (uses default)

systemd config

status commands

Status: systemctl status --full nginx.service

Stop: systemctl stop nginx.service

Start: systemctl start nginx.service

Restart: systemctl restart nginx.service

Restart: systemctl reload nginx.service reloads the config without restarting

Test config: nginx -t

Listening process: ss -tulnp | grep nginx

standard

Service: nginx.service

Address: 0.0.0.0

Port: 443

Exec: /usr/sbin/nginx -g 'daemon on; master_process on;'

Version: 1.18.0

docker

From the docker host view container status - docker ps --filter "name=trmm-
nginx"

View logs: docker compose logs tactical-nginx

"tail" logs: docker compose logs tactical-nginx | tail

Shell access: docker exec -it trmm-nginx /bin/bash

https://docs.tacticalrmm.com/howitallworks/ 5/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Tactical RMM (Django uWSGI) Service

Built on the Django framework, the Tactical RMM service is the heart of the system by
serving the API for the frontend and agents.

uWSGI config

uWSGI docs

status commands

Status: systemctl status --full rmm.service

Stop: systemctl stop rmm.service

Start: systemctl start rmm.service

Restart: systemctl restart rmm.service

journalctl:

"tail" the logs: journalctl --identifier uwsgi --follow

View the logs: journalctl --identifier uwsgi --since "30 minutes ago" |
less

Debug logs for 5xx errors will be located in

/rmm/api/tacticalrmm/tacticalrmm/private/log

standard

Service: rmm.service

Socket: /rmm/api/tacticalrmm/tacticalrmm.sock

uWSGI config: /rmm/api/tacticalrmm/app.ini

Log: None

Journal identifier: uwsgi

Version: 2.0.18

docker

From the docker host view container status - docker ps --filter "name=trmm-
backend"

View logs: docker compose logs tactical-backend

"tail" logs: docker compose logs tactical-backend | tail

Shell access: docker exec -it trmm-backend /bin/bash

https://docs.tacticalrmm.com/howitallworks/ 6/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Daphne: Django Channels Daemon

Daphne is the official ASGI HTTP / WebSocket server maintained by the Channels
project.

Daphne config

Django Channels configuration docs

status commands

Status: systemctl status --full daphne.service

Stop: systemctl stop daphne.service

Start: systemctl start daphne.service

Restart: systemctl restart daphne.service

journalctl (this provides only system start/stop logs, not the actual logs):

"tail" the logs: journalctl --identifier daphne --follow

View the logs: journalctl --identifier daphne --since "30 minutes ago" |
less

standard

Service: daphne.service

Socket: /rmm/daphne.sock

Exec: /rmm/api/env/bin/daphne -u /rmm/daphne.sock

tacticalrmm.asgi:application

Config: /rmm/api/tacticalrmm/tacticalrmm/local_settings.py

Log: /rmm/api/tacticalrmm/tacticalrmm/private/log/trmm_debug.log

docker

From the docker host view container status - docker ps --filter "name=trmm-
websockets"

View logs: docker compose logs tactical-websockets

"tail" logs: docker compose logs tactical-websockets | tail

Shell access: docker exec -it trmm-websockets /bin/bash

NATS Server Service

https://docs.tacticalrmm.com/howitallworks/ 7/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

NATS is a messaging bus for "live" communication between the agent and server.
NATS provides the framework for the server to push commands to the agent and
receive information back.

https://docs.tacticalrmm.com/howitallworks/ 8/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

NATS config

NATS server configuration docs

status commands

Status: systemctl status --full nats.service

Stop: systemctl stop nats.service

Start: systemctl start nats.service

Restart: systemctl restart nats.service

Reload: systemctl reload nats.service reloads the config without restarting

journalctl:

"tail" the logs: journalctl --identifier nats-server --follow

View the logs: journalctl --identifier nats-server --since "30 minutes

ago" | less

Listening process: ss -tulnp | grep nats-server

Checking for NATS or websocket problems sudo journalctl --no-pager -u nats

and sudo journalctl --no-pager -u nats-api

standard

Service: nats.service

Address: 0.0.0.0

Port: 4222 (standard), 9235 (websocket)

Exec: /usr/local/bin/nats-server --config /rmm/api/tacticalrmm/nats-

rmm.conf

Config: /rmm/api/tacticalrmm/nats-rmm.conf

TLS: /etc/letsencrypt/live/example.com/fullchain.pem

Log: None

Version: v2.3.3

docker

Get into bash in your docker with: docker exec -it trmm-nats /bin/bash

Log: nats-api -log debug

Shell access: docker exec -it trmm-nats /bin/bash

https://docs.tacticalrmm.com/howitallworks/ 9/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

NATS API Service

NATS API config

status commands

Status: systemctl status --full nats-api.service

Stop: systemctl stop nats-api.service

Start: systemctl start nats-api.service

Restart: systemctl restart nats-api.service

journalctl: This application does not appear to log anything.

standard

Service: nats-api.service

Exec: /usr/local/bin/nats-api --config /rmm/api/tacticalrmm/nats-api.conf

Config: /rmm/api/tacticalrmm/nats-api.conf

TLS: /etc/letsencrypt/live/example.com/fullchain.pem

Log: None

docker

Get into bash in your docker with: docker exec -it trmm-nats /bin/bash

Log: nats-api -log debug

Celery Service

Celery is a task queue focused on real-time processing and is responsible for

scheduling tasks to be sent to agents.

Log located at /var/log/celery

https://docs.tacticalrmm.com/howitallworks/ 10/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Celery config

Celery docs

Celery configuration docs

status commands

Status: systemctl status --full celery.service

Stop: systemctl stop celery.service

Start: systemctl start celery.service

Restart: systemctl restart celery.service

journalctl: Celery executes sh causing the systemd identifier to be sh , thus mixing the
celery and celerybeat logs together.
"tail" the logs: journalctl --identifier sh --follow

View the logs: journalctl --identifier sh --since "30 minutes ago" | less

Tail logs: tail -F /var/log/celery/w*-*.log

standard

Service: celery.service

Exec: /bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES -

-pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --
loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'

Config: /etc/conf.d/celery.conf

Log: /var/log/celery/w*-*.log

docker

From the docker host view container status - docker ps --filter "name=trmm-
celery"

View logs: docker compose logs tactical-celery

"tail" logs: docker compose logs tactical-celery | tail

Shell access: docker exec -it trmm-celery /bin/bash

Celery Beat Service

Celery Beat is a scheduler. It kicks off tasks at regular intervals, that are then executed
by available worker nodes in the cluster.

https://docs.tacticalrmm.com/howitallworks/ 11/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Celery Beat config

Celery beat docs

status commands

Status: systemctl status --full celerybeat.service

Stop: systemctl stop celerybeat.service

Start: systemctl start celerybeat.service

Restart: systemctl restart celerybeat.service

journalctl: Celery executes sh causing the systemd identifier to be sh , thus mixing the
celery and celerybeat logs together.

"tail" the logs: journalctl --identifier sh --follow

View the logs: journalctl --identifier sh --since "30 minutes ago" | less

Tail logs: tail -F /var/log/celery/beat.log

standard

Service: celerybeat.service

Exec: /bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} beat --

pidfile=${CELERYBEAT_PID_FILE} --logfile=${CELERYBEAT_LOG_FILE} --
loglevel=${CELERYD_LOG_LEVEL}'
Config: /etc/redis/redis.conf

Log: /var/log/celery/beat.log

docker

From the docker host view container status - docker ps --filter "name=trmm-
celerybeat"

View logs: docker compose logs tactical-celerybeat

"tail" logs: docker compose logs tactical-celerybeat | tail

Shell access: docker exec -it trmm-celerybeat /bin/bash

Redis Service

Redis is an in-memory data structure store used as a database, cache, and message
broker for Django / Celery.

Log located at /var/log/redis

https://docs.tacticalrmm.com/howitallworks/ 12/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Redis config

Redis docs

status commands

Status: systemctl status --full redis-server.service

Stop: systemctl stop redis-server.service

Start: systemctl start redis-server.service

Restart: systemctl restart redis-server.service

Tail logs: tail -F /var/log/redis/redis-server.log

standard

Service: redis-server.service

Log: /var/log/redis/redis-server.log

docker

From the docker host view container status - docker ps --filter "name=trmm-
redis"
View logs: docker compose logs tactical-redis

"tail" logs: docker compose logs tactical-redis | tail

Shell access: docker exec -it trmm-redis /bin/bash

MeshCentral

MeshCentral is used for "Take Control" (connecting to machine for remote access),
and 2 screens of the "Remote Background" (Terminal, and File Browser).

Config file location:

/meshcentral/meshcentral-data/config.json

Customize with care.

https://docs.tacticalrmm.com/howitallworks/ 13/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Info

Mesh usernames are CaSe sEnSiTive. Tactical will make sure it's all lower case to avoid
sync problems.

MeshCentral

MeshCentral docs

status commands

Status: systemctl status --full meshcentral

Stop: systemctl stop meshcentral

Start: systemctl start meshcentral

Restart: systemctl restart meshcentral

docker

From the docker host view container status - docker ps --filter "name=trmm-
meshcentral"
View logs: docker compose logs tactical-meshcentral

"tail" logs: docker compose logs tactical-meshcentral | tail

Shell access: docker exec -it trmm-meshcentral /bin/bash

Debugging

Open either "Take Control" or "Remote Background" to get mesh login token.

Open https://mesh.example.com to open native mesh admin interface.

Left-side "My Server" > Choose "Console" > type agentstats

To view detailed logging goto "Trace" > click Tracing button and choose categories.

If you run sudo systemctl status --full --no-pager meshcentral and you don't see
Active: active (running) since ...

You can manually run meshcentral using this command to see the full output with errors.

sudo systemctl stop meshcentral

cd /meshcentral/
/usr/bin/node node_modules/meshcentral

https://docs.tacticalrmm.com/howitallworks/ 14/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

MeshCentral Agent

Get Mesh Agent Version info with this command. Should match server version.

"C:\Program Files\Mesh Agent\MeshAgent.exe" -info"

Compare the hash with the tags in the repo at

https://github.com/Ylianst/MeshAgent/tags.

Checks / tasks / agent data uses regular http to Nginx.

Agent status uses NATS websockets.

Other Dependencies

Django - Framework to enable the server to interact with browser.

Django dependencies

future==0.18.2
loguru==0.5.3
msgpack==1.0.2
packaging==20.9
psycopg2-binary==2.9.1
pycparser==2.20
pycryptodome==3.10.1
pyotp==2.6.0
pyparsing==2.4.7
pytz==2021.1

qrcode - Creating QR codes for 2FA.

qrcode dependencies

requests==2.25.1
six==1.16.0
sqlparse==0.4.1

Twilio - Python SMS notification integration.

https://docs.tacticalrmm.com/howitallworks/ 15/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

twilio dependencies

urllib3==1.26.5
uWSGI==2.0.19.1
validators==0.18.2
vine==5.0.0
websockets==9.1
zipp==3.4.1

Windows Agent
Found in %programfiles%\TacticalAgent

The Tactical RMM agent runs under the SYSTEM security context.

When scripts / checks execute, they are:

1. Transferred from the server via NATS.

2. Saved to a randomly created file in C:\ProgramData\TacticalRMM .

3. Executed.

4. Return info is captured and returned to the server via NATS.

5. File in C:\ProgramData\TacticalRMM is removed automatically after execution /

timeout.

6. Command Parameters for scripts stay in memory

Also "Send Command" stay in memory as well.

Having said that...Windows logs all things PowerShell: Event Viewer > Microsoft >
Windows > PowerShell > Operational Log so be careful with fancy API calls and
auth token using agents for execution.

Warning

Remember: Auth tokens are Username/Password/2FA verification all rolled into a single
chunk of text!

RunAsUser functionality

Now that we know the agent runs under the SYSTEM security context and what that
means, there is an option to "RunAsUser" (Windows only).

There are multiple things to understand and consider.

https://docs.tacticalrmm.com/howitallworks/ 16/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

1. TRMMs native "RunAsUser" is only supported on workstations and non-RDP/terminal

services servers.

2. The user has to be logged in, if the computer is still sitting at the Login screen there
will be no active user to discover, and fail. If you're using fast user switching, it is the
active user that will be discovered and used.

There are two ways to do RunAsUser with tactical in relation to scripting.

1. The Tactical RMM "RunAsUser" checkbox associated with the script, and all code will
be run under the actively logged in user only with their security permissions. The user
access token that will be used is the limited user access token. You will not be able to
do any admin level stuff because TRMM's RunAsUser doesn't have a UAC elevation
capability to call and request a 2nd access token with admin privileges.

2. Using the PowerShell "RunAsUser" 3rd party module

Outbound Firewall Rules

If you have strict firewall rules these are the only outbound rules from the agent
needed for all functionality:

1. All agents have to be able to connect outbound to TRMM server on the 3 domain
names on port 443.

2. The agent uses https://icanhazip.tacticalrmm.io/ to get public IP info. If this

site is down for whatever reason, the agent will fallback to https://icanhazip.com
and then https://ifconfig.co/ip

Unsigned Agents

Unsigned agents require access to:

https://github.com/amidaware/rmmagent/releases/* for downloading / updating
agents.

Signed Agents

Signed agents require access to: https://agents.tacticalrmm.com for downloading

/ updating agents.

Agent Installation Process

Copies temp files to C:\ProgramData\TacticalRMM folder.

INNO setup installs app into %ProgramFiles%\TacticalAgent\ folder.

https://docs.tacticalrmm.com/howitallworks/ 17/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Agent Update Process

Downloads latest tacticalagent-vx.x.x.exe to %PROGRAMDATA\TacticalRMM% .

Executes the file (INNO setup exe).

Log file C:\ProgramData\TacticalRMM\tacticalagent_update_vX.X.X.txt is created.

Tactical Agent Debugging

Choose your method:

Windows Automatically

If the Tactical agent is connecting to your server, you can use the Community scripts:

TacticalRMM - TRMM Agent enable Debug Mode

TacticalRMM - TRMM Agent disable Debug Mode

and TacticalRMM - Get Agent Logs

Windows Manually

Open CMD as admin on the problem computer and stop the agent services:

net stop tacticalrmm

Run the tacticalrmm service manually with debug logging:

"C:\Program Files\TacticalAgent\tacticalrmm.exe" -m rpc -log debug -logto

stdout

Note

There's a Community script that will collect your agent log called TacticalRMM - Get
Agent Logs .

Linux

As root user, edit:

vi /etc/systemd/system/tacticalagent.service

https://docs.tacticalrmm.com/howitallworks/ 18/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Change

ExecStart=/usr/local/bin/tacticalagent -m svc

to

ExecStart=/usr/local/bin/tacticalagent -m svc -log debug

then

systemctl daemon-reload
systemctl restart tacticalagent.service

Mac

In terminal window:

sudo launchtl list | grep -e mesh -e tacticalagent

Mesh Agent Recovery

Use Agents right click menu > Agent recovery > Mesh Agent

Tactical Agent Recovery

Use Agents right click menu > Agent recovery > Tactical Agent

...OR

MeshCentral is online

Connect to Terminal (Admin Shell)

Run

net stop tacticalrmm

net start tacticalrmm

Check if Tactical RMM agent is online.

From Local Machine

https://docs.tacticalrmm.com/howitallworks/ 19/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Start / Restart Tactical RMM service from either services.msc or from Admin Command
prompt:

net stop tacticalrmm

net start tacticalrmm

Open C:\Program Files\TacticalAgent\agent.log to look for issues.

Windows Update Management

The current Tactical RMM Windows Update process is relatively simple atm. As of right
now, it is in the top 3 big items to be reworked.

TLDR: Tactical RMM Based Patching Recommendation

Use the Automation Policy > Patch Policy to apply it to machines. The Other
category is poorly named by Microsoft, those are the regular monthly updates and
should be auto-approved.

Be patient, and things will be patched (based on the policy).

Trying to immediately approve patches to many machines OR block specific patches

is a slow and manual process.

Note

If you want more control of Windows patching right now, look into a script-based
implementation of PSWindowsUpdate.

Be aware: When you install the Tactical RMM Agent on a Windows computer it sets
this:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU
AUOptions (REG_DWORD):
1: Keep my computer up to date is disabled in Automatic Updates.

If you want to resume normal Windows patching and disable Tactical RMM updating
functions, you should run this.

Where does it get updates from? TRMM gets the list of Windows updates using this
Microsoft API: https://docs.microsoft.com/en-us/windows/win32/api/_wua/

The Tactical RMM server updates an agent's patch list every 8 hours based on the
patch policy to check for what to update, and what's installed.

https://docs.tacticalrmm.com/howitallworks/ 20/21
09/08/2024, 12:12 How it all Works - Tactical RMM Documentation

Note

Currently if the agent is not online at the time the patch policy is set to install, there is no
"install as soon as it comes online".

Tip

Trying to get reboots to happen at specific times after Windows update? Set your Reboot
After Installation to: Never
Then create a task that reboots at your preferred date/time

Log Files

You can find 3 sets of detailed logs at

/rmm/api/tacticalrmm/tacticalrmm/private/log .

error.log Nginx log for all errors on all TRMM URL's: rmm, api and mesh

tail -f /rmm/api/tacticalrmm/tacticalrmm/private/log/error.log

access.log Nginx log for auditing access on all URL's: rmm, api and mesh (this is a
large file, and should be cleaned periodically)

tail -f /rmm/api/tacticalrmm/tacticalrmm/private/log/access.log

django_debug.log created by Django webapp

tail -f /rmm/api/tacticalrmm/tacticalrmm/private/log/django_debug.log

https://docs.tacticalrmm.com/howitallworks/ 21/21