Orpheus Architecture¶
This document describes the high-level architecture of the Orpheus wildlife monitoring and cross-species communication platform.
Overview¶
Orpheus is a Python monorepo designed for real-time wildlife monitoring on edge computing hardware (NVIDIA Jetson Orin NX). The platform follows a modular, service-oriented architecture that enables:
- Real-time audio/video processing with low-latency ML inference
- Distributed agent communication via a messaging backplane (NATS + JetStream by default; mosquitto/MQTT as the fallback — see ADR 0017)
- Edge deployment on resource-constrained hardware
- Cross-platform development (macOS for development, ARM Linux for production)
System Architecture¶
graph TB
subgraph Hardware["🔌 Hardware Layer"]
MIC[🎤 USB Microphone<br/>Behringer UMC404HD]
CAM[📷 Camera<br/>Amcrest/CSI]
end
subgraph Agents["🤖 Detection Agents"]
AUDIO[Audio Motion Agent<br/>orpheus-agent-audio-motion]
VIDEO[Video Motion Agent<br/>orpheus-agent-video-motion]
BIRD[Bird Detection Agent<br/>orpheus-agent-bird-detection]
CROW[Crow Detection Agent<br/>orpheus-agent-crow-detection]
EVENTS[Audio Events Agent<br/>orpheus-agent-audio-events]
PLAYBACK[Audio Playback Agent<br/>orpheus-agent-audio-playback]
CORR[Event Correlator<br/>orpheus-agent-event-correlator]
end
subgraph VideoCapture["📹 Video Capture"]
SNAP[Video Snapshotter<br/>orpheus-agent-video-snapshotter]
TIMELAPSE[Video Timelapser<br/>orpheus-agent-video-timelapser]
end
subgraph Services["⚙️ Core Services"]
BUS[Messaging Backplane<br/>orpheus-backplane — NATS :4222 default]
DASH[UI orpheus_ui<br/>FastAPI :8082]
end
subgraph Platform["📦 Platform Library"]
COMMON[orpheus-common<br/>Config • Event Bus • Storage • Logging]
end
subgraph Storage["💾 Storage"]
DATA[/data/orpheus/<br/>Audio clips, Logs, Events/]
end
MIC --> AUDIO
CAM --> VIDEO
CAM --> SNAP
SNAP -.->|JPEG files on disk| TIMELAPSE
AUDIO --> BUS
BUS -->|orpheus/detection/audio/events| BIRD
BUS -->|orpheus/detection/audio/events| EVENTS
BUS -->|orpheus/detection/bird/events| CROW
BUS -->|orpheus/detection/audio/events| CROW
VIDEO --> BUS
BIRD --> BUS
CROW --> BUS
EVENTS --> BUS
BUS --> DASH
BUS --> PLAYBACK
AUDIO --> DATA
VIDEO --> DATA
BIRD --> DATA
CROW --> DATA
SNAP --> DATA
TIMELAPSE --> DATA
COMMON --> AUDIO
COMMON --> VIDEO
COMMON --> BIRD
COMMON --> CROW
COMMON --> PLAYBACK
COMMON --> UI
style Hardware fill:#e1f5fe
style Agents fill:#fff3e0
style Services fill:#e8f5e9
style Platform fill:#f3e5f5
style Storage fill:#fce4ec
Solid edges are event-bus subscriptions — no agent calls another directly. The dashed edge is a filesystem handoff: the snapshotter writes JPEGs and the timelapser reads them, with no bus involvement (ADR 0002).
Data Flow¶
sequenceDiagram
participant M as 🎤 Microphone
participant A as Audio Agent
participant Q as Backplane Broker
participant D as Dashboard
participant S as Storage
M->>A: Audio Stream (48kHz)
loop Every frame (audio.buffer_duration_ms, 21ms default)
A->>A: Analyze audio level
alt Motion Detected
A->>S: Save audio clip (.flac)
A->>Q: Publish detection event
Q->>D: Forward event
D->>D: Update UI
end
end
A->>Q: Publish health status (30s heartbeat default)
Q->>D: Forward status
Component Architecture¶
Monorepo Structure¶
graph LR
subgraph Root["📁 orpheus/"]
subgraph P["platform/"]
COMMON["orpheus-common<br/>━━━━━━━━━━━━<br/>config.py<br/>mqtt.py<br/>logging.py<br/>storage/"]
end
subgraph S["services/"]
BUS["orpheus-backplane<br/>━━━━━━━━━━━<br/>NATS broker (default)"]
UI["orpheus_ui<br/>━━━━━━━━━━━━━<br/>FastAPI + React"]
end
subgraph A["agents/"]
AUDIO["orpheus-agent-<br/>audio-motion<br/>━━━━━━━━━━━<br/>Detection agent"]
end
end
COMMON --> DASH
COMMON --> AUDIO
AUDIO --> BUS
UI --> BUS
style P fill:#f3e5f5
style S fill:#e8f5e9
style A fill:#fff3e0
Platform Constraints¶
| Constraint | Requirement |
|---|---|
| Target Hardware | NVIDIA Jetson Orin NX (ARM) |
| Python Version | 3.9.5 (locked for Jetson compatibility) |
| Development Platforms | macOS (Apple Silicon), Ubuntu |
| ML Framework | PyTorch + CUDA (crow-detection, audio-events); ONNX Runtime on CPU (bird-detection); TFLite for BirdNET's geo-filter |
Audio Processing Pipeline¶
graph LR
subgraph Input["🎤 Audio Input"]
ALSA[ALSA Source<br/>USB Audio]
DEFAULT[Default Input<br/>System Mic]
SYNTH[Synthetic<br/>Test Signal]
end
subgraph Processing["⚡ Processing"]
PROC[Channel Processor]
DET[Detector Algorithm<br/>• Fixed Threshold<br/>• Adaptive Threshold]
BUF[Pre-buffer<br/>Ring Buffer]
end
subgraph Output["📤 Output"]
CLIP[Clip Saver<br/>.flac files]
PUB[Event Bus Publisher<br/>Events & Status]
end
ALSA --> PROC
DEFAULT --> PROC
SYNTH --> PROC
PROC --> DET
DET --> BUF
BUF --> CLIP
DET --> PUB
style Input fill:#e3f2fd
style Processing fill:#fff8e1
style Output fill:#e8f5e9
Dashboard Architecture¶
The dashboard is services/orpheus_ui: a React + TypeScript frontend built with
Vite, served as static files by a FastAPI backend that also exposes the API. The
backend reads the detection database directly and subscribes to the event bus for
live health and detection updates.
graph TB
subgraph Browser["Browser"]
UI[React + TypeScript UI]
end
subgraph Backend["FastAPI backend (:8082)"]
API[REST API<br/>/api/*]
AUTH[Auth<br/>JWT sessions]
STATIC[Built frontend<br/>static files]
end
subgraph Data["Data sources"]
DB[(DetectionDB<br/>SQLite)]
CONF[OrpheusConfig]
STORE[Clips and snapshots<br/>under the data root]
BUS[("Event bus<br/>NATS + JetStream")]
end
UI <--> API
UI --> STATIC
API --> AUTH
API --> DB
API --> CONF
API --> STORE
BUS -->|health, detections| API
style Browser fill:#e3f2fd
style Backend fill:#e8f5e9
style Data fill:#fff3e0
The backend can be pointed at a read-only replica instead of the live database
(ui.read_from_replica), so history browsing reads a snapshot rather than the
database the agents are writing to.
BUS Topic Structure¶
The orpheus/... hierarchy below is the wire contract on either backend: NATS
(default) mirrors it as subjects; the mosquitto fallback uses it as literal BUS
topics.
graph TD
ROOT[orpheus/]
ROOT --> AUDIO[audio/]
ROOT --> DETECTION[detection/]
ROOT --> ENTITIES[entities/]
ROOT --> SYSTEM[system/]
ROOT --> VIDEO[video/]
AUDIO --> A_EVENTS[motion/events<br/>Audio motion events]
AUDIO --> A_STATUS[motion/status<br/>Channel status]
AUDIO --> A_PLAYBACK[playback/request<br/>Playback requests]
DETECTION --> D_BIRD[bird/events<br/>BirdNET detections]
DETECTION --> D_CROW[crow/events<br/>Crow detections]
DETECTION --> D_AUDIO[audio/events<br/>PANNs sound-event detections]
ENTITIES --> E_ANIMAL[animal<br/>Correlated entity events]
SYSTEM --> S_HEALTH[*/health<br/>Agent health]
SYSTEM --> S_CONFIG[config_changed<br/>Config changes]
SYSTEM --> S_SAFETY[safety<br/>Circuit-breaker state]
DETECTION --> D_REPLAY[replay<br/>Replayed detections]
STATE[state/] --> S_LOC[location<br/>GPS fix, retained]
ENV[environment/] --> WX[weather<br/>Ecowitt readings — stubbed]
ACT[actuation/] --> A_PB[audio/playback<br/>Playback windows]
EU[entity-updates/] --> EU_A[animal<br/>Late-arrival enrichment]
VIDEO --> V_EVENTS[motion/events<br/>Motion events]
VIDEO --> V_STATUS[motion/status<br/>Camera status]
style ROOT fill:#1565c0,color:#fff
style AUDIO fill:#2196f3,color:#fff
style DETECTION fill:#9c27b0,color:#fff
style ENTITIES fill:#c2185b,color:#fff
style SYSTEM fill:#4caf50,color:#fff
style VIDEO fill:#ff9800,color:#fff
Configuration Flow¶
flowchart LR
subgraph Sources["Configuration Sources"]
YAML["config/orpheus.yaml<br/>(defaults)"]
ENV["Environment Variables<br/>ORPHEUS_*"]
DOTENV[".env file<br/>(development)"]
PROD["/opt/orpheus/config/<br/>(production)"]
end
subgraph Singleton["OrpheusConfig"]
LOAD[Load & Merge]
VALIDATE[Pydantic Validation]
CACHE[Singleton Instance]
end
subgraph Consumers["Config Consumers"]
AGENT[Audio Agent]
DASHBOARD[Dashboard]
STORAGE[Storage Manager]
end
YAML --> LOAD
ENV --> LOAD
DOTENV --> LOAD
PROD --> LOAD
LOAD --> VALIDATE
VALIDATE --> CACHE
CACHE --> AGENT
CACHE --> DASHBOARD
CACHE --> STORAGE
style Sources fill:#fff3e0
style Singleton fill:#e8f5e9
style Consumers fill:#e3f2fd
Deployment Architecture¶
graph TB
subgraph Jetson["🖥️ Jetson Orin NX"]
subgraph Systemd["systemd Services"]
S1[orpheus-backplane.service]
S2[orpheus-ui.service]
S3[orpheus-agent-audio-motion.service]
end
subgraph Hardware["Hardware"]
USB[USB Audio Interface]
NET[Network Interface]
end
end
subgraph Network["📡 Network"]
LAN[Local Network]
BROWSER[Web Browser<br/>Dashboard UI]
end
USB --> S3
S3 --> S1
S1 --> S2
S2 --> NET
NET --> LAN
LAN --> BROWSER
style Jetson fill:#e8f5e9
style Network fill:#e3f2fd
Directory Structure¶
orpheus/
├── platform/
│ └── orpheus-common/ # Shared platform library
│ ├── src/orpheus_common/
│ │ ├── config.py # Configuration management
│ │ ├── event_bus.py # EventBus ABC + factory
│ │ ├── event_bus_nats.py # NATS + JetStream backend
│ │ ├── mqtt.py # mosquitto fallback backend
│ │ ├── actor/ # Actor base every agent subclasses
│ │ ├── detection/ # Detection, Entity, DetectionDB, taxonomy
│ │ ├── logging.py # Structured logging
│ │ ├── storage/ # File storage utilities
│ │ ├── hardware/ # Hardware abstraction
│ │ └── diagnostics/ # Health monitoring
│ └── tests/
│
├── services/
│ ├── orpheus-backplane/ # Messaging backplane (NATS default; mosquitto fallback)
│ │ ├── config/
│ │ ├── scripts/
│ │ └── systemd/
│ │
│ ├── orpheus-gps/ # GPS time + location service
│ ├── orpheus-bluetooth-autoconnect/ # Audio-out routing
│ ├── orpheus-dashboard/ # RETIRED — superseded by orpheus_ui; not deployed
│ └── orpheus_ui/ # Web-based UI
│ ├── backend/ # FastAPI backend (port 8082)
│ ├── frontend/ # React + Vite frontend (port 5173 dev)
│ └── systemd/
│
├── agents/
│ ├── orpheus-agent-audio-motion/ # Audio motion detection agent
│ │ ├── src/orpheus_agent_audio_motion/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ ├── audio_source.py # Audio capture backends
│ │ │ ├── detector_algorithm.py # Detection algorithms
│ │ │ └── channel_processor.py # Per-channel processing
│ │ └── tests/
│ │
│ ├── orpheus-agent-video-motion/ # Video motion detection agent
│ │ ├── src/orpheus_agent_video_motion/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ ├── video_source.py # RTSP stream capture
│ │ │ └── motion_detector.py # Motion detection
│ │ └── tests/
│ │
│ ├── orpheus-agent-bird-detection/ # BirdNET species identification
│ │ ├── src/orpheus_agent_bird_detection/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ └── birdnet_model.py # BirdNET ONNX inference
│ │ └── tests/
│ │
│ ├── orpheus-agent-crow-detection/ # Crow vocalization analysis
│ │ ├── src/orpheus_agent_crow_detection/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ ├── embedder.py # AVES embedder (16kHz)
│ │ │ └── classifier.py # Multi-task classifier
│ │ └── tests/
│ │
│ ├── orpheus-agent-audio-events/ # General AudioSet sound classification
│ │ ├── src/orpheus_agent_audio_events/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ ├── model.py # PANNs sound-event detection
│ │ │ ├── post_processing.py # Frames → temporal intervals
│ │ │ └── audioset_ontology.py # AudioSet 527-class ontology
│ │ └── tests/
│ │
│ ├── orpheus-agent-audio-playback/ # Audio output agent
│ │ ├── src/orpheus_agent_audio_playback/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ └── playback.py # Audio playback manager
│ │ └── tests/
│ │
│ ├── orpheus-agent-video-snapshotter/ # Periodic camera snapshots
│ │ ├── src/orpheus_agent_video_snapshotter/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ └── config.py # Configuration loading
│ │ └── tests/
│ │
│ ├── orpheus-agent-video-timelapser/ # Timelapse generation
│ │ ├── src/orpheus_agent_video_timelapser/
│ │ │ ├── main.py # Agent entrypoint
│ │ │ └── config.py # Configuration loading
│ │ └── tests/
│ │
│ └── orpheus-agent-event-correlator/ # Fuses detections into entities
│ ├── src/orpheus_agent_event_correlator/
│ │ ├── main.py # Agent entrypoint
│ │ └── cluster_manager.py # Same-source grouping
│ └── tests/
│
├── config/ # orpheus.example.yaml — every knob, with comments
├── deploy/ # host-level drop-ins (journald log bounds)
├── docker/ # Dockerfiles + the Simulacrum's sim-source
├── scripts/ # dev-stack and other developer scripts
├── hardware/ # Hardware-specific configurations
├── artifacts/ # ML models, recordings (Git LFS)
├── tools/ # Development utilities
└── docs/ # Documentation
Configuration¶
Configuration is managed through a layered system:
- Base Config:
config/orpheus.yaml(defaults) - Environment Override:
ORPHEUS_*environment variables - Local Override:
.envfiles (development) - Instance Config:
/opt/orpheus/config/orpheus.yaml(production)
# Example orpheus.yaml structure
audio:
channels:
- id: mic_1
device: alsa://orpheus_umc?channel=1
enabled: true
detection:
algorithm: adaptive_threshold
threshold_db: -40.0
event_bus:
backend: "nats" # nats (default) | mqtt
nats_url: "nats://127.0.0.1:4222"
audio:
channels:
- id: 1
enabled: true
storage:
base_path: /data/orpheus
retention:
sweep_enabled: true
Agents¶
The Orpheus platform uses a layered agent architecture where Layer 1 agents detect motion/activity, and Layer 2 agents perform specialized analysis.
Layer 1: Motion Detection¶
Audio Motion Detection (orpheus-agent-audio-motion)¶
- Purpose: Detect audio activity above threshold across 4 microphone channels
- Input: Raw audio from Behringer UMC404HD (48kHz, 4 channels)
- Output: Audio clips + motion events via
orpheus/audio/motion/events - Algorithm: Fixed or adaptive threshold detection with pre-buffering
- Storage: FLAC audio clips in
/data/orpheus/audio/audio_motion/{channel}/
Video Motion Detection (orpheus-agent-video-motion)¶
- Purpose: Detect visual motion in camera feeds
- Input: RTSP streams from 4 Amcrest IP cameras
- Output: Motion events + video clips via
orpheus/video/motion/events - Algorithm: Frame differencing with configurable sensitivity
- Storage: MP4 video clips in
/data/orpheus/video/video_motion/{camera}/
Layer 2: Specialized Analysis¶
Bird Detection (orpheus-agent-bird-detection)¶
- Purpose: Identify bird species from audio clips
- Model: BirdNET ONNX (species classification)
- Input: Audio motion events from
orpheus/audio/motion/events - Output: Species detections via
orpheus/detection/bird/events - Data Format:
{
"event_id": "bird_det_...",
"timestamp": "2025-12-05T22:35:58Z",
"channel_id": "1",
"detections": [
{
"species_code": "amecro",
"species_common": "American Crow",
"confidence": 0.92,
"start_time": 1.2,
"end_time": 3.5
}
],
"audio_clip_path": "/data/orpheus/audio/..."
}
Crow Detection (orpheus-agent-crow-detection)¶
- Purpose: Detailed crow vocalization analysis (species, call type, quality)
- Models:
- AVES embedder (aves-base-bio.pt): 16kHz audio → 768-dim embeddings
- Multi-task classifier (mt_70.pt): species, call type, quality prediction
- Input: corvid signals from either upstream classifier —
orpheus/detection/bird/events(BirdNET naming a corvid) andorpheus/detection/audio/events(an AudioSet "Crow"/"Caw" tag). It does not subscribe to raw audio motion: the expensive AVES pass runs only on clips another model already flagged, deduped by clip path within a 30-second window so a second flag on the same clip does not start a second pass. - Processing: Resample 48kHz → 16kHz, extract embeddings, classify
- Output: Crow detections via
orpheus/detection/crow/events - Data Format:
{
"event_id": "crow_det_...",
"timestamp": "2025-12-05T22:35:58Z",
"channel_id": "1",
"detection": {
"species": "american_crow",
"call_type": "caw",
"quality_score": 0.87
},
"audio_clip_path": "/data/orpheus/audio/...",
"inference_time_ms": 145
}
- Storage: Detections stored in DetectionDB (SQLite) at
/data/orpheus/detections/
Audio Events (orpheus-agent-audio-events)¶
- Purpose: Tag everything else in the clip — dog barks, vehicles, voices, rain — and say when in the clip each sound occurred
- Model: PANNs
Cnn14_DecisionLevelMaxover the AudioSet 527-class ontology, with native frame-level (10 ms) outputs - Input: Audio motion events from
orpheus/audio/motion/events - Processing: Resample to 32 kHz mono, run sound-event detection, post-process
frames into intervals, emit one
Detectionper surviving label - Output: Sound-event detections via
orpheus/detection/audio/events - Data Format: one
Detection(detection_type="audio.classified")per label, carrying anaudiosettaxonomy reference (/m/...machine ID) per ADR 0011, aspecies_codeofaudioset_<machine_id>, andintervalsofTemporalInterval(start_seconds, end_seconds, confidence)
This is the third model in the audio chain: bird-detection and crow-detection answer "which bird, and what was it doing"; audio-events answers "what else was there". The correlator reconciles all three into one entity.
Layer 2.5: Fusion¶
Event Correlator (orpheus-agent-event-correlator)¶
The agent that turns three independent detection streams into one row per animal. Nothing else produces entities.
- Purpose: group observations that came from the same source — overlapping in-clip intervals, labels that denote the same thing — into a single entity carrying every classifier's evidence.
- Input: the detection subjects named in
correlation.input_topics; it processesspecies.detected,crow.analyzedandaudio.classified. - Output:
orpheus/entities/animal, plusorpheus/entity-updates/animalfor late-arriving evidence. Withpublish_entity_type_topicson, entities also route by type (orpheus/entities/animal/bird/crow). - See: ADR 0013 for what merge keys on, and ADR 0016 for the type taxonomy.
Layer 3: Output¶
Audio Playback (orpheus-agent-audio-playback)¶
- Purpose: Play audio responses via system speakers
- Input: Playback requests from
orpheus/audio/playback/request - Output: Audio via default ALSA output device
- Features: Sound registry, repeat counts, pause between repeats
- Use Cases: Wildlife callbacks, alert sounds, test signals
Video Capture & Processing¶
These agents handle video data capture and processing, independent of the motion detection layer.
Video Snapshotter (orpheus-agent-video-snapshotter)¶
- Purpose: Capture periodic still images from IP cameras
- Input: RTSP streams from configured cameras
- Output: JPEG files in
/data/orpheus/video/snapshots/{YYYY.MM.DD}/ - Configuration: Per-camera intervals (e.g.,
5m,10m) - Design: On-demand RTSP connections minimize resource usage
- See: ADR 0002: Video Snapshot Architecture
Video Timelapser (orpheus-agent-video-timelapser)¶
- Purpose: Generate timelapse videos from snapshots
- Input: JPEG snapshots from snapshotter
- Output: H.264 MP4 videos in
/data/orpheus/video/timelapses/{YYYY.MM.DD}/ - Tiers: Multiple lookback windows (24h, 12h, 6h, 1h, 30m, 10m)
- Encoding: mp4v + ffmpeg transcode for Jetson compatibility
- Filename Format:
{camera}.{label}.{tier}.{lookback}.{timestamp}.mp4 - See: ADR 0003: Timelapse Generation Architecture
- See: ADR 0004: Jetson Video Codec Strategy
Development Workflow¶
graph LR
subgraph Dev["💻 Development (Mac)"]
CODE[Write Code]
TEST[make test]
LINT[make lint]
FMT[make format]
end
subgraph CI["🔄 CI/CD"]
PR[Pull Request]
GHA[GitHub Actions]
COV[Coverage Check]
end
subgraph Prod["🚀 Production (Jetson)"]
DEPLOY[make services-install]
START[make services-start]
LOGS[make service-logs]
end
CODE --> TEST
TEST --> LINT
LINT --> FMT
FMT --> PR
PR --> GHA
GHA --> COV
COV --> DEPLOY
DEPLOY --> START
START --> LOGS
style Dev fill:#e3f2fd
style CI fill:#fff3e0
style Prod fill:#e8f5e9
Testing Strategy¶
- Unit Tests: Per-component with pytest
- Coverage Target: 70% for most components, with
orpheus-commonat 78% andaudio-motionat 72%.codecov.ymlis the source of truth; the CI thresholds live in theenv:block of.github/workflows/pr-tests.yml. - CI/CD: GitHub Actions on push/PR
- Platform Tests: Separate workflows for ARM validation
Future Architecture¶
graph TB
subgraph Current["✅ Current"]
AUDIO_NOW[Audio Detection]
BIRDNET[BirdNET Integration<br/>Species ID]
DASH_NOW[Dashboard]
MQTT_NOW[Messaging Backplane<br/>NATS + JetStream]
end
subgraph Planned["🔮 Planned"]
YOLO[YOLOv8 Video<br/>Object Detection]
ACTIVE[Active Inference<br/>Playback Response]
MULTI[Multi-Station<br/>groundwork shipped behind flags — ADR 0018;<br/>install profiles pending]
SPATIAL[Spatial Web<br/>GIS Integration]
end
AUDIO_NOW --> BIRDNET
DASH_NOW --> SPATIAL
MQTT_NOW --> MULTI
BIRDNET --> ACTIVE
YOLO --> ACTIVE
style Current fill:#e8f5e9
style Planned fill:#fff3e0
Contributing¶
When contributing, ensure:
- Changes work on Python 3.9.5 (Jetson constraint)
- No dependencies incompatible with ARM architecture
- Tests pass with
make test - Code is linted with
ruff - Documentation is updated
See CONTRIBUTING.md for detailed guidelines.