Longzhao Gong

M.S. Computer Engineering student working on embedded AI, edge intelligence, and autonomous systems.

Robotics software, multimodal perception, and edge AI deployment for a small autonomous racing platform

This project is an end-to-end autonomous racing system built around DonkeyCar, DonkeySim, Jetson Nano, and a Waveshare JetRacer-style vehicle. The goal was not just to train a reinforcement learning policy in simulation, but to carry that policy through the real deployment chain: sensors, model export, embedded inference, control adaptation, runtime logging, and safety validation.

I treated the project as a sim-to-real systems engineering problem. The final engineering stack connects camera input, 360-degree LiDAR, RP2040 telemetry, a multimodal recurrent PPO actor, action adaptation, Jetson-side monitoring, TensorRT inference, and shadow-mode validation.

Project type
Robotics software / Edge AI / Autonomous systems

Platform
Jetson Nano + DonkeyCar / Waveshare JetRacer

Core stack
Python, DonkeyCar, Stable-Baselines3, PyTorch, ONNX, TensorRT, ROS LiDAR

Status
V17 endpoint deployment chain completed and frozen in shadow mode. Active obstacle-avoidance performance remains a separate model-quality problem.

Model RecurrentPPO multimodal actor 6-channel semantic image, state vector, 72 LiDAR ranges + 72 validity sectors, and LSTM state

Deployment ONNX → TensorRT FP16 Actor-only export, TensorRT engine, CUDA runtime wrapper, and Jetson shadow backend

Runtime Vehicle-loop p95: 696.2 ms → 130.8 ms LiDAR sectorization, async logging, and telemetry cache moved blocking work out of the loop

Validation 20-minute TensorRT shadow run 1199.55 s, exit 0, no inference timeout, no LiDAR missing, no RP2040 missing

Endpoint Deployment Evidence

Area Result
TensorRT actor residual p50 25.856 ms → 12.479 ms
TensorRT actor residual p95 40.988 ms → 17.046 ms
V17 latency p50 after runtime optimization 168.570 ms → 87.153 ms
V17 latency p95 after runtime optimization 253.958 ms → 124.753 ms
Effective FPS mean after runtime optimization 4.270 → 10.641
Vehicle-loop p95 after runtime optimization 696.175 ms → 130.800 ms
DataCollector p99 after async logging + telemetry cache 537.42 ms → 5.67 ms
Final TensorRT shadow validation 1199.55 s, exit 0

The supported claim is that the Jetson endpoint deployment chain is observable, reproducible, safety-gated, and validated in shadow mode.

Engineering Problem

A simulator-trained racing policy cannot be dropped directly onto a physical Jetson-powered car. The failure modes are coupled:

  • simulation and real vehicle dynamics do not match;
  • camera, LiDAR, and telemetry streams arrive at different rates;
  • Jetson Nano has tight compute, memory, and runtime constraints;
  • DonkeyCar, Gym, Stable-Baselines3, TensorFlow/Keras, PyTorch, and Jetson Python versions create deployment friction;
  • logging and telemetry can block the vehicle loop if implemented naively;
  • a policy that looks stable in simulation can still be unsafe or unusable on real hardware.

The project therefore evolved from a reinforcement learning experiment into a full deployment-oriented autonomy stack.

System Architecture

DonkeyCar / DonkeySim
multi-scene PPO training</div>

Policy export
Keras, PyTorch weights, ONNX</div>

Jetson runtime
camera, LiDAR, RP2040</div>

V17 actor backend
PyTorch / TensorRT FP16</div>

Action + safety layer
adapter, gates, shadow logs</div>

The final V17 runtime path is:

CSI camera + 360° LiDAR /scan + RP2040 telemetry
        ↓
Semantic image preprocessing + LiDAR sectorization + state vector
        ↓
V17 RecurrentPPO actor backend
        ↓
Action adapter + deployment safety gate
        ↓
Shadow-mode logging or real actuator path
        ↓
CSV, summary JSON, telemetry, and reproducible reports

The deployed model is not a simple image-only CNN. It uses a dictionary observation with semantic image channels, ego-state features, LiDAR range/validity sectors, timing metadata, and persistent LSTM state.

My Engineering Contribution

Training Infrastructure

Built PPO training loops, parallel simulation workflows, curriculum logic, reward shaping, checkpointing, monitoring, and multi-scene sampling.

Deployment Interface

Created practical model bridges from PPO to Jetson, including PPO-to-Keras distillation, direct PyTorch policy-weight inference, and actor-only ONNX/TensorRT deployment.

Multimodal Perception

Integrated camera features, semantic image channels, LiDAR sectors, timing metadata, and vehicle telemetry into policy observations.

Sim-to-Real Alignment

Measured simulator-vs-real differences in steering, throttle, control frequency, LiDAR semantics, and vehicle response.

Runtime Optimization

Profiled the Jetson chain and moved bottlenecks out of the vehicle loop: LiDAR sectorization, async logging, and telemetry caching.

Safety and Validation

Added engine/metadata preflight checks, LiDAR/RP2040 gates, inference timeout counters, safety summaries, and shadow-mode validation.

K–12 Robotics Mentorship

This DonkeyCar work also became the technical basis for a K–12 robotics mentorship project at IntelliSys Lab. I guided high school students through a beginner-friendly DonkeyCar learning path: setting up the vehicle, understanding the camera-to-control loop, collecting driving data, training baseline models, and connecting small autonomous-car experiments to broader robotics and AI concepts.

The public project page is hosted by IntelliSys Lab K–12 Projects, and the teaching repository is available as a DonkeyCar beginner guide. I keep this section separate from the main RL deployment timeline because its value is different: it demonstrates technical mentoring, documentation, and translating an engineering system into accessible learning material.

Project Timeline

This timeline compresses the raw weekly engineering logs into a public engineering story. It intentionally uses large implementation milestones, not weekly status updates.

Jul 2025 Parallel DonkeySim training windows

1. Stable PPO Training Foundation

Built the first usable reinforcement learning baseline and stabilized the simulator training loop.

  • Fixed deadlocks, NumPy crashes, reconnect failures, and simulator recovery issues.
  • Added monitoring for reward, speed, deviation, collisions, and efficiency.
  • Implemented early safeguards against catastrophic forgetting and unstable CTE behavior.
  • Moved from isolated experiments toward parallel training and reproducible runs.

Jul–Aug 2025 YOLO perception overlay from follower car camera

2. Multi-Agent and Multimodal Exploration

Expanded the project from single-car lane following to multi-car interaction, camera perception, LiDAR, and overtaking logic.

  • Launched multi-car / multi-environment training experiments.
  • Added camera detection modules and early overtaking decision logic.
  • Integrated LiDAR as an additional observation channel.
  • Built a dual-car YOLO + Coach prototype for collision risk, following distance, and overtaking advice.

Sep 2025 Jetson camera and simulator validation view

3. Real-Car Deployment Contract

Validated the full DonkeyCar deployment path on Jetson before forcing reinforcement learning models into the vehicle stack.

  • Trained and deployed a compatible KerasLinear .h5 baseline.
  • Verified the CSICamera → model inference → Web Controller → real vehicle control path.
  • Clarified deployment contracts for image size, RGB preprocessing, steering range, throttle range, and model output order.
  • Used the baseline as the reference interface for later RL deployment.

Sep–Oct 2025 Distilled DonkeyCar model running in a simulated indoor track

4. RL-to-Jetson Model Bridge

Created practical deployment routes for reinforcement learning policies under Jetson Nano compatibility constraints.

  • Built a PPO-to-distilled-Keras .h5 pipeline for DonkeyCar compatibility.
  • Removed fragile TensorFlow Lambda dependencies for Jetson Nano deployment.
  • Rebuilt the RL environment around compatible DonkeyCar, Gym, SB3, Python, and Jetson versions.
  • Later adopted a direct .pth policy-weight inference route to avoid SB3/cloudpickle .zip deserialization problems.

Nov 2025

5. Multimodal PPO and Curriculum Learning

Moved beyond image-only policies into camera + LiDAR early fusion and curriculum-driven control improvement.

  • Built a 6-channel vision pipeline with RGB, yellow mask, Canny edges, and motion-related features.
  • Added LiDAR sector features and CNN + MLP fusion for policy input.
  • Introduced curriculum learning for speed, CTE tolerance, penalties, and safe-distance behavior.
  • Deployed the multimodal policy path to Jetson Nano for early real-car testing.

Feb–Mar 2026 Multi-scene semantic feature analysis across simulated tracks

6. System-Level Sim-to-Real Architecture

Shifted from “train a model” to “align the whole autonomy system.”

  • Added action safety layers: slew-rate limiting, low-pass filtering, smoothness penalties, and mismatch diagnostics.
  • Replaced direct actuation with high-level action targets executed by a lower-level controller.
  • Refactored reset/spawn logic to reduce invalid episodes and NPC-coupled failures.
  • Built FiLM/LSTM-based semantic observation and action adapter components for multi-scene training.
  • Validated dual-domain training across Waveshare and Generated Track domains before adding obstacle-oriented training.

Apr 2026 Real taped track with Jetson-based smart cars and obstacle boxes

7. Hardware Alignment and Edge Limits

Closed the loop back to the physical vehicle by measuring real hardware behavior and edge-device constraints.

  • Compared simulator and real-car steering, throttle, control frequency, and sensor behavior.
  • Debugged servo center, PWM frequency, motor response, and LiDAR representation issues.
  • Measured LiDAR timing and identified camera/LiDAR/control-loop mismatch.
  • Benchmarked an on-device LLM strategy sidecar and concluded it was not suitable for real-time racing control on Jetson Nano.

May 2026

8. V17 Endpoint Deployment Freeze

Completed the Jetson-side engineering deployment chain and froze the endpoint validation scope.

  • Exported the V17 actor through ONNX and built a TensorRT FP16 engine.
  • Implemented CUDA runtime inference without adding PyCUDA as a deployment dependency.
  • Moved LiDAR sectorization out of the main vehicle loop.
  • Converted DataCollector to asynchronous logging and cached Jetson telemetry in a background thread.
  • Added engine/metadata preflight, LiDAR/RP2040 freshness gates, inference timeout counters, and shadow non-takeover validation.
  • Completed repeated TensorRT shadow validation, including 180-second, 10-minute, and 20-minute runs.

Selected Engineering Results

Area Result
Model deployment Actor-only ONNX export and TensorRT FP16 runtime on Jetson Nano
Runtime profiling Separated actor inference from image preprocessing, LiDAR processing, logging, and telemetry overhead
TensorRT benefit Actor residual p95 reduced from 40.988 ms to 17.046 ms in optimized A/B testing
Logging DataCollector p99 reduced from 537.42 ms to 5.67 ms through async logging and telemetry caching
LiDAR processing 360-degree LiDAR sectorization moved out of the main vehicle loop
Safety checks Engine, metadata, LiDAR, RP2040, and inference-timeout gates added before or during runtime
Shadow validation Final endpoint chain validated in a 1199.55-second non-takeover TensorRT shadow run

Scope and Limitation

The V17 endpoint deployment milestone evaluates the Jetson-side engineering chain: model export, TensorRT inference, runtime stability, logging, safety gates, and shadow-mode reproducibility.

What This Project Demonstrates

This project demonstrates my ability to work across the full stack of an autonomous system:

  • reinforcement learning training and curriculum design;
  • multimodal perception and sensor preprocessing;
  • embedded AI deployment on Jetson Nano;
  • real-time runtime profiling and bottleneck isolation;
  • hardware debugging across camera, LiDAR, PWM, motor, and telemetry layers;
  • sim-to-real alignment for dynamics and sensor timing;
  • safety-oriented validation and reproducible experiment reporting;
  • technical mentorship and beginner-oriented robotics documentation.

The value of the project is not a single checkpoint. The value is the engineering process that turned a fragile research prototype into a measurable, debuggable, and deployable edge autonomy system.

Recognition

  • 05/28/2026: Received the 2026 Outstanding Master’s Student Award in Computer Engineering as a master’s student RA. The announcement was shared by IntelliSys Lab, with the SES community event video timestamp.

Updated: