Multi-Sensor Fusion | Chang Nie

Commercial Autonomous Mowing Robot: Multi-Modal Fusion & Terrain-Aware 3D Perception

Sun, 01 Jan 2023 00:00:00 +0000

Project context

Positec Group (WORX) Mass-Production Partnership

Overcoming the severe challenges of unstructured, deformable outdoor turf—including thick grass returns, non-planar slopes, hidden trenches, and dynamic pedestrian/pet traffic—this system brings automotive-grade autonomous driving technologies to outdoor mobile robotics, pioneering truly scene-aware, wire-free lawn robotics.

Key Deliverables & Field Validation: Commercial-grade integrated robotic hardware-software system; >98% obstacle avoidance success rate, terrain-adaptive 3D obstacle perception engine, positive/negative obstacle extraction, and field-proven autonomous mowing

My contributions

As project lead, developed slope-adaptive LiDAR point-cloud filtering and visual obstacle avoidance for Positec robotic lawn mowers, achieving >98% obstacle avoidance success rate and resolving point-cloud clutter filtering, positive/negative obstacle separation, and multimodal sensor fusion on rolling terrain.

Commercial and residential robotic mowers have traditionally relied on low-frequency inductive perimeter wires buried beneath turf edges, coupled with random-bounce navigation. Over expansive undulating estates, municipal parks, or irregular municipal greens, physical perimeter wiring incurs prohibitive civil trenching costs, difficult wire fault troubleshooting, and inherent random-walk inefficiencies characterized by high re-mow overlap and missed cutting patches. Boundary-wire-free autonomous perception, all-terrain topological navigation, and systematic parallel pattern mowing represent the defining technological paradigm for next-generation outdoor groundskeeping robotics.

This project partnered with global outdoor power equipment leader Positec Group (WORX) to engineer a production-ready, boundary-wire-free commercial mowing robot equipped with full-scene autonomous perception, self-adaptive mapping, and global coverage motion planning. As Project Tech Lead, I directed the top-level full-stack hardware-software architecture, cross-module subsystem integration, and field deployment; simultaneously, as principal perception lead, I directed algorithm development for multi-beam LiDAR terrain-adaptive point-cloud modeling, bifurcated positive/negative obstacle extraction, and spatial cross-modal vision-LiDAR fusion.

Key Perception Challenges in Undulating Turf

On structured urban asphalt, the ground surface is predominantly rigid, planar, and continuous; point cloud reflections protruding 10 cm above the surface unambiguously signify physical obstacles. However, when transitioning into unstructured, undulating outdoor turf, these foundational geometric assumptions break down completely:

Vegetation Clutter & Multiple-Scattering Ambiguity (“Grass vs. Obstacle”): Turfgrass blades are flexible, semi-translucent, non-rigid, and continuously wind-perturbed. Infrared laser pulses penetrating dense turf experience multiple diffuse scattering across grass tips, mid-stems, and underlying soil. In raw 3D point clouds, this manifests as a loose, porous noise blanket 20 to 35 cm thick. If points protruding above bare soil are treated as obstacles, the mower becomes paralyzed by false phantom obstacles; conversely, applying naive 30 cm geometric height filtering causes low garden hoses, tree roots, perimeter curbs, and recumbent domestic pets to be completely overlooked, risking blade damage and safety hazards.
Non-Planar Undulating Terrain & Local Tangent Variations: Real outdoor lawns feature rolling hills, swales, drainage mounds, and subtle undulating contours. Global planar ground extraction (such as standard RANSAC ground fitting) fails within 3 to 5 meters of the vehicle, mistaking normal uphill grass surfaces for solid vertical walls and triggering emergency halts.
The “Negative Obstacle” Crisis (Depressions, Drainage Ditches, and Step Drops): Commercial mowers feature compact chassis with drive wheel diameters of only 15 to 20 cm. On contiguous grass, 7 cm subsidence ruts, open drainage trenches, or lawn edges dropping off into patios will instantly trap front casters, strike cutting discs, or induce vehicle rollovers. Negative obstacles do not generate positive point-cloud reflections; rather, they present as voids of missing laser returns and ray-shadow gaps, rendering classical spatial clustering algorithms ineffective.
Severe Optical Variations in Outdoor Natural Environments: Lawn maintenance subjects camera sensors to harsh dynamic lighting transitions: unshielded midday sun, dappled tree shadows, and acute morning/evening glare. Active structured-light infrared sensors suffer complete pattern washout under direct solar irradiance, while passive stereo algorithms experience extensive disparity dropouts across repetitive, textureless green turf.
Ultra-Low Embedded Power & Thermal Budgets: Unlike passenger autonomous testbeds carrying water-cooled industrial PCs in vehicle trunks, autonomous mowers operate strictly on compact onboard lithium battery packs. The complete SLAM, perception, and coverage planning pipeline must execute deterministically at 20 Hz within a compute budget limited to ~15 Watts on embedded edge SoCs (such as NVIDIA Jetson modules).

System architecture

An autonomous mobile robot for unstructured lawns

Localization, perception, planning, and control share one onboard sensor and compute platform.

Onboard sensing and compute

01 · VisionMulti-camera and RGB-DLawn, objects, semantics, and appearance

02 · GeometryLiDARRange, terrain, and obstacle point clouds

03 · MotionRTK/GNSS · IMU · wheel odometryGlobal constraints and continuous local motion

Autonomy core

04Multi-sensor localizationVision, LiDAR, inertial, wheel, and satellite cues

05Visual objects and semanticsMowable area, object categories, and scene semantics

06 · Program leadership and technical focusSystem integration + LiDAR obstacle detection + multi-sensor obstacle fusionLead module integration and field commissioning; separate obstacles from uneven grass and unify 3D geometry, semantic segmentation, and visual detections for planning

Robot behavior

07Local environment modelTraversable lawn and dynamic/static obstacles

08Path and coverage planningSafe avoidance, coverage mowing, and return-to-charge

09Chassis and mowing controlTurn perception into real-time motion decisions

Core Algorithmic Architecture & Engineering Design

To achieve reliable autonomy under low-power compute and natural outdoor environmental degradation, we developed a full-stack autonomy pipeline spanning factor-graph SLAM, polar terrain manifold modeling, positive/negative obstacle decoupling, and cross-modal sensor fusion.

1. Factor-Graph Multi-Sensor Fusion Localization & Slip Adaptation

Maintaining tightly spaced, millimeter-parallel “boustrophedon” (lawn-striping) patterns over hours of continuous operation demands high-rate, drift-free 6-DOF pose estimation:

Heterogeneous Sensor Topology: The platform integrates dual-antenna RTK-GNSS, an automotive-grade 6-axis IMU, precision wheel encoders, and a multi-beam 3D LiDAR. In open sky, dual-antenna differential GNSS delivers centimeter-level position fixes and true orientation heading;
Slip-Adaptive Wheel Odometry Factor: On dew-soaked morning grass or damp sloping soil, mower wheels frequently encounter micro-slip and wheelspin. The system continuously cross-examines wheel acceleration against axial IMU accelerations, establishing a slip-residual monitoring mechanism that dynamically downweights wheel kinematic constraints to prevent spurious odometry drift;
Unified Smoothing Backend via GTSAM: Implemented in GTSAM, the sliding-window optimization backend simultaneously resolves RTK position factors, IMU pre-integration factors, slip-adaptive kinematic odometry factors, and LiDAR planar surface constraints over the trajectory. When transitioning under dense tree canopies or adjacent to tall buildings where GNSS signals degrade, inertial and LiDAR odometry sustain continuous localization without trajectory jumps.

2. Terrain-Adaptive LiDAR Point-Cloud Modeling & Decoupled Hazard Extraction

Extracting physical hazards out of hundreds of thousands of diffuse grass returns forms the central algorithmic breakthrough of the perception pipeline:

Terrain-adaptive point-cloud pipeline: from polar grid partitioning and slope continuity estimation to clustering, temporal tracking, and obstacle dispatch.Open full size

We built a local polar manifold terrain analysis and ray-gap detection pipeline:

Structured Polar Elevation Grid: Raw 3D point clouds within a 10-meter operational radius are projected into an egocentric polar coordinate grid. Dividing space into fine radial range cells and azimuth sectors, the system computes lowest elevation, highest elevation, point count density, and elevation variance per cell;
Radial Tangent Ground Manifold Propagation: Bypassing global ground plane assumptions, local tangent planes are propagated outward along each azimuth ray starting from the tire-ground contact footprint. Enforcing smoothness and bounded gradient step constraints, the algorithm conforms to undulating terrain contours while dynamically isolating soil levels from overlying turfgrass;
Dual-Mode Positive Hazard Classification (Protruding Obstacles): Operating across manicured lawns and wild overgrown grass:
1. Manicured Turf Mode: Point clusters exceeding 16 cm in elevation with surface normals diverging from local ground normals are classified as obstacles (tree trunks, stones, fences);
2. Overgrown Meadow Mode: Elevation thresholds adaptively elevate to 35 cm, combined with vertical point density entropy. Because grass stems produce uniformly dispersed vertical returns whereas rigid solid objects concentrate dense returns, entropy testing reliably separates dense obstacles from tall grass;
Negative Hazard Extraction (Pits, Trenches, and Cliffs): Depressions lack direct reflective returns. We model the expected line-of-sight laser propagation geometry. When encountering a ditch or step drop ($\ge 7\text{ cm}$ depth), the laser beam either passes over the depression or strikes the distant lip at an anomalous radial distance discontinuity, creating a ray-shadow gap. Detecting this discontinuity paired with abrupt elevation drops marks the negative hazard boundary, projecting a repulsive field into the local costmap;
Two-Pass Connected Component Clustering & Kalman Tracking: Surviving obstacle cells undergo fast raster clustering into 3D bounding boxes. Bounding boxes are tracked across frames via Hungarian matching and Kalman filtering, eliminating transient wind-blown grass blades that persist for only 1–2 cycles.

LiDAR delivers precise metric geometry and 3D bounding dimensions, but cannot infer object semantics (distinguishing an innocent toy from a sleeping pet). Cameras capture rich color semantics but lack reliable depth on textureless lawns.

Multi-modal obstacle fusion pipeline: aligning 3D point clusters with 2D semantic masks to output verified, category-aware navigation obstacles.Open full size

We deployed TensorRT-accelerated lightweight object detection networks on the embedded GPU, processing at 20 Hz to detect pedestrians, domestic animals (dogs, cats), hoses, garden tools, and flowerbed edges. Reprojecting 3D LiDAR clusters onto 2D camera pixels via the camera-LiDAR calibration matrix yields three complementary outputs:

Category-Aware High-Confidence Fused Targets: 3D LiDAR bounding projections match 2D visual bounding detections with high Intersection-over-Union (IoU). The target receives unambiguous category labeling (e.g., sleeping pet) and millimeter 3D coordinates. The motion planner assigns highest avoidance priority, generating a generous $>1.0\text{ m}$ standoff safety corridor;
Point-Cloud-Only Geometric Obstacles: Unclassified visual targets (e.g., untextured grey boulders, tree stumps, ground trenches) confirmed by strong LiDAR geometry. The planner handles them as rigid obstacles, adopting standard 0.3 m edge clearances;
Vision-Only Distant Precautionary Candidates: Visual detections of humans or pets at 10–15 meters where LiDAR point returns remain sparse (1–2 points). The system preemptively curtails speed, maintaining tracking focus until approaching within dense LiDAR verification range.

Field association example: visual object semantics and 3D point-cloud supports lock together in real-time navigation frames.

4. Autonomous Coverage Planning & Anti-Sideslip Control

Boustrophedon Complete Coverage Path Planning (CCPP): Decomposing complex yards into convex polygonal cells, the planner computes parallel sweeping tracks with continuous turn loops along boundaries, minimizing deadhead transit;
Gravity-Compensated Slope Anti-Sideslip Control: On lateral grass slopes reaching 20° (36% grade), gravity pulls the mower downhill, producing severe trajectory lateral slip. The wheel controller actively monitors chassis roll and pitch from the fused IMU and injects lateral velocity feedforward compensation into differential drive outputs, cancelling downhill sliding drift and preserving razor-straight cutting tracks on steep embankments.

Outdoor Field Playback

Field Playback: Terrain-Adaptive Point-Cloud PerceptionReal-world field operation: red 3D point clusters mark detected low-profile obstacles and terrain boundaries, while front/rear cameras provide real-time visual confirmation.

The field recordings above demonstrate real-world autonomous closed-loop operation on undulating villa lawns: synchronized dual-camera video feeds align with 32-beam 3D LiDAR point clouds. Red highlighted clusters cleanly reject tall grass clutter while locking onto low-profile rocks, flowerbed curbs, and terrain drop-offs under dappled sunlight and changing outdoor conditions.

Engineering Summary & Technical Takeaways

Deploying autonomous mowers into real-world unstructured greenery highlights primary engineering insights:

Leveraging Physics Priors Against Natural Ambiguity: When dealing with diffuse foliage returns and non-planar slopes, generic data-driven models or flat-ground assumptions inevitably fail. Understanding laser scattering mechanics within vegetation and parameterizing radial slope continuity allows clean clutter rejection at minimal compute cost.
Negative Obstacle Detection as an Essential Safety Prerequisite: For low-clearance mobile bases, ground trenches and step drops are as hazardous as rigid vertical walls. Elevating negative hazard detection to equal architectural priority ensures safe off-road operations.
Balancing Algorithmic Sophistication with Edge Thermal Limits: In battery-powered commercial robotics, algorithms detached from hardware constraints cannot reach production. Orchestrating lightweight neural networks with compact geometric point-cloud processing enables robust 20 Hz real-time loops on compact low-power SoCs.
Active Gravity Feedforward Beats Passive Integral Trimming: On steep, slippery outdoor grass, passive integral controllers take seconds to wind up, resulting in significant downhill lane departure. Directly feeding physical gravity projections into dynamic steering compensation keeps tracking tight across slope transitions.

Full-Stack Autonomous Driving Research Platform: Hardware-Software Co-Design & Real-Vehicle Verification

Sat, 01 Jan 2022 00:00:00 +0000

Autonomous driving research requires validation beyond idealized simulator environments to directly address real-world ground friction, physical sensor noise, bus communication latency, and actuator non-linearities. In 2019, commercial autonomous testbeds were prohibitively expensive and strictly closed-source, precluding in-depth research into chassis dynamics and custom low-level sensor drivers. To establish an accessible, controllable physical research platform, we undertook the ground-up hardware and software construction of an autonomous driving research testbed.

Between 2019 and 2022, we designed and built two complementary research vehicles: a low-speed pure-electric passenger vehicle configured for campus road cruising, and an omnidirectional research AGV featuring 4-wheel independent driving and 4-wheel independent steering (4WID-4WIS). The engineering effort spanned mechanical mounting fabrication, high/low-voltage isolated power distribution, reverse engineering of proprietary CAN bus protocols, development of Linux SocketCAN drivers, microsecond-level hardware time synchronization across heterogeneous sensors, and a full-stack ROS-based software autonomy suite encompassing perception, SLAM, planning, and control. The platform successfully completed closed-loop verification across diverse representative operational scenarios under realistic campus road traffic.

Project context

Autonomous Driving Research Core · Ground-Up Hardware/Software R&D

To evaluate autonomous driving algorithms on physical hardware rather than purely in simulation, we built full-stack research vehicle platforms from the ground up. Covering chassis adaptation, power distribution, SocketCAN drivers, and ROS autonomy modules, the platform provided an experimental testbed for ongoing laboratory research.

Platform Deliverables & Multi-Scenario Real-Vehicle Demonstrations: Two operational drive-by-wire autonomous research platforms; 300+ hours of road testing, 8 real-vehicle closed-loop demonstrations (<5 cm lateral tracking error, <60 ms AEB braking response), and a complete custom ROS autonomy stack.

My contributions

As project lead, built two generations of drive-by-wire autonomous vehicle research platforms (passenger cart and 4WIS AGV) from scratch, accumulating 300+ hours of physical road testing; responsible for drive-by-wire modification, power isolation, CAN bus reverse engineering, and multi-scenario closed-loop validation.

Two Complementary Research Chassis

To comprehensively ground our research across disparate operational domains, we developed two distinct physical platforms:

Low-Speed Campus Research Cart: Configured with passenger-carrying seating, a standard automotive steering column, and hydraulic braking. Its open frame and high payload capacity accommodate heavy sensor payloads: a 32-beam spinning 3D LiDAR, forward stereo and surround wide-angle cameras, dual-antenna RTK-GNSS, a high-rate IMU, and a ruggedized industrial PC equipped with high-performance discrete GPUs. This vehicle serves as the primary testbed for large-scale metric-topological SLAM, multi-modal perception, traffic interaction, and cooperative V2X platooning.
Compact 4-Wheel Independent Drive & Steer (4WID-4WIS) AGV: Equipped with eight independent brushless DC servomotors (four driving wheels, four steering hubs). Bypassing the non-holonomic kinematic limits of traditional Ackermann steering, the vehicle dynamically reconfigures its steering mode in software: standard front-wheel Ackermann, diagonal crab-steering, pure lateral translation, and zero-radius pivot spinning around its chassis center. This platform provides an agile testbed for high-speed obstacle avoidance maneuvers and extreme tight-space parking.

Both vehicles enforce a foundational design paradigm: exposing algorithms directly to real-world tire-ground compliance, CAN transmission jitter, and actuator deadbands, thereby establishing a physical benchmark for subsequent algorithmic robustness.

Full-Stack Autonomous Driving Architecture

Multi-Sensor Fusion → State Estimation → Planning → Drive-by-Wire Actuation

Full-stack onboard ROS autonomy connects perception, SLAM, and planning; the drive-by-wire chassis executes closed-loop control over CAN.

Sensing and computation

01Observe surroundingsCamera · LiDAR · ultrasonics

02Positioning inputsRTK GNSS · IMU · LiDAR map

03Onboard computationIndustrial PC · ROS · HMI

Autonomy and actuation

04Perception and localizationObjects · lanes · obstacles · vehicle pose

05Decision and planningReference route · driving objective

06Control and drive-by-wireMotion commands · CAN · chassis actuation

Vehicle motion → new observations and feedback → next control updateMap-based and RTK route following use their respective localization inputs.

Ground-Up Electrical & Sensor Topology

Without commercial turnkey kits, integrating disparate sensors and actuators into a fail-safe vehicular platform required overcoming fundamental electrical isolation, extrinsic calibration, and temporal synchronization challenges.

Subsystem	Configuration & Role	Core Engineering Challenges Overcome
Drive-by-Wire & CAN Bus	Electronic steering motor, throttle potentiometer, and hydraulic brake booster	Reverse-engineered proprietary cyclic DBC protocols; built 100 Hz deterministic Linux SocketCAN drivers with heartbeat fail-safe routines
3D LiDAR	32-beam / 16-beam mechanical spinning LiDAR for 3D mapping and obstacle ranging	Custom CNC aluminum vibration dampers isolating chassis road shock; calibrated beam reflectivity intensity decay curves
HDR Industrial Cameras	Narrow-FOV (60°) and wide-angle (120°) cameras for lane segmentation and object vision	Mitigated severe lens flare transitioning into shaded tree canopies; designed rigid camera-LiDAR multi-modal calibration targets
RTK-GNSS Differential GPS	Dual-antenna receiver delivering centimeter-level 3D coordinates and true heading	Optimized 1.5 m baseline antenna placement; mitigated carrier-phase slip and multipath jumps beside dense concrete campus structures
Automotive IMU	6-axis tactical-grade inertial unit providing 200 Hz linear acceleration and yaw rates	Mechanically aligned IMU sensing center with chassis rear-axle centerline; performed online thermal bias drift calibration
Ultrasonic Transceiver Array	8 bumper-mounted ultrasonic sensors for near-field blind-spot guarding (0.2–2.5 m)	Optimized acoustic beam-cone overlap geometry, enforcing sequential time-multiplexed firing to eliminate cross-talk
Power Distribution & Isolation	48V traction battery pack converted to isolated 12V/24V clean bus	Custom electrical distribution enclosure with TVS diodes and optocouplers, suppressing motor back-EMF voltage surges and ground loops

1. Hardware Hard-Triggering & Microsecond Temporal Synchronization

At vehicle speeds of 25 km/h, a 100 ms timing desynchronization between sensors induces an spatial projection error approaching 0.7 meters. Relying on software system timestamps (ROS Time) introduces non-deterministic OS scheduling jitter incompatible with feature-level sensor fusion.

We engineered a multi-tier hardware synchronization architecture:

PTP and PPS Nanosecond Clock Locking: The RTK-GNSS receiver, upon locking satellite constellations, generates a precision Pulse Per Second (PPS) square wave and GPRMC NMEA timing packets. The 32-beam LiDAR ingests the PPS signal, synchronizing its internal clock to absolute UTC time within nanoseconds;
FPGA/Microcontroller External Optical Exposure Triggering: A dedicated microcontroller captures the PPS pulse and synthesizes a phase-locked 20 Hz square wave sent to the external trigger pins of industrial cameras. This forces camera sensor exposure to coincide exactly with the moment the LiDAR spinning beam sweeps through the forward field of view;
Driver nodes read hardware frame timestamps directly from peripheral registers, compressing inter-sensor temporal alignment error below 1 millisecond.

2. Rigid Extrinsic Joint Calibration (LiDAR-Camera-IMU)

To eliminate spatial misalignments:

The rigid transformation between the camera optical center and LiDAR coordinate frame is optimized using a custom calibration board featuring hollow circular cutouts, simultaneously minimizing 3D laser point-to-plane fitting errors and 2D visual circular re-projection errors;
IMU-to-LiDAR spatial extrinsics are identified through continuous sinusoidal steering maneuvers and rapid acceleration profiles in open areas, parameterizing trajectories via continuous B-splines to minimize angular velocity and linear acceleration residuals, reaching millimeter and 0.1-degree alignment accuracy.

3. Proprietary CAN Bus Reverse Engineering & Deterministic SocketCAN Drivers

With native chassis controller (VCU) documentation withheld, we captured raw bus traffic using CAN analyzers during manual driving. By correlating driver throttle inputs, pedal pressure, steering wheel sweeps, and wheel encoder pulses, we systematically isolated and mapped the cyclic DBC communication matrix.

We subsequently built an industrial-grade Linux SocketCAN driver:

Implemented a 100 Hz deterministic periodic thread transmitting steering angle, throttle voltage, and brake pressure setpoints;
Integrated a dual-directional watchdog supervisor: if the VCU misses valid heartbeat packets for $>100\text{ ms}$, onboard firmware immediately disengages autonomous drive and engages gentle mechanical braking, preventing runaway faults caused by upstream software crashes.

Full-Stack ROS Autonomy Architecture

The software stack operates on Ubuntu Linux and ROS with optimized shared-memory IPC, partitioned into three core decoupled pipelines:

1. Localization & High-Precision Mapping Pipeline

Tightly Coupled LiDAR-Inertial SLAM: Extracting geometric edge and planar features from raw 32-beam point clouds, the frontend fuses high-rate IMU pre-integration priors within a factor graph optimization backend. Between keyframes, relative motion increments and bias updates are factored alongside LiDAR point-to-plane geometric residuals into an incremental smoothing and mapping (iSAM2) factor graph, rendering globally consistent 3D point-cloud maps of campus roadways;
50 Hz Online NDT Localization: Within the pre-built metric voxel map, incoming point clouds are registered using the Normal Distributions Transform (NDT) at 50 Hz. Each spatial voxel approximates local point distributions via 3D Gaussians. Even when traversing covered pedestrian corridors and dense underpasses where GNSS signals vanish, NDT scan registration maintains continuous state estimation without trajectory drift or discrete position jumps;
Dual-Antenna Differential Kinematic Heading: Fixed dual antennas establish an instantaneous physical baseline across the vehicle roof, determining true vehicular heading even at zero velocity without dead-reckoning integration drift.

2. Environmental Perception & Scene Understanding Pipeline

Bird’s-Eye-View Visual Lane Tracking: Monocular camera images undergo distortion rectification and Inverse Perspective Mapping (IPM). A lightweight segmentation network extracts lane boundary pixels, fitting polynomial curves to output continuous road curvature and lateral offset for steering regulation;
Multimodal 3D Object Detection & Tracking: 2D deep learning detectors classify pedestrians, bicycles, and vehicles in image space. Detected bounding frustums are back-projected into spatial point clouds to segment candidate point clusters and fit 3D oriented bounding boxes. An Interacting Multiple Model Unscented Kalman Filter (IMM-UKF) bridges constant-velocity and constant-turn-rate motion hypotheses to publish stable obstacle velocity vectors.

3. Hierarchical Planning & High-Rate Actuation

Hierarchical Decision State Machine: Finite state machines (FSM) govern macroscopic mission transitions across cruise, dynamic yield, intersection negotiation, bay parking, and emergency stop. Local trajectory generators sample candidate polynomial trajectories in the Frenet frame to dodge static obstacles and slow pedestrians;
Dual-Loop Dynamic Tracking Control: Lateral control combines Pure Pursuit lookahead with Linear Time-Varying Model Predictive Control (LTV-MPC) over a dynamic bicycle model, enforcing steering angle limits and tire lateral slip angle saturation bounds to eliminate overshoot during high-curvature turns. Longitudinal control employs closed-loop PID with speed feedforward, transmitting steering and braking commands at 100 Hz over CAN.

Seven Real-Vehicle Closed-Loop Scenarios

Real-Time 3D LiDAR SLAM MappingOnboard 3D LiDAR incrementally constructing metric point-cloud maps of complex campus terrain with crisp structural fidelity.

High-Precision SLAM Waypoint TrackingAutonomous waypoint navigation localized entirely within self-built 3D LiDAR point-cloud maps.

Vision-Based Lane Centering on 4WIS AGVMonocular front-facing camera segmenting lane boundaries to drive smooth centering control at speed.

Dynamic Multi-Object Roadside TrackingSynchronized camera-LiDAR fusion tracking pedestrians, vehicles, and dynamic campus traffic.

Autonomous Emergency Braking (AEB)Sub-millisecond emergency brake actuation upon sudden obstacle intrusion across the vehicle path.

Autonomous Precision Bay ParkingCoordinated ultrasonic-vision trajectory optimization backing the vehicle into narrow parking stalls.

V2X Multi-Vehicle PlatooningDirect inter-vehicle ad-hoc wireless communications coordinating two autonomous cars in synchronized following along curves.

The seven video recordings document extensive field testing across campus thoroughfares:

Real-Time 3D LiDAR Mapping: The passenger vehicle cruises smoothly under dense tree canopies, incrementally rendering metric 3D point-cloud geometry of building facades and road curbstones;
High-Precision SLAM Waypoint Following: Operating with GNSS antennas disconnected, the vehicle navigates autonomously through long, winding campus avenues guided solely by real-time NDT point-cloud matching;
Vision Lane Centering on 4WIS AGV: The forward camera parses faded white road lines, while the 4WIS chassis dynamically modulates steer angles to remain centered in the travel lane;
Dynamic Roadside Object Tracking: Onboard detectors identify pedestrians crossing intersections and oncoming bicycles, maintaining 3D bounding boxes and trajectory predictions;
Autonomous Emergency Braking (AEB) Verification: When an obstacle mannequin is rapidly thrust into the vehicle path, perception computes Time-to-Collision ($\text{TTC} < 1.2\text{s}$) within 60 ms, triggering full hydraulic clamp lockup to halt the vehicle 0.8 meters before the target;
Precision Bay Parking: Ultrasonic sensors measure bay clearance, allowing the planner to execute a two-stage reverse trajectory that parks the 4WIS AGV neatly aligned with painted stall markers;
V2X Connected Platooning: Over 5.8 GHz ad-hoc wireless mesh links, lead-vehicle acceleration and braking intentions are shared at 100 Hz, enabling two autonomous vehicles to maintain tight, zero-lag platoon spacing through multi-turn courses.

Engineering Summary & Technical Takeaways

Building two complete physical autonomous platforms from bare metal established foundational systems engineering insights:

Physical Ground Truth Governs Algorithmic Success: Simulators hide bus dropouts, high-frequency frame vibration, electrical ground loops, and motor deadbands. Only by directly debugging power distribution surges and actuator dead zones on real iron can one design resilient control architectures with adequate feedforward and fail-safe watchdogs.
Deterministic Timing Underpins Sensor Fusion: Sophisticated geometric algorithms fail if multi-modal sensor streams suffer millisecond-scale timestamp jitter. Enforcing hardware-level PPS clock discipline and external exposure triggering is a prerequisite for reliable real-world perception.
Reusable Modular System Foundation: The full-stack hardware-software architecture established in this project provided directly transferable foundational modules for subsequent laboratory research and industry partnerships in mobile robotics and autonomous valet parking.
Hardware-Level Heartbeat Watchdogs as Safety Primitives: Upstream Linux software stacks inevitably experience occasional thread freezes, driver crashes, or kernel scheduling delays. Guaranteeing that the low-level chassis microcontroller automatically cuts throttle and engages gradual hydraulic braking when cyclic CAN heartbeats drop for 100 ms is what keeps physical autonomous prototypes safe during high-speed live road trials.

Multi-Sensor Fusion | Chang Nie

Commercial Autonomous Mowing Robot: Multi-Modal Fusion & Terrain-Aware 3D Perception

Project context

My contributions

Key Perception Challenges in Undulating Turf

Core Algorithmic Architecture & Engineering Design

1. Factor-Graph Multi-Sensor Fusion Localization & Slip Adaptation

2. Terrain-Adaptive LiDAR Point-Cloud Modeling & Decoupled Hazard Extraction

3. Cross-Modal Vision-LiDAR Spatial Fusion & Semantic Augmentation

4. Autonomous Coverage Planning & Anti-Sideslip Control

Outdoor Field Playback

Engineering Summary & Technical Takeaways

Full-Stack Autonomous Driving Research Platform: Hardware-Software Co-Design & Real-Vehicle Verification

Project context

My contributions

Two Complementary Research Chassis

Ground-Up Electrical & Sensor Topology

1. Hardware Hard-Triggering & Microsecond Temporal Synchronization

2. Rigid Extrinsic Joint Calibration (LiDAR-Camera-IMU)

3. Proprietary CAN Bus Reverse Engineering & Deterministic SocketCAN Drivers

Full-Stack ROS Autonomy Architecture

1. Localization & High-Precision Mapping Pipeline

2. Environmental Perception & Scene Understanding Pipeline

3. Hierarchical Planning & High-Rate Actuation

Seven Real-Vehicle Closed-Loop Scenarios

Engineering Summary & Technical Takeaways