Pedestrian Detection with Day/Night Cameras and Radar Fusion
Dor Peleg3 min read
Key facts
RILLOO led the software development team of a multidisciplinary in-vehicle situational awareness system that detects and tracks pedestrians around the clock. The system fuses day cameras, night cameras and radar, and runs AI inference on NVIDIA Jetson AGX edge hardware inside custom electrical and mechanical systems.
- Industry
- Automotive
- Challenge
- Reliably detect and track pedestrians around a vehicle day and night, in real time, on edge hardware, while integrating software with complex custom electrical and mechanical subsystems.
- Solution
- A real-time software stack on NVIDIA Jetson AGX that runs detection models, fuses day camera, night camera and radar data into one picture, and tracks each pedestrian over time, built in C, C++ and Python by a RILLOO-led software team.
- Results
- Around-the-clock pedestrian detection from combined day camera, night camera and radar data
- Real-time AI inference running on board the vehicle, with no dependence on a cloud connection
- Software delivered as one integrated system with custom electrical and mechanical subsystems
- Technologies
- CC++PythonNVIDIA Jetson AGXEdge AIComputer visionRadarSensor fusionMulti-object tracking
Knowing who is moving around a vehicle is the core of situational awareness on the road. Pedestrians are among the hardest targets to get right: they are small, they move unpredictably, they appear in groups, and they must be detected just as reliably at night, in glare or in poor weather as on a clear afternoon. RILLOO led the software development of an in-vehicle system built to do exactly that, combining day cameras, night cameras and radar into one real-time picture of every pedestrian around the vehicle.
Day/night camera and radar fusion detecting and tracking pedestrians around a vehicle
The Project
The system was a multidisciplinary effort. Alongside the software, it included custom electrical and mechanical subsystems designed specifically for the product. RILLOO led the software development team and owned the software side end to end: from the low-level code that talks to sensors and hardware, through the AI models that find pedestrians, to the tracking and fusion logic that turns raw detections into reliable situational awareness.
Working inside a complex automotive hardware program meant the software had to fit real-world constraints from the start: fixed compute and power budgets, tight timing between sensors, and close coordination with the electrical and mechanical teams as the hardware evolved.
Why Three Sensors
No single sensor sees everything:
- Day cameras give rich color and detail, ideal for recognizing people in good light.
- Night cameras keep the system working in darkness and low light, when a day camera loses most of its value.
- Radar measures range and velocity directly and is far less affected by darkness, glare, rain or fog, but on its own it does not tell you what an object is.
By fusing all three, the system combines the recognition power of cameras with the robust range and motion data of radar, so it can keep detecting pedestrians across lighting and weather conditions.
Main Challenges
Edge AI
All detection had to run on board the vehicle, on an NVIDIA Jetson AGX edge computer, in real time and without depending on a cloud connection. That meant choosing and optimizing models to make full use of the Jetson’s GPU within the available compute and power budget while keeping detection accuracy high, and writing performance-critical code paths in C and C++ to keep latency low and predictable.
Tracking
Detecting a pedestrian in a single frame is not enough. Situational awareness means knowing where each person is, where they are heading and whether they are the same person seen a moment ago. The tracking layer follows each pedestrian over time, keeps a stable identity through brief occlusions and missed detections, and estimates motion so the system understands how the scene is changing.
Sensor Fusion
Day camera, night camera and radar each produce data at different rates, resolutions and coordinate frames. The fusion layer aligns them in time and space, associates detections from different sensors with the same physical person, and weighs each sensor according to how much it can be trusted in the current conditions. The result is a single, consistent view that is more reliable than any one sensor alone.
Technologies
- C and C++ for real-time, performance-critical code: sensor interfaces, edge inference, tracking and fusion
- Python for model development, data processing, analysis and tooling
- NVIDIA Jetson AGX for edge AI inference on board the vehicle
- Computer vision on day and night camera streams
- Radar processing, multi-object tracking and sensor fusion
Results
- Pedestrian detection that works around the clock, by combining day camera, night camera and radar data
- Real-time AI inference running fully on board the vehicle
- A software stack delivered as one integrated system together with custom electrical and mechanical subsystems
Building software for a system like this takes more than good models. It takes engineering discipline across hardware boundaries, real-time constraints and multiple teams. If you are building a vehicle perception, sensor fusion or edge AI product, we would be glad to talk.
