At the recent Embedded Vision Summit, Expedera chief scientist and co-founder Sharad Chole detailed LittleNPU, our new AI processing approach for always-sensing smartphones, security cameras, doorbells, and other consumer devices. Always-sensing cameras persistently sample and analyze visual data to identify specific triggers relevant to the user experience. Always-sensing enables a host of new use cases, such as:
- Facial recognition for secure access
- “Find a face” detection, which turns the device’s display on, down, or off for power saving
- Gesture recognition enabling innovative UX control and operability
- Motion detection for bandwidth-friendly security
- Object detection, which captures events or triggers
- “Shoulder surfing” alerts for better privacy
Like always-listening applications like Siri and Alexa, always-sensing cameras enable a seamless, more natural user experience. Always-sensing audio and video must be processed locally on the device to provide the rapid response times necessary for a satisfactory user experience. However, always-sensing cameras require specialized processing due to the quantity and complexity of data generated. Since ISPs/DSPs can’t deliver the processing necessary in a power- or area-friendly way, OEMs are turning to specialized AI engines.
Power, area, and privacy are real concerns with always-sensing. Let’s start with power. Despite ongoing improvements in energy storage density, next-generation devices always place increased demands on batteries. Even wall-powered devices face scrutiny, with consumers, businesses, and governments demanding lower power consumption. To succeed, the always-sensing subsystem (camera, processor NPU, and associated memory) must consume the lowest amount of power possible. Moving to area requirements—smaller is better, both from a design perspective and as a function of the cost of silicon. Area needs to be considered as a function of the subsystem, not just the processor, but also the required memory. Privacy and data security are also significant concerns; always-sensing systems need to be architected to securely capture and process data from the camera without storing or exposing it. Additionally, the always-sensing subsystem must work hand in hand with the device’s security protocols to best protect user data.
So how can always-sensing be enabled in a power, latency, and privacy-friendly method? AI processing is best done via NPUs (Neural Processing Units). And while many existing Application Processors (APs) have NPUs inside of them, those NPUs aren’t the ideal vehicle for always-sensing. We’ll explain…
A typical AP is a mix of heterogeneous computing cores, including CPUs, ISPs, GPU/DSPs, and NPUs. Each processor is designed for specific computing and potentially large processing loads. For example, a typical general-purpose NPU might provide 5-10 TOPS of performance, with a typical power consumption of around 4 TOPS/W and about 40% utilization. However, it is inefficient because it must be somewhat overdesigned to handle worst-case workloads.
Here’s the rub, though—as discussed, always-sensing processing requires the lowest, most efficient power consumption possible. Because of that, always-sensing neural networks are specifically created to require minimal processing, typically measured in GOPS — GOPS being one-thousandth of TOPS. While the NPU in an existing AP is capable of always-sensing AI processing, it’s not the right choice for various reasons. First, power consumption will be higher, which is a non-starter for an always-on feature since it translates directly to reduced battery life. Second, low latency is essential for instant user interaction responses. Since AP-based NPU is typically busy with other tasks, other processes can increase latency and negatively impact the user experience. Finally, privacy concerns essentially preclude using the application processor. This is because the always-sensing camera data needs to be isolated from the rest of the system and must not be stored within the device or transmitted off the device. This is necessary to limit the exposure of that data and reduce the chances of a nefarious party stealing the data.
The solution, then, is a dedicated NPU specifically designed and implemented to process always-sensing networks with an absolute minimum of area, power, and latency: the LittleNPU. While the LittleNPU offers a comparatively modest 500 GOPS to 1 TOPS processing, this level of processing is ideally matched to always-sensing needs. It uses Expedera’s market-leading 18 TOPS/W OriginTM architecture, ensuring that power consumption is at the absolute minimum, preserving battery life. We estimate that most always-sensing networks may require only 10-20mW active power consumption. LittleNPUs are ideal from a latency perspective as well. Since they are implemented to process only the always-sensing networks, there is no contention with other processes within the device. This enables the absolute lowest latency and best user experience. Finally, all camera data for always-sensing is kept within the boundary of the always-sensing LittleNPU subsystem, providing better user privacy and a smaller footprint for security implementation. Still, this approach does not absolutely guarantee security, and appropriate device-level hardware and software security measures should be enacted to protect this information.
Expedera’s Origin E1 LittleNPU IP can be implemented in multiple ways. For example, many OEMs implement a near-sensor processing architecture using a dedicated chip that places the LittleNPU with the image sensor into a single subsystem (#1 below). Others may go the co-processor path, where an NPU is combined with an ISP on a discreet chip containing all necessary memory within the device (#2). Ultimately, we foresee that a LittleNPU will be fully integrated into the AP, where the subsystem occupies a dedicated area and has its own power island and memory (#3).
LittleNPU is shown in red.
Always-sensing is a logical evolution of smartphones, security cameras, home appliances, and other consumer devices. However, to be successful, it must be architected to preserve battery life and privacy while delivering key user experience improvements.
For more information on LittleNPU solutions, visit Expedera’s E1 page or contact us, and we’ll be in touch!