Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators

Research Goal: Spatio-Temporal Convolutional Neural Networks (ST-CNN) allow extending CNN capabilities from image processing to consecutive temporal-pattern recognition. Generally, state-of-the-art (SotA) ST-CNNs inflate the feature maps and weights from well-known CNN backbones to represent the additional time dimension. However, edge computing applications would suffer tremendously from such large computation/memory overhead. Fortunately, the overlapping nature of ST-CNN enables various optimizations, such as the dilated causal convolution structure and Depth-First (DF) layer fusion to reuse the computation between time steps and CNN sliding windows, respectively.

This research is being carried out in a joint project with Bosch within the EU Marie-Curie Project I-SPOT, within the application domain of automotive acoustic perception.

Gap in SotA: A large amount of computations can be saved from such joint workload topology-scheduling optimization without the need for any network retraining. However, they all lack the full consideration of ST-CNN features: On the one hand, the dilated causal convolution structure is rigid when optimizing a batch of frames. On the other hand, the existing Depth-First (DF) layer fusion optimizers are not capable of handling inter-frame causal overlaps and exploring better layer-fusion ranges. 

Recent results: This project has yielded a unified optimization framework, ACCO, to perform joint design space exploration (DSE) on the workload transformation, hardware computation scheduling, and memory allocation. ACCO proposes the formal extension of the DF design space towards ST-CNN workloads with an automatic layer-fusion transformation. The performance is verified on four case studies, including ablation studies and SotA comparisons to demonstrate ACCO’s strength, optimizing representative ST-CNN models for both the single and batch frame(s).