Methodology
NPU Methodology
We used a fairly standard digital methodology to design our chip as shown in Figure 1. This methodology is customized to some extent to satisfy our needs. We've been heavily relying on this methodology to realize our design, by maximizing the percentage of work that can be completed by CAD tools and minimizing manual interference. Utilizing CAD tools enabled us to achieve more complex designs, but there was a learning curve for each new tool thus we did put considerable amount of effort to figure out how to use them. Each block in Figure 1 is explained in the following section. We also included some lessons learnt when implementing this methodology.
Figure 1. Block diagram for methodology
NPU Concept: The idea of NPU comes from the fact that we want to accomplish an AI image recognition task. The entire derivation process has been described in detail in System Overview tab.
TensorFlow Model: We constructed the MLP (Multi-layer Perceptron) NN model in this step. TensorFlow(TF) has good support for many different NN structures and it is fairly easy to build the model if the target is clear. TF also has good utility for training and validation based on existing dataset. In this case we used the MNIST dataset, which includes tens of thousands of images containing a single hand-written number. There is a "normalization" logic behind all images (e.g. all 28*28, mass center overlaps with image center etc.). Please refer the online resouces for the full description.
TF model is a non-trivial step, because its results will directly affect the following steps and any high-level change in this step can cause problems in the following stages. There will be some back-and-forth since it is impossible for designer to confirm the validity of a NN structure without the verification using actual hardware design. Also the designer may need to process the dataset by themselves to suit the need of circuits. In our case we shrank the size of the image to 16*16 and do the normalization again before training and validation. Any operation like this will affect the final accuracy, PPA metrics of circuits, system-level implementation and many other factors. We highly recommend the designer remembering the "big picture" of the entire system in this step, otherwise either the accuracy derived in this stage is not informative at all, or a fancy model in TF cannot be implemented in hardware. We recommend selecting a fairly reasonable design point first, going through the following MATLAB and RTL stages to verify the actual accuracy using selected number representation, programming the RTL to have a feel of the actual circuit, maybe also pass the behavioral model to synthesizer and then to the PNR tool to have a first-order idea of the required area, power etc. so that designer will know how to tweak the TF model based on the time/area/power/ budget. For example designers figure out they still have lots of silicon area for this design point, they can scale up the design and maybe implement more parallelism, more complex datapath, more complex NN structure etc. This was actually what we did when designing the chip and we found this particular effective and efficient for non-expert designers like us.
Essentially TF provides a high-level, well-supported Python environment for designer to manipulate the NN structures, do training process and have initial validation results. Our validation result (recognition accuracy) in TF is based on 32-bit floating number, while the actual hardware used 8-bit, fixed-point, signed integer for computation, hence inherently there is a big gap. First-order validation can still be obtained from TF, and we bridged this gap using the following MATLAB fixed-point integer modelling process. We have not tried to use the Python environment to do the fixed-point modelling, but since there are lots of packages we guess it is OK to replace MATLAB with Python to complete the architecting process.
MATLAB Model: We used MATLAB fixed-point number support to model the complete architecture of our chip. This step will provide final accuracy and insights to the following RTL design. We manually created each layer in MATLAB (nested for loops) and digitized the parameters from TF (weights and biases). Once these layers are executed in the correct sequence and proper activation function (ReLU in this case) is applied, the entire model emulated the architecture of the chip. There is still some nuance since there is no timing, pipelining and bandwidth limitation in MATLAB. We recommend being particularly careful when switching between different number of bits for numbers in MATLAB. Some transition may be easy in software but difficult in hardware, and it can cause misrepresentation of the actual hardware.
Block Diagram & Datapath Structure: We created block diagrams of our chip in this step, based on the architecture established in MATLAB. This is another difficult transition since there are infinite number of possibilities and everything is limited by the computation resouce and available die area/IOs. We started from just one essential MAC module and some reset mechanism for bias. This is followed by a ReLU module to apply activation. These two blocks can already achieve the inference process of our NN. We expanded to the full architecture (covered in detail in Architecture) from this starting point. We recommend specifying as much detail as possible in this stage so that the following RTL programming will be easier.
RTL Coding: We used Verilog to describe the behavior of our blocks. This step is fairly easy if all functions, bitwidth, pipelining scheme, number representation etc. were determined in the block diagram step. Timing is a critical part in this step as the design needs to make sure every block will receive the correct signal at correct cycle, and all clock windows are maximally used for largest throughput. We designed the hardware structure first and proposed a timing diagram for correct operation. We then designed a FSM system to automatically provide the control signals to computation circuits regarding correct timing requirement. Since our datapath is designed to be general-purpose, our FSM is a programmable FSM that is adaptive to our current NN structure.
TB Verification: Our Verilog testbench is used for behavioral model, synthesized netlist, placed and routed netlist and extracted netlist. We used a statistical approach to guarantee our design by incorporating a large number of testing vectors (~10k images) and letting TB to automatically compare testing results to our golden model from MATLAB. Our system has multiple computing modes so we exhausted the possible paths for different modes in TB. Each specific test will be covered in Verification.
Synthesized Netlist: We used Genus to synthesize the behavioral model from RTL stage and used the same TB to make sure the setup and hold are still valid after back-annotation. Existing scripts from Prof. Kinget's lab were used for this process. Our target frequency spec is 10 MHz and we synthesized the netlist using 100 MHz for some margin. Since the design is pipelined, this frequency goal is easy for synthesizer to reach. The only concern in this stage is whether the behavioral model is synthesizable or not. We used typical datapath coding style when creating the RTL model for FSM, multiplier, adder, MUX, registers etc., hence we did not have issue about non-synthesizable design. Due to the fact that it is fairly easy for synthesizer to close timing, no custom datapath structures were passed to the synthesizer. We accepted the netlist solution from synthesizer without further modification.
PNR Digital Core: We used Innovus to automatically place and route the digital core based on the netlist from the synthesizer. Again, existing scripts from Prof.Kinget's lab were used, but this is not a trivial stage, in terms of both time and effort. We have not used PNR tool to generate layout before so it took us considerable amount of time to figure out how to use the TCL API to interact with Innovus. We compiled our entire design into one layout block and controlled the placing density to be ~30% for more decap. Generally we were not constrained by area too much. For non-expert designer, we recommend using an existing set of scripts as a starting point and modify the scripts per design requirement. There are thousands of settings in innovus, writing scripts from scratch will easily lead to malfunctional layout. Detail about the layout is covered in Layout.
Custom Circuits: We did design some custom circuits for our system, including buffers and a Schmitt trigger. Buffers were designed for buffering signals coming into and out of the chip. They are typical designs using a chain of inverters, but we have not used standard cells for this so that we can have more practice. Schmitt trigger was designed for the clock signal. A voltage hysteresis was created for the externally injected clock signal so that the internal clock tree will not be triggered multiple times if there are some glitches in the input clock source. These are all fully custom circuits. We designed the Virtuoso schematic and layout manually, and made sure that the driving strength, slew rate, voltage hysteresis, PVT variation satisfy our requirement. These custom circuits will be covered in Custom Circuits.
Integrated Core: We combined the digitally PNRed computing core with our custom circuits to get the integrated core. This integration was completed manually in Virtuoso layout design , and the routing between blocks, between pads, between decaps were also done manually for the final GDS. We used most of our remaining area for decaps constructed from MOM caps. The custom ESD circuits were provided by Prof.Kinget's lab for IO EDS protection. These ESD blocks were placed under the IO pads so not taking additional area. We used this combined layout for the final DRC and LVS sign-off.
AMS Verification: AMS stands for Analog and Mixed Signal. It is a simulator that allows the co-simulation of analog and digital circuits. Specifically for us, we incorporated this level of verification due to the large number of testing vectors we required. Without AMS, we had to manually set the piecewise linear voltage source after combining the custom circuits with our PNRed digital core. This was certainly unfeasible for thousands of testing images. In AMS, we created a stimulus provider block in Virtuoso schematic and initiated the block using the Verilog testbench that we designed before. The TB was modified so that it can act as a stimulus generator to provide testing vectors to our integrated chip. This enabled us to perform complex verification to the fully integrated chip (digital core + custom circuit + ESD + Decap). We confirmed the final sign-off before tape-out if calibre parasitic extracted netlist can pass this verification test. The detail is covered in Verification.