Menu
Groq Adds Responsiveness to
Inference Performance to Lower TCO
Running a batch size of one, which refers to computations on a single image or sample during inference processing, is a valuabl particularly those that require real-time responsiveness. However, small batch sizes and batch size 1 introduce a number of performance and responsiveness complexities to machine
learning applications, particularly with conventional inference platforms based on GPUs.
