{"id":387,"date":"2023-11-02T06:29:53","date_gmt":"2023-11-02T10:29:53","guid":{"rendered":"https:\/\/staging.oddbee.com\/centml\/?p=387"},"modified":"2023-11-16T09:35:09","modified_gmt":"2023-11-16T14:35:09","slug":"maximizing-llm-training-and-inference-efficiency-using-centml-on-oci","status":"publish","type":"post","link":"https:\/\/staging.oddbee.com\/centml\/maximizing-llm-training-and-inference-efficiency-using-centml-on-oci\/","title":{"rendered":"Maximizing LLM training and inference efficiency using CentML on OCI"},"content":{"rendered":"<p><em>The authors want to thank Shang Wang, CTO at CentML, Xin Li, and Zhanda Zhu, research engineers at CentML, for their contributions.<\/em><\/p>\n<p>In partnership with CentML, Oracle has developed innovative solutions to meet the growing demand for high-performance NVIDIA GPUs for machine learning (ML) model training and inference. Utilizing CentML\u2019s state-of-the-art ML optimization software and Oracle Cloud Infrastructure (OCI), the collaboration has achieved significant performance improvements for both training and inference tasks, specifically with the LLaMa-V2 and Falcon-40B models.<\/p>\n<p>Key findings indicate that OCI outperforms other public cloud service providers\u00a0because of reduced virtualization overhead, and CentML\u2019s optimizations significantly enhance performance metrics across the board. The breakthroughs have demonstrated that it is possible to achieve compelling performance even on more readily available NVIDIA Tensor Core GPUs like the\u00a0<a href=\"https:\/\/www.nvidia.com\/en-us\/data-center\/products\/a10-gpu\/\" target=\"_blank\" rel=\"noopener\">NVIDIA<\/a><a href=\"https:\/\/www.nvidia.com\/en-us\/data-center\/products\/a10-gpu\/\" target=\"_blank\" rel=\"noopener\">\u00a0A10<\/a>\u00a0Tensor Core GPU with 24GB of\u00a0 memory, offering cost-effective alternatives to top-of-the-line GPUs, such as the\u00a0<a href=\"https:\/\/www.nvidia.com\/en-us\/data-center\/a100\/\" target=\"_blank\" rel=\"noopener\">NVIDIA A100<\/a>\u00a0and\u00a0<a href=\"https:\/\/www.nvidia.com\/en-us\/data-center\/h100\/\" target=\"_blank\" rel=\"noopener\">H100<\/a>\u00a0Tensor Core GPUs.<\/p>\n<p><iframe loading=\"lazy\" title=\"Quick Guide on Hidet\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/wuS9Q6sJBEE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<h2>Background<\/h2>\n<p>The emergence of ChatGPT and LLaMa series large language models (LLMs) has demonstrated the potential of generative AI, serving as the catalyst for the ongoing AI revolution. Oracle has collaborated with customers to help them navigate their artificial intelligence (AI) and ML journey and achieve their objectives. The surge in AI has resulted in an insatiable demand for high-end GPUs, like the NVIDIA A100 and H100 GPUs, to support LLM training and inference. Given this surge, AI\/ML developers need to plan for a number of factors outside of just compute, including time-to-market constraints. As a result, Oracle, in partnership with CentML, has embarked on a mission to offer customers a greater variety of NVIDIA GPUs for AI training and inference. From the H100 GPU to the A10 GPU, developers can pick and choose the NVIDIA-powered instance that is best in -line with their own go-to-market strategy to position themselves for success.<\/p>\n<p>The combination of\u00a0<a href=\"https:\/\/centml.ai\/\" target=\"_blank\" rel=\"noopener\">CentML\u2019s<\/a>\u00a0unprecedented ML-model performance optimization software with OCI\u2019s Gen-2 cloud has\u00a0achieved remarkable performance results for NVIDIA A10 24GB GPU- optimized LLM training and inference.<\/p>\n<p>Oracle and CentML\u2019s teams have collaborated on the following use cases:<\/p>\n<ul>\n<li>Accelerate LLaMA-V2 (7B) serving<\/li>\n<li>Fine-tune training Falcon-40B on OCI instances powered by NVIDIA A10 GPUs<\/li>\n<\/ul>\n<h2>Accelerate LLaMA-V2 (7B) serving<\/h2>\n<p>Using\u00a0<a href=\"https:\/\/hidet.org\/\" target=\"_blank\" rel=\"noopener\">CentML\u2019s deep learning compiler<\/a>, we accelerated LLaMA-v2 only days after its release, demonstrating compelling performance improvements.<\/p>\n<p>In the LLama 2 inference serving performance test, measuring latency and throughput on OCI, we used the following benchmark parameters:<\/p>\n<ul>\n<li>FP16<\/li>\n<li>Prompt Length: Approximately 10 tokens<\/li>\n<li>Generated Sequence Length: 255 tokens<\/li>\n<\/ul>\n<figure id=\"attachment_388\" aria-describedby=\"caption-attachment-388\" style=\"width: 1024px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-388 size-full\" src=\"https:\/\/staging.oddbee.com\/centml\/wp-content\/uploads\/2023\/11\/Medium.jpeg\" alt=\"\" width=\"1024\" height=\"750\" srcset=\"https:\/\/staging.oddbee.com\/centml\/wp-content\/uploads\/2023\/11\/Medium.jpeg 1024w, https:\/\/staging.oddbee.com\/centml\/wp-content\/uploads\/2023\/11\/Medium-300x220.jpeg 300w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption id=\"caption-attachment-388\" class=\"wp-caption-text\">This chart shows time to complete work in seconds, where lower is better. NVIDIA A10 GPU instances from OCI deliver more compelling inference performance than competing NVIDIA A100 GPU instances from other cloud providers.<\/figcaption><\/figure>\n<p>From these tests, we gather the following takeaways:<\/p>\n<ul>\n<li>OCI is faster than other public cloud services out-of-the-box, because it eliminates virtualization overhead.<\/li>\n<li>CentML optimizations can improve LLaMA-v2 inference serving performance by 48% on OCI over its competitor and further optimize performance on OCI by 36%.<\/li>\n<li>CentML optimizations on NVIDIA A10 GPUs achieved 1.2 times better performance than NVIDIA A100 GPU performance without CentML optimizations.<\/li>\n<\/ul>\n<p>Equally as exciting as the performance results we achieved for inference serving is the performance we attained for LLM fine-tune training on the NVIDIA A10 24 GB GPU.<\/p>\n<h2>Fine-tuning Falcon-40B on OCI GU1 instance powered by NVIDIA A10 GPUs<\/h2>\n<p>As mentioned, access to top-of-the-line NVIDIA GPUs, like the A100 and H100, is even more acutely felt for fine-tuning workloads. The reason that everyone is after top-of-the-line GPUs is that you can fit the model into the memory of the chip. But what if the size of the model exceeds the available memory of the GPU? Can you run the fine-tuning workload on GPUs that are more readily available?<\/p>\n<p>Applying CentML\u2019s advanced techniques for data and model parallelism, we successfully ran a fine-tuning training workload of the Falcon-40B model on NVIDIA A10 GPUs. We established the following test case criteria for evaluation:<\/p>\n<h3>Benchmark metadata<\/h3>\n<ul>\n<li>Model: Falcon-40B<\/li>\n<li>4 GPUs<\/li>\n<li>Sequence length generated: 2,048<\/li>\n<\/ul>\n<h3>OCI GM 4 instance (powered by NVIDIA A100 GPUs) baseline test<\/h3>\n<ul>\n<li>4 x NVIDIA A100 40-GB GPUs with NVIDIA NVLink technology<\/li>\n<li>Data- parallel fine-tuning<\/li>\n<li>Per GPU throughput: 1,324 samples\/hour<\/li>\n<\/ul>\n<h3>OCI GU1 instance (powered by NVIDIA A10 GPUs) baseline test with Hugging Face native model parallelism<\/h3>\n<ul>\n<li>4 x NVIDIA A100 40-GB GPUs with NVIDIA NVLink technology<\/li>\n<li>Popular existing frameworks like Deepspeed and Megatron require users to modify the model implementation of Falcon-40B and don\u2019t support important functionality like low-bit precision and LoRA.<\/li>\n<li>Hugging Face two-way model parallelism<\/li>\n<li>Per GPU throughput: 191 samples\/hour, which is almost seven times slower than the NVIDIA A100 baseline<\/li>\n<\/ul>\n<h3>OCI GU1 instance (powered by NVIDIA A10 GPUs) baseline with CentML training framework<\/h3>\n<ul>\n<li>4 x NVIDIA A10 -24GB GPUs connected by PCIe<\/li>\n<li>No modification to the Falcon-40B model<\/li>\n<li>Per GPU throughput using two-way pipeline parallelism: 296 samples\/hour<\/li>\n<\/ul>\n<h2>Results<\/h2>\n<p>In the following table, understand the following acronyms:<\/p>\n<ul>\n<li>DP: Data- parallel fine-tuning using HuggingFace Trainer<\/li>\n<li>MP: Model- parallel fine-tuning using Huggingface Trainer<\/li>\n<li>MP+TP: Model- and data- parallel fine-tuning using open-source libraries<\/li>\n<li>CentML: A mixture of parallelization and optimization strategies devised by CentML<\/li>\n<\/ul>\n<table>\n<tbody>\n<tr>\n<td><b><strong>A100 GPU (DP)\u00a0<\/strong><\/b><\/td>\n<td><b><strong>A10 GPU (MP)<\/strong><\/b><\/td>\n<td><b><strong>A10 GPU (MP+TP)<\/strong><\/b><\/td>\n<td><b><strong>A10 GPU (CentML)<\/strong><\/b><\/td>\n<\/tr>\n<tr>\n<td>1,324 samples\/hour<\/td>\n<td>109 samples\/hour<\/td>\n<td>191 samples\/hour<\/td>\n<td>296 samples\/hour<\/td>\n<\/tr>\n<tr>\n<td>$12.20\/hr<\/td>\n<td>$8\/hr<\/td>\n<td>$8\/hr<\/td>\n<td>$8\/hr<\/td>\n<\/tr>\n<tr>\n<td>108.50 Samples\/dollar<\/td>\n<td>13.61 Samples\/dollar<\/td>\n<td>23.90 Samples\/dollar<\/td>\n<td>37.04 Samples\/dollar<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>We have the following key takeaways from our exceptional results:<\/p>\n<ul>\n<li>CentML training framework doesn\u2019t require model changes and automatically scales with model size.<\/li>\n<li>Using CentML on OCI, you can perform model fine-tuning that scales from 7B to 60B parameters.<\/li>\n<li>Consider running LLM\u2019s like Falcon-40B at scale on NVIDIA A10 GPUs \u00a0on OCI with CentML services.<\/li>\n<\/ul>\n<p>Through the application of two-way pipeline parallelism, the NVIDIA A10 GPU showcases a throughput of 296 samples\/hour. While slower than the NVIDIA A100 GPU baseline, this result reveals the untapped potential of the NVIDIA A10 GPU infrastructure.<\/p>\n<p>When combined with CentML\u2019s optimization, the NVIDIA A10 GPU\u2019s performance offers a remarkable 37.04 samples\/dollar ratio. This advantage can be a game-changer for organizations looking to maximize performance and flexibility while facing the need to scale their models immediately.<\/p>\n<h2>Conclusion<\/h2>\n<p>In an industry constrained by the limited availability of high-performance GPUs, Oracle and CentML provide a groundbreaking solution to your performance, cost, and scalability challenges. Our collaboration unlocks the untapped potential of more readily available GPUs, such as the NVIDIA A10 GPU, offering you optimized performance at a remarkable cost-benefit ratio.<\/p>\n<p>Act now to use the unmatched capabilities of CentML\u2019s optimization techniques and Oracle Cloud Infrastructure\u2019s robust NVIDIA GPU-powered cloud infrastructure.\u00a0<a href=\"mailto:support@centml.ai?subject=Re%3A%20Maximizing%20LLM%20Training%20and%20Inference%20Efficiency%20Using%20CentML%20on%20OCI\">Contact us today<\/a>\u00a0to discover how you can achieve your unique training and inference goals while maximizing performance, cost efficiency, and flexibility.<\/p>\n<h2>Authors<\/h2>\n<p><strong>Sanjay Basu PhD<\/strong><br \/>\nSenior Director &#8211; Gen AI\/GPU Cloud Engineering<\/p>\n<p>Sanjay focuses on the advanced services like Generative AI, Machine-Learning, GPU Engineering, Blockchain, Microservices, Industrial IoT, 5G core along with Cloud Security and Compliance. He has double masters in Computer Science and Systems Design. His PhD was in Organizational Behaviour and Applied Neuroscience. Currently, he is pursuing his second PhD in AI. His focus of research is Retentive Networks.<\/p>\n<hr \/>\n<p><strong>Ron Caputo<\/strong><br \/>\nOracle AI Cloud Sales Director<\/p>\n<p>Ron is a Global Field Sales Director selling Oracle&#8217;s GPU Cloud and OCI services to foundational AI\/ML companies.<\/p>\n<hr \/>\n<p><strong>Akbar Nurlybaev<\/strong><br \/>\nCo-founder and COO, CentML<\/p>\n<p>Co-founder and COO at CentML. We Make Machine Learning Training Affordable. C100 2023 Fellow.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The authors want to thank Shang Wang, CTO at CentML, Xin Li, and Zhanda Zhu, research engineers at CentML, for [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":459,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[4],"tags":[],"class_list":["post-387","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-case-studies"],"acf":[],"_links":{"self":[{"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/posts\/387","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/comments?post=387"}],"version-history":[{"count":3,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/posts\/387\/revisions"}],"predecessor-version":[{"id":412,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/posts\/387\/revisions\/412"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/media\/459"}],"wp:attachment":[{"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/media?parent=387"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/categories?post=387"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staging.oddbee.com\/centml\/wp-json\/wp\/v2\/tags?post=387"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}