P2PLLMEdge: Peer-to-Peer Framework for Localized Large Language Models using CPU only Resource-Constrained Edge

Abstract

In this research, we present P2PLLMEdge, a pioneering peer-to-peer framework designed to enable localized Large Language Models (LLMs) to operate efficiently in resource-constrained edge environments, exemplified by devices such as the Raspberry Pi 4B and CPU-only laptops. The framework addresses critical challenges, including limited computational capacity, network overhead, and scalability, by leveraging lightweight RESTful communication protocols, model-specific quantization, and decentralized task distribution. Key results demonstrate that P2PLLMEdge achieves substantial performance improvements. On average, Peer 2 (CPU-only laptop) achieves a 44.7% reduction in total duration (tpeer2, total = 15.87 × 109 ns) compared to Peer 1 (Raspberry Pi 4B, tpeer1, total = 28.18 × 109 ns). The framework processes tokens at a rate of 21.77 tokens/second on advanced LLMs like Granite3.1-moe:1b, significantly outperforming the baseline. Peer 1, employing quantized LLMs such as smolm2:360m-instruct-q8_0, reduces prompt evaluation duration by 23.2% (tpeer1, prompt_eval = 0.76 × 109 ns) compared to larger models like qwen2.5:0.5binstruct (tpeer1, prompt_eval =0.99 × 109 ns). Peer 2 demonstrates superior summarization capabilities, with evaluation durations (tpeer2, eval) reduced by 72.8% (tpeer2, eval = 5.15 × 109 ns) for explanation-type prompts relative to Peer 1 (tpeer1, eval = 18.93 × 109 ns). The framework also achieves significant network efficiency, reducing inter-peer communication durations by up to 44.9% (tpeer2, network = 25.83 × 109 ns
vs. tpeer1, network = 46.92 × 109 ns). Peer-to-peer synergy ensures seamless task execution, where Peer 1 generates text and offloads computationally intensive summarization tasks to Peer 2, achieving a balance between performance and resource utilization. The novelty of P2PLLMEdge lies in its ability to seamlessly integrate lightweight LLMs with decentralized edge devices, achieving advanced natural language processing functionalities entirely on edge devices traditionally deemed unsuitable for such tasks. This framework provides an adaptable, and cost-effective approach for deploying quantized LLM-driven applications. Future directions include scaling the framework to multi-peer environments, optimizing task scheduling algorithms, and exploring integration with heterogeneous LLM-enabled systems. The codes are available on https://github.com/ParthaPRay/peer_to_peer_local_llm_interaction.