Yi Zhu

🌟 Boost Meta's Llama 3.3 70B inference throughput by 3x with Tensor RT-LLM optimizations including speculative decoding, in-flight batching, KV caching, custom FP8 quantization, and more.🦙 Technical deep dive ➡️

8

𝐌𝐮𝐬𝐭-𝐫𝐞𝐚𝐝, 𝐟𝐫𝐞𝐞 𝐛𝐨𝐨𝐤 𝐭𝐡𝐚𝐭 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐬 𝐚 𝐩𝐚𝐭𝐡𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐆𝐀𝐈 𝐭𝐨 𝐀𝐆𝐈! ​ Yann LeCun has expressed skepticism about the potential of LLMs to achieve Artificial General Intelligence (AGI), citing limitations in memory, planning, and grounding in real-world understanding. However, we argue that 𝗟𝗟𝗠 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗟𝗖𝗜)—particularly with multiple multimodal LLMs working together—offers a promising architecture to address these challenges and progress toward AGI. ​ Two key architectural innovations underlie LCI: ​ 𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐔𝐧𝐜𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐚𝐧𝐝 𝐂𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐑𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐧𝐧𝐢𝐧𝐠: Inspired by the human mind’s dual-layer structure, LCI divides its architecture into foundational and adaptive layers. The foundation layer, akin to unconscious processing, leverages extensive datasets to build robust pattern recognition, encoding essential responses much like human instincts. The adaptive layer, mirroring conscious thought, enables rapid adaptation and contextual reasoning. This dual-layer approach allows LLMs to perform complex tasks with foundational knowledge while adjusting to new information on the fly, much as a child learns new concepts within a developed cognitive framework. ​ 𝐃𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐑𝐨𝐥𝐞𝐬 𝐟𝐨𝐫 𝐄𝐚𝐜𝐡 𝐋𝐋𝐌-𝐀𝐠𝐞𝐧𝐭: Each LLM-agent is assigned a specific role, such as executive (knowledge processing), legislative (behavior guardrails), and judicial (adapting to cultural norms). ​ Additionally, our validation of the LCI framework has been thoroughly tested through real-world deployment in applications across various domains, including healthcare, sales planning, investment, and debiasing news. These empirical deployments highlight the practical viability and adaptability of the LCI framework, showcasing its ability to enhance reasoning and decision-making through collaborative intelligence. ​ The eleven aphorisms presented in this free book illustrate the philosophical foundation of the LCI framework, backed by empirical studies. Through LCI, we envision a clear pathway to AGI, rooted in collaborative intelligence and flexible, context-aware adaptation. https://lnkd.in/gn74DkNj (free) https://lnkd.in/gszSvfqP ($6.98 printing fee)

9

We are thrilled to announce our latest paper, now available on [arXiv](https://lnkd.in/grgvEftD)! 📚 🔬 Title: DPSNN: Spiking Neural Network for Low-Latency Streaming Speech Enhancement In our recent work, we tackle a crucial issue in speech enhancement SNN solutions: latency. While neuromorphic hardware is designed for low latency, some algorithms and implementations can still introduce significant delays. In speech enhancement, the Short-Time Fourier Transform (STFT)—a common preprocessing step in frequency-domain approaches—can be a significant source of latency. Inspired by the success of high-performance, low-latency deep learning models, we have developed a novel time-domain SNN framework that achieves the very low latency required for applications like hearing aids. Key Contributions of Our Paper: 1. Innovative Solution: We introduce a novel two-phase time-domain streaming SNN framework that effectively addresses latency while ensuring high accuracy and power efficiency. 2. Latency Optimization: Traditional methods often suffer from latency due to long sampling windows, such as 32ms. Our time-domain approach significantly reduces this latency, meeting the stringent requirements of real-time applications like hearing aids, which demand latencies under 5ms. 3. Competitive Performance: Our framework not only reduces latency but also achieves competitive performance compared to current SNN models, pushing the boundaries of what’s possible in speech enhancement. Explore the full details of our work on [arXiv](https://lnkd.in/grgvEftD) and discover how our innovations are advancing the practical applications of neuromorphic computing in this vital field. We look forward to your feedback and discussions! #SpeechEnhancement #SNN #LatencyReduction #DeepLearning #NeuralNetworks #RealTimeProcessing #AI #TechInnovation #arXiv #HearingAids

25

2 Comments

Check out the latest research on high-performance graph processing from a hardware/software co-design perspective! This paper dives deep into innovative strategies to optimize computational efficiency. https://lnkd.in/gUXumc5x 廖小飞 Hai Jin Long Zheng Zhiyuan Shao

Dear Colleagues, We kindly invite you to read our recent paper, which delves into the development and application of MobiScout, an iOS application designed to monitor users' driving habits and physiological conditions on the road. This app offers a cost-effective, next-generation solution for data collection and analysis in naturalistic driving studies. MobiScout captures real-time data, encompassing physiological information, using sensors on mobile phones, smartwatches, and Bluetooth-enabled OBD equipment. The software records data such as speed, braking, pulse rate, and acceleration, while the phone's camera captures both the interior and exterior views of the car. Data can either be streamed to cloud storage in real-time or stored locally in areas without WIFI. The acquired data serves to enhance our understanding of typical traffic behavior, driver performance, and surrounding conditions. If your research aligns with our work, we would greatly appreciate your citation of our paper. Your support and acknowledgment would be immensely valuable to us. Thank you for your time and consideration. #Swift, #IOS, #OBD, #Naturalistic_DrivingStory #UI #paper #hci #research

8

1. 🚀 TextGrad uses text feedback for automatic differentiation, offering a new method for prompt optimization in large language models, effective in Q&A, molecular optimization, and radiotherapy planning. 2. 🔧 DSPy provides the algorithmic foundation for TextGrad to optimize LLMs' prompts and weights, reducing the need for manual adjustments. 3. 🌐 Automatic differentiation is crucial in deep learning, and TextGrad applies it to the text domain, optimizing through text feedback. 4. ✨ ProTeGi leverages natural language gradients and prompt editing to effectively optimize LLMs' prompts, finding the optimal configuration. 5. ❓ Despite TextGrad's innovation, its practical application may be limited by the performance of the teacher model and the need for extensive API calls. #AI #TextGrad #Innovation #DSPy #LLMOptimization #Algorithm #AutoDiff #DeepLearning #ProTeGi #PromptOptimization #AIOptimization #AIChallenges #TechLimitations

1

OpenCoder: An Open-Source Code LLM Redefining Model Efficiency 🌟 The OpenCoder team has launched a game-changer in the world of code language models. Unlike many LLMs that focus on sheer size, OpenCoder brings something unique to the table: a data-focused training approach. By putting quality before volume, OpenCoder’s 1.5B and 8B models rival some of the biggest players, like DeepSeekCoder and QwenCoder, with only a fraction of their token count. The core strength of OpenCoder lies in its meticulous data processing pipeline and two-stage fine-tuning. With a training dataset composed of 2.5 trillion tokens, 90% raw code and 10% code-related web data, OpenCoder refines every aspect of its corpus. Advanced filtering and deduplication strategies, applied to 607 programming languages, optimize model efficiency and precision, ensuring high-quality data drives its training. Key Features: - Complete Training Transparency: Open-source datasets and training pipelines enable reproducibility and further research. - Instruction Fine-Tuning: Two-stage tuning sharpens both core CS concepts and real-world coding skills, prepping OpenCoder for production-level code generation. - Available on Ollama: Accessibility in deployment means more avenues for experimentation and improvement. OpenCoder's data-driven strategy showcases the potential of a leaner approach. By focusing on quality data rather than quantity alone, OpenCoder achieves file-level deduplication that outperforms traditional repo-level methods, making it both efficient and robust. 🚀 Read more here: https://lnkd.in/eEnG3yJA #OpenSource #MachineLearning #CodeLLM #AI #ModelEfficiency #DataQuality #Innovation #CodeGeneration #OpenCoder #TechNews #AIResearch #Programming #LargeLanguageModels #ArtificialIntelligence

1

Our paper "EffFix: Efficient and Effective Repair of Pointer Manipulating Programs " has been recently accepted for publication in ACM Transactions on Software Engineering and Methodology (TOSEM). Kudos to Yuntong Zhang, Andreea Costea, Davin McCall, Abhik Roychoudhury. This work proposes a novel program repair technique using probabilistic context-free grammar guided by static analysis. The key idea is to integrate static analysis and program repair such that the repair engine can incrementally learns what a correct patch may look like based on its effect on the symbolic heap. Experimental evaluation of fixing real world memory errors in medium to large scale subjects like OpenSSL and Linux Kernel, shows the efficiency and effectiveness of EffFix. In particular, EffFix has a fix ratio of 66% for memory leak bugs and 83% for Null Pointer Dereferences for the considered dataset. Pre-print: https://lnkd.in/gRrV43Pc #publication #TOSEM #programrepair

54

We are running the next ASTRA-sim tutorial at MICRO 2024! In this tutorial we will introduce MLCommons Chakra - a distributed ML workload schema that we have been collaboratively developing since the last 1+ year. We will present mechanisms to collect execution traces from real multi-node AI clusters and feed them to ASTRA-sim to study the impact of next generation compute and/or network fabrics. We will also introduce recent updates to ASTRA-sim: including (1) Chakra-ASTRA-sim interface, (2) the ability to evaluate *custom* collective algorithms on diverse network fabrics, and (3) study LLM-specific optimizations. Resources: - Tutorial Website: https://lnkd.in/eGy9Gs-z - ASTRA-sim: https://lnkd.in/eX-2KzJN - Chakra: https://lnkd.in/e968Xsga Presenters: Tushar Krishna (GT) William Won (GT) Joongun Park (GT) Taekyung Heo (NVIDIA) Vinay Ramakrishnaiah (AMD)

67

1 Comment

Predictions based on Machine Learning and Statistics often rely on maximal likelihood, which can trap LLMs in biases inherited from the majority opinions of training data. This problem has been substantially addressed through conditional statistics, allowing for mitigation of biases and preventing the cancellation of minority perspectives. https://lnkd.in/gMXKqwyB

7

Here is our new paper accepted in Neurocomputing journal. SITU: Stochastic input encoding and weight update thresholding for efficient memristive neural network in-situ training The Analog-to-Digital Converter (ADC) sensing and conductance update are the most power-demanding processes in the in-situ training of memristive neural networks. In this work, we propose a new thresholded weight update method in conjunction with stochastic input encoding to reduce the ADC sensing requirement and weight updates. This leads to better power and area efficiency for simple neural network models developed in situ on memristive crossbars. This is an interesting research work by Xuening Dong and collaboration between University of Toronto, James Cook University , and York University. https://lnkd.in/gY4-GgtC

57

1 Comment

Excited to share my latest research paper preprint on arXiv. 🎮🖥️ Adrenaline: Adaptive Rendering Optimization System for Scalable Cloud Gaming (https://lnkd.in/e8RwSPKD) In this work, we tackle the scalability challenge of cloud gaming services at the edge cloud, introducing Adrenaline - a new system that optimizes game rendering quality by considering both user-side service quality and server-side cost. Key highlights: • Cloud gaming at the edge cloud provides low-latency connections, but fixed compute capacity poses challenges for multi-user serving. • High-quality game frames generated on the server may be ineffective due to lossy compression during transmission by a user's network condition. • Adrenaline adaptively optimizes content generation costs on the server while considering user-side quality, reducing resource waste. • Our network-condition-aware optimization achieves up to 24% higher service quality and serves 2x more users with the same resource footprint. • Easy integration with modern game engines and existing cloud gaming service architectures Check out more details with the full paper! #CloudGaming #EdgeComputing #ResearchPaper

35

2 Comments

#ICLR2024 check out our poster #148 on "Memorization in Self-Supervised Learning Improves Downstream Generalization" from 16:30 to 18:30. @fraboeni @NicolasPapernot We propose a framework for defining memorization in SSL and design new metrics based on th alignment to measure it.

26

I am happy to share the presentation I uploaded on ResearchGate created as part of Master Engineering Systems at HAN University of Applied Sciences. It discusses the CrossVit: Cross-Attention Multi-Scale vision transformer for image classification, by starting by explaining the Cross-attention mechanism passing by the transformer in attention is all you need research paper then ViT, DeiT and elaborating on the CrossVit transformer in detail. Link to the presentation: https://lnkd.in/edg97Uba Link to the research paper: https://lnkd.in/eNFndV59 #Transformers #MachineLearning #Msc #CapitaSelecta

3

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

Journal Impact Factor JCR-WoS: 2.0 (2023) Q1 CiteScore 2023: 3.1 Q1

11

📢 Our new paper "Higher-Order GNNs Meet Efficiency: Sparse Sobolev Graph Neural Networks" has been accepted in IEEE Transactions on Signal and Information Processing over Networks! This work introduces sparse Sobolev graph neural networks (S2-GNN) to address a key challenge in scaling up GNNs. While traditional GNNs typically rely on immediate node neighbors, limiting their effectiveness in capturing complex graph relationships, S2-GNN employs higher-order, sparse filters to efficiently handle large graphs. Our key contributions: Spectral insights: We reveal a spectral similarity between higher-order Laplacians and their sparse approximations, which supports the design of high-performing yet efficient GNNs. Architecture: S2-GNN leverages cascades of sparse filters to enable accurate updates in node embeddings without excessive computational costs. Robustness: We provide stability analysis of S2-GNN, demonstrating robustness to graph perturbations. Extensive benchmarking: S2-GNN consistently outperforms or competes with leading GNN methods across diverse datasets and tasks. We hope this work contributes to advancing scalable, high-efficiency GNN architectures for real-world applications. 🔗 Preprint: https://lnkd.in/eAJJ9SAD 🔗 Code: https://lnkd.in/eWK7XZM2 Thanks to my co-authors Aref Einizade, Andjela Todorovic, Jhon Castro-Correa, Mohsen Badiey, Thierry Bouwmans, and Fragkiskos Malliaros. This work was supported by ANR, the French National Research Agency, Office of Naval Research, Hi! PARIS Center - AI for Science, Business & Society, Télécom Paris, Institut Polytechnique de Paris, University of Delaware, La Rochelle Université, CentraleSupélec, Université Paris-Saclay, and Inria. #GraphNeuralNetworks #SparseGNN

64

9 Comments

💥💥💥 GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, Ruiming Tang Abstract Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), facilitate intelligent agents being capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions by simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data, frameworks, and applications. We begin by discussing representative datasets and benchmarks. Next, we summarize a unified framework that captures the essential components used in prior research, accompanied by a taxonomy. Additionally, we explore commercial applications of (M)LLM-based GUI agents. Drawing from existing work, we identify several key challenges and propose future research directions. We hope this paper will inspire further developments in the field of (M)LLM-based GUI agents. 👉 https://lnkd.in/dtCrG5DB #machinelearning

1

Read our latest published research paper about SuperCoder

27