FastChat 竟然真的可以单机用,却也没法用
<p><a href="https://github.com/lm-sys/FastChat" rel="nofollow">https://github.com/lm-sys/FastChat</a> 我单机用了 51g 内存跑起来了,就是速度慢得离谱,毕竟是 CPU,用英文对话可以一个字一个字得往外蹦,差不多一个词 10s,我这还是 8700,要是换个更好的可能速度更快. </p>
<p>中文就更慢了,不知道是卡了还是怎么回事,20 分钟不出一句话,huggingface 里的 demo 倒是可以中文.</p>
<p>这是跑了几分钟的结果</p>
<pre><code class="language-shell">Human: write a vue demo
Assistant: Here is a simple demo of a Vue.js component that displays "Hello, World!" in the center of the screen:
<template>
<div>
<h1>Hello, World!</h1>
</div>
</template>
<script>
export default {
mounted() {
console.log('The component has been mounted.');
}
}
</script>
This component uses Vue's template syntax to define the HTML structure of the component, and the script section contains
the Vue logic for the component. The `mounted` function is called when the component is mounted, and it logs a message to the console. When the component is used in a Vue application, the `Hello, World!` message will be displayed in the center of the screen.
(用时 10 分钟)
Human: "你好" translate to English
Assistant: "你好" is the Chinese word for "hello." In English, it is pronounced as "you hao."
(用时 2 分钟)
Human: <水浒传>是谁写的
Assistant: ravinder singh
Human: Who is that?
Assistant: Ravinder Singh is an Indian author. He is known for writing the novel "The Water's Edge," which tells the story of a young woman's journey to find love and fulfillment in the midst of challenging circumstances. The novel has been
translated into multiple languages and has received critical acclaim.
(用时 5 分钟)
</code></pre>
<p>后面结果全错了,小模型想比大模型确实在"知识量"上有差距,不过我对未来的 LLM 非常有信心,我觉得显卡厂可以出一张 AI 专门卡,超大显存就为加载 LLM,但是不带训练功能,只为个人用户本地运行大语言模型</p>
<p>PS:模型还需要先从 LLaMA 加权重转换,这时候内存占用超过 60g 了,差一点都不行,压缩倒是 51g 就可以了.</p>
收藏(0)
分享
相关标签：
注意：本文归作者所有，未经作者允许，不得转载
