Faster AI chat model
- Thread starter janubaba
- Start date
jskinnerd
Regular Member
- Jul 21, 2023
- 205
- 299
Computation time follows a linear correlation to the number of parameters of said model. (Assuming equivalent hardware)
That being said, out of Llama 3 (70 B) and GPT-3 (175 B), Llama 3 would on average be more than twice as fast than the latter. Of course it should be tested experimentally for your use-case.
But regarding the speed of the response from clicking the button to receiving the message, you're now talking about network latency and bandwidth, which is is the main bottleneck here.
This is also highly subjective because it depends on your location from the server. I'd focus more on which model produces the result you're looking for over which website is faster.
That being said, out of Llama 3 (70 B) and GPT-3 (175 B), Llama 3 would on average be more than twice as fast than the latter. Of course it should be tested experimentally for your use-case.
But regarding the speed of the response from clicking the button to receiving the message, you're now talking about network latency and bandwidth, which is is the main bottleneck here.
This is also highly subjective because it depends on your location from the server. I'd focus more on which model produces the result you're looking for over which website is faster.
DarkerAds
BANNED
- May 1, 2023
- 568
- 1,040
running a LLM model on your own rig is the fastest way
samlonking8
BANNED - Failed to resolve a dispute
- Oct 1, 2023
- 152
- 51
google is faster compare to others. also i never used bing ai
bullsbears
Power Member
- Feb 17, 2014
- 646
- 223
In terms of response, I found 4o mini to be faster than 4o. Quality wise 4o is better.
Larry Igna
Power Member
- Nov 25, 2016
- 663
- 717
In terms of speed, groq.com is probably fastest and it runs open-source models only. They have specialized hardware and they claim to be the fastest in AI inference. Whatever.. they have a free tier so you can always try for yourself.
As for the big guns, Gemini from Google is also fast, at least compared to ChatGPT free. The quality is similar to open models but below ChatGPT on certain tasks. Sometimes better, sometimes worse, it depends on your use case.
If you only need speed, run a TinyLlama or other similar small model on vast.ai . But quality is submediocre.
As for the big guns, Gemini from Google is also fast, at least compared to ChatGPT free. The quality is similar to open models but below ChatGPT on certain tasks. Sometimes better, sometimes worse, it depends on your use case.
If you only need speed, run a TinyLlama or other similar small model on vast.ai . But quality is submediocre.
PURSUIT MEDIA
Elite Member
- Dec 27, 2019
- 3,133
- 827
I think this is based on preference , I mean speed can be variable, and it’s a good idea to test different models to see which one best fits your needs.
- May 13, 2013
- 6,960
- 1,959
Not anymore. Most of the time, I leave the tab to do something else until it finishes.
FrennyJay99
BANNED
- Jan 1, 2024
- 115
- 22
paid chatgpt is faster
carrotoverrated
Newbie
- Aug 26, 2024
- 17
- 3
Look into Groq. They have near instant inference speed for <10B parameter models with their specialized hardware. 10x inference speeds across the board compared to the competition. There are other labs working on it but nothing much as easily accessible.
