steve123_
Registered Member
- Nov 29, 2020
- 66
- 59
AI writing tools are a common discussion topic here on BHW; @MisterF periodically mentions that he uses Jarvis as a part of his content creation process and @jonnyah also mentioned how he is using GPT-3 to generate content for one of his websites.
The problem that I have ran into with many of the GPT-3 based writing tools on the market is that many do not have API access. Those with API access usually charge a fee + monthly subscription on top of the initial subscription price which discourages me from trying mass automation experiments.
Fortunately, there is a group of researchers developing GPT-3 alternatives that are open source and available for everyone on GitHub. This group, EleutherAI, is most popular for their GPT-3-like model GPT-Neo. This uses a transformer (machine learning model) which is notably also used by OpenAI for GPT-3 and Google AI. I was only able to find a few mentions of GPT-Neo on BHW. I would guess the reason is because GPT-Neo underperformed when compared to OpenAi's GPT-3.
This chart is taken from the official GPT-Neo GitHub page. It is sorted from least performant in linguistic reasoning to the highest. To put it simply, GPT-Neo performs worse than even the lowest quality OpenAI GPT-3 model, Ada. Refer to the image below to see the four models that OpenAI offers, all with varying ability and pricing.
The EleutherAI team has recently released a model they call GPT-J-6B which has caught my attention. Connor Leahy, one of the founding members of EleutherAI, said that his team believes that their new GPT-J-6B model is "sized similar to the Curie model of OpenAI" with around six billion parameters. Parameters are basically how the AI was trained (Larger parameter database = more sophisticated AI). Take a look at this video that compares GPT-2, GPT-3, and GPT-J-6B responses to general questions. Read the description of the video for important information about his test.
The video shows that GPT-J-6B outperformed both GPT-2 and GPT-3, however, it is important to remember that this only shows the ability of GPT-J-6B with general questions. The EleutherAI team has an official page in which you can try out the content generation capabilities of their model. I have found that you really need to mess around with the TOP-P and Temperature sliders to get good outputs. These two sliders seem to be very important in determining whether the model will create gibberish or readable content. Take a look at these images below.
These are using the default TOP-P and Temperature settings. We can obviously see that this is inaccurate and somewhat random. Romney and Obama are most definitely not the current presidents of the United States.
Changing only the TOP-P value from 0.8 to 0.77 results in a much better output. While I did not do multiple trials and run a proper experiment, it is safe to assume that these two sliders are essential in determining the quality of the output. This is even written about in their official GitHub pages. Unfortunately the answer is wrong about Trump but I would guess that this is due to the data that the model has been trained on.
The most appealing thing about GPT-J-6B is that it is open source. With the ability to implement this model into your content generation work flow, and even the ability to train the model with your own data with sufficient resources, I really do think that this is something that some of the more code-savvy users on BHW take a look at.
If any of you guys have ideas about possible implementations and projects, I would love to hear them! AI is so fascinating and I am so interested in the bright future it has.
The problem that I have ran into with many of the GPT-3 based writing tools on the market is that many do not have API access. Those with API access usually charge a fee + monthly subscription on top of the initial subscription price which discourages me from trying mass automation experiments.
Fortunately, there is a group of researchers developing GPT-3 alternatives that are open source and available for everyone on GitHub. This group, EleutherAI, is most popular for their GPT-3-like model GPT-Neo. This uses a transformer (machine learning model) which is notably also used by OpenAI for GPT-3 and Google AI. I was only able to find a few mentions of GPT-Neo on BHW. I would guess the reason is because GPT-Neo underperformed when compared to OpenAi's GPT-3.
This chart is taken from the official GPT-Neo GitHub page. It is sorted from least performant in linguistic reasoning to the highest. To put it simply, GPT-Neo performs worse than even the lowest quality OpenAI GPT-3 model, Ada. Refer to the image below to see the four models that OpenAI offers, all with varying ability and pricing.
The EleutherAI team has recently released a model they call GPT-J-6B which has caught my attention. Connor Leahy, one of the founding members of EleutherAI, said that his team believes that their new GPT-J-6B model is "sized similar to the Curie model of OpenAI" with around six billion parameters. Parameters are basically how the AI was trained (Larger parameter database = more sophisticated AI). Take a look at this video that compares GPT-2, GPT-3, and GPT-J-6B responses to general questions. Read the description of the video for important information about his test.
The video shows that GPT-J-6B outperformed both GPT-2 and GPT-3, however, it is important to remember that this only shows the ability of GPT-J-6B with general questions. The EleutherAI team has an official page in which you can try out the content generation capabilities of their model. I have found that you really need to mess around with the TOP-P and Temperature sliders to get good outputs. These two sliders seem to be very important in determining whether the model will create gibberish or readable content. Take a look at these images below.
These are using the default TOP-P and Temperature settings. We can obviously see that this is inaccurate and somewhat random. Romney and Obama are most definitely not the current presidents of the United States.
Changing only the TOP-P value from 0.8 to 0.77 results in a much better output. While I did not do multiple trials and run a proper experiment, it is safe to assume that these two sliders are essential in determining the quality of the output. This is even written about in their official GitHub pages. Unfortunately the answer is wrong about Trump but I would guess that this is due to the data that the model has been trained on.
The most appealing thing about GPT-J-6B is that it is open source. With the ability to implement this model into your content generation work flow, and even the ability to train the model with your own data with sufficient resources, I really do think that this is something that some of the more code-savvy users on BHW take a look at.
If any of you guys have ideas about possible implementations and projects, I would love to hear them! AI is so fascinating and I am so interested in the bright future it has.
