Let's Make An AI Content Generator Based On GPT-2 (The OpenAI Model)

The Doctor

Senior Member
Joined
Dec 18, 2010
Messages
1,110
Reaction score
474
1556422286.png

GPT-2 (Recently discussed here) is currently the state-of-the-art machine learning model for text generation agents. The previous was probably BERT. OpenAI decided to not release their fully-trained model. Instead, they released a retarded (Literally) version. In order to train a model comparable to the one they didn't release would cost a lot of money - or would it? I may have a way to do it for free if I can jerry-rig Google Colab.

This thread is for the discussion of how to approximate, to the best of my/our ability, a kick-ass GPT-2.

I have setup a Github to accompany this thread here: https://github.com/TheDoctorAI/aiwriter

Contributions welcomed.
 
Awesome idea and possibly a nice upcoming share.

You got me hooked
 
They didn't release the trained model for obvious reasons. Someone said that even if we trained the bot it will not produce content with SEO in mind, and I have to say that content can be fooled because the algorithm is just an algorithm. Take the lorem ipsum rhinoplasty, for example, the guy figured that if you put certain words X amount of times in X places you will rank.
 
I had downloaded this on a Ubuntu VM. The text generated is mostly fiction and made-up. I mean if I wanted to write an article on something then how do I know if the algorithm is churning out text based on the actual knowledge of the given topic with facts and research or just making things up and writing fictional stories. Anyway someone has made it simpler to retrain this with colaboratory notebook on a GPU (https://www.reddit.com/r/MachineLea...ython_package_to_easily_retrain_openais_gpt2/).
I am not a programmer but just try to read information and tutorials available on the net till the point I am able to comprehend. I usually watch Siraj videos to remain updated about the AI world (https://www.youtube.com/channel/UCWN3xxRkmTPmbKwht9FuE5A) but mostly things go above my head. :confused:
 
Well, based on the fact that Google loves written content I think it has a really good use case, as far as you don't want to sell as content.
All of the big media companies utilize AI for writing, but I don't think it should be the game of big fishes.
Is it too dangerous? Just for big companies, not for people.
 
I've gotta put a pin in this one for a while. I've just got too much going on at the moment. I'll continue to update the AI-Trader thread though.
 
Time to take the pin out. OpenAI has released the more advanced/trained model: https://openai.com/blog/gpt-2-6-month-follow-up/ | demo based on it: https://talktotransformer.com/

As I have been so busy these past few months, I was unaware of this until watching the recently-posted 2 minute papers:



Things have changed for me. I now have programmers at my disposal and some ambitious business-people in my circle. Let's see if we can get this done.

Aside: The open-source crypto trading bot project has been suspended for now since I was recently hired by a hedge fund.
 
I have been experiementing with this for some time now. As someone pointed out above they have now released the larger 774M model which is chunky. He is an example;

=================

Stephen King is taking his thrill ride through the existential terror of "The Shining" to Broadway.

The producer Kasper Culkin is partnering with producer Craig Silverstein in a new production of "The Shining" — a high-concept affair that again thrusts King into the center of one of the most divisive and anxiously debated collections of horror tales such as "It" or "The Shining."

The nonprofits that originally commissioned the Stanley Kubrick adaptation are behind it, along with New York-based American Theatre Wing.

"It's a thrill to be working with a group of people who are so passionate about this show, and in such great state of mind," King told The Post Tuesday. "We're building a workingist theater here up in Chinatown, and people are bringing in more and more of these actors unheard of. It's beautiful to grow concert series and musicals here. It's fantastic."

Added Culkin: "It's extremely thrilling because we have found ... another audience who love this and are doing all of the right things to help us get there."

The production will premiere Sept. 22 during a sold-out reprising of "Falls Home," the Tony-winning play by the revival's original book-turner Renée Elise Goldsberry.

As for an untitled King miniseries, the master author declined comment, but his film "Rise of the Planet of the Apes" is currently undergoing reviews as it ghosts the tightly wound horror for a November 30 bow.

The Hollywood Reporter first said in January that Culkin, Silverstein and Hawkins Productions co-president Jeremy Kleiner were planning to execute the new production in New York, as well as have it showcased around the world.

But it was unclear if Silverstein and his minority investors had been seen any of the needed details, such as the staging location, that had generated interest before about a foot on Broadway.

A rep for the director, Lucas Haden Church, affirmed the planning.

The team has also heard formal offers to develop secret deals, some perhaps made by crossover investors, that might increase its theater count, so it can keep its creative control.

===================================

Its not bad but does draw upon some truths within its writing structure (there are plans for a SK broadway production). The trick here is to fine tune to your particular niche and this is currently what I am working on. Makes sense to gather a *hit ton of good quality material from lets say the toenail fungus niche, and format it nicely in order to run. You can then create different models for your different niches....

Word of warning here though. I see they (The GPT-2 folks) have also created a means to identify AI generated content. I guess this makes sense given how good this could get - I can see in the future this code to sniff out GPT-2 content will be provided to the big G in order to filter any content the crawler detects as AI generated :(
 
I have been experiementing with this for some time now. As someone pointed out above they have now released the larger 774M model which is chunky. He is an example;

=================

Stephen King is taking his thrill ride through the existential terror of "The Shining" to Broadway.

The producer Kasper Culkin is partnering with producer Craig Silverstein in a new production of "The Shining" — a high-concept affair that again thrusts King into the center of one of the most divisive and anxiously debated collections of horror tales such as "It" or "The Shining."

The nonprofits that originally commissioned the Stanley Kubrick adaptation are behind it, along with New York-based American Theatre Wing.

"It's a thrill to be working with a group of people who are so passionate about this show, and in such great state of mind," King told The Post Tuesday. "We're building a workingist theater here up in Chinatown, and people are bringing in more and more of these actors unheard of. It's beautiful to grow concert series and musicals here. It's fantastic."

Added Culkin: "It's extremely thrilling because we have found ... another audience who love this and are doing all of the right things to help us get there."

The production will premiere Sept. 22 during a sold-out reprising of "Falls Home," the Tony-winning play by the revival's original book-turner Renée Elise Goldsberry.

As for an untitled King miniseries, the master author declined comment, but his film "Rise of the Planet of the Apes" is currently undergoing reviews as it ghosts the tightly wound horror for a November 30 bow.

The Hollywood Reporter first said in January that Culkin, Silverstein and Hawkins Productions co-president Jeremy Kleiner were planning to execute the new production in New York, as well as have it showcased around the world.

But it was unclear if Silverstein and his minority investors had been seen any of the needed details, such as the staging location, that had generated interest before about a foot on Broadway.

A rep for the director, Lucas Haden Church, affirmed the planning.

The team has also heard formal offers to develop secret deals, some perhaps made by crossover investors, that might increase its theater count, so it can keep its creative control.

===================================

Its not bad but does draw upon some truths within its writing structure (there are plans for a SK broadway production). The trick here is to fine tune to your particular niche and this is currently what I am working on. Makes sense to gather a *hit ton of good quality material from lets say the toenail fungus niche, and format it nicely in order to run. You can then create different models for your different niches....

Word of warning here though. I see they (The GPT-2 folks) have also created a means to identify AI generated content. I guess this makes sense given how good this could get - I can see in the future this code to sniff out GPT-2 content will be provided to the big G in order to filter any content the crawler detects as AI generated :(

Are you going to be offering a service for this....or is it just for your own use?.

Also, you mention that the GPT-2 developers have coded into it a means of discovery....what type of warning is this?....

Thanks.
 
No, just for my own uses i’m afraid at present.

It’s all open source though so get yourself to the links above and with a bit of testing anyone can get it up and running.

They have been very light on details but I think it’s a reverse engineering of the AI which makes sense that somehow there are markers generated which can be identified but as I say, they aren’t giving much detail in order to keep it in the back pocket probably. They seem obsessed with control given the power of it which is understandable to avoid fake news/content polluting the intraweb!
 
No, just for my own uses i’m afraid at present.

It’s all open source though so get yourself to the links above and with a bit of testing anyone can get it up and running.

They have been very light on details but I think it’s a reverse engineering of the AI which makes sense that somehow there are markers generated which can be identified but as I say, they aren’t giving much detail in order to keep it in the back pocket probably. They seem obsessed with control given the power of it which is understandable to avoid fake news/content polluting the intraweb!
It's by Harvard & IBM Watson... you can find more here:
https://www.technologyreview.com/f/614021/a-new-tool-uses-ai-to-spot-text-written-by-ai/
 
This is a legitimately incredible idea. And a fascinating concept to follow! I look forward to keeping up with it! You're a boss
 
Wouldn't the detection stop working once the generation becomes advanced enough to actually replicate how humans write like?

"it exploits the fact that AI text generators rely on statistical patterns in text, as opposed to the actual meaning of words and sentences"
You assume

- AI generation models will improve, yet the detection won't? Obviously the current way of detecting it would fail if these issues were addressed without making any changes to the detection model. You can be sure that both will advance and then it remains to be seen how they can detect and also when will we actually see content that does not just make sense grammatically and uses proper punctuation, but also is fully coherent.
 
This will never be shared as the person who develops it will have a goldmine, don't be ridiculous. There are Ai writing tools out there in the wild gaming the system as we speak. It will only be publicly available once AI is commonplace enough that it is no longer an edge. I'm a developer. I could make AI Writer. Real AI Writer is a whole different story, the people who are at the top of the market are at the top of the game period.
 
Last edited:
It is super expensive to train these RIGHT. I know a funded team that has been training up their model for a while now. The content still takes 30 min of human editing per 1000 words to make HIGH quality. They will eventually get there to where maybe you need 10min of human editing. But its expensive.
 
I have been doing some experimenting with the algorithm. I have noted something interesting. I am able to output a list of probable text continuations, giving each a probability score of being the best one. Within that list, one of those sentences is almost always the perfect sentence to continue with - like really frackin good. The problem is that often, the best continuation isn't rated with the highest probability. I could choose manually but that would sort of defeat the purpose. It would allow me to write an article in like a minute that would normally take an hour but I don't want any input to be necessary. More to come.
 
Back
Top