Help needed from machine learning developer

Hey guys,

I decided not to post this in "Hire a freelancer" section because for now I just need to understand if it's even possible to implement this.

So here it goes:

Let's take an example of Quora. Duplicate question is a big issue on Quora as there are tens of different ways to ask the same thing. Here's an example;

Q. How old are you?
Q. What is your name

Despite having all different words, the question is essentially the same.
Now Quora has a feature of merging such questions however it's not right neither sustainable to allow the users to merge anything they want. This has opened a gateway for aged users to limit their competition by merging other questions with their own.

Now, is it possible to somehow detect the similarities between the questions and only allow unique questions to be posted? This should obviously require ML.

Bump......

USA282 said:

Hey guys,

I decided not to post this in the "Hire a freelancer" section because, for now, I need to understand if it's even possible to implement this.

So here it goes:

Let's take an example of Quora. Duplicate question is a big issue on Quora as there are tens of different ways to ask the same thing. Here's an example;

Q. How old are you?
Q. What is your name

Despite having all different words, the question is essentially the same.
Now Quora has a feature of merging such questions. However, it's not right neither sustainable to allow the users to merge anything they want. This has opened a gateway for aged users to limit their competition by joining other questions with their own.

Now, is it possible to somehow detect the similarities between the questions and only allow unique items to be posted? This should require ML.

Click to expand...

It would come under natural language processing, so the computer has to know the meaning of each question that has been asked. Quora might have an algorithm for detecting them.
It's a bit too complicated for a beginner, I suppose. You might need a large dataset with similar questions. Try doing projects on natural language processing on python.

I am not a machine learning programmer, but...

Q. How old are you?
Q. What is your name?

Click to expand...

these two are different questions.. #justsaying

Gogol said:

I am not a machine learning programmer, but...




these two are different questions.. #justsaying

Click to expand...

Hahah thanks for pointing out, I meant "what is your age"

USA282 said:

Hahah thanks for pointing out, I meant "what is your age"

Click to expand...

Okay, that makes sense.

You should check word2vec or other vector similarity algo's

Hi Amateur ML learner this side , Basically you need Natural language processing we have different set of algorithms over here in your case i believe NER name entity recognition tagging should be used to identify the context of every words after cleaning the data and i believe there must .In the world of Deep learning now there are few open source pre-trained model gets introduced which you can customise and train.See these Link google BERT :
https://www.quora.com/How-do-I-cluster-texts-using-BERT

https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a
Good Luck