How To Extract Data From A List Of URLs

[princeofmtl]

I have a list of 2000+ URLs that have only a small amount of text data in them, is there a way to automate the process and put the data into a spreadsheet? Any input would be greatly appreciated, i have been looking everywhere!
Thank you!

 

[BlackBDO]

Creating a small bot for this should cost you max $5-7.

 

[tb303]

what is the url structure look like?

if its something like url.com/page.php?param1=text1&param2=text2&etc.... then you could do it quickly with excel

Open the file in notepad and search&replace all "?" for "&" and save it as a temp file. Then open that temp file in excel, select Other as the Delimiter and put "&" in the box. You should get each parameter as a column then that looks like

param1=text1     param2=text2

You could also do a search/replace all for "=" to "&" before importing it to get it like this

param1     text1     param2     text2

for other url structures you can try different search/replaces to make it easily splitable eg "/" - just make sure the char you replace with is not valid in the text you want to extract.

 

[nobodyelsein]

I imagine scrapebox could likely do this quite easily?

Is this just a one-off or an ongoing project?

 

[thetrustedzone]

i shared tool from long time .... may help you in this https://www.blackhatworld.com/seo/get-deep-extractor.731801/ , you could use your own regex to extract what you want .....

 

[HoNeYBiRD]

It can be done even with iMacros looping through your list of urls.
I just put this macro together really quick:

VERSION BUILD=10022823
TAB T=1
TAB CLOSEALLOTHERS
SET !ERRORIGNORE NO
SET !DATASOURCE c:\urls.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !EXTRACT_TEST_POPUP NO
ADD !EXTRACT {{!COL1}}
URL GOTO={{!COL1}}
TAG POS=1 TYPE=HTML ATTR=* EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\ FILE=data.csv

Instructions:
Create an urls.csv file in the root of your c: drive (or alter the location of the data source in the macro), put one url per line in your .csv.
Play the macro in loop, if you have 2,000 urls, put 2000 in the loop counter.
A data.csv file will be saved into the root of your c: drive with the urls in the first column and with the belonging extracted data in the second column.

The macro will extract everything which is on a page, ads too, so there might be redundant data.

 

[princeofmtl]

Creating a small bot for this should cost you max $5-7.

Where can a find someone to make me a bot?

 

[princeofmtl]

It can be done even with iMacros looping through your list of urls.
I just put this macro together really quick:

VERSION BUILD=10022823
TAB T=1
TAB CLOSEALLOTHERS
SET !ERRORIGNORE NO
SET !DATASOURCE c:\urls.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !EXTRACT_TEST_POPUP NO
ADD !EXTRACT {{!COL1}}
URL GOTO={{!COL1}}
TAG POS=1 TYPE=HTML ATTR=* EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\ FILE=data.csv

Instructions:
Create an urls.csv file in the root of your c: drive (or alter the location of the data source in the macro), put one url per line in your .csv.
Play the macro in loop, if you have 2,000 urls, put 2000 in the loop counter.
A data.csv file will be saved into the root of your c: drive with the urls in the first column and with the belonging extracted data in the second column.

The macro will extract everything which is on a page, ads too, so there might be redundant data.

Looks great however when I run it I am getting
Error -1100: Wrong format of SAVEAS TYPE=EXTRACT FOLDER=C:\ FILE=data.csv command, at line: 12
What am I doing wrong?

 

[HoNeYBiRD]

Looks great however when I run it I am getting
Error -1100: Wrong format of SAVEAS TYPE=EXTRACT FOLDER=C:\ FILE=data.csv command, at line: 12
What am I doing wrong?

Nothing, i effed it up. lol

When i made the macro, i worked with different file paths, which i stripped down to c:\, when i posted the code above. It wasn't a good idea, it appears. With Win 10 i can't seem to write the root of c: for some reason (i guess security reason). What you need to do is alter the FOLDER=c:\ parameter in the last line to another location on your hard drive and it should work. Just make it to FOLDER=c:\Users\YourWindowsUsername\Desktop\ or wherever you want to save the .csv and the macro should work.

Or wait, maybe that's not even it, because i'm not getting the error. The macro simply don't run for me in the form i originally posted.

Have you modified the path already in the last line? Because if you did and if there's any space in the path, you should insert <SP> in the code, where the space(s) can be found.

Let' say you want to save the .csv file to the following path:
c:\Users\YourWindowsUsername\My Documents\
then when you put it in the code, it should look like:
c:\Users\YourWindowsUsername\My<SP>Documents\

Wherever a space is in the path (either in the folder name or the file name), you should insert <SP> instead of it into the code.

 

[aminima]

Where can a find someone to make me a bot?

I can help.
I am on fiverr too under the same username.

 

[Weirdest]

I can help.
I am on fiverr too under the same username.

Can you make an apk bot that will run on my Android phone? I need a bot to do multiple signups

 

[aminima]

Please, i'm not in the liberty of discussing this things here.

 

[fastlinks]

i can make bot, skype: ipowerhost2