Filtering a list of URLs

nahaczyku

nahaczyku

Newbie
Joined
Oct 12, 2009
Messages
35
Reaction score
8
I have a link list as is

Code: 
http://www.example.com/category/post
http://www.example.com/
http://www.example.com/category/index.php
http://www.example.com/
http://www.example.com/category/post

I would like to get only
Code: 
http://www.example.com/category/post
http://www.example.com/category/index.php
http://www.example.com/category/post

Do you know any simple solutions?
 
Advertise on BHW
Sort by alphabet and remove the rest.
 
If it's not the same domain every time, you can try to remove lines, which are less than x characters long.
Use Notepad++, select the Replace function and check "Regular expression" in the fieldset 'Search mode'.
Find the following: ^.{0,25}$
Replace with nothing.
This matches lines with 25 or less characters and remove them.
You obviously need to adjust the number until all root domains are removed.

If it's the same domain, you just need to remove the duplicate lines as it was mentioned above. You can use Notepad++ for that purpose too and the appropriate function of the TextFX plugin.
 
const urls = URLS GO HERE const filtered = []; for (const url of urls.split('/n')) { url.includes('category') && void filtered.push(url); } void console.log(filtered);
 
I'm sorry but I was not very precise. These are not the same domains


Code: 
http://www.example1.com/post/post
http://www.example222.com/
http://www.example3.com/seo/index.php
http://www.example42.com/
http://www.example5.com/lib/post

Can this be done in notepad ++ or pspad?
 
Last edited:
Copy that into excel and remove duplicates.

Or copy into excel, sort by alphabet and keep only the long URLs.

Or put it into Scrapebox if you have it and remove duplicate URLs.
 
Scrapebox doesn't have such a feature yet. The idea is to remove only clean domains with no subpages. If someone is so nice and gives a rule in notepad ++, I will be very grateful :)
 
In that case, use the function in Scrapebox of "Remove URLs NOT containing" the following:

.com/1
.com/a
.com/b
.com/c
.com/d

etc etc.

This should remove all the root URLs and keep the deeper level URLs.
 
What should be the value in the mask field?
 
nahaczyku said:
What should be the value in the mask field?
Click to expand...

Put all your URLs you want to filter into Scrapebox "URLs Harvested" field.

Create a .txt file with the following:

.com/1
.com/a
.com/b
.com/c
.com/d

(Make sure you have all the TLDs and all the numbers/letters combinations you want there)

so com, net, org, all numbers and all letters.

Then just select the option Remove/Filter -> Remove URLs not containing entries from and select your .txt file. And that's it.
 
if you use linux its very easy, just put your list into a text file i.e. list.txt and then:
Code: 
cat list.txt | sort -uf > sorted.txt
 
Thank you, this solution does work, it also does not solve the problem of subdomains.

It will take some time to create such a list of all TLDs Do you have a ready-made list like this?

If anyone knows how to do it in notepad ++ I will also be grateful
 
@Ipnp

Unfortunately, this expression removes all .com domains and their subpages

Code: 
http.*?(.com)(.*)

There is also the question of removing the blank line after the address.
 
Created this HTML page. Try it.
https://vishalbty.github.io/cleanerbhw/index.html
 
@Vuex
The lists are 200MB and more, so the browser output is out of the question, Anyway, thanks for the link :)
 
const filtered = [];

for (const url of urls.split('/n')) {
If(url.split('.').pop().split('/').length >= 2) {
void filtered.push(url);
}
}

// split url at last tld (capture subdomains)
// then split path with /. If url has path the array length should be at least 2
 
Yes, the shell would be more efficient, I only use it occasionally. however, on a daily basis, I work on windows and it is more convenient to process files in notepad ++, moreover, editing such a list takes up to a dozen or so seconds.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top