how to mirror website ?
- Thread starter autodream
- Start date
- Jan 17, 2015
- 5,263
- 4,956
Two options;
1. The easy way via RSS, just use a triggered event like IFTTT
2. The hard way, with a perpetual spider scrapper than runs once every few minutes, with Scrapy you can do this fairly affordable. If the site is not yours, you may end get banned your IP at some point (specially if you are scrapping a site in a protected hosting).
1. The easy way via RSS, just use a triggered event like IFTTT
2. The hard way, with a perpetual spider scrapper than runs once every few minutes, with Scrapy you can do this fairly affordable. If the site is not yours, you may end get banned your IP at some point (specially if you are scrapping a site in a protected hosting).
autodream
Regular Member
- Aug 17, 2017
- 241
- 35
i dont mean this ,what i need is
the same url structure as the original site
it is a script do this - when a user has certain query the script do search on original site then brings results to the main site and caches it for future use to save rescources
with some find and replace rules
- Jan 17, 2015
- 5,263
- 4,956
Ok, I get it. I don't know how to replicate those mirror sites you are looking for. There must be a straight forward method. Maybe if you ask the straight question someone else with specific experience on that can help you.
This are the kind of topics that users tend to search for on the dark web.
This are the kind of topics that users tend to search for on the dark web.