scrape and loop results count

shaymiller · Jan 30, 2020

Hi - I’m scraping multiples pages from same site. Page-1, Page-2, etc. how can I insert this into MySQL database in the same order?

I’m able scrape page numbers and count # into MySQL. But one every new page, the count # restarts. So if one page has 10 results per page, next page count should start at 11, not 1

basically top songs site. it has 20 pages. page 1 would have highest rank and page would have lowest rank

using php simple dom to scrape

GILGAME$H · Jan 30, 2020

Should be able to do this with a unique key, i.e for database "site.com" you have a unique key for each database entry that gets incremented on each insertion. So, first time you insert a result, it would be key-id=1, second time key-id=2, so on and so forth.

I'm fairly certain MySQL has this built-in.

shaymiller · Jan 30, 2020

GILGAME$H said:
Should be able to do this with a unique key, i.e for database "site.com" you have a unique key for each database entry that gets incremented on each insertion. So, first time you insert a result, it would be key-id=1, second time key-id=2, so on and so forth.

I'm fairly certain MySQL has this built-in.

I just thought about using "id" as the unique key. my database already does have a id for each row. Here is the issue, lets says, let next week new track is added as #1. when I rerun my code, that #1 would be row #99, right?

url=page number where it scraped it from
number= count results from scraping (i++)
id=unique key incremental

Thanks

GILGAME$H · Jan 30, 2020

If you know how to do functions, do:

Code:

rank = (page_number * 10 - 10) + page_order

So let's say you have a blank column named rank, a page_number column that has the page number (1, 2, 3, etc.), and a column that has the song's order for that page (1, 2, 3... 10).

If the "page number" is 1, then every song's "rank" is listed as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Because (1 * 10 - 10) = 0.

Now, let's say the "page number" is 2, then every song's "rank" is now listed as 11, 12, 13... 20. Because (2 * 10 - 10) = 10, we add that to the song's "page order."

You feel?

shaymiller · Jan 30, 2020

GILGAME$H said:
If you know how to do functions, do:

Code:

rank = (page_number * 10 - 10) + page_order

So let's say you have a blank column named rank, a page_number column that has the page number (1, 2, 3, etc.), and a column that has the song's order for that page (1, 2, 3... 10).

If the "page number" is 1, then every song's "rank" is listed as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Because (1 * 10 - 10) = 0.

Now, let's say the "page number" is 2, then every song's "rank" is now listed as 11, 12, 13... 20. Because (2 * 10 - 10) = 10, we add that to the song's "page order."

You feel?

I worked!!! ThANKS alot

scrape and loop results count

shaymiller

Regular Member

GILGAME$H

BANNED

shaymiller

Regular Member

GILGAME$H

BANNED

shaymiller

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World