1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Huge lists bringing FL to it's knees. Scrape first in another program? Other solutions?

Discussion in 'Twitter' started by punkinhead, Sep 9, 2015.

  1. punkinhead

    punkinhead Regular Member

    Joined:
    Feb 19, 2015
    Messages:
    437
    Likes Received:
    28
    I'm moving over to much bigger vps, so that should help, but I could swear FL is just inefficient in how it deals with lists as they grow. It basically slows to a dead stop as the blacklist, etc grows. Dev has told me to delete scraped lists and so on to speed up. Real system killer, though, appears to be all the scraping it does per account hundreds of times a day.

    So... the last bit, at least, I could address if I scrape everything ahead of time. If, for instance, I scraped all 3M followers of a user ahead of time, then just fed that list without scraping for my bot accts to follow, it should be much quicker, no?

    I'm generally aware of issues that larger lists should be broken up, etc... but need specifics. If you are scraping your lists ahead of time, what program are you using, and are you getting the full list or only top few thousand? If full list, what's the right way at size to chunk it before feeding it to FL?
     
  2. punkinhead

    punkinhead Regular Member

    Joined:
    Feb 19, 2015
    Messages:
    437
    Likes Received:
    28
    Anyone? Beuller?

    Who's scraping ahead of time, and what are you using, and why?