Puppeteer often fails waitForNavigation with Timeout error on Twitter

kexkex · Sep 27, 2020

Hey. I am playing around with Puppeteer for Twitter automation. I have discovered that often the navigation will timeout. For example:

Code:

puppeteer.launch({ headless: false }).then(async browser => {
    try {
        const page = await browser.newPage();

        await page.goto('https://twitter.com/home');
        
        // This sometimes fails with timeout error
        await page.waitForNavigation({ waitUntil: 'networkidle2' });

        if (page.url() === 'https://twitter.com/login') {
            await login(page);
        }

    } catch(error) {
        console.log(error);
    }
});

The page will just hang and I'll get the error TimeoutError

I have tried changing the waitUntil parameter and it doesn't seem to make a difference. I have also set await page.setDefaultNavigationTimeout(0); this of course stops the error appearing but the page just never responds.

Has anyone else faced this problem? Is Twitter detecting me? Or have I missed something? I am using puppeteer-extra with puppeteer-extra-plugin-stealth using the default settings.

thanks

Heresiarch · Oct 1, 2020

It probably means that some network connections never stop activity. Instead of networkIdle you can try something like waitTillElementLoads(visible=True). I don't remember the exact name of this function, but you got the idea.

WPAutoGuy · Oct 1, 2020

Have you tried changing the waitUntil parameter to 'domcontentloaded'?

More info: https://pptr.dev/#?product=Puppeteer&version=v2.1.1&show=api-pagewaitfornavigationoptions

Puppeteer often fails waitForNavigation with Timeout error on Twitter

kexkex

Newbie

Heresiarch

Junior Member

WPAutoGuy

Power Member

Main Menu

Marketplace

Making Money

BlackHat World