[GUIDE] Handling multiple Puppeteer/Playwright instances (Node.js)

Digital Finger

Regular Member
Joined
Feb 12, 2019
Messages
223
Reaction score
190
Following my last guide, I received numerous inquiries about controlling and automating multiple instances concurrently.

This guide will mostly comprise of code, but the overall sentiment and structure can be applied to any language.

The provided code is written in TypeScript, which offers better clarity compared to raw JavaScript.

Firstly, you must instantiate a class, which I've named `BrowserInstance`. This class will encapsulate every browser instance you create.

Please let me know if you'd like me to build a plugin for `puppeteer-extra` that can be installed via npm, etc.

Ideally, you can add more functionalities to each instance. For the purpose of this guide, I've only added a sample close function.


JavaScript:
import { executablePath, Page, Browser } from "puppeteer";

export class BrowserInstance {
  id: string;
  browser: Browser;
  page: Page;

  constructor(id: string, browser: Browser, page: Page) {
    this.id = id;
    this.browser = browser;
    this.page = page;
  }

  async close() {
    try {
      await this.browser.close();
    } catch (error) {
      console.error(`Error closing browser: ${error}`);
      // Handle the error here
    }
  }
}

Building upon my previous guide, which uses `puppeteer-with-fingerprints`, this guide will leverage a custom plugin provided by `puppeteer-with-fingerprints`. This enables us to create multiple instances, each with unique fingerprints.

Additionally, I've employed an event emitter to track browser instances on a global scale. This is an optional feature; feel free to remove it if it doesn't align with your needs.

Types:
JavaScript:
import { HTTPRequest, Page } from "puppeteer";

export interface BrowserManagerConfig {
  storagePath?: string;
}

export interface CreateBrowserConfig {
  id: string;
  proxy?: Proxy;
  fingerprints?: string;
  existingProfileKey?: string;
}

export interface CreateProfileArgs {
  id: string;
}

export interface GetProfileArgs {
  id: string;
}

export interface Proxy {
  ip: string;
  port: string;
  username?: string;
  password?: string;
}

JavaScript:
import EventEmitter from "events";
import { join } from "path";
import { access, mkdir, remove } from "fs-extra";
import puppeteer from "puppeteer-extra";
import { createPlugin } from "puppeteer-with-fingerprints";


// BrowserManager class extends EventEmitter to handle events
export class BrowserManager extends EventEmitter {
  // Define class properties
  private STORAGE_PATH: string;
  browserInstances: BrowserInstance[] = [];
  handledRequestUrls = [];

  // Constructor for the class
  constructor({ storagePath }: BrowserManagerConfig) {
    super();
    // Use the provided storagePath or a default one
    this.STORAGE_PATH = storagePath ?? DEFAULT_PROFILE_STORAGE_PATH;
  }

  // Method to create a new browser instance
  async createBrowser(createBrowserConfig: CreateBrowserConfig) {
    // Destructure the config object to get necessary parameters
    const {
      id,
      proxy,
      fingerprints,
      existingProfileKey,
    } = createBrowserConfig;

    // Check if a browser with the provided id already exists
    const isBrowserActive = this.browserInstances.find(
      (browser) => browser.id === id
    );

    // If the browser already exists, throw an error
    if (isBrowserActive) {
      throw new Error("Browser ID already active");
    }

    // Create a new plugin for the browser
    const plugin = createPlugin({
      launch: (config) => puppeteer.launch(config),
    });

    try {
      // Use the provided fingerprints if any
      if (fingerprints) {
        plugin.useFingerprint(JSON.stringify(fingerprints));
      }

      // Set up proxy if provided
      if (proxy) {
        const { username, password, ip, port } = proxy;

        plugin.useProxy(`${ip}:${port}@${username}:${password}`, {
          detectExternalIP: true,
          changeGeolocation: true,
          changeBrowserLanguage: true,
          changeTimezone: true,
          changeWebRTC: true,
        });
      }

      // If an existing profile key is provided, get the profile path
      let profilePath: string;
      if (existingProfileKey) {
        profilePath = await this.getProfile({ id: existingProfileKey });
      }

      // Set up the browser configuration
      const browserConfig = {
        headless: false,
        args: [
          "--no-sandbox",
          "--disable-setuid-sandbox",
          "--disable-session-crashed-bubble",
          "--noerrdialogs",
        ],
        executablePath: executablePath(),
        ...(existingProfileKey ? { userDataDir: profilePath } : {}),
      };

      // Launch the browser with the provided configuration
      const browser = await plugin.launch(browserConfig);

      // Open a new page in the browser
      const page = await browser.newPage();

      // Create a new browser instance and add it to the list
      const browserInstance = new BrowserInstance(id, browser, page);
      this.browserInstances.push(browserInstance);

      // Emit an event to indicate the change in active browsers
      this.emit(
        "activeBrowsersChanged",
        this.browserInstances.map((instance) => instance.id)
      );

      // Return the new browser instance
      return browserInstance;
    } catch (error) {
      // If there's an error, throw it
      throw new Error(`Error creating browser: ${error}`);
    }
  }

  // Method to close a specific browser instance
  async closeBrowser(id: string): Promise<void> {
    // Find the index of the browser in the list
    const index = this.browserInstances.findIndex((b) => b.id === id);
    // If the browser exists, close it and remove it from the list
    if (index !== -1) {
      await this.browserInstances[index].close();
      this.browserInstances.splice(index, 1);

      // Emit an event to indicate the change in active browsers
      this.emit(
        "activeBrowsersChanged",
        this.browserInstances.map((instance) => instance.id)
      );
    }
  }

  // Method to close all browser instances
  async closeAll(): Promise<void> {
    // Use Promise.all to close all browsers concurrently
    await Promise.all(
      this.browserInstances.map((browserInstance) => {
        return browserInstance.browser.close();
      })
    );

    // Clear the list of browser instances
    this.browserInstances = [];

    // Emit an event to indicate the change in active browsers
    this.emit("activeBrowsersChanged", []);
  }

  // Method to get a browser instance by its id
  getBrowserById(id: string): BrowserInstance {
    // Find and return the browser instance with the given id
    return this.browserInstances.find((instance) => instance.id === id);
  }

  // Method to get a profile by its id
  async getProfile({ id }: GetProfileArgs): Promise<string> {
    // Construct the directory path for the profile
    const directory = join(this.STORAGE_PATH, id);

    try {
      // Try to access the directory
      await access(directory);
    } catch {
      // If the directory does not exist, throw an error
      throw new Error(`Profile ${id} does not exist.`);
    }

    // Return the directory path
    return directory;
  }

  // Method to delete a profile by its id
  async deleteProfile({ id }: GetProfileArgs): Promise<boolean> {
    // Construct the directory path for the profile
    const directory = join(this.STORAGE_PATH, id);

    try {
      // Try to remove the directory
      await remove(directory);
    } catch {
      // If the directory does not exist, throw an error
      throw new Error(`Profile ${id} does not exist.`);
    }

    // Return true to indicate successful deletion
    return true;
  }

  // Method to create a new profile
  async createProfile({ id }: CreateProfileArgs): Promise<string> {
    // Construct the directory path for the profile
    const directory = join(this.STORAGE_PATH, id);

    try {
      // Try to access the directory
      await access(directory);
    } catch {
      // If the directory does not exist, create it
      await mkdir(directory, { recursive: true });
    }

    // Return the directory path
    return directory;
  }
}

It's a fair bit of code and I could breakdown "createBrowser" however for the purpose of understanding the process I have decided to structure it in this way.

Let me know if you have any questions i'd be happy to help.
 
Looks great!, could you explain how to actually implement it? should I save each block of code in a seperate file and import? or all in one, Also I encountered in some bugs I don't know if just my configuration.
some bugs:
Cannot find name 'DEFAULT_PROFILE_STORAGE_PATH'.
Variable 'profilePath' is used before being assigned.
Type 'BrowserInstance | undefined' is not assignable to type 'BrowserInstance'.
Type 'undefined' is not assignable to type 'BrowserInstance'.
 
Looks great!, could you explain how to actually implement it? should I save each block of code in a seperate file and import? or all in one, Also I encountered in some bugs I don't know if just my configuration.
some bugs:
Cannot find name 'DEFAULT_PROFILE_STORAGE_PATH'.
Variable 'profilePath' is used before being assigned.
Type 'BrowserInstance | undefined' is not assignable to type 'BrowserInstance'.
Type 'undefined' is not assignable to type 'BrowserInstance'.

It isn’t the full code, it’s just so I can show you how and what it looks like to implement.
 
It isn’t the full code, it’s just so I can show you how and what it looks like to implement.
Got it, I got it all working actually but one problem, when I try to run multiple browsers with fingerprints I load manually I get: Error: Lock is not acquired/owned by you, have you encountered anything like that?
 
Got it, I got it all working actually but one problem, when I try to run multiple browsers with fingerprints I load manually I get: Error: Lock is not acquired/owned by you, have you encountered anything like that?

No this should work fine, it might not be puppeteer related.

Contact me on telegram @digitalfinger and push the code to a repo I might be able to take a look for you.
 
No this should work fine, it might not be puppeteer related.

Contact me on telegram @digitalfinger and push the code to a repo I might be able to take a look for you.
got it all working, multiple instances and all and again thank you lol, you really helped me.
one last(?) thing, using headless mode I sometime still get recapcha for some reason any idea on why would that happen? I use a different chrome profile every time with randomized history would that cause that? or I just look up for making my code run slower?
 
got it all working, multiple instances and all and again thank you lol, you really helped me.
one last(?) thing, using headless mode I sometime still get recapcha for some reason any idea on why would that happen? I use a different chrome profile every time with randomized history would that cause that? or I just look up for making my code run slower?

I don't know without taking a deeper look into it, could be the proxy it could be a confluence of factors.

However, if you do have any captcha problems I suggest NopeCHA. It's not supported by puppeteer captcha solver but you can make your own wrapper for it or install the extension on the browser.
 
Got it, I got it all working actually but one problem, when I try to run multiple browsers with fingerprints I load manually I get: Error: Lock is not acquired/owned by you, have you encountered anything like that?
How did you solve it bro ? Facing same issue
 
Is pupeter less detectable than selenium?

Browserleaks.com/canvas do you get 100% unique canvas here with pupetter?
 
Back
Top