Selenium webdriver move mouse from one point to another

I would like to move the mouse from point A (random) to point B (an element) on the webpage using selenium and chrome driver.

As in simulating the mouse moving rather than just jumping from one point to another.

I have the coordinates of both the starting location and the destination element.

Unfortunately, there doesn't seem to be a way of moving the cursor to a specific point, it seems my options are MoveToElement and moveByOffset. Both of which work with offsets so you can only move a certain distance from the element you are trying to click on.

Moving the cursor

Code:

Actions builder = new Actions(driver);
builder.MoveToElement(element, offsetX, offsetY).Perform();

I'm using the code below to get the starting location/coordinates and the current coordinates

Code:

IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
js.ExecuteScript("mousePosX =  0; mousePosY = 0; onmousemove = function(e){ mousePosX = e.clientX; mousePosY = e.clientY; console.log('Y:' + mousePosY + ' X:' + mousePosX); }; ");

IWebElement currentElement = driver.SwitchTo().ActiveElement();
builder.MoveToElement(currentElement, randomStartX, randomStartY).Perform();

int mousePosX = convertJSMousePos(driver.ExecuteScript("return mousePosX;"));
int mousePosY = convertJSMousePos(driver.ExecuteScript("return mousePosY;"));

Getting the destination coordinates

Code:

IWebElement link = driver.FindElement(By.LinkText("more info"));
System.Drawing.Point location = link.Location;

int targetXPostion = location.X;
int targetYPostion = location.Y;

I know the solution will be a loop that gradually reduces the offset until the X/Y target destination is reached or the destination element is in focus. It sounds simple but my attempts at slotting this all together have failed miserably so far.

Don't know why I'm struggling with this one, normally I find this shit easy as writing English.

Anyone done this before?

I don't have tried it but ;

have you tried to do an A* between your start pos and your end pos, and moving to every point in your solution path ?

Of course, you may have to convert absolute position to relative position as the move functions are working with offset but that should be fairly simple as you know your starting point coords and each step of the path.

That might not simulate the continuity of a regular mouse movement, and performance might be terrible, but I think it will be as far as someone can get using only Selenium as-is.

Sheepp said:

I don't have tried it but ;

have you tried to do an A* between your start pos and your end pos, and moving to every point in your solution path ?

Of course, you may have to convert absolute position to relative position as the move functions are working with offset but that should be fairly simple as you know your starting point coords and each step of the path.

That might not simulate the continuity of a regular mouse movement, and performance might be terrible, but I think it will be as far as someone can get using only Selenium as-is.

Click to expand...

Thanks man. I'm surprised I can't find a good example online anywhere.

When I had to solve the same issue I did it natively and chose to take real samples and then compose the new movement based on that to make it more believable for detection systems.

As for the continuity you have to keep in mind that timer granularity in browsers is not even close to native so it's not a huge problem, you might be able to get by using the following snippet :


I am in the process of releasing some code related to this, hopefully it will be ready within the following days, I'll try to prioritize the code I wrote to take the samples as that is something you can adapt and reuse easily.

verrocchio said:

When I had to solve the same issue I did it natively and chose to take real samples and then compose the new movement based on that to make it more believable for detection systems.

As for the continuity you have to keep in mind that timer granularity in browsers is not even close to native so it's not a huge problem, you might be able to get by using the following snippet :

Code:

def pregenerated_delay(delay_length):
    def delay():
        time.sleep(delay_length)

    return delay

action_chain = ActionChains(browser)
for offset in range(100):
    action_chain._actions.append(pregenerated_delay(0.1))
    action_chain.move_to_element_with_offset(element, offset, offset)

action_chain.perform()

I am in the process of releasing some code related to this, hopefully it will be ready within the following days, I'll try to prioritize the code I wrote to take the samples as that is something you can adapt and reuse easily.

Click to expand...

Thanks mate I really appreciate that