How is this server flagging me?

Nighcore2 · Jul 14, 2019

Im getting flagered in response headers
'User-Agent': 'python-requests/2.21.0'

Code:

import requests
from proxy import proxy  as ez #returns random proxy from my list


def getcookies(proxy):
    proxies = {
        'https': proxy  # returns random proxy
    }

    headers = {

        'cache-control':'max-age=0',
        'save-data': 'on',
        'upgrade-insecure-requests': '1',
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Safari/537.36',
        'accept': 'itext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
        'accept-encoding': 'gzip,deflate,br',
        'accept-language': 'es-ES,es;q=0.9,en;q=0.8',
    }
    session = requests.session()

    session.get('https://www.forocoches.com',proxies=proxies, headers=headers)
    z = session.cookies.get_dict()
    print(session.headers)

output:

Code:

#response headers, i get flaggered in user-agent
{'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

#cookies:
{'__cfduid': 'd686cd68b195e814b4d3eb50131f803a81563144210', 'bblastactivity': '0', 'bblastvisit': '1563144211'}

original http request:

Code:

HTTP-Trace-Version: 1.0
Generator: Charles/4.2.8

Method: GET
Protocol-Version: HTTP/2.0
Protocol: https
Host: m.forocoches.com
File: /codigo/
Remote-Address: 104.20.49.239
Client-Address: 192.168.1.67
Start-Time: 2019-07-14T22:37:59.144+02:00
Request-Begin-Time: 2019-07-14T22:37:59.144+02:00
Request-Time: 2019-07-14T22:37:59.144+02:00
Response-Time: 2019-07-14T22:37:59.177+02:00
End-Time: 2019-07-14T22:37:59.177+02:00
Request-Header-Size: 60
Response-Header-Size: 60
Request-Body-Size: 0
Response-Body-Size: 1506
Request-Body-Decoded: false
Response-Body-Decoded: false
Request-Header:<<--EOF-1563144017949-
:method: GET
:authority: m.forocoches.com
:scheme: https
:path: /codigo/
cache-control: max-age=0
save-data: on
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 9; POCOPHONE F1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: es-ES,es;q=0.9,en;q=0.8
cookie: __cfduid=d3ec973c3bb639a5e7a56b850865b89d81563136490
cookie: __utma=119157022.2140298539.1563136491.1563136491.1563136491.1
cookie: __utmc=119157022
cookie: __utmz=119157022.1563136491.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: __utmt=1
cookie: __utmb=119157022.1.10.1563136491
cookie: bblastvisit=1563136509
cookie: bblastactivity=0
cookie: _ga=GA1.2.2140298539.1563136491
cookie: _gid=GA1.2.923970786.1563136510
cookie: __gads=ID=d3ddd522c95b6d76:T=1563136510:S=ALNI_MY0T0Bq7gDd96rZVH1UdWpF34nE-g
cookie: _fbp=fb.1.1563136510092.311329930
cookie: euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: cmp_nodisplay=1
cookie: cmp_enforcedlocal=1
cookie: ___tg_ses=EE426D22EDDB7444.2
cookie: ___tg_vis=EE426D22EDDB7444.1563136520461
cookie: ___tg_ses_sec=125:1563136520461
cookie: ___tg_vis_sec=125:1563136520461
cookie: tg-refr=https://m.forocoches.com/foro/showthread.php?t=4133485
cookie: __utma=101525213.2140298539.1563136491.1563136525.1563136525.1
cookie: __utmc=101525213
cookie: __utmz=101525213.1563136525.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: sas_euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: __utmb=101525213.4.10.1563136525


--EOF-1563144017949-
Response-Header:<<--EOF-1563144017950-
:status: 200
date: Sun, 14 Jul 2019 20:37:59 GMT
content-type: text/html
x-litespeed-cache: hit,private
vary: Accept-Encoding
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f664c356e91d66d-MAD
content-encoding: br

Sebastian Rooks · Jul 15, 2019

Passing the headers through like you're doing, doesn't effect the session object itself.

You can update the session object headers instead of including headers=headers in your GET request to produce the output you're expecting, but it won't make a difference from the server's perspective.

It also might be worth pointing out that you're not actually handling the server response.

Code:

session.headers.update(headers)

r = session.get(<params>)
print(r.headers)   # for the server response or
r.request.headers  # for the headers you actually sent to the server.

While we're in here, I'd always recommend throwing in a timeout=20, or whatever you're into. Or one day you'll find that your script hangs forever.

edited 8k times because I don't know how to use code tags or form sentences.

Nighcore2 · Jul 15, 2019

thanks for extended response

Pepe García · Sep 7, 2019

Nighcore2 said:
thanks for extended response

Pole

ContentExpert · Sep 7, 2019

Pepe García said:
Pole

You bumped the thread for this response?

PIEGE · Sep 10, 2019

Nighcore2 said:

Im getting flagered in response headers
'User-Agent': 'python-requests/2.21.0'

Code:

import requests
from proxy import proxy  as ez #returns random proxy from my list


def getcookies(proxy):
    proxies = {
        'https': proxy  # returns random proxy
    }

    headers = {

        'cache-control':'max-age=0',
        'save-data': 'on',
        'upgrade-insecure-requests': '1',
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Safari/537.36',
        'accept': 'itext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
        'accept-encoding': 'gzip,deflate,br',
        'accept-language': 'es-ES,es;q=0.9,en;q=0.8',
    }
    session = requests.session()

    session.get('https://www.forocoches.com',proxies=proxies, headers=headers)
    z = session.cookies.get_dict()
    print(session.headers)

output:

Code:

#response headers, i get flaggered in user-agent
{'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

#cookies:
{'__cfduid': 'd686cd68b195e814b4d3eb50131f803a81563144210', 'bblastactivity': '0', 'bblastvisit': '1563144211'}

original http request:

Code:

HTTP-Trace-Version: 1.0
Generator: Charles/4.2.8

Method: GET
Protocol-Version: HTTP/2.0
Protocol: https
Host: m.forocoches.com
File: /codigo/
Remote-Address: 104.20.49.239
Client-Address: 192.168.1.67
Start-Time: 2019-07-14T22:37:59.144+02:00
Request-Begin-Time: 2019-07-14T22:37:59.144+02:00
Request-Time: 2019-07-14T22:37:59.144+02:00
Response-Time: 2019-07-14T22:37:59.177+02:00
End-Time: 2019-07-14T22:37:59.177+02:00
Request-Header-Size: 60
Response-Header-Size: 60
Request-Body-Size: 0
Response-Body-Size: 1506
Request-Body-Decoded: false
Response-Body-Decoded: false
Request-Header:<<--EOF-1563144017949-
:method: GET
:authority: m.forocoches.com
:scheme: https
:path: /codigo/
cache-control: max-age=0
save-data: on
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 9; POCOPHONE F1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: es-ES,es;q=0.9,en;q=0.8
cookie: __cfduid=d3ec973c3bb639a5e7a56b850865b89d81563136490
cookie: __utma=119157022.2140298539.1563136491.1563136491.1563136491.1
cookie: __utmc=119157022
cookie: __utmz=119157022.1563136491.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: __utmt=1
cookie: __utmb=119157022.1.10.1563136491
cookie: bblastvisit=1563136509
cookie: bblastactivity=0
cookie: _ga=GA1.2.2140298539.1563136491
cookie: _gid=GA1.2.923970786.1563136510
cookie: __gads=ID=d3ddd522c95b6d76:T=1563136510:S=ALNI_MY0T0Bq7gDd96rZVH1UdWpF34nE-g
cookie: _fbp=fb.1.1563136510092.311329930
cookie: euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: cmp_nodisplay=1
cookie: cmp_enforcedlocal=1
cookie: ___tg_ses=EE426D22EDDB7444.2
cookie: ___tg_vis=EE426D22EDDB7444.1563136520461
cookie: ___tg_ses_sec=125:1563136520461
cookie: ___tg_vis_sec=125:1563136520461
cookie: tg-refr=https://m.forocoches.com/foro/showthread.php?t=4133485
cookie: __utma=101525213.2140298539.1563136491.1563136525.1563136525.1
cookie: __utmc=101525213
cookie: __utmz=101525213.1563136525.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: sas_euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: __utmb=101525213.4.10.1563136525


--EOF-1563144017949-
Response-Header:<<--EOF-1563144017950-
:status: 200
date: Sun, 14 Jul 2019 20:37:59 GMT
content-type: text/html
x-litespeed-cache: hit,private
vary: Accept-Encoding
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f664c356e91d66d-MAD
content-encoding: br

this site is protected by cloudflare; "https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" <-

the problem with detection in this case goes further than just your mismatched user-agent.

for instance, you don't appear to be including your referrer .. but if this site checks your referrer header and doesnt see one, may reject the request

example from source : : : 'cookie: tg-refr=https://m.forocoches.com/foro/showthread.php?t=4133485'

depending on your goal, you may want to change your approach.

How is this server flagging me?

Nighcore2

Registered Member

Sebastian Rooks

Junior Member

Nighcore2

Registered Member

Pepe García

Junior Member

ContentExpert

Elite Member

PIEGE

Newbie

Main Menu

Marketplace

Making Money

BlackHat World