How is this server flagging me?

Im getting flagered in response headers
'User-Agent': 'python-requests/2.21.0'



output:

original http request:

Passing the headers through like you're doing, doesn't effect the session object itself.

You can update the session object headers instead of including headers=headers in your GET request to produce the output you're expecting, but it won't make a difference from the server's perspective.

It also might be worth pointing out that you're not actually handling the server response.


While we're in here, I'd always recommend throwing in a timeout=20, or whatever you're into. Or one day you'll find that your script hangs forever.

edited 8k times because I don't know how to use code tags or form sentences.

thanks for extended response

Nighcore2 said:

thanks for extended response

Click to expand...

Pole

Pepe García said:

Pole

Click to expand...

You bumped the thread for this response?

Nighcore2 said:

Im getting flagered in response headers
'User-Agent': 'python-requests/2.21.0'

Code:

import requests
from proxy import proxy  as ez #returns random proxy from my list


def getcookies(proxy):
    proxies = {
        'https': proxy  # returns random proxy
    }

    headers = {

        'cache-control':'max-age=0',
        'save-data': 'on',
        'upgrade-insecure-requests': '1',
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Safari/537.36',
        'accept': 'itext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
        'accept-encoding': 'gzip,deflate,br',
        'accept-language': 'es-ES,es;q=0.9,en;q=0.8',
    }
    session = requests.session()

    session.get('https://www.forocoches.com',proxies=proxies, headers=headers)
    z = session.cookies.get_dict()
    print(session.headers)

output:

Code:

#response headers, i get flaggered in user-agent
{'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

#cookies:
{'__cfduid': 'd686cd68b195e814b4d3eb50131f803a81563144210', 'bblastactivity': '0', 'bblastvisit': '1563144211'}

original http request:

Code:

HTTP-Trace-Version: 1.0
Generator: Charles/4.2.8

Method: GET
Protocol-Version: HTTP/2.0
Protocol: https
Host: m.forocoches.com
File: /codigo/
Remote-Address: 104.20.49.239
Client-Address: 192.168.1.67
Start-Time: 2019-07-14T22:37:59.144+02:00
Request-Begin-Time: 2019-07-14T22:37:59.144+02:00
Request-Time: 2019-07-14T22:37:59.144+02:00
Response-Time: 2019-07-14T22:37:59.177+02:00
End-Time: 2019-07-14T22:37:59.177+02:00
Request-Header-Size: 60
Response-Header-Size: 60
Request-Body-Size: 0
Response-Body-Size: 1506
Request-Body-Decoded: false
Response-Body-Decoded: false
Request-Header:<<--EOF-1563144017949-
:method: GET
:authority: m.forocoches.com
:scheme: https
:path: /codigo/
cache-control: max-age=0
save-data: on
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 9; POCOPHONE F1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: es-ES,es;q=0.9,en;q=0.8
cookie: __cfduid=d3ec973c3bb639a5e7a56b850865b89d81563136490
cookie: __utma=119157022.2140298539.1563136491.1563136491.1563136491.1
cookie: __utmc=119157022
cookie: __utmz=119157022.1563136491.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: __utmt=1
cookie: __utmb=119157022.1.10.1563136491
cookie: bblastvisit=1563136509
cookie: bblastactivity=0
cookie: _ga=GA1.2.2140298539.1563136491
cookie: _gid=GA1.2.923970786.1563136510
cookie: __gads=ID=d3ddd522c95b6d76:T=1563136510:S=ALNI_MY0T0Bq7gDd96rZVH1UdWpF34nE-g
cookie: _fbp=fb.1.1563136510092.311329930
cookie: euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: cmp_nodisplay=1
cookie: cmp_enforcedlocal=1
cookie: ___tg_ses=EE426D22EDDB7444.2
cookie: ___tg_vis=EE426D22EDDB7444.1563136520461
cookie: ___tg_ses_sec=125:1563136520461
cookie: ___tg_vis_sec=125:1563136520461
cookie: tg-refr=https://m.forocoches.com/foro/showthread.php?t=4133485
cookie: __utma=101525213.2140298539.1563136491.1563136525.1563136525.1
cookie: __utmc=101525213
cookie: __utmz=101525213.1563136525.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
cookie: sas_euconsent=BOjs7RLOjs7RLBcAa_ESCb-AAAAo1rv___7__9_-____9uz7Ov_v_f__33e8779v_h_7_-___u_-3zd4u_1vf99yfm1-7ctr3tp_87uesm_Xur__59__3z3_9phPrsk89r6337AgAAA
cookie: __utmb=101525213.4.10.1563136525


--EOF-1563144017949-
Response-Header:<<--EOF-1563144017950-
:status: 200
date: Sun, 14 Jul 2019 20:37:59 GMT
content-type: text/html
x-litespeed-cache: hit,private
vary: Accept-Encoding
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f664c356e91d66d-MAD
content-encoding: br

Click to expand...

this site is protected by cloudflare; "https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" <-

the problem with detection in this case goes further than just your mismatched user-agent.

for instance, you don't appear to be including your referrer .. but if this site checks your referrer header and doesnt see one, may reject the request

example from source : : : 'cookie: tg-refr=https://m.forocoches.com/foro/showthread.php?t=4133485'

depending on your goal, you may want to change your approach.