kirkonpolttaja
Senior Member
- Feb 6, 2010
- 1,030
- 674
This is from 2007 and has some good information for your own bot creation eventough you dont read it all the way through.
Good code snippets and explanations
Here is chapter infos:
VT as it is PDF:
Good code snippets and explanations
Here is chapter infos:
Chapter 1: The Structure of HTTP Requests ....................................................35
Chapter 2: Examining HTTP Traffic ...................................................................73
Chapter 3: Simple Requests .............................................................................93
Chapter 4: Beyond Simple Requests ................................................................127
Chapter 5: Secure HTTP Requests ...................................................................153
Chapter 6: Extracting Data ...............................................................................171
Chapter 7: Responding to Forms .....................................................................251
Chapter 8: Handling Sessions and Cookies .....................................................285
Chapter 9: Using JavaScript .............................................................................307
Chapter 10: Working with AJAX Sites ..............................................................339
Chapter 11: Handling Web Services .................................................................365
Chapter 12: Working with RSS Feeds ...............................................................395
Chapter 13: Using a Spider ..............................................................................419
Chapter 14: Inside the Heaton Research Spider ..............................................459
Chapter 15: Using a SQL Workload ...................................................................519
Chapter 16: Well Behaved Bots ........................................................................563
Appendix A: Downloading Examples ...............................................................581
Appendix B: Setting Up Examples ..................................................................585
Appendix C: Useful Classes, Methods and Functions ....................................591
Appendix D: Setting Up your Database ...........................................................595
Appendix E: HTTP Response Codes .................................................................601
XIV HTTP Programming Recipes for C# Bots
XV
Contents
Introduction ......................................................................................................XXXI
Chapter 1: The Structure of HTTP Requests ....................................................35
The HTTP Recipes Examples Site ..............................................................35
The Structure of Surfing .............................................................................38
Examining HTTP Requests .........................................................................38
HTTP Requests and Responses .................................................................42
HTTP Headers .............................................................................................47
Recipes .......................................................................................................50
Summary ....................................................................................................71
Chapter 2: Examining HTTP Traffic ...................................................................73
Using a Network Analyzer ..........................................................................73
Understanding WireShark ..........................................................................74
Recipes .......................................................................................................85
Summary ....................................................................................................90
Chapter 3: Simple Requests .............................................................................93
Constructing URLs ......................................................................................94
What is a URL? ...........................................................................................95
Encoding Special Characters into a URL ...................................................97
Reading from URLs .....................................................................................98
Recipes .......................................................................................................101
Summary ....................................................................................................124
Chapter 4: Beyond Simple Requests ................................................................127
Using HttpWebRequest ...............................................................................127
Using HttpWebResponse ............................................................................130
Other Useful Options ..................................................................................132
Recipes .......................................................................................................133
Summary ....................................................................................................150
Chapter 5: Secure HTTP Requests ...................................................................153
Using HTTPS in C# ......................................................................................153
Understanding HTTP Authentication ..........................................................158
Recipes .......................................................................................................161
Summary ....................................................................................................167
Chapter 6: Extracting Data ...............................................................................171
Peekable Stream ........................................................................................172
Parsing HTML .............................................................................................179
Encapsulating HTML Tags ..........................................................................200
Contents
XVI HTTP Programming Recipes for C# Bots
Recipes .......................................................................................................205
Summary ....................................................................................................248
Chapter 7: Responding to Forms .....................................................................251
Understanding HTML Forms .......................................................................251
POST or GET ................................................................................................256
Multipart POST ...........................................................................................258
Processing Forms .......................................................................................261
Recipes .......................................................................................................268
Summary ....................................................................................................282
Chapter 8: Handling Sessions and Cookies .....................................................285
URL Variables for State ..............................................................................285
Cookies for State ........................................................................................287
Recipes .......................................................................................................289
Summary ....................................................................................................305
Chapter 9: Using JavaScript .............................................................................307
Understanding JavaScript ..........................................................................307
Common JavaScript Techniques ................................................................308
Interpreting JavaScript ..............................................................................317
Recipes .......................................................................................................318
Summary ....................................................................................................335
Chapter 10: Working with AJAX Sites ..............................................................339
Understanding AJAX ..................................................................................340
Recipes .......................................................................................................346
Summary ....................................................................................................362
Chapter 11: Handling Web Services .................................................................365
Notable Public Web Services .....................................................................365
Using the Google API ..................................................................................366
Hybrid Bots .................................................................................................367
Understanding SOAP ..................................................................................367
Recipes .......................................................................................................375
Summary ....................................................................................................392
Chapter 12: Working with RSS Feeds ...............................................................395
Using RSS with a Web Browser .................................................................395
RSS Format .................................................................................................397
Parsing RSS Files .......................................................................................401
Recipes .......................................................................................................411
Summary ....................................................................................................415
Chapter 13: Using a Spider ..............................................................................419
XVII
Using the Heaton Research Spider ............................................................420
Recipes .......................................................................................................428
Summary ....................................................................................................456
Chapter 14: Inside the Heaton Research Spider ..............................................459
The Spider Class .........................................................................................461
Other Important Classes in the Heaton Research Spider .........................475
Workload Management ..............................................................................498
Implementing a Memory Based WorkloadManager ..................................504
Summary ....................................................................................................516
Chapter 15: Using a SQL Workload ...................................................................519
Repeatable Statements ..............................................................................520
Implementing a SQL Based Workload Manager ........................................531
Summary ....................................................................................................560
Chapter 16: Well Behaved Bots ........................................................................563
Using a CAPTCHA .......................................................................................564
User Agent Filtering ....................................................................................566
Robots Exclusion Standard ........................................................................567
Using Filters with the Heaton Research Spider .........................................569
Implementing a robots.txt Filter ................................................................570
Summary ....................................................................................................579
Appendix A: Downloading Examples ...............................................................581
Appendix B: Setting Up Examples ..................................................................585
Running from a Command Prompt ..........................................................586
Running from Visual Studio .......................................................................587
Compiling without Visual Studio ................................................................588
Appendix C: Useful Classes, Methods and Functions ....................................591
Reusable Functions and Methods .............................................................591
Reusable Classes .......................................................................................591
All Recipes ..................................................................................................592
Appendix D: Setting Up your Database ...........................................................595
DDL for MySQL ............................................................................................595
DDL for Microsoft Access ...........................................................................596
DDL for Oracle ...........................................................................................597
OLEDB for .NET ...........................................................................................599
Appendix E: HTTP Response Codes .................................................................601
1xx Informational .......................................................................................601
2xx Success ................................................................................................601
3xx Redirection ...........................................................................................601
XVIII HTTP Programming Recipes for C# Bots
4xx Client Error ...........................................................................................602
5xx Server Error ..........................................................................................602
Code:
http://www.mediafire.com/view/?19dq1kmr6yavunvVT as it is PDF:
Code:
https://www.virustotal.com/file/eda687a6042a7675fa31b9fbcf3ef09980ef5fac670a4765124857c6781956f7/analysis/1338112980/#%20Bots%20%282007%29.pdf
