What kind of useless garbage UA

L

lanmaok

Registered Member
Joined
Oct 25, 2024
Messages
81
Reaction score
20
My website is run by the nginx program. In order to save broadband resources, how can I stop useless garbage crawlers? What are some web garbage crawlers?
 
Advertise on BHW
you can use block the specific crawlers in your robots.txt, hope that helps brother :)
 
bbrokeaf said:
you can use block the specific crawlers in your robots.txt, hope that helps brother :)
Click to expand...
Robots.txt cannot be completely trusted. There are many unknown crawlers that completely ignore robots.txt, so they must be blocked from the nginx server to make the website more secure.
 
Brother, what kind of garbage robots are there? Ua or features
 
lanmaok said:
Brother, what kind of garbage robots are there? Ua or features
Click to expand...
Lots of crawlers, most popular ones are from google, bing, yahoo, and social media sites. The question is why you consider them garbage.
lanmaok said:
Robots.txt cannot be completely trusted. There are many unknown crawlers that completely ignore robots.txt, so they must be blocked from the nginx server to make the website more secure.
Click to expand...
Go to your nginx server block configuration and add a 'deny' directive. Target shady user agent strings, i will list some resources for that below, and restart and clear cache.

I will apply a list of known malicious IP's so you can block them in your server.

Hope this helps :)



IP Ban List 1
185.234.216.59
212.83.185.105
194.143.136.122
113.160.133.8
194.44.38.51
176.8.55.92
131.253.18.12
131.253.18.11
46.149.184.5
46.211.241.39
109.196.187.208
185.242.113.224
46.101.197.155
108.28.164.248
117.215.207.136
41.33.58.100
190.14.228.17
149.202.62.12
212.156.149.146
149.56.99.38
149.56.99.37
27.106.62.214
91.140.233.130
163.172.211.135
41.200.247.222
181.189.148.98
58.137.148.186
67.205.149.140
185.62.189.56
200.5.112.180
94.183.112.60
85.90.244.52
89.36.208.231
163.172.43.52
185.86.149.97
177.85.98.227
81.170.184.90
201.110.183.154
196.200.16.23
163.172.38.173
158.58.170.186
146.0.77.50
163.172.158.208
185.69.168.112
120.50.19.10
87.185.31.200
59.99.43.205
163.172.154.105
163.172.209.46
146.185.139.55
185.107.70.202
159.203.30.48
181.143.243.98
116.68.103.36
188.161.150.22
41.33.197.132
117.239.224.138
14.176.1.82
213.74.133.14
59.93.29.14
41.138.60.225
163.172.152.231
197.211.53.25
113.176.70.96
5.255.80.27
89.33.246.114
197.254.106.218
162.210.173.109
212.47.248.81
119.93.79.68
98.124.243.32
59.93.84.191
119.17.192.102
178.62.18.173
178.159.36.185
89.35.178.104
192.64.119.234
201.76.186.230
138.97.240.250
151.100.179.50
46.28.111.122
217.12.201.109
116.68.98.252
107.182.131.117
46.101.138.211
101.98.11.146
187.44.138.226
31.31.72.43
64.137.178.3
193.138.219.231
186.205.89.48
185.100.87.63
84.251.91.165
27.50.94.251
31.16.91.237
108.61.187.24
212.47.247.226
92.222.92.152
104.236.58.27
37.187.247.3
101.0.54.130
93.219.113.201
87.120.254.200
51.254.215.7
178.32.53.124
181.143.153.250
185.61.138.104
104.168.177.9
185.86.13.130
139.59.9.200
198.134.125.78
116.107.184.31
118.69.176.205
27.116.21.219
181.143.253.106
85.207.155.39
185.128.40.220
163.172.143.114
5.196.58.96
94.26.2.74
103.214.54.82
45.55.178.34
167.114.238.104
185.100.85.191
95.31.22.193
92.222.103.234
46.148.26.78
45.79.85.112
46.39.102.250
82.221.129.96
199.175.49.21
72.34.42.146
45.56.90.85
91.234.33.250
212.47.227.72
49.88.121.163
149.202.44.177
193.200.241.195
209.66.119.150
149.202.42.188
207.244.78.3
166.62.28.107
188.126.81.155
185.76.35.11
128.153.145.125
5.28.62.85
78.39.117.17
89.238.212.68
113.161.8.108
178.32.53.94
118.174.3.106
158.69.194.36
108.167.180.12
185.38.151.64
186.42.192.22
45.33.48.204
108.61.123.73
82.211.19.129
216.17.99.183
158.69.208.131
69.195.129.71
223.81.192.191
87.255.51.230
199.87.154.251
37.48.109.107
46.166.188.208
92.222.28.243
185.104.120.7
217.160.2.23
193.232.240.11
193.111.136.162
122.52.159.39
146.185.128.226
107.170.1.205
46.166.190.182
109.188.126.11
94.103.175.86
5.199.171.58
185.27.134.131
146.185.135.43
193.110.157.49
31.172.30.4
81.30.158.81
104.237.152.195
212.83.40.239
46.28.110.136
91.219.236.232
62.210.87.71
62.210.87.30
65.181.112.128
212.47.238.193
174.127.99.132
109.232.223.115
210.92.18.118
178.32.53.131
212.83.40.238
141.138.141.208
162.247.72.27
71.19.157.127
187.217.183.146
68.206.21.41
185.17.184.228
217.12.204.104
185.53.178.12
116.68.102.237
198.50.200.139
31.192.228.185
162.247.72.216
59.93.21.183
77.76.228.4
201.87.244.10
108.167.140.20
189.236.206.143
200.50.124.165
139.0.6.91
109.188.125.3
109.188.124.25
109.188.125.13
210.245.54.103
218.65.6.122
184.168.221.17
201.234.180.155
109.188.127.28
198.61.227.6
197.159.9.29
31.41.59.29
178.72.78.52
115.127.13.235
201.43.143.92
49.89.225.74
49.71.142.141
117.196.54.136
82.191.238.172
180.178.189.50
117.196.17.4
178.72.70.160
46.153.152.177
180.178.188.222
180.178.189.76
164.132.172.220
95.38.74.75
85.174.205.255
180.178.188.150
117.199.234.103
27.128.226.95
178.72.70.167
178.72.78.136
178.72.77.167
94.156.58.198
91.187.121.119
78.106.220.129
193.84.22.89
103.23.135.78
178.72.78.126
91.222.170.103
200.74.158.82
27.109.119.60
124.109.38.217
183.91.7.106
178.72.71.54
95.141.142.46
143.215.130.33
59.94.134.222
202.168.229.110
59.99.41.204
79.56.192.128
59.94.132.44
180.178.190.37
164.132.108.171
117.207.237.135
213.178.34.190
178.72.70.110
117.205.4.136
118.97.45.30
178.72.68.90
180.178.190.150
109.237.97.141
178.72.78.47
164.132.200.121
117.195.83.8
180.148.209.190
124.43.64.233
115.178.27.33
180.178.189.34
95.25.71.31
115.197.203.117
111.125.66.146
2.95.30.122
2.95.58.158
59.97.169.159
202.166.164.46
223.179.146.27
186.33.86.218
59.99.41.45
116.73.206.203
178.72.71.126
178.72.68.8
103.21.58.121
185.9.19.107
194.36.111.59
141.8.225.244
54.72.9.115
198.54.117.210
50.62.250.1
50.63.202.38
209.99.64.20
50.63.202.94
184.168.221.49
184.168.221.62
50.63.202.54
50.63.202.35
184.168.221.72
199.59.243.120
50.63.202.71
184.168.221.78
108.179.192.37
72.52.4.91
184.168.221.36
192.64.119.13
50.63.202.32
50.63.202.37
89.187.85.11
192.185.141.89
162.255.119.250
185.53.179.29
185.27.134.201
185.27.134.93
185.27.134.144
185.27.134.120
185.27.134.140
116.202.101.219
84.38.185.51
111.15.166.68
27.124.40.170
164.132.48.230
85.202.163.127
118.69.176.13
111.90.150.192
27.78.40.128
46.43.90.85
212.92.23.152
60.214.127.246
95.220.140.101
95.165.65.191
60.2.165.42
59.92.123.104
218.22.253.37
175.107.32.186
182.71.75.106
164.132.56.147
162.144.107.64
121.58.230.5
49.207.185.223
212.71.237.140
61.184.176.231
210.13.99.66
181.2.151.236
122.176.142.45
1.203.178.11
107.172.15.135
116.202.26.141
87.97.126.177
213.152.161.149
95.26.213.236
111.249.147.29
216.158.226.206
181.162.165.211
187.7.71.55
49.124.151.30
218.161.68.57
177.72.87.7
122.160.25.50
109.120.150.70
114.96.71.150
111.249.161.208
46.44.59.217
82.193.109.199
189.46.3.128
2.133.174.46
114.36.106.192
77.126.99.230
116.203.165.79
83.25.1.32
60.247.92.186
45.78.7.252
222.211.65.72
117.81.74.8
218.83.244.226
192.160.102.170
64.42.179.43
95.37.138.179
95.37.167.168
95.32.73.28
95.86.215.118
95.235.112.20
95.233.29.90
95.80.175.176
95.220.190.101
95.32.195.21
95.173.236.123
95.237.197.200
95.168.107.19
95.155.49.249
95.140.215.7
95.110.86.251
94.50.161.245
95.125.165.97
94.241.253.55
94.41.17.242
94.64.221.49
94.241.195.252
94.21.120.255
93.81.98.56
94.248.18.163
93.118.168.201
93.181.41.128
93.178.95.54
92.241.122.55
92.49.181.230
92.36.224.5
93.124.93.250
92.36.188.4
92.252.170.100
92.248.180.60
92.36.144.151
92.249.153.233
92.81.222.217
91.229.208.69
91.83.180.220
91.138.66.143
91.241.142.0
91.189.246.149
90.188.102.124
90.150.203.82
89.254.252.71
89.254.238.172
89.146.162.46
89.178.212.162
88.252.92.100
89.214.58.109
89.109.46.75
89.109.198.250
87.5.93.208
87.4.83.20
87.20.115.157
87.5.52.103
87.116.229.214
87.117.11.11
87.18.24.249
85.26.235.65
85.14.31.38
84.53.198.249
84.53.198.216
84.53.198.206
85.105.25.190
84.53.198.169
84.224.223.109
84.46.246.236
84.237.136.6
84.46.243.248
84.53.198.8
84.53.198.117
84.53.198.146
84.51.85.58
83.4.115.7
84.232.53.24
83.30.139.240
83.30.100.7
83.242.255.253
83.11.14.37
82.55.198.210
82.55.111.25
81.198.112.7
81.163.41.212
81.163.43.74
81.0.82.211
80.80.173.58
79.37.104.153
79.172.119.174
79.172.112.96
79.33.217.58
79.25.196.125
79.22.9.136
79.47.152.132
79.186.103.217
79.139.195.70
79.43.32.232
79.186.12.127
79.172.103.125
79.17.140.92
79.126.114.218
78.92.63.85
78.92.32.202
79.126.71.62
79.127.113.235
79.126.115.110
78.92.117.219
79.11.222.142
78.84.96.46
78.29.21.78
77.49.154.157
78.131.111.129
78.110.65.106
78.110.64.127
77.238.211.117
77.222.114.11
77.222.109.187
77.222.104.56
77.222.112.210
76.226.89.100
69.57.239.254
64.42.179.59
62.240.7.8
63.230.130.197
62.240.7.15
62.182.82.10
61.64.108.29
61.64.147.29
61.3.98.45
61.231.225.152
61.227.188.173
61.227.199.249
61.216.176.84
61.227.102.1
61.224.229.125
61.223.230.117
61.224.46.121
61.224.231.87
61.224.230.197
61.140.62.212
61.0.86.98
61.0.93.181
61.1.236.174
60.191.240.10
60.190.132.58
60.243.214.101
60.17.116.220
60.167.134.204
59.99.146.117
59.99.66.250
59.99.65.80
59.99.51.101
59.99.133.107
59.99.48.85
59.99.131.137
59.99.132.74
59.98.138.100
59.98.137.113
59.99.137.182
59.98.138.177
59.96.137.71
59.99.129.125
59.98.192.230
59.98.193.99
59.96.137.143
59.98.192.91
59.96.138.165
59.99.192.232
59.94.249.174
59.95.70.82
59.98.169.50
59.95.177.208
59.95.222.240
59.94.249.239
59.94.113.29
59.92.169.178
59.92.41.227
59.92.175.245
59.93.17.3
59.92.72.91
59.92.175.2
59.92.175.244
59.92.171.112
59.92.174.62
59.92.167.216
59.92.167.111
59.92.160.103
59.89.24.220
59.92.162.122
59.89.41.95
59.89.231.194
59.91.239.84
59.89.226.122
59.52.102.95
59.88.233.221
59.88.233.30
59.88.235.124
59.55.82.14
59.177.73.228
59.104.139.154
59.115.182.224
58.222.106.106
5.39.220.78
5.175.78.135
5.139.53.203
49.144.194.71
49.207.183.11
49.207.184.137
49.124.131.156
49.158.2.61
46.71.142.133
46.71.131.122
46.71.233.117
46.71.174.132
46.71.26.84
46.71.65.188
46.71.84.178
46.72.73.105
46.72.54.250
46.70.84.79
46.70.4.143
46.98.204.56
46.70.236.111
46.70.250.147
46.33.222.241
46.57.115.143
46.185.239.5
46.175.98.11
46.51.218.71
46.149.184.14
46.130.1.134
46.100.175.253
45.64.239.182
43.228.130.169
42.239.152.246
42.115.169.229
42.119.30.70
42.113.97.218
42.113.188.145
41.74.140.185
41.77.209.15
41.77.11.130
41.72.105.130
41.238.246.238
41.232.231.88
41.233.252.193
41.200.203.220
41.129.13.243
39.42.146.120
39.40.98.94
39.33.233.208
39.33.25.94
37.114.136.15
37.114.134.250
37.114.175.4
37.114.167.149
36.237.167.133
36.232.89.119
36.238.5.57
36.224.156.58
36.224.81.213
36.225.148.145
31.23.110.185
31.166.139.119
31.16.91.225
31.181.130.248
31.170.175.237
31.134.240.48
31.171.155.102
27.7.216.149
27.7.216.128
27.79.75.112
27.72.76.10
27.6.216.55
27.6.216.144
27.78.40.57
27.6.216.19
27.79.209.40
27.6.204.55
27.254.41.5
27.254.142.31
27.227.186.160
27.2.115.84
27.17.34.234
223.240.85.75
222.254.149.100
222.253.182.135
222.124.113.18
221.212.178.157
220.136.186.156
220.142.228.215
220.143.22.166
220.142.160.139
220.134.90.231
220.135.249.206
220.143.9.92
219.85.237.136
220.134.209.246
218.29.112.130
218.250.245.3
218.161.53.31
218.173.85.39
217.66.152.128
217.118.83.189
213.80.203.243
213.167.223.58
213.167.211.138
213.242.42.245
212.36.28.70
212.156.86.62
212.107.140.2
210.89.62.169
210.89.63.5
210.56.26.113
209.141.43.173
203.99.53.107
203.73.34.18
203.73.105.211
202.89.73.138
202.83.57.56
202.83.56.8
202.83.56.148
203.115.85.76
202.176.184.161
202.164.139.14
202.142.125.200
202.142.90.46
201.220.190.254
201.191.196.6
201.76.5.13
200.93.108.125
201.140.122.13
200.93.80.29
200.27.140.130
200.161.220.14
2.92.237.166
2.40.63.222
2.193.198.117
2.192.70.48
2.192.134.22
2.192.226.10
2.192.226.204
2.192.32.76
2.194.6.100
2.179.81.45
2.181.250.84
199.249.230.27
2.140.97.77
198.199.114.53
198.199.98.117
196.219.235.42
195.175.109.150
194.44.26.226
194.44.26.254
194.125.225.210
192.241.220.23
191.5.238.188
190.99.94.66
190.99.94.207
190.7.20.162
190.51.71.146
190.239.247.10
190.249.214.33
190.51.97.67
190.36.3.145
190.219.16.202
190.77.219.24
190.232.70.241
190.144.61.51
190.196.234.3
190.141.164.140
189.41.6.117
189.69.216.183
189.79.71.150
189.217.192.53
189.234.89.171
189.174.120.134
189.15.67.238
189.15.171.37
188.59.156.18
189.111.19.110
188.49.236.109
188.48.168.15
188.171.3.117
187.27.240.7
188.143.48.249
187.27.194.181
187.27.213.245
187.74.108.90
187.75.13.33
188.164.220.196
187.37.189.233
187.23.14.109
187.16.154.231
187.13.7.131
186.9.239.190
186.9.239.203
186.89.176.66
186.50.129.139
186.37.207.17
186.89.124.35
186.37.120.212
186.219.255.16
186.46.136.214
186.210.63.9
186.143.5.153
186.206.150.168
186.224.164.154
186.142.135.44
186.143.1.30
186.141.7.250
186.142.134.6
186.143.3.216
186.142.135.181
186.142.134.188
186.141.7.127
186.142.133.185
186.142.134.222
186.142.135.101
183.82.33.78
183.82.205.110
186.141.1.120
186.140.134.225
186.140.134.111
186.10.62.179
186.140.1.52
186.10.26.26
186.140.1.129
186.140.135.239
186.140.135.31
186.140.1.130
186.140.1.112
186.140.0.158
186.140.0.53
186.140.0.230
186.140.0.133
183.88.80.183
183.250.42.47
185.106.144.10
182.240.227.236
182.177.241.248
182.177.128.238
182.177.228.201
182.177.224.116
182.177.212.149
182.177.168.182
182.177.204.130
182.177.160.99
182.177.194.120
182.177.144.28
182.177.136.100
181.64.13.140
181.82.231.114
181.0.4.70
181.0.9.150
181.17.27.194
181.0.8.84
181.0.8.192
181.48.116.115
181.0.27.139
181.0.28.70
180.242.18.193
181.0.16.21
181.0.1.100
180.253.166.104
180.246.181.197
180.123.156.215
178.72.78.206
178.72.77.213
178.72.77.161
178.72.75.134
178.72.78.166
178.72.76.188
178.72.76.186
178.72.77.149
178.72.81.216
178.72.68.81
178.72.81.19
178.72.68.249
178.44.228.75
178.72.71.4
178.72.75.76
178.72.81.189
178.72.68.219
178.72.81.185
178.72.70.117
178.72.81.136
178.72.68.190
178.72.77.101
178.72.69.9
178.72.75.52
178.44.11.155
178.72.70.101
178.72.81.117
178.72.75.54
178.72.77.86
178.72.78.104
178.72.70.11
178.72.81.10
178.72.78.106
178.72.76.42
178.44.171.12
178.90.12.23
178.72.76.237
178.72.77.82
178.72.71.15
178.72.77.59
178.72.78.77
178.72.75.231
178.72.69.24
178.72.75.211
178.72.78.71
178.72.68.132
178.72.68.110
178.72.68.114
178.72.76.226
178.72.71.135
178.46.211.109
178.218.104.252
178.72.78.246
178.72.77.43
178.206.10.102
178.173.196.186
178.173.142.117
178.72.68.108
178.72.70.96
178.217.206.195
178.72.68.1
178.72.69.122
178.72.69.110
178.72.70.37
178.205.12.156
178.72.70.61
178.205.35.51
178.204.137.178
178.44.245.36
178.44.249.135
178.214.184.2
178.204.0.60
178.214.164.240
177.84.41.92
178.147.19.177
177.9.224.19
177.106.235.65
177.106.5.168
177.106.235.228
177.106.165.35
177.106.166.128
177.191.207.250
177.39.230.88
177.106.121.206
177.106.128.237
177.105.224.111
177.20.166.83
176.8.16.206
176.113.123.253
175.107.2.211
175.107.0.85
175.107.1.79
175.107.2.210
175.182.109.94
175.140.230.152
175.107.14.26
175.107.1.223
175.107.11.211
175.107.11.155
173.249.18.113
175.137.182.94
175.107.13.214
175.107.2.6
173.44.55.179
171.226.168.150
171.241.225.182
171.237.86.103
171.229.86.16
171.226.177.37
171.236.38.54
164.132.113.57
151.31.14.80
154.72.169.206
154.72.171.234
154.72.168.218
150.185.252.222
151.80.29.232
14.97.218.174
14.194.238.230
14.192.208.212
14.170.159.223
14.160.70.182
14.175.24.84
14.186.41.224
14.181.228.107
14.189.91.225
14.186.173.66
14.169.2.201
14.102.20.214
14.172.178.125
14.177.154.48
128.71.181.9
125.228.68.53
125.229.88.18
125.227.234.187
125.75.37.98
125.45.88.14
125.228.170.125
125.26.162.204
125.224.167.122
125.228.144.245
125.42.214.117
125.42.215.240
125.42.215.90
125.231.160.162
124.31.107.234
124.44.90.184
124.234.246.154
124.227.218.51
124.31.104.236
124.248.174.106
124.106.226.35
124.106.227.70
123.185.223.254
123.185.223.11
123.21.233.207
123.21.141.18
123.26.16.117
123.194.32.138
123.204.6.71
122.236.203.141
122.194.11.79
122.118.65.75
122.121.74.117
121.236.107.34
120.59.41.165
119.228.135.144
120.57.117.227
119.234.20.97
117.6.127.31
118.160.99.63
118.160.214.92
118.70.187.90
118.170.37.23
118.96.148.217
118.172.144.193
118.167.198.79
118.168.161.73
117.80.17.138
117.68.194.44
117.3.66.37
117.253.244.31
117.253.244.222
117.253.240.215
117.247.27.113
117.253.106.144
117.247.163.239
117.247.164.130
117.253.99.156
117.239.4.146
117.253.105.198
117.253.104.231
117.253.104.236
117.241.185.5
117.222.178.189
117.222.179.245
117.248.132.110
117.221.122.233
117.241.113.187
117.241.113.76
117.242.249.242
117.220.101.232
117.220.77.98
117.219.119.124
117.217.234.90
117.221.93.11
117.219.113.161
117.216.29.22
117.217.233.151
117.216.1.192
117.216.29.148
117.215.249.179
117.216.25.60
117.216.28.248
117.216.0.209
117.215.147.117
117.215.146.224
117.215.146.65
117.216.25.55
117.216.25.254
117.217.232.128
117.216.0.77
117.217.232.174
117.216.5.255
117.216.25.115
117.212.173.94
117.216.16.246
117.211.44.90
117.212.162.242
117.215.145.5
117.214.152.50
117.211.40.208
117.211.40.167
117.211.40.163
117.216.30.151
117.215.144.195
117.211.44.209
117.212.170.39
117.216.26.55
117.216.20.24
117.216.26.174
117.215.1.80
117.216.30.1
117.216.3.250
117.211.40.134
117.216.2.212
117.211.43.69
117.212.18.65
117.211.37.144
117.212.173.140
117.212.169.239
117.211.43.73
117.211.43.48
117.211.37.188
117.211.37.18
117.211.40.114
117.214.210.236
117.211.36.4
117.211.39.3
117.212.172.242
117.211.47.85
117.211.43.24
117.214.246.148
117.211.39.166
117.212.172.139
117.211.6.8
117.211.39.164
117.211.37.139
117.214.249.237
117.211.47.194
117.211.39.130
117.211.42.38
117.211.33.214
117.211.39.131
117.211.33.227
117.212.164.95
117.212.168.174
117.212.164.77
117.212.163.212
117.211.41.30
117.212.164.134
117.211.46.36
117.212.160.113
117.212.171.193
117.211.32.61
117.211.46.205
117.211.46.138
117.211.38.165
117.211.41.216
117.211.38.162
117.211.11.158
117.204.157.152
117.211.41.102
117.211.40.81
117.204.140.40
117.205.145.65
117.205.145.131
117.205.151.16
117.201.72.173
117.211.32.12
117.203.8.69
117.208.74.8
117.207.10.128
117.197.3.153
117.205.111.31
117.199.200.39
117.200.135.67
117.200.230.161
117.194.206.115
117.198.171.165
117.194.205.95
117.198.173.183
117.197.24.62
117.197.25.234
117.197.1.102
117.198.249.175
117.197.6.75
117.194.165.167
117.194.205.235
117.198.249.246
117.197.149.86
117.197.15.106
117.207.35.195
117.201.77.45
117.197.0.59
117.194.205.71
117.207.239.176
117.201.17.229
117.197.173.250
117.197.9.62
117.196.75.43
117.198.246.106
117.199.43.31
117.200.94.254
117.198.161.181
117.195.211.11
117.197.9.32
117.197.50.245
117.200.90.210
117.194.203.226
117.194.203.132
117.200.91.25
117.197.84.133
117.200.92.185
117.200.90.251
117.198.244.75
117.197.5.225
117.197.8.105
117.194.201.119
117.199.216.195
117.194.97.30
117.197.168.209
117.195.109.99
117.197.169.77
117.198.243.14
117.195.147.98
117.197.4.35
117.194.150.57
117.197.154.180
117.194.67.123
117.194.228.170
117.194.196.218
117.197.7.41
117.194.198.155
117.197.11.5
117.194.224.225
117.194.174.100
117.196.19.184
117.193.109.60
117.194.206.26
116.68.101.133
116.203.185.64
116.105.16.89
115.242.247.86
114.40.143.132
114.40.129.215
114.40.143.25
114.42.182.53
114.42.206.246
114.40.166.88
114.42.223.58
114.38.211.239
114.38.200.66
114.39.35.43
114.46.210.78
114.27.97.120
114.38.104.126
114.39.30.228
114.39.201.157
114.24.218.221
114.39.160.78
114.45.21.176
114.39.165.137
114.37.38.66
114.27.84.41
114.27.8.52
114.27.131.39
114.46.142.70
114.37.138.235
114.39.154.218
114.37.12.90
114.38.7.67
114.39.11.252
114.37.1.19
114.25.111.73
114.41.223.78
114.41.222.161
114.24.19.63
114.27.214.233
114.37.118.151
115.114.40.105
114.38.59.200
114.41.11.248
114.36.189.221
114.27.203.206
114.41.168.121
114.43.183.143
114.27.168.132
114.38.45.121
114.47.175.187
114.42.65.24
114.38.45.71
114.32.213.135
114.32.184.121
114.27.19.95
114.38.40.180
114.38.38.140
115.165.214.110
114.93.187.166
114.239.102.50
114.27.140.236
114.25.36.170
114.25.77.203
114.25.36.216
114.47.165.172
114.25.21.249
114.25.181.172
114.239.50.230
114.143.182.226
113.26.203.0
113.88.239.29
113.172.132.245
113.175.9.202
113.174.43.253
113.20.118.50
113.193.82.33
113.183.60.34
113.172.96.242
113.172.222.120
113.163.202.118
113.162.244.1
113.162.244.36
113.167.37.174
113.162.78.58
113.160.56.58
112.66.110.12
112.78.3.81
112.66.108.52
112.72.153.32
112.205.66.86
112.104.36.49
111.252.231.8
111.254.235.214
111.243.139.73
111.254.203.217
111.254.209.236
111.92.76.126
111.252.157.39
111.246.95.177
111.253.217.204
111.255.12.24
111.255.13.73
111.253.198.241
111.243.39.224
111.254.46.163
111.242.136.247
111.250.161.218
111.254.46.140
111.254.34.174
111.249.205.219
110.77.236.58
110.137.195.231
109.122.20.70
109.184.102.88
109.175.4.86
109.173.62.108
109.125.161.182
109.125.164.207
109.125.160.189
109.125.164.133
109.168.188.48
109.115.187.31
109.125.161.105
109.191.53.189
109.111.8.12
109.184.58.67
109.197.81.46
109.122.22.254
106.51.5.134
104.131.110.81
106.51.19.224
104.254.90.251
105.110.68.85
103.49.228.10
103.21.77.110
103.225.95.34
1.53.41.81
1.53.143.215
1.85.217.4
1.52.125.113
101.0.34.57
1.85.217.29
1.85.219.155
101.109.202.20
1.85.42.195
1.170.164.166
1.170.115.86
1.170.65.131
1.170.109.74
1.170.62.178
1.170.104.243
1.170.236.170
1.170.235.242
1.173.163.236
1.173.161.115
1.161.152.212
1.170.2.102
1.160.170.139
1.160.226.63
1.168.194.29
1.172.221.37
1.168.184.76
1.162.224.94
108.179.252.83
37.208.97.2
2.132.63.15
188.19.200.244
176.226.153.174
176.221.29.123
1.234.70.26
185.220.101.29
171.25.193.78
171.25.193.235
185.220.101.1
185.220.101.21
185.220.101.38
185.220.101.45
192.42.116.24
171.25.193.20
178.20.55.16
176.121.81.51
185.100.87.139
185.220.101.8
185.220.101.13
185.220.101.25
185.220.101.28
185.220.101.3
185.220.101.60
185.177.151.34
23.129.64.216
104.244.78.233
176.97.113.53
109.69.67.17
166.70.207.2
176.58.100.98
185.220.101.9
185.220.101.20
185.220.101.18
185.220.101.30
185.220.101.33
185.220.101.6
162.247.74.7
23.129.64.213
158.69.201.47
185.220.101.17
185.220.101.26
185.220.101.22
185.220.101.15
185.220.100.254
185.220.101.27
192.42.116.20
185.233.100.23
185.220.101.53
185.220.101.69
23.129.64.217
185.34.33.2
185.220.101.5
185.220.101.2
192.42.116.13
185.220.101.52
185.220.101.56
198.96.155.3
185.100.87.192
185.220.101.34
185.220.101.36
195.176.3.24
195.176.3.23
192.42.116.19
185.220.101.65
185.130.44.108
171.25.193.77
185.129.62.62
162.210.173.17
185.117.215.9
185.220.101.10
185.220.101.46
103.28.52.93
185.220.101.62
82.221.128.191
204.85.191.9
162.247.72.199
128.31.0.13
185.220.101.12
185.220.101.35
185.220.101.37
185.220.101.31
185.220.101.68
23.129.64.211
138.59.18.110
37.48.120.196
185.220.101.0
185.220.101.40
192.42.116.18
185.220.101.7
192.42.116.25
162.247.74.217
162.247.74.200
185.220.101.64
204.8.156.142
89.234.157.254
216.239.90.19
185.220.101.14
185.191.204.254
185.220.100.255
185.220.100.253
192.42.116.14
192.42.116.27
74.82.47.194
185.220.101.50
185.220.101.49
185.220.101.4
185.220.100.252
192.42.116.23
185.220.101.55
185.220.101.67
91.203.145.116
23.129.64.150
185.42.170.203
80.241.60.207
185.220.101.16
204.194.29.4
171.25.193.25
217.170.201.71
80.67.172.162
45.79.144.222
185.220.101.24
200.122.181.2
162.247.74.213
185.220.101.70
23.129.64.171
23.129.64.173
104.244.74.97
95.128.43.164
160.119.249.240
192.42.116.15
192.42.116.22
192.42.116.26
162.247.74.27
185.220.101.48
204.85.191.8
125.212.241.131
23.129.64.219
23.129.64.214
23.129.64.212
23.129.64.218
185.100.85.132
185.220.101.32
162.247.74.206
185.220.101.54
82.221.131.5
23.129.64.215
212.21.66.6
93.95.228.125
162.247.74.204
23.129.64.172
185.220.101.61
176.97.114.202
35.0.127.52
185.100.87.183
185.220.101.42
185.220.101.44
71.19.144.106
185.113.128.30
192.42.116.28
162.247.74.202
66.146.193.33
185.220.101.58
217.12.221.131
54.36.108.162
109.169.33.163
82.221.139.190
77.81.247.72
185.220.101.41
185.227.68.78
173.249.57.253
162.247.74.74
162.247.74.216
185.220.101.47
37.228.129.5
185.220.101.66
149.56.44.47
185.129.62.63
91.92.109.43
185.56.171.94
185.220.101.57
95.143.193.125
192.42.116.17
162.247.74.201
185.220.101.51
198.50.191.95
207.192.70.250
185.100.86.182
223.240.90.213
122.160.156.85
202.157.76.122
 
bbrokeaf said:
Lots of crawlers, most popular ones are from google, bing, yahoo, and social media sites. The question is why you consider them garbage.

Go to your nginx server block configuration and add a 'deny' directive. Target shady user agent strings, i will list some resources for that below, and restart and clear cache.

I will apply a list of known malicious IP's so you can block them in your server.

Hope this helps :)



IP Ban List 1
185.234.216.59
212.83.185.105
194.143.136.122
113.160.133.8
194.44.38.51
176.8.55.92
131.253.18.12
131.253.18.11
46.149.184.5
46.211.241.39
109.196.187.208
185.242.113.224
46.101.197.155
108.28.164.248
117.215.207.136
41.33.58.100
190.14.228.17
149.202.62.12
212.156.149.146
149.56.99.38
149.56.99.37
27.106.62.214
91.140.233.130
163.172.211.135
41.200.247.222
181.189.148.98
58.137.148.186
67.205.149.140
185.62.189.56
200.5.112.180
94.183.112.60
85.90.244.52
89.36.208.231
163.172.43.52
185.86.149.97
177.85.98.227
81.170.184.90
201.110.183.154
196.200.16.23
163.172.38.173
158.58.170.186
146.0.77.50
163.172.158.208
185.69.168.112
120.50.19.10
87.185.31.200
59.99.43.205
163.172.154.105
163.172.209.46
146.185.139.55
185.107.70.202
159.203.30.48
181.143.243.98
116.68.103.36
188.161.150.22
41.33.197.132
117.239.224.138
14.176.1.82
213.74.133.14
59.93.29.14
41.138.60.225
163.172.152.231
197.211.53.25
113.176.70.96
5.255.80.27
89.33.246.114
197.254.106.218
162.210.173.109
212.47.248.81
119.93.79.68
98.124.243.32
59.93.84.191
119.17.192.102
178.62.18.173
178.159.36.185
89.35.178.104
192.64.119.234
201.76.186.230
138.97.240.250
151.100.179.50
46.28.111.122
217.12.201.109
116.68.98.252
107.182.131.117
46.101.138.211
101.98.11.146
187.44.138.226
31.31.72.43
64.137.178.3
193.138.219.231
186.205.89.48
185.100.87.63
84.251.91.165
27.50.94.251
31.16.91.237
108.61.187.24
212.47.247.226
92.222.92.152
104.236.58.27
37.187.247.3
101.0.54.130
93.219.113.201
87.120.254.200
51.254.215.7
178.32.53.124
181.143.153.250
185.61.138.104
104.168.177.9
185.86.13.130
139.59.9.200
198.134.125.78
116.107.184.31
118.69.176.205
27.116.21.219
181.143.253.106
85.207.155.39
185.128.40.220
163.172.143.114
5.196.58.96
94.26.2.74
103.214.54.82
45.55.178.34
167.114.238.104
185.100.85.191
95.31.22.193
92.222.103.234
46.148.26.78
45.79.85.112
46.39.102.250
82.221.129.96
199.175.49.21
72.34.42.146
45.56.90.85
91.234.33.250
212.47.227.72
49.88.121.163
149.202.44.177
193.200.241.195
209.66.119.150
149.202.42.188
207.244.78.3
166.62.28.107
188.126.81.155
185.76.35.11
128.153.145.125
5.28.62.85
78.39.117.17
89.238.212.68
113.161.8.108
178.32.53.94
118.174.3.106
158.69.194.36
108.167.180.12
185.38.151.64
186.42.192.22
45.33.48.204
108.61.123.73
82.211.19.129
216.17.99.183
158.69.208.131
69.195.129.71
223.81.192.191
87.255.51.230
199.87.154.251
37.48.109.107
46.166.188.208
92.222.28.243
185.104.120.7
217.160.2.23
193.232.240.11
193.111.136.162
122.52.159.39
146.185.128.226
107.170.1.205
46.166.190.182
109.188.126.11
94.103.175.86
5.199.171.58
185.27.134.131
146.185.135.43
193.110.157.49
31.172.30.4
81.30.158.81
104.237.152.195
212.83.40.239
46.28.110.136
91.219.236.232
62.210.87.71
62.210.87.30
65.181.112.128
212.47.238.193
174.127.99.132
109.232.223.115
210.92.18.118
178.32.53.131
212.83.40.238
141.138.141.208
162.247.72.27
71.19.157.127
187.217.183.146
68.206.21.41
185.17.184.228
217.12.204.104
185.53.178.12
116.68.102.237
198.50.200.139
31.192.228.185
162.247.72.216
59.93.21.183
77.76.228.4
201.87.244.10
108.167.140.20
189.236.206.143
200.50.124.165
139.0.6.91
109.188.125.3
109.188.124.25
109.188.125.13
210.245.54.103
218.65.6.122
184.168.221.17
201.234.180.155
109.188.127.28
198.61.227.6
197.159.9.29
31.41.59.29
178.72.78.52
115.127.13.235
201.43.143.92
49.89.225.74
49.71.142.141
117.196.54.136
82.191.238.172
180.178.189.50
117.196.17.4
178.72.70.160
46.153.152.177
180.178.188.222
180.178.189.76
164.132.172.220
95.38.74.75
85.174.205.255
180.178.188.150
117.199.234.103
27.128.226.95
178.72.70.167
178.72.78.136
178.72.77.167
94.156.58.198
91.187.121.119
78.106.220.129
193.84.22.89
103.23.135.78
178.72.78.126
91.222.170.103
200.74.158.82
27.109.119.60
124.109.38.217
183.91.7.106
178.72.71.54
95.141.142.46
143.215.130.33
59.94.134.222
202.168.229.110
59.99.41.204
79.56.192.128
59.94.132.44
180.178.190.37
164.132.108.171
117.207.237.135
213.178.34.190
178.72.70.110
117.205.4.136
118.97.45.30
178.72.68.90
180.178.190.150
109.237.97.141
178.72.78.47
164.132.200.121
117.195.83.8
180.148.209.190
124.43.64.233
115.178.27.33
180.178.189.34
95.25.71.31
115.197.203.117
111.125.66.146
2.95.30.122
2.95.58.158
59.97.169.159
202.166.164.46
223.179.146.27
186.33.86.218
59.99.41.45
116.73.206.203
178.72.71.126
178.72.68.8
103.21.58.121
185.9.19.107
194.36.111.59
141.8.225.244
54.72.9.115
198.54.117.210
50.62.250.1
50.63.202.38
209.99.64.20
50.63.202.94
184.168.221.49
184.168.221.62
50.63.202.54
50.63.202.35
184.168.221.72
199.59.243.120
50.63.202.71
184.168.221.78
108.179.192.37
72.52.4.91
184.168.221.36
192.64.119.13
50.63.202.32
50.63.202.37
89.187.85.11
192.185.141.89
162.255.119.250
185.53.179.29
185.27.134.201
185.27.134.93
185.27.134.144
185.27.134.120
185.27.134.140
116.202.101.219
84.38.185.51
111.15.166.68
27.124.40.170
164.132.48.230
85.202.163.127
118.69.176.13
111.90.150.192
27.78.40.128
46.43.90.85
212.92.23.152
60.214.127.246
95.220.140.101
95.165.65.191
60.2.165.42
59.92.123.104
218.22.253.37
175.107.32.186
182.71.75.106
164.132.56.147
162.144.107.64
121.58.230.5
49.207.185.223
212.71.237.140
61.184.176.231
210.13.99.66
181.2.151.236
122.176.142.45
1.203.178.11
107.172.15.135
116.202.26.141
87.97.126.177
213.152.161.149
95.26.213.236
111.249.147.29
216.158.226.206
181.162.165.211
187.7.71.55
49.124.151.30
218.161.68.57
177.72.87.7
122.160.25.50
109.120.150.70
114.96.71.150
111.249.161.208
46.44.59.217
82.193.109.199
189.46.3.128
2.133.174.46
114.36.106.192
77.126.99.230
116.203.165.79
83.25.1.32
60.247.92.186
45.78.7.252
222.211.65.72
117.81.74.8
218.83.244.226
192.160.102.170
64.42.179.43
95.37.138.179
95.37.167.168
95.32.73.28
95.86.215.118
95.235.112.20
95.233.29.90
95.80.175.176
95.220.190.101
95.32.195.21
95.173.236.123
95.237.197.200
95.168.107.19
95.155.49.249
95.140.215.7
95.110.86.251
94.50.161.245
95.125.165.97
94.241.253.55
94.41.17.242
94.64.221.49
94.241.195.252
94.21.120.255
93.81.98.56
94.248.18.163
93.118.168.201
93.181.41.128
93.178.95.54
92.241.122.55
92.49.181.230
92.36.224.5
93.124.93.250
92.36.188.4
92.252.170.100
92.248.180.60
92.36.144.151
92.249.153.233
92.81.222.217
91.229.208.69
91.83.180.220
91.138.66.143
91.241.142.0
91.189.246.149
90.188.102.124
90.150.203.82
89.254.252.71
89.254.238.172
89.146.162.46
89.178.212.162
88.252.92.100
89.214.58.109
89.109.46.75
89.109.198.250
87.5.93.208
87.4.83.20
87.20.115.157
87.5.52.103
87.116.229.214
87.117.11.11
87.18.24.249
85.26.235.65
85.14.31.38
84.53.198.249
84.53.198.216
84.53.198.206
85.105.25.190
84.53.198.169
84.224.223.109
84.46.246.236
84.237.136.6
84.46.243.248
84.53.198.8
84.53.198.117
84.53.198.146
84.51.85.58
83.4.115.7
84.232.53.24
83.30.139.240
83.30.100.7
83.242.255.253
83.11.14.37
82.55.198.210
82.55.111.25
81.198.112.7
81.163.41.212
81.163.43.74
81.0.82.211
80.80.173.58
79.37.104.153
79.172.119.174
79.172.112.96
79.33.217.58
79.25.196.125
79.22.9.136
79.47.152.132
79.186.103.217
79.139.195.70
79.43.32.232
79.186.12.127
79.172.103.125
79.17.140.92
79.126.114.218
78.92.63.85
78.92.32.202
79.126.71.62
79.127.113.235
79.126.115.110
78.92.117.219
79.11.222.142
78.84.96.46
78.29.21.78
77.49.154.157
78.131.111.129
78.110.65.106
78.110.64.127
77.238.211.117
77.222.114.11
77.222.109.187
77.222.104.56
77.222.112.210
76.226.89.100
69.57.239.254
64.42.179.59
62.240.7.8
63.230.130.197
62.240.7.15
62.182.82.10
61.64.108.29
61.64.147.29
61.3.98.45
61.231.225.152
61.227.188.173
61.227.199.249
61.216.176.84
61.227.102.1
61.224.229.125
61.223.230.117
61.224.46.121
61.224.231.87
61.224.230.197
61.140.62.212
61.0.86.98
61.0.93.181
61.1.236.174
60.191.240.10
60.190.132.58
60.243.214.101
60.17.116.220
60.167.134.204
59.99.146.117
59.99.66.250
59.99.65.80
59.99.51.101
59.99.133.107
59.99.48.85
59.99.131.137
59.99.132.74
59.98.138.100
59.98.137.113
59.99.137.182
59.98.138.177
59.96.137.71
59.99.129.125
59.98.192.230
59.98.193.99
59.96.137.143
59.98.192.91
59.96.138.165
59.99.192.232
59.94.249.174
59.95.70.82
59.98.169.50
59.95.177.208
59.95.222.240
59.94.249.239
59.94.113.29
59.92.169.178
59.92.41.227
59.92.175.245
59.93.17.3
59.92.72.91
59.92.175.2
59.92.175.244
59.92.171.112
59.92.174.62
59.92.167.216
59.92.167.111
59.92.160.103
59.89.24.220
59.92.162.122
59.89.41.95
59.89.231.194
59.91.239.84
59.89.226.122
59.52.102.95
59.88.233.221
59.88.233.30
59.88.235.124
59.55.82.14
59.177.73.228
59.104.139.154
59.115.182.224
58.222.106.106
5.39.220.78
5.175.78.135
5.139.53.203
49.144.194.71
49.207.183.11
49.207.184.137
49.124.131.156
49.158.2.61
46.71.142.133
46.71.131.122
46.71.233.117
46.71.174.132
46.71.26.84
46.71.65.188
46.71.84.178
46.72.73.105
46.72.54.250
46.70.84.79
46.70.4.143
46.98.204.56
46.70.236.111
46.70.250.147
46.33.222.241
46.57.115.143
46.185.239.5
46.175.98.11
46.51.218.71
46.149.184.14
46.130.1.134
46.100.175.253
45.64.239.182
43.228.130.169
42.239.152.246
42.115.169.229
42.119.30.70
42.113.97.218
42.113.188.145
41.74.140.185
41.77.209.15
41.77.11.130
41.72.105.130
41.238.246.238
41.232.231.88
41.233.252.193
41.200.203.220
41.129.13.243
39.42.146.120
39.40.98.94
39.33.233.208
39.33.25.94
37.114.136.15
37.114.134.250
37.114.175.4
37.114.167.149
36.237.167.133
36.232.89.119
36.238.5.57
36.224.156.58
36.224.81.213
36.225.148.145
31.23.110.185
31.166.139.119
31.16.91.225
31.181.130.248
31.170.175.237
31.134.240.48
31.171.155.102
27.7.216.149
27.7.216.128
27.79.75.112
27.72.76.10
27.6.216.55
27.6.216.144
27.78.40.57
27.6.216.19
27.79.209.40
27.6.204.55
27.254.41.5
27.254.142.31
27.227.186.160
27.2.115.84
27.17.34.234
223.240.85.75
222.254.149.100
222.253.182.135
222.124.113.18
221.212.178.157
220.136.186.156
220.142.228.215
220.143.22.166
220.142.160.139
220.134.90.231
220.135.249.206
220.143.9.92
219.85.237.136
220.134.209.246
218.29.112.130
218.250.245.3
218.161.53.31
218.173.85.39
217.66.152.128
217.118.83.189
213.80.203.243
213.167.223.58
213.167.211.138
213.242.42.245
212.36.28.70
212.156.86.62
212.107.140.2
210.89.62.169
210.89.63.5
210.56.26.113
209.141.43.173
203.99.53.107
203.73.34.18
203.73.105.211
202.89.73.138
202.83.57.56
202.83.56.8
202.83.56.148
203.115.85.76
202.176.184.161
202.164.139.14
202.142.125.200
202.142.90.46
201.220.190.254
201.191.196.6
201.76.5.13
200.93.108.125
201.140.122.13
200.93.80.29
200.27.140.130
200.161.220.14
2.92.237.166
2.40.63.222
2.193.198.117
2.192.70.48
2.192.134.22
2.192.226.10
2.192.226.204
2.192.32.76
2.194.6.100
2.179.81.45
2.181.250.84
199.249.230.27
2.140.97.77
198.199.114.53
198.199.98.117
196.219.235.42
195.175.109.150
194.44.26.226
194.44.26.254
194.125.225.210
192.241.220.23
191.5.238.188
190.99.94.66
190.99.94.207
190.7.20.162
190.51.71.146
190.239.247.10
190.249.214.33
190.51.97.67
190.36.3.145
190.219.16.202
190.77.219.24
190.232.70.241
190.144.61.51
190.196.234.3
190.141.164.140
189.41.6.117
189.69.216.183
189.79.71.150
189.217.192.53
189.234.89.171
189.174.120.134
189.15.67.238
189.15.171.37
188.59.156.18
189.111.19.110
188.49.236.109
188.48.168.15
188.171.3.117
187.27.240.7
188.143.48.249
187.27.194.181
187.27.213.245
187.74.108.90
187.75.13.33
188.164.220.196
187.37.189.233
187.23.14.109
187.16.154.231
187.13.7.131
186.9.239.190
186.9.239.203
186.89.176.66
186.50.129.139
186.37.207.17
186.89.124.35
186.37.120.212
186.219.255.16
186.46.136.214
186.210.63.9
186.143.5.153
186.206.150.168
186.224.164.154
186.142.135.44
186.143.1.30
186.141.7.250
186.142.134.6
186.143.3.216
186.142.135.181
186.142.134.188
186.141.7.127
186.142.133.185
186.142.134.222
186.142.135.101
183.82.33.78
183.82.205.110
186.141.1.120
186.140.134.225
186.140.134.111
186.10.62.179
186.140.1.52
186.10.26.26
186.140.1.129
186.140.135.239
186.140.135.31
186.140.1.130
186.140.1.112
186.140.0.158
186.140.0.53
186.140.0.230
186.140.0.133
183.88.80.183
183.250.42.47
185.106.144.10
182.240.227.236
182.177.241.248
182.177.128.238
182.177.228.201
182.177.224.116
182.177.212.149
182.177.168.182
182.177.204.130
182.177.160.99
182.177.194.120
182.177.144.28
182.177.136.100
181.64.13.140
181.82.231.114
181.0.4.70
181.0.9.150
181.17.27.194
181.0.8.84
181.0.8.192
181.48.116.115
181.0.27.139
181.0.28.70
180.242.18.193
181.0.16.21
181.0.1.100
180.253.166.104
180.246.181.197
180.123.156.215
178.72.78.206
178.72.77.213
178.72.77.161
178.72.75.134
178.72.78.166
178.72.76.188
178.72.76.186
178.72.77.149
178.72.81.216
178.72.68.81
178.72.81.19
178.72.68.249
178.44.228.75
178.72.71.4
178.72.75.76
178.72.81.189
178.72.68.219
178.72.81.185
178.72.70.117
178.72.81.136
178.72.68.190
178.72.77.101
178.72.69.9
178.72.75.52
178.44.11.155
178.72.70.101
178.72.81.117
178.72.75.54
178.72.77.86
178.72.78.104
178.72.70.11
178.72.81.10
178.72.78.106
178.72.76.42
178.44.171.12
178.90.12.23
178.72.76.237
178.72.77.82
178.72.71.15
178.72.77.59
178.72.78.77
178.72.75.231
178.72.69.24
178.72.75.211
178.72.78.71
178.72.68.132
178.72.68.110
178.72.68.114
178.72.76.226
178.72.71.135
178.46.211.109
178.218.104.252
178.72.78.246
178.72.77.43
178.206.10.102
178.173.196.186
178.173.142.117
178.72.68.108
178.72.70.96
178.217.206.195
178.72.68.1
178.72.69.122
178.72.69.110
178.72.70.37
178.205.12.156
178.72.70.61
178.205.35.51
178.204.137.178
178.44.245.36
178.44.249.135
178.214.184.2
178.204.0.60
178.214.164.240
177.84.41.92
178.147.19.177
177.9.224.19
177.106.235.65
177.106.5.168
177.106.235.228
177.106.165.35
177.106.166.128
177.191.207.250
177.39.230.88
177.106.121.206
177.106.128.237
177.105.224.111
177.20.166.83
176.8.16.206
176.113.123.253
175.107.2.211
175.107.0.85
175.107.1.79
175.107.2.210
175.182.109.94
175.140.230.152
175.107.14.26
175.107.1.223
175.107.11.211
175.107.11.155
173.249.18.113
175.137.182.94
175.107.13.214
175.107.2.6
173.44.55.179
171.226.168.150
171.241.225.182
171.237.86.103
171.229.86.16
171.226.177.37
171.236.38.54
164.132.113.57
151.31.14.80
154.72.169.206
154.72.171.234
154.72.168.218
150.185.252.222
151.80.29.232
14.97.218.174
14.194.238.230
14.192.208.212
14.170.159.223
14.160.70.182
14.175.24.84
14.186.41.224
14.181.228.107
14.189.91.225
14.186.173.66
14.169.2.201
14.102.20.214
14.172.178.125
14.177.154.48
128.71.181.9
125.228.68.53
125.229.88.18
125.227.234.187
125.75.37.98
125.45.88.14
125.228.170.125
125.26.162.204
125.224.167.122
125.228.144.245
125.42.214.117
125.42.215.240
125.42.215.90
125.231.160.162
124.31.107.234
124.44.90.184
124.234.246.154
124.227.218.51
124.31.104.236
124.248.174.106
124.106.226.35
124.106.227.70
123.185.223.254
123.185.223.11
123.21.233.207
123.21.141.18
123.26.16.117
123.194.32.138
123.204.6.71
122.236.203.141
122.194.11.79
122.118.65.75
122.121.74.117
121.236.107.34
120.59.41.165
119.228.135.144
120.57.117.227
119.234.20.97
117.6.127.31
118.160.99.63
118.160.214.92
118.70.187.90
118.170.37.23
118.96.148.217
118.172.144.193
118.167.198.79
118.168.161.73
117.80.17.138
117.68.194.44
117.3.66.37
117.253.244.31
117.253.244.222
117.253.240.215
117.247.27.113
117.253.106.144
117.247.163.239
117.247.164.130
117.253.99.156
117.239.4.146
117.253.105.198
117.253.104.231
117.253.104.236
117.241.185.5
117.222.178.189
117.222.179.245
117.248.132.110
117.221.122.233
117.241.113.187
117.241.113.76
117.242.249.242
117.220.101.232
117.220.77.98
117.219.119.124
117.217.234.90
117.221.93.11
117.219.113.161
117.216.29.22
117.217.233.151
117.216.1.192
117.216.29.148
117.215.249.179
117.216.25.60
117.216.28.248
117.216.0.209
117.215.147.117
117.215.146.224
117.215.146.65
117.216.25.55
117.216.25.254
117.217.232.128
117.216.0.77
117.217.232.174
117.216.5.255
117.216.25.115
117.212.173.94
117.216.16.246
117.211.44.90
117.212.162.242
117.215.145.5
117.214.152.50
117.211.40.208
117.211.40.167
117.211.40.163
117.216.30.151
117.215.144.195
117.211.44.209
117.212.170.39
117.216.26.55
117.216.20.24
117.216.26.174
117.215.1.80
117.216.30.1
117.216.3.250
117.211.40.134
117.216.2.212
117.211.43.69
117.212.18.65
117.211.37.144
117.212.173.140
117.212.169.239
117.211.43.73
117.211.43.48
117.211.37.188
117.211.37.18
117.211.40.114
117.214.210.236
117.211.36.4
117.211.39.3
117.212.172.242
117.211.47.85
117.211.43.24
117.214.246.148
117.211.39.166
117.212.172.139
117.211.6.8
117.211.39.164
117.211.37.139
117.214.249.237
117.211.47.194
117.211.39.130
117.211.42.38
117.211.33.214
117.211.39.131
117.211.33.227
117.212.164.95
117.212.168.174
117.212.164.77
117.212.163.212
117.211.41.30
117.212.164.134
117.211.46.36
117.212.160.113
117.212.171.193
117.211.32.61
117.211.46.205
117.211.46.138
117.211.38.165
117.211.41.216
117.211.38.162
117.211.11.158
117.204.157.152
117.211.41.102
117.211.40.81
117.204.140.40
117.205.145.65
117.205.145.131
117.205.151.16
117.201.72.173
117.211.32.12
117.203.8.69
117.208.74.8
117.207.10.128
117.197.3.153
117.205.111.31
117.199.200.39
117.200.135.67
117.200.230.161
117.194.206.115
117.198.171.165
117.194.205.95
117.198.173.183
117.197.24.62
117.197.25.234
117.197.1.102
117.198.249.175
117.197.6.75
117.194.165.167
117.194.205.235
117.198.249.246
117.197.149.86
117.197.15.106
117.207.35.195
117.201.77.45
117.197.0.59
117.194.205.71
117.207.239.176
117.201.17.229
117.197.173.250
117.197.9.62
117.196.75.43
117.198.246.106
117.199.43.31
117.200.94.254
117.198.161.181
117.195.211.11
117.197.9.32
117.197.50.245
117.200.90.210
117.194.203.226
117.194.203.132
117.200.91.25
117.197.84.133
117.200.92.185
117.200.90.251
117.198.244.75
117.197.5.225
117.197.8.105
117.194.201.119
117.199.216.195
117.194.97.30
117.197.168.209
117.195.109.99
117.197.169.77
117.198.243.14
117.195.147.98
117.197.4.35
117.194.150.57
117.197.154.180
117.194.67.123
117.194.228.170
117.194.196.218
117.197.7.41
117.194.198.155
117.197.11.5
117.194.224.225
117.194.174.100
117.196.19.184
117.193.109.60
117.194.206.26
116.68.101.133
116.203.185.64
116.105.16.89
115.242.247.86
114.40.143.132
114.40.129.215
114.40.143.25
114.42.182.53
114.42.206.246
114.40.166.88
114.42.223.58
114.38.211.239
114.38.200.66
114.39.35.43
114.46.210.78
114.27.97.120
114.38.104.126
114.39.30.228
114.39.201.157
114.24.218.221
114.39.160.78
114.45.21.176
114.39.165.137
114.37.38.66
114.27.84.41
114.27.8.52
114.27.131.39
114.46.142.70
114.37.138.235
114.39.154.218
114.37.12.90
114.38.7.67
114.39.11.252
114.37.1.19
114.25.111.73
114.41.223.78
114.41.222.161
114.24.19.63
114.27.214.233
114.37.118.151
115.114.40.105
114.38.59.200
114.41.11.248
114.36.189.221
114.27.203.206
114.41.168.121
114.43.183.143
114.27.168.132
114.38.45.121
114.47.175.187
114.42.65.24
114.38.45.71
114.32.213.135
114.32.184.121
114.27.19.95
114.38.40.180
114.38.38.140
115.165.214.110
114.93.187.166
114.239.102.50
114.27.140.236
114.25.36.170
114.25.77.203
114.25.36.216
114.47.165.172
114.25.21.249
114.25.181.172
114.239.50.230
114.143.182.226
113.26.203.0
113.88.239.29
113.172.132.245
113.175.9.202
113.174.43.253
113.20.118.50
113.193.82.33
113.183.60.34
113.172.96.242
113.172.222.120
113.163.202.118
113.162.244.1
113.162.244.36
113.167.37.174
113.162.78.58
113.160.56.58
112.66.110.12
112.78.3.81
112.66.108.52
112.72.153.32
112.205.66.86
112.104.36.49
111.252.231.8
111.254.235.214
111.243.139.73
111.254.203.217
111.254.209.236
111.92.76.126
111.252.157.39
111.246.95.177
111.253.217.204
111.255.12.24
111.255.13.73
111.253.198.241
111.243.39.224
111.254.46.163
111.242.136.247
111.250.161.218
111.254.46.140
111.254.34.174
111.249.205.219
110.77.236.58
110.137.195.231
109.122.20.70
109.184.102.88
109.175.4.86
109.173.62.108
109.125.161.182
109.125.164.207
109.125.160.189
109.125.164.133
109.168.188.48
109.115.187.31
109.125.161.105
109.191.53.189
109.111.8.12
109.184.58.67
109.197.81.46
109.122.22.254
106.51.5.134
104.131.110.81
106.51.19.224
104.254.90.251
105.110.68.85
103.49.228.10
103.21.77.110
103.225.95.34
1.53.41.81
1.53.143.215
1.85.217.4
1.52.125.113
101.0.34.57
1.85.217.29
1.85.219.155
101.109.202.20
1.85.42.195
1.170.164.166
1.170.115.86
1.170.65.131
1.170.109.74
1.170.62.178
1.170.104.243
1.170.236.170
1.170.235.242
1.173.163.236
1.173.161.115
1.161.152.212
1.170.2.102
1.160.170.139
1.160.226.63
1.168.194.29
1.172.221.37
1.168.184.76
1.162.224.94
108.179.252.83
37.208.97.2
2.132.63.15
188.19.200.244
176.226.153.174
176.221.29.123
1.234.70.26
185.220.101.29
171.25.193.78
171.25.193.235
185.220.101.1
185.220.101.21
185.220.101.38
185.220.101.45
192.42.116.24
171.25.193.20
178.20.55.16
176.121.81.51
185.100.87.139
185.220.101.8
185.220.101.13
185.220.101.25
185.220.101.28
185.220.101.3
185.220.101.60
185.177.151.34
23.129.64.216
104.244.78.233
176.97.113.53
109.69.67.17
166.70.207.2
176.58.100.98
185.220.101.9
185.220.101.20
185.220.101.18
185.220.101.30
185.220.101.33
185.220.101.6
162.247.74.7
23.129.64.213
158.69.201.47
185.220.101.17
185.220.101.26
185.220.101.22
185.220.101.15
185.220.100.254
185.220.101.27
192.42.116.20
185.233.100.23
185.220.101.53
185.220.101.69
23.129.64.217
185.34.33.2
185.220.101.5
185.220.101.2
192.42.116.13
185.220.101.52
185.220.101.56
198.96.155.3
185.100.87.192
185.220.101.34
185.220.101.36
195.176.3.24
195.176.3.23
192.42.116.19
185.220.101.65
185.130.44.108
171.25.193.77
185.129.62.62
162.210.173.17
185.117.215.9
185.220.101.10
185.220.101.46
103.28.52.93
185.220.101.62
82.221.128.191
204.85.191.9
162.247.72.199
128.31.0.13
185.220.101.12
185.220.101.35
185.220.101.37
185.220.101.31
185.220.101.68
23.129.64.211
138.59.18.110
37.48.120.196
185.220.101.0
185.220.101.40
192.42.116.18
185.220.101.7
192.42.116.25
162.247.74.217
162.247.74.200
185.220.101.64
204.8.156.142
89.234.157.254
216.239.90.19
185.220.101.14
185.191.204.254
185.220.100.255
185.220.100.253
192.42.116.14
192.42.116.27
74.82.47.194
185.220.101.50
185.220.101.49
185.220.101.4
185.220.100.252
192.42.116.23
185.220.101.55
185.220.101.67
91.203.145.116
23.129.64.150
185.42.170.203
80.241.60.207
185.220.101.16
204.194.29.4
171.25.193.25
217.170.201.71
80.67.172.162
45.79.144.222
185.220.101.24
200.122.181.2
162.247.74.213
185.220.101.70
23.129.64.171
23.129.64.173
104.244.74.97
95.128.43.164
160.119.249.240
192.42.116.15
192.42.116.22
192.42.116.26
162.247.74.27
185.220.101.48
204.85.191.8
125.212.241.131
23.129.64.219
23.129.64.214
23.129.64.212
23.129.64.218
185.100.85.132
185.220.101.32
162.247.74.206
185.220.101.54
82.221.131.5
23.129.64.215
212.21.66.6
93.95.228.125
162.247.74.204
23.129.64.172
185.220.101.61
176.97.114.202
35.0.127.52
185.100.87.183
185.220.101.42
185.220.101.44
71.19.144.106
185.113.128.30
192.42.116.28
162.247.74.202
66.146.193.33
185.220.101.58
217.12.221.131
54.36.108.162
109.169.33.163
82.221.139.190
77.81.247.72
185.220.101.41
185.227.68.78
173.249.57.253
162.247.74.74
162.247.74.216
185.220.101.47
37.228.129.5
185.220.101.66
149.56.44.47
185.129.62.63
91.92.109.43
185.56.171.94
185.220.101.57
95.143.193.125
192.42.116.17
162.247.74.201
185.220.101.51
198.50.191.95
207.192.70.250
185.100.86.182
223.240.90.213
122.160.156.85
202.157.76.122
Click to expand...
Thank you for your reply. However, this is not ideal. Because the IP of the robot will change after a period of time. Do you know the iconic features of those garbage robots? For example, ua:Google
 
lanmaok said:
Thank you for your reply. However, this is not ideal. Because the IP of the robot will change after a period of time. Do you know the iconic features of those garbage robots? For example, ua:Google
Click to expand...
Identification, Content Specificity, Robots.txt compliance, Device and Platform Information, Compression Support, and Versioning
 
lanmaok said:
Thank you for your reply. However, this is not ideal. Because the IP of the robot will change after a period of time. Do you know the iconic features of those garbage robots? For example, ua:Google
Click to expand...
I've never found any issues with generic crawlers on my servers, they consume minimal resources. If you feel your server is clogged, maybe its a matter of configuration, just increase your workers, add more hardware resources

If you are obssessed with those crawlers, why don't you get their footprint by yourself? Just get your nginx logs, and filter out the most common user agents for your regular visitors.

Anyway here is a site that offers most common UA for both browsers and crawlers, but still I would only focus on browsers and filter out. You will end finding out that its not as terrible as you believe.
 
bbrokeaf said:
Identification, Content Specificity, Robots.txt compliance, Device and Platform Information, Compression Support, and Versioning
Click to expand...
Yes, as long as you know his unique characteristics, you can use nginx to block him.
if ($http_user_agent ~* "Baiduspider") {

return 403;
 
SirLouen said:
I've never found any issues with generic crawlers on my servers, they consume minimal resources. If you feel your server is clogged, maybe its a matter of configuration, just increase your workers, add more hardware resources

If you are obssessed with those crawlers, why don't you get their footprint by yourself? Just get your nginx logs, and filter out the most common user agents for your regular visitors.

Anyway here is a site that offers most common UA for both browsers and crawlers, but still I would only focus on browsers and filter out. You will end finding out that its not as terrible as you believe.
Click to expand...

SirLouen said:
I've never found any issues with generic crawlers on my servers, they consume minimal resources. If you feel your server is clogged, maybe its a matter of configuration, just increase your workers, add more hardware resources

If you are obssessed with those crawlers, why don't you get their footprint by yourself? Just get your nginx logs, and filter out the most common user agents for your regular visitors.

Anyway here is a site that offers most common UA for both browsers and crawlers, but still I would only focus on browsers and filter out. You will end finding out that its not as terrible as you believe.
Click to expand...
Thank you for your reply. This inspires me a lot.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top