Python 的 requests 库是最常用的 HTTP 请求库,设置代理只需一个 proxies 参数。
HTTP / HTTPS 代理
import requests
# 代理格式:协议://账号:密码@IP:端口
proxy = "http://用户名:密码@IP:端口"
proxies = {
"http": proxy,
"https": proxy,
}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(resp.json()) # 返回的 origin 即为代理出口 IP
SOCKS5 代理
socks5 需要先安装 requests[socks]:
pip install "requests[socks]"
import requests
proxies = {
"http": "socks5://用户名:密码@IP:端口",
"https": "socks5://用户名:密码@IP:端口",
}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(resp.json())
批量采集:轮换 IP
import requests, random
ips = ["IP1:端口", "IP2:端口", "IP3:端口"]
def fetch(url):
ip = random.choice(ips)
proxies = {"http": f"http://用户名:密码@{ip}", "https": f"http://用户名:密码@{ip}"}
return requests.get(url, proxies=proxies, timeout=10)
for url in urls:
r = fetch(url)
print(r.status_code)
设置超时
timeout 并做好重试,能显著提升采集稳定性。