crm开发定制【Python】【爬虫】爬取小说5000章，遇到的爬虫问题与解决思路

问题分析

回顾

crm开发定制之前写了一个爬取小说crm开发定制网站的爬虫，crm开发定制操作流程如下：

crm开发定制先爬取小说介绍页，crm开发定制获取所有章节信息(章节名称，crm开发定制章节对应阅读链接)，crm开发定制然后使用多线程的方式(pool = Pool(50))，crm开发定制通过章节的阅读链接爬crm开发定制取章节正文并保存为本地markdown文件。(crm开发定制代码见文末 run01.python)

爬取100章，用了10秒

限制爬取101章，crm开发定制从运行程序到结束程序，用时9秒

Redis+MongoDB，无多线程

最近学了Redis和MongoDB，crm开发定制要求爬取后将章节链接放在redis，crm开发定制然后通过读取redis的章节链接来进行爬取。(代码见文末run02.python)

…不用测试了，一章一章读真的太慢了！

爬取101章用时两分钟！

Redis+MongoDB+多线程

爬取101章，只需8秒！

爬取4012章，用时1分10秒！

问题与解析

懒得打字，我就录成视频发在小破站上面了。(小破站搜：萌狼蓝天)

其他的去我小破站主页翻

代码20221020

run01.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8. import datetime
  9. import re
  10. import random
  11. from multiprocessing import Pool
  12.  
  13. import requests
  14. import bs4
  15. import os
  16.  
  17. os.environ['NO_PROXY'] = "www.lingdianksw8.com"
  18.  
  19.  
  20. def Log_text(lx="info", *text):
  21.     lx.upper()
  22.     with open("log.log", "a+", encoding="utf-8") as f:
  23.         f.write("[" + str(datetime.datetime.now()) + "]" + "[" + lx + "]")
  24.         for i in text:
  25.             f.write(i)
  26.     f.close()
  27.  
  28.  
  29. # 调试输出
  30. def log(message, i="info"):
  31.     if type(message) == type(""):
  32.         i.upper()
  33.         print("[", i, "] [", str(type(message)), "]", message)
  34.     elif type(message) == type([]):
  35.         count = 0
  36.         for j in message:
  37.             print("[", i, "] [", str(count), "] [", str(type(message)), "]", j)
  38.             count += 1
  39.     else:
  40.         print("[", i, "]  [", str(type(message)), "]", end=" ")
  41.         print(message)
  42.  
  43.  
  44. # 获取源码
  45. def getCode(url, methods="post"):
  46.     """
  47.     获取页面源码
  48.     :param methods: 请求提交方式
  49.     :param url:书籍首页链接
  50.     :return:页面源码
  51.     """
  52.     # 设置请求头
  53.     user_agent = [
  54.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  55.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
  56.         "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  57.         "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
  58.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
  59.         "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
  60.         "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
  61.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
  62.         "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
  63.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
  64.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
  65.         "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
  66.         "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
  67.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
  68.         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
  69.         "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
  70.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
  71.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
  72.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
  73.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
  74.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",
  75.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  76.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
  77.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  78.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",
  79.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  80.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  81.         "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  82.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  83.         "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
  84.         "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",
  85.         "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",
  86.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
  87.         "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10"
  88.     ]
  89.     headers = {
  90.         'User-Agent': random.choice(user_agent),
  91.         # "user-agent": user_agent[random.randint(0, len(user_agent) - 1)]
  92.     }
  93.  
  94.     # 获取页面源码
  95.     result = requests.request(methods, url, headers=headers, allow_redirects=True)
  96.     log("cookie" + str(result.cookies.values()))
  97.     tag = 0
  98.     log("初始页面编码为：" + result.encoding)
  99.     if result.encoding != "gbk":
  100.         log("初始页面编码非gbk，需要进行重编码操作", "warn")
  101.         tag = 1
  102.     try:
  103.         result = requests.request(methods, url, headers=headers, allow_redirects=True, cookies=result.cookies)
  104.     except:
  105.         return "InternetError"
  106.     result_text = result.text
  107.     # print(result_text)
  108.     if tag == 1:
  109.         result_text = recoding(result)
  110.         log("转码编码完成，当前编码为gbk")
  111.     return result_text
  112.  
  113.  
  114. def recoding(result):
  115.     try:
  116.         result_text = result.content.decode("gbk",errors='ignore')
  117.     except:
  118.         # UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 6917:
  119.         try:
  120.             result_text = result.content.decode("").encode("unicode_escape").decode("gbk",errors='ignore')
  121.         except:
  122.             try:
  123.                 result_text = result.content.decode("gb18030",errors='ignore')
  124.             except:
  125.                 result_text = result.text
  126.     return result_text
  127.  
  128.  
  129. # 分析数据
  130. def getDict(code):
  131.     """
  132.     分析网页源码，获取数据，并存储为以字典元素构成的列表返回
  133.     :param code:网页源码
  134.     :return:List
  135.     """
  136.     # 通过正则的方式缩小范围
  137.     code = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  138.     # log(code)
  139.     # obj = bs4.BeautifulSoup(markup=code,features="html.parser")
  140.     obj = bs4.BeautifulSoup(markup=code, features="lxml")
  141.     # log("输出结果")
  142.     # log(obj.find_all("a"))
  143.     # 通过上面调试输出可知得到的是个列表
  144.     tag = obj.find_all("a")
  145.     log("tag长度为：" + str(len(tag)))
  146.     result = []
  147.     count = 0
  148.     for i in range(len(tag)):
  149.         count += 1
  150.         link = tag[i]["href"]
  151.         text = tag[i].get_text()
  152.         result.append({"title": text, "link": "https://www.lingdianksw8.com" + link})
  153.     return result
  154.  
  155.  
  156. # 文章内容
  157. def getContent(url):
  158.  
  159.     code = getCode(url, "get")
  160.     if code=="InternetError":
  161.         return "InternetError",""
  162.     try:
  163.         code = code.replace("<br />", "\")
  164.         code = code.replace("&nbsp;", " ")
  165.         code = code.replace("        ", " ")
  166.     except Exception as e:
  167.     # AttributeError: 'tuple' object has no attribute 'replace'
  168.         Log_text("error","[run01-161~163]"+str(e))
  169.     # with open("temp.txt","w+",encoding="utf-8") as f:
  170.     #     f.write(code)
  171.     obj = bs4.BeautifulSoup(markup=code, features="lxml")
  172.     titile = obj.find_all("h1")[0].text
  173.     try:
  174.         content = obj.find_all("div", attrs={"class": "showtxt"})[0].text
  175.     except:
  176.         return None, None
  177.     # with open("temp.txt", "w+", encoding="utf-8") as f:
  178.     #     f.write(content)
  179.     # log(content)
  180.     try:
  181.         g = re.findall(
  182.             "(:.*?https://www.lingdianksw8.com.*?天才一秒记住本站地址：www.lingdianksw8.com。零点看书手机版阅读网址：.*?.com)",
  183.             content, re.S)[0]
  184.         log(g)
  185.         content = content.replace(g, "")
  186.     except:
  187.         Log_text("error", "清除广告失败！章节" + titile + "(" + url + ")")
  188.     log(content)
  189.     return titile, content
  190.  
  191.  
  192. def docToMd(name, title, content):
  193.     with open(name + ".md", "w+", encoding="utf-8") as f:
  194.         f.write("## " + title + "/n" + content)
  195.     f.close()
  196.     return 0
  197.  
  198.  
  199. # 多线程专供函数 - 通过链接获取文章
  200. def thead_getContent(link):
  201.     # 根据链接获取文章内容
  202.     Log_text("info", "尝试获取" + str(link))
  203.     title, content = getContent(str(link))  # 从文章内获取到标题和内容
  204.     Log_text("success", "获取章节" + title + "完成")
  205.     docToMd(title, title, content)
  206.     Log_text("success", "写出章节" + title + "完成")
  207.  
  208.  
  209. # 操作汇总
  210. def run(url):
  211.     with open("log1.log", "w+", encoding="utf-8") as f:
  212.         f.write("")
  213.         f.close()
  214.     Log_text("info", "开始获取小说首页...")
  215.     code = getCode(url)
  216.     Log_text("success", "获取小说首页源代码完成，开始分析...")
  217.     index = getDict(code)  # 获取到[{章节名称title:链接link}]
  218.     links = []
  219.     # lineCount限制要爬取的数量
  220.     lineCount = 0
  221.     for i in index:
  222.         if lineCount > 10:
  223.             break
  224.         lineCount += 1
  225.         links.append(i["link"])
  226.  
  227.     print("链接状态")
  228.     print(type(links))
  229.     print(links)
  230.     Log_text("success", "分析小说首页完成，数据整理完毕，开始获取小说内容...")
  231.     pool = Pool(50)  # 多线程
  232.     pool.map(thead_getContent, links)
  233.  
  234.  
  235. if __name__ == '__main__':
  236.     start = datetime.datetime.today()
  237.     Log_text("===【日志】[多线程-]开始新的测试 =|=|=|= " + str(start))
  238.     run(r"https://www.lingdianksw8.com/31/31596")
  239.     # getContent("http://www.lingdianksw8.com/31/31596/8403973.html")
  240.     end = datetime.datetime.today()
  241.     Log_text("===【日志】[多线程]测试结束 =|=|=|= " + str(end))
  242.     Log_text("===【日志】[多线程]测试结束 =|=|=|= 用时" + str(end - start))
  243.     print("")

run02.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8.  
  9. """
  10. 1.通过run01获取章节的链接，将链接存储到Redis中
  11. 2.从Redis获取章节链接并爬取
  12. """
  13. import re
  14.  
  15. import pymongo
  16. from lxml import html
  17. import run01 as xrilang
  18. import redis
  19. import datetime
  20. client = redis.StrictRedis()
  21.  
  22. def getLinks():
  23.      xrilang.Log_text("===【日志】开始获取章节名称和链接")
  24.      code = xrilang.getCode("https://www.lingdianksw8.com/61153/61153348/","get")
  25.      source = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  26.      selector = html.fromstring(source)
  27.      title_list = selector.xpath("//dd/a/text()")
  28.      url_list = selector.xpath("//dd/a/@href")
  29.      client.flushall() # 清空Redis全部内容，避免重复运行造成的数据重复
  30.      xrilang.Log_text("===【日志】开始获取标题")
  31.      for title in title_list:
  32.         xrilang.log(title)
  33.         client.lpush('title_queue', title)
  34.      xrilang.Log_text("===【日志】开始获取章节链接")
  35.      for url in url_list:
  36.         xrilang.log(url)
  37.         client.lpush('url_queue', url)
  38.      xrilang.log(client.llen('url_queue'))
  39.      xrilang.Log_text("===【日志】获取章节链接结束，共"+str(client.llen('url_queue'))+"条")
  40. def getContent():
  41.     xrilang.Log_text("===【日志】开始获取章节内容")
  42.     database = pymongo.MongoClient()['book']
  43.     collection = database['myWifeSoBeautifull']
  44.     startTime=datetime.datetime.today()
  45.     xrilang.log("开始"+str(startTime))
  46.     linkCount=0
  47.     datas=[]
  48.     while client.llen("url_queue")>0:
  49.         # 爬取101章
  50.         if linkCount >10:
  51.             break
  52.         linkCount += 1
  53.         url = client.lpop("url_queue").decode()
  54.         title = client.lpop("title_queue").decode()
  55.         xrilang.log(url)
  56.         # 获取文章内容并保存到数据库
  57.         content_url = "https://www.lingdianksw8.com"+url
  58.         name,content = xrilang.getContent(content_url)
  59.         if name!=None and content!=None:
  60.             datas.append({"title":title,"name":name,"content":content})
  61.     collection.insert_many(datas)
  62.  
  63. if __name__ == '__main__':
  64.     start = datetime.datetime.today()
  65.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]开始新的测试 =|=|=|= " + str(start))
  66.     getLinks()
  67.     getContent()
  68.     end = datetime.datetime.today()
  69.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]测试结束 =|=|=|= " + str(end))
  70.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]测试结束 =|=|=|= 用时" + str(end-start))
  71.     print("")

run03.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8.  
  9. """
  10. 1.通过run01获取章节的链接，将链接存储到Redis中
  11. 2.从Redis获取章节链接并爬取
  12. """
  13. import re
  14. import time
  15. from multiprocessing.dummy import Pool
  16.  
  17. import pymongo
  18. from lxml import html
  19. import run01 as xrilang
  20. import redis
  21. import datetime
  22. client = redis.StrictRedis()
  23. database = pymongo.MongoClient()['book']
  24. collection = database['myWifeSoBeautifull']
  25.  
  26.  
  27. def getLinks():
  28.      xrilang.Log_text("===【日志】开始获取章节名称和链接")
  29.      code = xrilang.getCode("https://www.lingdianksw8.com/61153/61153348/","get")
  30.      source = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  31.      selector = html.fromstring(source)
  32.      url_list = selector.xpath("//dd/a/@href")
  33.      client.flushall() # 清空Redis全部内容，避免重复运行造成的数据重复
  34.      xrilang.Log_text("===【日志】开始获取章节链接")
  35.      i=0
  36.      for url in url_list:
  37.         xrilang.log(url)
  38.         client.lpush('url_queue', url)
  39.         i+=1
  40.         client.lpush('sort_queue', i) # 解决多线程爬虫导致的顺序问题
  41.      xrilang.log(client.llen('url_queue'))
  42.      xrilang.Log_text("===【日志】获取章节链接结束，共"+str(client.llen('url_queue'))+"条")
  43. def getContent(durl):
  44.     url = durl["url"]
  45.     isort=durl["isort"]
  46.     content_url = "https://www.lingdianksw8.com" + url
  47.     title, content = xrilang.getContent(content_url)
  48.  
  49.     if title != "InternetError":
  50.         if title != None and content != None:
  51.             xrilang.log("获取"+title+"成功")
  52.             collection.insert_one({"isort":isort,"title": title,  "content": content})
  53.         else:
  54.             # 没有成功爬取的添加回redis，等待下次爬取
  55.             client.lpush('url_queue', url)
  56.             client.lpush('sort_queue', isort)  # 解决多线程爬虫导致的顺序问题
  57.             # 等待5秒
  58.             time.sleep(1000)
  59.     else:
  60.         # 没有成功爬取的添加回redis，等待下次爬取
  61.         client.lpush('url_queue', url)
  62.         client.lpush('sort_queue', isort)  # 解决多线程爬虫导致的顺序问题
  63.         # 等待5秒
  64.         time.sleep(5000)
  65. def StartGetContent():
  66.     xrilang.Log_text("===【日志】开始获取章节内容")
  67.     startTime = datetime.datetime.today()
  68.     xrilang.log("开始"+str(startTime))
  69.     urls=[]
  70.     # xrilang.log(client.llen("url_queue"))
  71.     while client.llen("url_queue")>0:
  72.         url = client.lpop("url_queue").decode()
  73.         isort=  client.lpop("sort_queue").decode()
  74.         #urls.append(url)
  75.         urls.append({"url":url,"isort":isort})
  76.     # xrilang.log(urls)
  77.     pool = Pool(500)  # 多线程
  78.     pool.map(getContent,urls)
  79.     endTime=datetime.datetime.today()
  80.     xrilang.log("【结束】"+str(endTime))
  81.     xrilang.Log_text("===【日志】开始获取章节结束，用时"+str(endTime-startTime))
  82. if __name__ == '__main__':
  83.     start = datetime.datetime.today()
  84.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]开始新的测试 =|=|=|= " + str(start))
  85.     getLinks()
  86.     StartGetContent()
  87.     end = datetime.datetime.today()
  88.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]测试结束 =|=|=|= " + str(end))
  89.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]测试结束 =|=|=|= 用时" + str(end-start))
  90.     print("")

mongoQ.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/10/20
  7. import pymongo
  8.  
  9. database = pymongo.MongoClient()['book']
  10. collection = database['myWifeSoBeautifull']
  11. result = collection.find().collation({"locale":"zh", "numericOrdering":True}).sort("isort")
  12.  
  13.  
  14. with open("list.txt", "a+", encoding="utf-8") as f:
  15.     for i in result:
  16.         f.writelines(i["isort"]+" "+i["title"]+"\")

代码20221019

run01.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8. import datetime
  9. import re
  10. import random
  11. from multiprocessing import Pool
  12.  
  13. import requests
  14. import bs4
  15. import os
  16.  
  17. os.environ['NO_PROXY'] = "www.lingdianksw8.com"
  18.  
  19.  
  20. def Log_text(lx="info", *text):
  21.     lx.upper()
  22.     with open("log.log", "a+", encoding="utf-8") as f:
  23.         f.write("[" + str(datetime.datetime.now()) + "]" + "[" + lx + "]")
  24.         for i in text:
  25.             f.write(i)
  26.     f.close()
  27.  
  28.  
  29. # 调试输出
  30. def log(message, i="info"):
  31.     if type(message) == type(""):
  32.         i.upper()
  33.         print("[", i, "] [", str(type(message)), "]", message)
  34.     elif type(message) == type([]):
  35.         count = 0
  36.         for j in message:
  37.             print("[", i, "] [", str(count), "] [", str(type(message)), "]", j)
  38.             count += 1
  39.     else:
  40.         print("[", i, "]  [", str(type(message)), "]", end=" ")
  41.         print(message)
  42.  
  43.  
  44. # 获取源码
  45. def getCode(url, methods="post"):
  46.     """
  47.     获取页面源码
  48.     :param methods: 请求提交方式
  49.     :param url:书籍首页链接
  50.     :return:页面源码
  51.     """
  52.     # 设置请求头
  53.     user_agent = [
  54.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  55.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
  56.         "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  57.         "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
  58.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
  59.         "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
  60.         "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
  61.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
  62.         "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
  63.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
  64.         "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
  65.         "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
  66.         "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
  67.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
  68.         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
  69.         "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
  70.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
  71.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
  72.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
  73.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
  74.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",
  75.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  76.         "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
  77.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  78.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",
  79.         "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
  80.         "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
  81.         "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  82.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
  83.         "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
  84.         "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",
  85.         "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",
  86.         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
  87.         "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10"
  88.     ]
  89.     headers = {
  90.         'User-Agent': random.choice(user_agent),
  91.         # "user-agent": user_agent[random.randint(0, len(user_agent) - 1)]
  92.     }
  93.  
  94.     # 获取页面源码
  95.     result = requests.request(methods, url, headers=headers, allow_redirects=True)
  96.     log("cookie" + str(result.cookies.values()))
  97.     tag = 0
  98.     log("初始页面编码为：" + result.encoding)
  99.     if result.encoding == "gbk" or result.encoding == "ISO-8859-1":
  100.         log("初始页面编码非UTF-8，需要进行重编码操作", "warn")
  101.         tag = 1
  102.     try:
  103.         result = requests.request(methods, url, headers=headers, allow_redirects=True, cookies=result.cookies)
  104.     except:
  105.         return "InternetError",""
  106.     result_text = result.text
  107.     # print(result_text)
  108.     if tag == 1:
  109.         result_text = recoding(result)
  110.         log("转码编码完成，当前编码为gbk")
  111.     return result_text
  112.  
  113.  
  114. def recoding(result):
  115.     try:
  116.         result_text = result.content.decode("gbk",errors='ignore')
  117.     except:
  118.         # UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 6917:
  119.         try:
  120.             result_text = result.content.decode("").encode("unicode_escape").decode("gbk",errors='ignore')
  121.         except:
  122.             try:
  123.                 result_text = result.content.decode("gb18030",errors='ignore')
  124.             except:
  125.                 result_text = result.text
  126.     return result_text
  127.  
  128.  
  129. # 分析数据
  130. def getDict(code):
  131.     """
  132.     分析网页源码，获取数据，并存储为以字典元素构成的列表返回
  133.     :param code:网页源码
  134.     :return:List
  135.     """
  136.     # 通过正则的方式缩小范围
  137.     code = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  138.     # log(code)
  139.     # obj = bs4.BeautifulSoup(markup=code,features="html.parser")
  140.     obj = bs4.BeautifulSoup(markup=code, features="lxml")
  141.     # log("输出结果")
  142.     # log(obj.find_all("a"))
  143.     # 通过上面调试输出可知得到的是个列表
  144.     tag = obj.find_all("a")
  145.     log("tag长度为：" + str(len(tag)))
  146.     result = []
  147.     count = 0
  148.     for i in range(len(tag)):
  149.         count += 1
  150.         link = tag[i]["href"]
  151.         text = tag[i].get_text()
  152.         result.append({"title": text, "link": "https://www.lingdianksw8.com" + link})
  153.     return result
  154.  
  155.  
  156. # 文章内容
  157. def getContent(url):
  158.  
  159.     code = getCode(url, "get")
  160.     try:
  161.         code = code.replace("<br />", "\")
  162.         code = code.replace("&nbsp;", " ")
  163.         code = code.replace("        ", " ")
  164.     except Exception as e:
  165.     # AttributeError: 'tuple' object has no attribute 'replace'
  166.         Log_text("error","[run01-161~163]"+str(e))
  167.     # with open("temp.txt","w+",encoding="utf-8") as f:
  168.     #     f.write(code)
  169.     obj = bs4.BeautifulSoup(markup=code, features="lxml")
  170.     titile = obj.find_all("h1")[0].text
  171.     try:
  172.         content = obj.find_all("div", attrs={"class": "showtxt"})[0].text
  173.     except:
  174.         return None, None
  175.     # with open("temp.txt", "w+", encoding="utf-8") as f:
  176.     #     f.write(content)
  177.     # log(content)
  178.     try:
  179.         g = re.findall(
  180.             "(:.*?https://www.lingdianksw8.com.*?天才一秒记住本站地址：www.lingdianksw8.com。零点看书手机版阅读网址：.*?.com)",
  181.             content, re.S)[0]
  182.         log(g)
  183.         content = content.replace(g, "")
  184.     except:
  185.         Log_text("error", "清除广告失败！章节" + titile + "(" + url + ")")
  186.     log(content)
  187.     return titile, content
  188.  
  189.  
  190. def docToMd(name, title, content):
  191.     with open(name + ".md", "w+", encoding="utf-8") as f:
  192.         f.write("## " + title + "/n" + content)
  193.     f.close()
  194.     return 0
  195.  
  196.  
  197. # 多线程专供函数 - 通过链接获取文章
  198. def thead_getContent(link):
  199.     # 根据链接获取文章内容
  200.     Log_text("info", "尝试获取" + str(link))
  201.     title, content = getContent(str(link))  # 从文章内获取到标题和内容
  202.     Log_text("success", "获取章节" + title + "完成")
  203.     docToMd(title, title, content)
  204.     Log_text("success", "写出章节" + title + "完成")
  205.  
  206.  
  207. # 操作汇总
  208. def run(url):
  209.     with open("log1.log", "w+", encoding="utf-8") as f:
  210.         f.write("")
  211.         f.close()
  212.     Log_text("info", "开始获取小说首页...")
  213.     code = getCode(url)
  214.     Log_text("success", "获取小说首页源代码完成，开始分析...")
  215.     index = getDict(code)  # 获取到[{章节名称title:链接link}]
  216.     links = []
  217.     # lineCount限制要爬取的数量
  218.     lineCount = 0
  219.     for i in index:
  220.         if lineCount > 100:
  221.             break
  222.         lineCount += 1
  223.         links.append(i["link"])
  224.  
  225.     print("链接状态")
  226.     print(type(links))
  227.     print(links)
  228.     Log_text("success", "分析小说首页完成，数据整理完毕，开始获取小说内容...")
  229.     pool = Pool(50)  # 多线程
  230.     pool.map(thead_getContent, links)
  231.  
  232.  
  233. if __name__ == '__main__':
  234.     start = datetime.datetime.today()
  235.     Log_text("===【日志】[多线程-]开始新的测试 =|=|=|= " + str(start))
  236.     run(r"https://www.lingdianksw8.com/31/31596")
  237.     # getContent("http://www.lingdianksw8.com/31/31596/8403973.html")
  238.     end = datetime.datetime.today()
  239.     Log_text("===【日志】[多线程]测试结束 =|=|=|= " + str(end))
  240.     Log_text("===【日志】[多线程]测试结束 =|=|=|= 用时" + str(end - start))
  241.     print("")

run02.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8.  
  9. """
  10. 1.通过run01获取章节的链接，将链接存储到Redis中
  11. 2.从Redis获取章节链接并爬取
  12. """
  13. import re
  14.  
  15. import pymongo
  16. from lxml import html
  17. import run01 as xrilang
  18. import redis
  19. import datetime
  20. client = redis.StrictRedis()
  21.  
  22. def getLinks():
  23.      xrilang.Log_text("===【日志】开始获取章节名称和链接")
  24.      code = xrilang.getCode("https://www.lingdianksw8.com/61153/61153348/","get")
  25.      source = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  26.      selector = html.fromstring(source)
  27.      title_list = selector.xpath("//dd/a/text()")
  28.      url_list = selector.xpath("//dd/a/@href")
  29.      client.flushall() # 清空Redis全部内容，避免重复运行造成的数据重复
  30.      xrilang.Log_text("===【日志】开始获取标题")
  31.      for title in title_list:
  32.         xrilang.log(title)
  33.         client.lpush('title_queue', title)
  34.      xrilang.Log_text("===【日志】开始获取章节链接")
  35.      for url in url_list:
  36.         xrilang.log(url)
  37.         client.lpush('url_queue', url)
  38.      xrilang.log(client.llen('url_queue'))
  39.      xrilang.Log_text("===【日志】获取章节链接结束，共"+str(client.llen('url_queue'))+"条")
  40. def getContent():
  41.     xrilang.Log_text("===【日志】开始获取章节内容")
  42.     database = pymongo.MongoClient()['book']
  43.     collection = database['myWifeSoBeautifull']
  44.     startTime=datetime.datetime.today()
  45.     xrilang.log("开始"+str(startTime))
  46.     linkCount=0
  47.     datas=[]
  48.     while client.llen("url_queue")>0:
  49.         # 爬取101章
  50.         if linkCount >10:
  51.             break
  52.         linkCount += 1
  53.         url = client.lpop("url_queue").decode()
  54.         title = client.lpop("title_queue").decode()
  55.         xrilang.log(url)
  56.         # 获取文章内容并保存到数据库
  57.         content_url = "https://www.lingdianksw8.com"+url
  58.         name,content = xrilang.getContent(content_url)
  59.         if name!=None and content!=None:
  60.             datas.append({"title":title,"name":name,"content":content})
  61.     collection.insert_many(datas)
  62.  
  63. if __name__ == '__main__':
  64.     start = datetime.datetime.today()
  65.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]开始新的测试 =|=|=|= " + str(start))
  66.     getLinks()
  67.     getContent()
  68.     end = datetime.datetime.today()
  69.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]测试结束 =|=|=|= " + str(end))
  70.     xrilang.Log_text("===【日志】[redis+MongoDB无多线程]测试结束 =|=|=|= 用时" + str(end-start))
  71.     print("")

run03.py

  1. # -*- coding: UTF-8 -*-
  2. # 开发人员：萌狼蓝天
  3. # 博客：Https://mllt.cc
  4. # 笔记：Https://cnblogs.com/mllt
  5. # 哔哩哔哩/微信公众号：萌狼蓝天
  6. # 开发时间：2022/9/28
  7. # https://www.lingdianksw8.com/31/31596/
  8.  
  9. """
  10. 1.通过run01获取章节的链接，将链接存储到Redis中
  11. 2.从Redis获取章节链接并爬取
  12. """
  13. import re
  14. import time
  15. from multiprocessing.dummy import Pool
  16.  
  17. import pymongo
  18. from lxml import html
  19. import run01 as xrilang
  20. import redis
  21. import datetime
  22. client = redis.StrictRedis()
  23. database = pymongo.MongoClient()['book']
  24. collection = database['myWifeSoBeautifull']
  25.  
  26.  
  27. def getLinks():
  28.      xrilang.Log_text("===【日志】开始获取章节名称和链接")
  29.      code = xrilang.getCode("https://www.lingdianksw8.com/61153/61153348/","get")
  30.      source = re.findall("正文卷</dt>(.*?)</dl>", code, re.S)[0]
  31.      selector = html.fromstring(source)
  32.      url_list = selector.xpath("//dd/a/@href")
  33.      client.flushall() # 清空Redis全部内容，避免重复运行造成的数据重复
  34.      xrilang.Log_text("===【日志】开始获取章节链接")
  35.      i=0
  36.      for url in url_list:
  37.         xrilang.log(url)
  38.         client.lpush('url_queue', url)
  39.         i+=1
  40.         client.lpush('sort_queue', i) # 解决多线程爬虫导致的顺序问题
  41.      xrilang.log(client.llen('url_queue'))
  42.      xrilang.Log_text("===【日志】获取章节链接结束，共"+str(client.llen('url_queue'))+"条")
  43. def getContent(durl):
  44.     url = durl["url"]
  45.     isort=durl["isort"]
  46.     content_url = "https://www.lingdianksw8.com" + url
  47.     title, content = xrilang.getContent(content_url)
  48.  
  49.     if title != None and content != None:
  50.         if (title != "InternetError"):
  51.             xrilang.log("获取"+title+"成功")
  52.             collection.insert_one({"isort":isort,"title": title,  "content": content})
  53.         else:
  54.             # 没有成功爬取的添加回redis，等待下次爬取
  55.             client.lpush('url_queue', url)
  56.             client.lpush('sort_queue', isort)  # 解决多线程爬虫导致的顺序问题
  57.             # 等待5秒
  58.             time.sleep(5000)
  59. def StartGetContent():
  60.     xrilang.Log_text("===【日志】开始获取章节内容")
  61.     startTime = datetime.datetime.today()
  62.     xrilang.log("开始"+str(startTime))
  63.     urls=[]
  64.     # xrilang.log(client.llen("url_queue"))
  65.     while client.llen("url_queue")>0:
  66.         url = client.lpop("url_queue").decode()
  67.         isort=  client.lpop("sort_queue").decode()
  68.         #urls.append(url)
  69.         urls.append({"url":url,"isort":isort})
  70.     # xrilang.log(urls)
  71.     pool = Pool(500)  # 多线程
  72.     pool.map(getContent,urls)
  73.     endTime=datetime.datetime.today()
  74.     xrilang.log("【结束】"+str(endTime))
  75.     xrilang.Log_text("===【日志】开始获取章节结束，用时"+str(endTime-startTime))
  76. if __name__ == '__main__':
  77.     start = datetime.datetime.today()
  78.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]开始新的测试 =|=|=|= " + str(start))
  79.     getLinks()
  80.     StartGetContent()
  81.     end = datetime.datetime.today()
  82.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]测试结束 =|=|=|= " + str(end))
  83.     xrilang.Log_text("===【日志】[redis+MongoDB+多线程]测试结束 =|=|=|= 用时" + str(end-start))
  84.     print("")
网站建设定制开发 软件系统开发定制 定制软件开发 软件开发定制 定制app开发 app开发定制 app开发定制公司 电商商城定制开发 定制小程序开发 定制开发小程序 客户管理系统开发定制 定制网站 定制开发 crm开发定制 开发公司 小程序开发定制 定制软件 收款定制开发 企业网站定制开发 定制化开发 android系统定制开发 定制小程序开发费用 定制设计 专注app软件定制开发 软件开发定制定制 知名网站建设定制 软件定制开发供应商 应用系统定制开发 软件系统定制开发 企业管理系统定制开发 系统定制开发