爬虫实战-爬取兴盛优选首页所有商品数据并写入excel

亦凉 2021-10-29 12:42 200阅读 0赞

兴盛优选这个小程序很火，小编今天就教大家如何爬取兴盛优选的所有商品数据信息吧

1.首先获取windowid

这个请求是获取所有的windows的id的 这个请求实际返回windows   brandHouseWindows classifyWindows   这三个其实都返回了windowId 通过观察我们可知 windows里面的是包含所有的 所以我们只需要对他进行遍历即可

[https://mall.xsyxsc.com/user/product/indexSortWindows][https_mall.xsyxsc.com_user_product_indexSortWindows]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70][]

2.查看获取商品信息请求  这里其实有三个接口获取商品数据 分别如下【查看下方可知 ，我们一开始获取的windosid是用于这里获取商品数据的】

[https://mall.xsyxsc.com/user/product/activityProducts][https_mall.xsyxsc.com_user_product_activityProducts]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 1][]

[https://mall.xsyxsc.com/user/product/classifyProducts][https_mall.xsyxsc.com_user_product_classifyProducts]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 2][]

[https://mall.xsyxsc.com/user/brandhouse/window/getProducts][https_mall.xsyxsc.com_user_brandhouse_window_getProducts]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 3][]

相对于前面二个接口 第三个可能会稍微麻烦一点，因为他有pageIndex  pageSize参数 为了避免翻页 小编将pageSize修改成1000，发送请求  于是发现 他们做了控制 好像必须传入10的大小，好吧 不能投机取巧了我们还是老实的写个翻页吧  小编的思路是 直接来个很大的循环 然后当响应数据中 data.records的数据是空的时候 就跳出循环。![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 4][]

好了看到这里我们大致可以知道  获取商品信息总共分三个接口，他们的请求地址，以及请求参数都不一样！

3.确定思路

首先获取所有的windowid  然后根据windowid去遍历商品数据，而具体使用那个获取商品数据的接口  我们可以根据[https://mall.xsyxsc.com/user/product/indexSortWindows][https_mall.xsyxsc.com_user_product_indexSortWindows]接口返回的windowType来进行判断，之后将获取到的商品信息存进一个列表，之后写入excel中

好了 分析到这里  小编就分享一波源码给大家 当然self.userkey你需要去替换成你的 万一你用我的作死的访问导致小编被拉黑了就得不偿失了。

# author:Administrator
    # datetime:2019/5/28 12:32
    # project_name:python_code_warehouse
    # file_name:xingshengyouxuan
    # email:1163739580@qq.com
    import requests
    from requests.packages.urllib3.exceptions import InsecureRequestWarning
    import pandas as pd
    import datatime
    import time
    import os
    
    requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
    
    
    def myprint(*con):
        print("RunInfo----" + str(datetime.datetime.now().replace(microsecond=0)) + ":" + "".join(*con))
    
    
    class XingShengYouXuna(object):
        def __init__(self):
            self.userkey = "填你自己的"
            self.base_param = {'userKey': self.userkey}
            self.header = {
                'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 MicroMessenger/7.0.4(0x17000428) NetType/WIFI Language/zh_CN'}
            self.url = "https://mall.xsyxsc.com"
            self.sku_info_list = []
            self.sku_name = {}  # 用于去重
    
        def get_windows_id(self):
            """获取首页返回的windowId
    
            :return: 首页window信息 key为id&tpye value为标题
            """
            index_window_info = {}
            body = {'areaId': 101,
                    'storeId': '66880000082975'}
            body.update(self.base_param)
            resp = requests.post(url=self.url + "/user/product/indexSortWindows", data=body, verify=False).json().get(
                "data")
            """其实这个接口返回了
               windows brandHouseWindows  classifyWindows
               细心观察可知  windows才是全部的 其次分类界面所有的分类也是属于其中
               """
            for window in resp.get("windows"):
                myprint("搜索到标题为'{}'  windows_id为'{}'  类型为'{}'".format(window.get("windowName"), window.get("windowId"),
                                                                      window.get("windowType")))
                index_window_info[str(window.get("windowId")) + "&" + window.get("windowType")] = window.get(
                    "windowName")
            myprint("合计搜索到的windows个数为{}".format(len(index_window_info)))
            return index_window_info
    
        def load_index_sku(self, **kwargs):
            """将首页的所有商品信息添加到sku列表中
    
            :param kwargs: 函数 get_windows_id的返回值
            """
            for id_type in kwargs:
                windowid, windowtype = id_type.split("&")
                if windowtype in ("ACTIVITY", "CLASSIFY"):
                    url = self.url + "/user/product/{}Products".format(windowtype.lower())
                    if windowtype == "ACTIVITY":
                        body = {'windowId': windowid, 'openBrandHouse': 'OPEN', 'storeId': '66880000082975',
                                'areaId': '101'}
                    else:
                        body = {'windowId': windowid, 'areaId': '101', 'storeId': '66880000082975', 'excludeAct': 'N'}
                else:
                    url = self.url + "/user/brandhouse/window/getProducts"
                    body = {'windowId': windowid, 'areaId': '101', 'storeId': '66880000082975', 'excludeAct': 'N',
                            'pageIndex': '1', 'pageSize': '10'}
    
                body.update(self.base_param)
                sku = {}
                if windowtype == "BRAND_HOUSE":
                    pass
                    for pageIndex in range(1, 10000):
                        body.update({'pageIndex': pageIndex})
                        resp = requests.post(url=url, data=body, verify=False).json().get("data")
                        if resp.get("records") is None:
                            del url, body
                            break
                        for com_info in resp.get("records"):
                            if com_info.get("prName") not in self.sku_name.keys():
                                self.sku_name[com_info.get("prName")] = ""  # 用于去重
                                sku["prName"] = com_info.get("prName")  # 名称
                                sku["saleAmt"] = com_info.get("saleAmt")  # 价格
                                sku["marketAmt"] = com_info.get("marketAmt")  # 原价
                                sku["tmBuyStart"] = com_info.get("tmBuyStart")  # 开始购买时间
                                sku["tmBuyEnd"] = com_info.get("tmBuyEnd")  # 结束购买时间
                                myprint("活动标题'{}'下找到商品'{}'".format(kwargs.get(id_type), sku.get("prName")))
                                self.sku_info_list.append(sku)
                                sku = {}
                            else:
                                myprint(
                                    "活动标题'{}'下找到商品'{} 此商品已经存在 直接过滤'".format(kwargs.get(id_type), com_info.get("prName")))
                else:
                    resp = requests.post(url=url, data=body, verify=False).json()
                    for com_info in resp.get("data"):
                        if com_info.get("prName") not in self.sku_name.keys():
                            self.sku_name[com_info.get("prName")] = ""  # 用于去重
                            sku["prName"] = com_info.get("prName")  # 名称
                            sku["saleAmt"] = com_info.get("saleAmt")  # 价格
                            sku["marketAmt"] = com_info.get("marketAmt")  # 原价
                            sku["tmBuyStart"] = com_info.get("tmBuyStart")  # 开始购买时间
                            sku["tmBuyEnd"] = com_info.get("tmBuyEnd")  # 结束购买时间
                            myprint("活动标题'{}'下找到商品'{}'".format(kwargs.get(id_type), sku.get("prName")))
                            self.sku_info_list.append(sku)
                            sku = {}
                        else:
                            myprint("活动标题'{}'下找到商品'{} 此商品已经存在 直接过滤'".format(kwargs.get(id_type), com_info.get("prName")))
                myprint("累计搜索到{}个商品.".format(len(self.sku_info_list)))
    
        def to_excel_data(self):
            self.load_index_sku(**self.get_windows_id())
            sku_name = []
            sku_price = []
            sku_raw_price = []
            sku_buy_start = []
            sku_buy_end = []
            sku_excel_dict = {}
    
            for sku in self.sku_info_list:
                sku_name.append(sku.get("prName"))
                sku_price.append(sku.get("saleAmt"))
                sku_raw_price.append(sku.get("marketAmt"))
                sku_buy_start.append(sku.get("tmBuyStart"))
                sku_buy_end.append(sku.get("tmBuyEnd"))
            sku_excel_dict["商品名称"] = sku_name
            sku_excel_dict["目前售价"] = sku_price
            sku_excel_dict["原始价格"] = sku_raw_price
            sku_excel_dict["开始购买时间"] = sku_buy_start
            sku_excel_dict["结束购买时间"] = sku_buy_end
            file_path = r'兴盛优选商品数据_%s.xlsx' % (time.strftime('%Y-%m-%d', time.localtime(time.time())))
            writer = pd.ExcelWriter(file_path)
            df = pd.DataFrame(sku_excel_dict)
            df.to_excel(writer, columns=sku_excel_dict.keys(), index=False, encoding='utf-8',
                        sheet_name='兴盛优选商品数据')
            writer.save()
            myprint("写入excel成功 写入路径为{}".format(os.getcwd() + r"/" + file_path))
    
    
    if __name__ == "__main__":
        xiejiangpeng = XingShengYouXuna()
        xiejiangpeng.to_excel_data()

最后 分享上运行截图

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 5][]

生成的excel截图为![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 6][]

[https_mall.xsyxsc.com_user_product_indexSortWindows]: https://mall.xsyxsc.com/user/product/indexSortWindows
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70]: /images/20211029/48cb43e2ca1c470ca601460eea50ca4b.png
[https_mall.xsyxsc.com_user_product_activityProducts]: https://mall.xsyxsc.com/user/product/activityProducts
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 1]: /images/20211029/cc4aa418c58b46e3b63d4c8c7da14b5f.png
[https_mall.xsyxsc.com_user_product_classifyProducts]: https://mall.xsyxsc.com/user/product/classifyProducts
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 2]: /images/20211029/e1f0e34c5cd54771b9c5e26a79d09c8e.png
[https_mall.xsyxsc.com_user_brandhouse_window_getProducts]: https://mall.xsyxsc.com/user/brandhouse/window/getProducts
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 3]: /images/20211029/2d60e8f74eb24c0ea6cbd5960e32e7cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 4]: /images/20211029/988c1e9a479542288c8cbfc16bcfb478.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 5]: /images/20211029/0d55b028abad425dbe6d7b62238eeaed.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 6]: /images/20211029/60cc84d5db95441da735b11bad691e38.png