秋天的第一杯奶茶该买哪家?Python 爬取美团网红奶茶店告诉你
图1 奶茶发展史
数据获取
# 构建抓取URL
def get_url_1():
for city,city_code in city_dict.items():
for block_dict in area_dict[city]:
for children in block_dict['children']:
for page in range(1,33):
block_code = children['id']
offset = 32 * (page-1)
# print(city, area, block, block_code)
url = 'https://apimobile.meituan.com/group/v4/poi/pcsearch/{}?uuid=6ddabcb37fdd4a8e9cdf.1599125825.1.0.0&userid=280531290&limit=32&offset={}&cateId=-1&q=奶茶果汁&areaId={}&sort=solds'.format(city_code,offset,block_code)
redis_db.sadd('meituan_milk', url)
数据清洗
数据清洗部分,主要清洗了奶茶店铺名称,虽然数据均来源于美团网,但是同一个奶茶品牌会有多种格式,如1点点和1點點,大卡司和大卡司DAKASI。由于奶茶品牌数量众多,并且真假难辨,所以只能进行针对性清洗,对部分名气高的奶茶品牌名称要保证其统一。
# 清洗字段
def clean(x):
title = re.sub(u"(.*?)", "", x['title'])
title = title.replace('點點','点点').replace('(','').replace(')','')
title = title.replace('一点点','1点点')
if '一杯会说话的茶' in title:
title = '1314一杯会说话的茶'
elif '大卡司' in title:
title = '大卡司DAKASI'
elif '1点点' in title:
title = '1点点'
elif '都可' in title:
title = 'CoCo都可'
elif '书亦烧仙草' in title:
title = '书亦烧仙草'
elif '蜜雪冰城' in title:
title = '蜜雪冰城'
elif 'royal' in title or 'Royal' in title or 'ROYAL' in title:
title = 'Royaltea皇茶'
elif 'ALS' in title:
title = 'ALS GONG CHA贡茶'
elif 'GONG' in title:
title = '贡茶'
elif '茶百道' in title:
title = '茶百道'
elif '吾饮良品' in title:
title = '吾饮良品'
elif '悸动烧仙草' in title:
title = '悸动烧仙草'
elif '沪上阿姨' in title:
title = '沪上阿姨'
elif '7分甜' in title:
title = '7分甜'
elif '古茗' in title:
title = '古茗'
elif '奈雪' in title:
title = '奈雪の茶'
elif '悦色' in title:
title = '茶颜悦色'
else:
pass
return title
df['title'] = df.apply(clean, axis=1)
数据可视化
当笔者在制作可视化图表的时候,会发现有些奶茶品牌的名称极为相似,让人有一种傻傻分不清楚的感觉。
图2 类似的奶茶品牌名称
图3 热门城市奶茶店铺数量情况
图4 奶茶品牌店铺数量 TOP15
图5 特色奶茶分布情况
图6 大众奶茶分布情况
图7 奶茶品牌均价情况
结语
Gordon,对爬虫/机器学习/数据建模/可视化均有所涉猎的数据分析师。
Alice,在读研究僧一枚,跨行新手数据分析师。
更多精彩推荐