如何用python爬取抖音视频列表

导语

如果看到特别感兴趣的抖音vlogger的视频,想全部dump下来,如何操作呢?下面介绍介绍如何使用python导出特定用户所有视频信息

img

抓包分析

  • Chrome Deveploer Tools Chrome 浏览器开发者工具

在抖音APP端,复制vlogger主页地址, 比如: http://v.douyin.com/kGcU4y/, 在PC端用chrome浏览器打卡,并模拟手机,这里选择iPhone, 然后把复制的主页地址,放到浏览器进行访问,页面跳转到https://www.iesdouyin.com/share/user/110677980134

  • 下拉主页, 选择Network=>XHR 选项卡, 看到类似请求
:authority: www.iesdouyin.com
:method: GET
:path: /web/api/v2/aweme/post/?user_id=110677980134&sec_uid=&count=21&max_cursor=1561112910000&aid=1128&_signature=3Xf-nxAQgGfUO4SKisB.Ld13.o&dytk=061ae6e81229e178146aa674327eba89
:scheme: https
accept: application/json  
accept-encoding: gzip, deflate, br  
accept-language: zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,zh-TW;q=0.6,da;q=0.5  
cookie: tt_webid=6690145457198417412; _ga=GA1.2.605400954.1557670882; _ba=BA0.2-20181226-5199e-GIJXgXk9ajNkyFhmv7Wy; _gid=GA1.2.1914501522.1562857517  
referer: https://www.iesdouyin.com/share/user/110677980134  
user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1  
x-requested-with: XMLHttpRequest  

返回数据截图

img

通过分析ajax请求的URL https://www.iesdouyin.com/web/api/v2/aweme/post/?user_id=110677980134&sec_uid=&count=21&max_cursor=1559299764000&aid=1128&_signature=3Xf-nxAQgGfUO4SKisB.Ld13.o&dytk=061ae6e81229e178146aa674327eba89 得出请求参数主要包含:

| 字段 | 类型 | 说明 | |------|------------|------------| | userid | int | 抖音账号的ID | | count | int | 返回的数据条数,就用默认值21 | | maxcursor| int | 请求的游标,每次请求带上上次请求返回的maxcursor | |aid|int|使用默认值11128| |signature|string|每次请求带上的参数签名| |dytk|string|每次请求带上的一个参数|

参数的获取方法: - userid 通过访问主页地址,重定向之后的URL地址(https://www.iesdouyin.com/share/user/110677980134)上可以获取 userid - dytk 在主页的网页源码中包含此参数,网页源码片段

(function() {
    $(function(){
        __M.require('douyin_falcon:page/reflow_user/index').init({
            uid: "110677980134",
            dytk: '061ae6e81229e178146aa674327eba89'
        });
    });
})();

通过正则获取到此参数

  • _signature 获取比较复杂,抖音对前端的js代码进行了混淆压缩,不易直接分析出算法过程,不过可以执行签名算法代码,并返回对应的签名结果。
  • 执行js代码的可以使用nodejs或者selenium webdriver,这里推荐使用selenium webdriver , nodejs的js执行环境与浏览器有区别,计算出的签名结果,并不能通过验证,selenium webdriver 可以调用本地浏览器,计算出的签名可以跟浏览器直接访问访问计算出的签名一致。
  • 格式化之后的js代码,点击查看, 执行js方法_bytedAcrawler.sign("110677980134")对参数进行签名

代码实现导出主页视频列表

def get_user_video_list_by_uid(user_id, cursor=0):  
    url = 'https://www.iesdouyin.com/web/api/v2/aweme/post/?'
    sign, dytk = signature(user_id)
    tk_logger.info("sign:%s,dytk:%s" % (sign, dytk))
    if sign is None or dytk is None:
        tk_logger.log("sign [%s] or dytk [%s] is none" % (sign, dytk))
        return None
    headers = dict_merge(CHROME_HEADER, {
        "Accept": "application/json",
        "X-Requested-With": "XMLHttpRequest",
    })
    params = {
        "user_id": user_id,
        "count": "21",
        "max_cursor": cursor,
        "aid": "1128",
        "_signature": sign,
        "dytk": dytk
    }
    res = requests.get(url, headers=headers, params=params)
    tk_logger.info("request url: %s" % res.url)
    content = res.content.decode("utf8")
    jsn = json.loads(content)
    return jsn

获取的视频列表信息 视频列表

获取视频信息代码片段

def get_video_detail_by_id(video_id):  
    url = "https://aweme-hl.snssdk.com/aweme/v1/aweme/detail/?version_code=6.5.0&pass-region=1&pass-route=1&js_sdk_version=1.16.2.7&app_name=aweme&vid=9D5F078E-A1A9-4F64-81C7-F89CA6A3B1DC&app_version=6.5.0&device_id=34712926793&channel=App%20Store&mcc_mnc=46011&aid=1128&screen_width=750&openudid=263bd93f02801d126ca004edccbff8f6e1b19f51&os_api=18&ac=WIFI&os_version=12.3.1&device_platform=iphone&build_number=65014&device_type=iPhone9,1&iid=74239983401&idfa=F39B285A-4B4F-4874-9D7E-C728A892BF6D"
    data = {"aweme_id": video_id}
    headers = {
        "sdk-version": "1",
        "x-Tt-Token": "00fc1e7950db67b5f43a312e9265cdfee513ea70c36d918c871f3bb553347f3db50ffca143b8722327b345816a75efca071d",
        "User-Agent": "Aweme 6.5.0 rv:65014 (iPhone; iOS 12.3.1; en_CN) Cronet",
        "Content-Type": "application/x-www-form-urlencoded",
        "Cookie": "tt_webid=6636348554880222728; __tea_sdk__user_unique_id=6636348554880222728; odin_tt=76d9b82d6e6f2ddfc99719a5b5d44a7d703cf977f0f7bddf8537f93920d57cb9ec33162ee47868b760f6b09e69209bb2f90bad220b75678af850a0dfa9f056e2; install_id=74239983401; ttreq=1$dab0516952a4157c0c11d4993533c09d6e45fc94; sid_guard=fc1e7950db67b5f43a312e9265cdfee5%7C1559955316%7C5184000%7CWed%2C+07-Aug-2019+00%3A55%3A16+GMT; uid_tt=0afcb06309f632d872799ec0ac3b2c80; sid_tt=fc1e7950db67b5f43a312e9265cdfee5; sessionid=fc1e7950db67b5f43a312e9265cdfee5",
        "X-Khronos": "1559956401",
        "X-Gorgon": "8300000000002e40eee38cad71d14037bd1385d18bc973f094f5",
    }
    ret = {}
    res = requests.post(url, data=data, headers=headers)
    if res.status_code == 200:
        # tk_logger.info("video detail raw:%s" % res.content.decode("utf8"))
        jsn = json.loads(res.content)
        detail = jsn.get("aweme_detail", {})
        video_info = get_video_info(detail)
        user_info = get_user_info(detail)
        play_addr = get_play_address(detail)
        video_cover = get_video_cover(detail)
        ret["video_info"] = video_info
        ret["user_info"] = user_info
        ret["play_addr"] = play_addr
        ret["video_cover"] = video_cover
    else:
        raise TKException("get video detail failed [%s][%d]" % (url, res.status_code))
    return ret

下载视频代码片段

detail =  get_video_detail_by_id(video_id)  
def download_video(detail):  
    url = detail.get("play_addr", {}).get("url_list", [])
    if len(url) == 0:
        raise TKException("cannot get video url list [%s]" % detail)

    url = url[0]
    folder = DOWNLOAD_DIR + '/' + detail.get('user_info', {}).get("uid", "unknown")
    if not os.path.exists(folder):
        os.mkdir(folder)
    video_id = detail.get('video_info', {}).get('statistics', {}).get('aweme_id')
    # filename = "%s/%s" % (folder, detail.get("video_info", {}).get("desc", video_id) + ".mp4")
    filename = "%s/%s" % (folder, video_id + ".mp4")
    tk_logger.info("download video %s" % url)
    if os.path.isfile(filename):
        file_size = get_remote_file_size(url)
        if file_size == os.path.getsize(filename):
            tk_logger.info("file already downloaded, skip ...")
            return
        else:
            tk_logger.info("download file , file size:%d" % file_size)
    res = requests.get(url, headers=IOS_HEADER)
    if res.status_code == 200:
        with open(filename, "wb") as fp:
            for chunk in res.iter_content(chunk_size=1024):
                fp.write(chunk)
    else:
        raise TKException("download video [%s] failed [%d]" % (url, res.status_code))

下载视频

img

声明

此教程仅为交流学习,不作商业用途