[搜片神器]直接从DHT网络下载BT种子的方法

DHT抓取程序开源地址：https://github.com/h31h31/H31DHTDEMO

数据处理程序开源地址:https://github.com/h31h31/H31DHTMgr

DHT系列文章:

1.[搜片神器] 之P2P中DHT网络爬虫原理

2.[搜片神器]之DHT网络爬虫的代码实现方法

3.[搜片神器]之DHT网络爬虫的C++程序初步开源

--------------------------------------------------------------------------------------------------------------------

看懂此文章须要提早看明白上面的系列文章,还须要你有TCP网络编程和bencode编码方法基础上,若是都看不明白,能够到娱乐区http://www.sosobta.com  去看看,休息下...

 

在介绍了这么多期文章后,最后介绍BT网络里面一个比较重要种子下载协议,方便你们知道如何从DHT网络直接下载种子的问题.

先说下咱们目前下载电影等文件是如何下载的,好比咱们有个BT种子,就能够去下载对应的文件,但若是咱们只有个文件名字,如何去找BT种子呢?

首先咱们能够去经过搜索获得磁链接,而后就由此字符串去下载对应的种子文件和电影等信息,但若是没有网站让你下载种子,咱们又当如何去搜索这个种子呢?

目前咱们下载BT种子有两种方式:                                                                           

1. 经过HTTP直接从WEB服务器上下载,这种直接方便,好比从迅雷服务器上下载种子,
2. 再就是经过BT软件从网络里面去获取BT网络里面专门有个下载种子的协议文件,只能下载种子,而后种子下载好后就能够交给BT软件来下载数据了.

如何从DHT网络下载种子,必须先看两个协议文章:

http://www.bittorrent.org/beps/bep_0009.html

http://www.bittorrent.org/beps/bep_0010.html

这里面有介绍,但仍是须要说明一下如何操做的流程方便你们更好的理解.

 咱们的代码流程必须仍是基于 DHT抓取程序开源地址：https://github.com/h31h31/H31DHTDEMO 之上,由于是从DHT网络里面获取数据,

须要咱们在此之上操做后续流程.

以前的DHT有SEARCH的相关代码来搜索这个HASH对应的哪些IP在提供下载.

        /* This is how you trigger a search for a torrent hash.  If port (the second argument) is non-zero, it also performs an announce.
           Since peers expire announced data after 30 minutes, it's a good idea to reannounce every 28 minutes or so. */
        if(searching) {
            //m_dht.dht_random_bytes((void*)hashList[2],20);
            if(m_soListen >= 0)
                m_dht.dht_search(hashList[2], 0, AF_INET, DHT_callback, this);
            if(s6 >= 0)
                m_dht.dht_search(hashList[2], 0, AF_INET6, DHT_callback, this);
            searching = 0;
        }

搜索到对方返回的IP信息和端口号后,你们能够分析dht.c里面的函数代码dht_periodic(const void *buf, size_t buflen,const struct sockaddr *fromAddr, int fromlen,time_t *tosleep,dht_callback *callback, void *closure)函数里面的ANNOUNCE_PEER返回请求里面带有对方代表本身此BT种子对应的认证码peerid.

dht_periodic(const void *buf, size_t buflen,const struct sockaddr *fromAddr, int fromlen,time_t *tosleep,dht_callback *callback, void *closure)
函数里面的ANNOUNCE_PEER

        case ANNOUNCE_PEER:
            _dout("Announce peer!From IP:%s:%d\n",inet_ntoa(tempip->sin_addr),tempip->sin_port);
            new_node(id, fromAddr, fromlen, 1);

            if(id_cmp(info_hash, zeroes) == 0) 
            {
                _dout("Announce_peer with no info_hash.\n");
                send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with no info_hash");
                break;
            }
            if(!token_match(token, token_len, fromAddr)) {
                _dout("Incorrect token for announce_peer.\n");
                send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with wrong token");
                break;
            }
            if(port == 0) {
                _dout("Announce_peer with forbidden port %d.\n", port);
                send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with forbidden port number");
                break;
            }
            if(callback) 
            {
                (*callback)(closure, DHT_EVENT_ANNOUNCE_PEER_VALUES, info_hash,(void *)fromAddr, port,id);//此ID就是peerid,
            }

知道了对应的IP,端口号,还有种子ID号,就能够向对方发送请求了.

获取HASH是经过UDP网络,但下载BT种子是经过TCP来处理,至关于别人是TCP服务器,咱们链接过去,直接下载对应PEERID的种子就好了.

BT种子在DHT网络下载流程                                                                  

 先看http://www.bittorrent.org/beps/bep_0010.html协议介绍,咱们必须先握手  

• handshake message

 

 此包构造比较简单,按照格式进行组装就好了,而后发送出去,对方就会回应本身是什么客户端的软件提供种子下载.

void CH31BTMgr::Encode_handshake()
{
	//a byte with value 19 (the length of the string that follows);
	//the UTF-8 string "BitTorrent protocol" (which is the same as in ASCII);
	//eight reserved bytes used to mark extensions;
	//the 20 bytes of the torrent info hash;
	//the 20 bytes of the peer ID.
	char btname[256];
	memset(btname,0,sizeof(btname));
	sprintf(btname,"BitTorrent protocol");
	char msg[1280];
	memset(msg,0,sizeof(msg));
	msg[0]=19;
	memcpy(&msg[1],btname,19);
	char ext[8];
	memset(ext,0,sizeof(ext));
	ext[5]=0x10;

	memcpy(&msg[20],ext,8);
	memcpy(&msg[28],m_hash,20);
	memcpy(&msg[48],m_peer_id,20);
	int res1=Write(msg, 68);//TCP发送消息
}

在发送握手后,咱们能够接着发送种子数据请求包,须要学习http://www.bittorrent.org/beps/bep_0009.html 里面的内容:　　

extension header
The metadata extension uses the extension protocol (specified in BEP 0010 ) to advertize its existence. It adds the "ut_metadata" entry to the "m" dictionary in the extension header hand-shake message. This identifies the message code used for this message. It also adds "metadata_size" to the handshake message (not the "m" dictionary) specifying an integer value of the number of bytes of the metadata.

Example extension handshake message:

{'m': {'ut_metadata', 3}, 'metadata_size': 31235}
extension message
The extension messages are bencoded. There are 3 different kinds of messages:

0 request 
1 data 
2 reject 
The bencoded messages have a key "msg_type" which value is an integer corresponding to the type of message. They also have a key "piece", which indicates which part of the metadata this message refers to.

In order to support future extensability, an unrecognized message ID MUST be ignored.

这就须要会bencode的相关代码,这个你们能够网上搜索进行编译,若是实现搞不定,能够留下邮箱我将此类代码发送给你,其实也是网上收集整理的.

void CH31BTMgr::Encode_Ext_handshake()
{
    entry m;
    m["ut_metadata"] = 0;
    entry e;
    e["m"]=m;

    char msg[200];
    char* header = msg;
    char* p = &msg[6];
    int len = bencode(p, e);
    int total_size = 2 + len;
    namespace io = detail;
    io::write_uint32(total_size, header);
    io::write_uint8(20, header);
    io::write_uint8(0, header);

    int res1=Write(msg, len + 6);
}

若是别人回应的是2,那就直接退出吧,说明别人拒绝了你.

若是回应是1,则返回的是数据区,每块是16K大小,最后一包不是.

 

data
The data message adds another entry to the dictionary, "total_size". This key has the same semantics as the "metadata_size" in the extension header. This is an integer.

The metadata piece is appended to the bencoded dictionary, it is not a part of the dictionary, but it is a part of the message (the length prefix MUST include it).

If the piece is the last piece of the metadata, it may be less than 16kiB. If it is not the last piece of the metadata, it MUST be 16kiB.

Example:

{'msg_type': 1, 'piece': 0, 'total_size': 3425}
d8:msg_typei1e5:piecei0e10:total_sizei34256eexxxxxxxx...
The x represents binary data (the metadata).

下面给出如何进行提交我须要第几包的数据代码:

void CH31BTMgr::write_metadata_packet(int type, int piece)
{
    ASSERT(type >= 0 && type <= 2);
    ASSERT(piece >= 0);

    entry e;
    e["msg_type"] = type;
    e["piece"] = piece;

    char const* metadata = 0;
    int metadata_piece_size = 0;

    if (type == 1)
    {
        e["total_size"] = 14132;
        int offset = piece * 16 * 1024;
        //metadata = m_tp.metadata().begin + offset;
        metadata_piece_size = (std::min)(int(14132 - offset), 16 * 1024);
    }

    char msg[200];
    char* header = msg;
    char* p = &msg[6];
    int len = bencode(p, e);
    int total_size = 2 + len + metadata_piece_size;
    namespace io = detail;
    io::write_uint32(total_size, header);
    io::write_uint8(20, header);
    io::write_uint8(m_message_index, header);

    int res1=Write(msg, len + 6);
}

在接收到一包请求后咱们才能够继续下一包的请求,下面给了咱们如何解析这一包的问题代码:

// 处理一个完整的包数据
bool CH31BTMgr::DeCodeFrameData(char * buffer,int buflen)
{
    char * p = (char *)mhFindstr(buffer, buflen, "ut_metadatai", 12);
    if(p) 
    {
        m_message_index=atoi(&p[12]);
        if(m_message_index==2)
        {
            return false;
        }
        write_metadata_packet(0,0);
        char filename[256];
        memset(filename,0,sizeof(filename));
        sprintf(filename,"%s\\torrent.txt",m_workPath);
        DelFile(filename);
    } 

    p = (char *)mhFindstr(buffer, buflen, "metadata_sizei", 14);
    if(p) 
    {
        m_metadata_size=atoi(&p[14]);
        m_fileCnt=(int)(m_metadata_size/16384)+1;
    } 

    p = (char *)mhFindstr(buffer, buflen, "msg_typei", 9);
    if(p) 
    {
        int type1=atoi(&p[9]);
        if(type1==1)
        {
            p = (char *)mhFindstr(buffer, buflen, "piecei", 6);
            if(p) 
            {
                int piece=atoi(&p[6]);
                p = (char *)mhFindstr(buffer, buflen, "total_sizei", 11);
                if(p) 
                {
                    int total_size=atoi(&p[11]);
                    p = (char *)mhFindstr(buffer, buflen, "ee", 2);
                    if(p) 
                    {
                        //保存数据
                        FILE* pfile=NULL;
                        char filename[256];

                        memset(filename,0,sizeof(filename));
                        sprintf(filename,"%s\\torrent.txt",m_workPath);
                        char openmethod[5]="a";
                        if(piece==0)
                            sprintf(openmethod,"w");
                        if((pfile=fopen(filename,openmethod))!=NULL)
                        {
                            if((piece+1)*16*1024<total_size)
                            {
                                fseek(pfile,(piece)*16*1024,SEEK_SET);
                                fwrite(&p[2],1,16*1024,pfile);
                                write_metadata_packet(0,piece+1);
                                fclose(pfile);
                            }
                            else
                            {
                                fwrite(&p[2],1,total_size-(piece)*16*1024,pfile);
                                fclose(pfile);
                                ManageTorrentFileToRealFile(filename);
                            }
                        }
                    }
                }
            }
        }
        else if(type1==2)
        {
            return false;
        }
    } 
    
    return true;
}

void * mhFindstr(const void *haystack, size_t haystacklen,const void *needle, size_t needlelen)
{
const char *h =(const char *) haystack;
const char *n =(const char *) needle;
size_t i;

/* size_t is unsigned */
if(needlelen > haystacklen)
return NULL;

for(i = 0; i <= haystacklen - needlelen; i++) {
if(memcmp(h + i, n, needlelen) == 0)
return (void*)(h + i);
}
return NULL;
}

 

接下来讲下如何进行快速调试的问题:                                                                                        

       第一次调试也很天真的等着DHT网络上的数据过来,须要等好久,并且调试老是发现别人不回应,要么就是拒绝,通过一段时间后,

问朋友老是不对问题,结果是协议没有构造对.下面就须要注意的地方总结下:

1.必定要接收到别的人PEERID后才可以与别人交流,否则别人确定不理你;

2.构造协议调试不可以在外网络上调试,最好你们将mono-monotorrent源代码下载回来,调试分析下,本地开启服务器;

3.经过本地与mono-monotorrent进行调试,你就能够分析出是哪里不对的问题,是否是协议哪些封装得不对的问题.

4.经过DHT网络下载回来的种子确定是最新的,WEB下载的可能尚未呢..

5.经过协议下载回来的种子好像没有announce-list,不知道为何不提供一些内容,可能还有些什么关键地方没有下载,分析mono-monotorrent代码里面就是不提供下载,但愿高手指点.

6.TCPClient接收数据区须要开到16K以上,这样方便处理,固然若是会先后拼接包就更好.

7.若是须要bencode相关的编码C++代码,能够在此留言或者给h31h31#163.com发邮件.

 

若是此文章看不太明白,请先看看以前的文章,分析调试下代码,再来学习此文章可能就比较懂一些.

但愿有了解的朋友更好的交流和进步.在此留言学习讨论.

 

但愿你们多多推荐哦...你们的推荐才是下一篇介绍的动力...