天然语言处理----处理原始文本

时间 2019-11-26

标签天然语言处理原始文本繁體版

原文原文链接

本文主要介绍编程访问网络文本的几种方式。html

1. 访问网络资源python

>>> from urllib import urlopen
>>> url='http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.astype.html'
>>> raw=urlopen(url).read()
>>> type(raw)
<type 'str'>
>>> len(raw)
16429
>>> raw[:75]
'\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"\n  "http://'

View Code

若是Python没法正确自动检测出Internet代理，可使用下面方法手动指定。正则表达式

>>> proxies={'http': 'http://www.someproxy.com:3128'}
>>> raw=urlopen(url, proxies=proxies).read（）

2. 访问博客编程

在Universal Feed Parser的第三方python库的帮助下，能够访问博客的内容。网络

>>> import feedparser
>>> llog=feedparser.parse('http://weibo.com/ttarticle/p/show?id=2309404116343489194022')
>>> llog.keys()
['feed', 'status', 'version', 'encoding', 'bozo', 'headers', 'href', 'namespaces', 'entries', 'bozo_exception']
>>> type(llog['feed'])
<class 'feedparser.FeedParserDict'>
>>> llog['feed'].keys()
['meta', 'summary']
>>> llog['feed']['meta']
{'content': u'text/html; charset=gb2312', 'http-equiv': u'Content-type'}
>>> llog['feed']['summary']
u'<span id="message"></span>\n\n&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;'

View Code

3. 处理htmlide

通常有三种方式：正则匹配， nltk.clean_html(), BeautifulSoup. 正则表达式比较繁琐，而nltk.clean_html（）如今已经不支持了，比较简单经常使用的是用BeautifulSoup包。ui

from bs4 import BeautifulSoup

html_doc=''' 
    <html><head><title>The Document's story</title></head>
    <html><head><title>The Dormouse's story</title></head>
    <body>
    <p class="title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
        and they lived at the bottom of a well.</p>

    <p class="story">...</p>
    </body></html>
'''
soup = BeautifulSoup(html_doc, 'html.parser')
content=soup.get_text()
print content

运行结果以下：url

runfile('D:/my project/e_book/XXMLV-2/4.Python_代码/test.py', wdir='D:/my project/e_book/XXMLV-2/4.Python_代码')

The Document's story
The Dormouse's story

The Dormouse's story
Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
        and they lived at the bottom of a well.
...