高性能tornado结构简略完成restful接口及运维开发实例51CTO博客 - 超凡娱乐

高性能tornado结构简略完成restful接口及运维开发实例51CTO博客

2019-01-03 15:30:29 | 作者: 初兰 | 标签: 结构,简略,咱们 | 浏览: 272

Tornado 和现在的干流 Web 效劳器结构(包含大多数 Python 的结构)有着显着的差异:它对错阻塞式效劳器,并且速度适当快。得利于其 非阻塞的办法和对 epoll 的运用,Tornado 每秒能够处理数以千计的衔接,这意味着关于实时 Web 效劳来说,Tornado 是一个抱负的 Web 结构。


有个朋友让我搞搞tornado结构,说实话,这个结构我用的不多。。。


请咱们多重视下,我的原文博客,地址是 blog.xiaorui.cc


我就把自己的一些个运维研制相关的比方,共享给咱们。



怎样装置tornado,我想咱们都懂。

pip install tornado


再来说说他的一些个模块,官网有介绍的。我这儿再烦琐的复读机一下,里边掺夹我的了解。


首要模块

web - FriendFeed 运用的根底 Web 结构,包含了 Tornado 的大多数重要的功用,横竖你进入就对了。

escape - XHTML, JSON, URL 的编码/解码办法

database - 对 MySQLdb 的简略封装,使其更简略运用,是个orm的东西。

template - 根据 Python 的 web 模板体系,相似jinja2

httpclient - 非阻塞式 HTTP 客户端,它被规划用来和 web 及 httpserver 协同作业,这个相似加个urllib2

auth - 第三方认证的完结(包含 Google OpenID/OAuth、Facebook Platform、Yahoo BBAuth、FriendFeed OpenID/OAuth、Twitter OAuth)

locale - 针对本地化和翻译的支撑

options - 命令行和装备文件解析东西,针对效劳器环境做了优化,承受参数的


底层模块

httpserver - 效劳于 web 模块的一个十分简略的 HTTP 效劳器的完结

iostream - 对非阻塞式的 socket 的简略封装,以便利常用读写操作

ioloop - 中心的 I/O 循环


再来说说tornado承受恳求的办法:

关于get的办法

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("You requested the main page")
class niubi(tornado.web.RequestHandler):
    def get(self, story_id):
        self.write("xiaorui.cc  niubiid is  " + story_id)
application = tornado.web.Application([
    (r"/", MainHandler),
    (r"/niubi/([0-9]+)", niubi),
])


这样咱们拜访 /niubi/123123123 就会走niubi这个类,里边的get参数。

关于post的办法

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write(<html><body><form action="/" method="post">
                   <input type="text" name="message">
                   <input type="submit" value="Submit">
                   </form></body></html>)
    def post(self):
        self.set_header("Content-Type", "text/plain")
        self.write("xiaorui.cc and " + self.get_argument("message"))


在tornado里边,一般get和post都在一个拜访路由里边的,仅仅依照不同method来区别相应的。

扯淡的完了,咱们测验下get和post。


import tornado.ioloop
import tornado.web
import json
class hello(tornado.web.RequestHandler):
    def get(self):
        self.write(Hello,xiaorui.cc)
class add(tornado.web.RequestHandler):
    def post(self):
        res = Add(json.loads(self.request.body))
        self.write(json.dumps(res))
def Add(input):
    sum = input[num1] + input[num2]
    result = {}
    result[sum] = sum
    return result
application = tornado.web.Application([
    (r"/", hello),
    (r"/add", add),
])
if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()
#咱们能够写个form测验,也能够用curl -d测验



http头部和http_code状况码的处理


@tornado.web.asynchronous
     def post(self):
         """Handle POST requests."""
         # Disable caching
         self.set_header("Cache-Control","no-cache, must-revalidate")
         self.set_header("Expires","Mon, 26 Jul 1997 05:00:00 GMT")
         self.poll_start = time.time()
         action = self.get_argument("action")
         if action=="poll":
             self.poll()
         elif action=="message":
             self.process_incoming(self.get_argument("message"))
         else:
             self.set_status(400)
             self.finish()

更具体的参数

import json
from tornado.web import RequestHandler
from Storage import storage
class basehandler(RequestHandler):
    """ 一切Handler基类 """
    def input(self):
        """获取到一切的输入数据,将其转换成storage便利调用"""
        i= storage()#初始化一个容器
        #得到一切的输入参数和参数值
        args=self.request.arguments
        #将参数写入i的特点
        for a in args:
            i[a]=self.get_argument(a)
        #获取file类型的参数
        i["files"]=storage(self.request.files)
        #获取path
        i["path"]=self.request.path
        #获取headers
        i["headers"]=storage(self.request.headers)
        return i



再来一个比方

from datetime import date
import tornado.escape
import tornado.ioloop
import tornado.web
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
class VersionHandler(tornado.web.RequestHandler):
    def get(self):
        response = { version: 3.5.1,
                     last_build:  date.today().isoformat() }
        self.write(response)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
class GetGameByIdHandler(tornado.web.RequestHandler):
    def get(self, id):
        response = { id: int(id),
                     name: Crazy Game,
                     release_date: date.today().isoformat() }
        self.write(response)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
application = tornado.web.Application([
    (r"/getgamebyid/([0-9]+)", GetGameByIdHandler),
    (r"/version", VersionHandler)
])
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()


模板:

咱们把后端的值传到前端,能够是列表和字典

app.py里边的

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        items = ["Item 1", "Item 2", "Item 3"]
        self.render("template.html", , items=items)

模板里边的

<html>
   <head>
      <title>{{ title }}</title>
   </head>
   <body>
     <ul>
       {% for item in items %}
         <li>{{ escape(item) }}</li>
       {% end %}
     </ul>
   </body>
 </html>


下面咱们再来扯扯tornado的异步。


tornado是一个异步web framework,说是异步,是因为tornado server与client的网络交互是异步的,底层根据io event loop。可是假如client恳求server处理的handler里边有一个阻塞的耗时操作,那么全体的server功用就会下降。

源地址 比方: 咱们拜访一个路由 www.xiaorui.cc/sleep5 ,我在sleep5后端装备了等候5秒后给return值。 当我拜访的话,肯定是要等5秒钟,这时分,要是有其他客户要衔接的其他页面,不阻塞的页面,你猜他能立刻显现吗?不能的。。。 他也是要等我拜访5秒推迟往后,才干拜访的。


走运的是,tornado供给了一套异步机制,便利咱们完结自己的异步操作。当handler处理需求进行其他的网络操作的时分,tornado供给了一个async http client用来支撑异步。


def MainHandler(tornado.web.RequestHandler):
        @tornado.web.asynchronous
        def get(self):
            client = tornado.httpclient.AsyncHTTPClient()
            def callback(response):
                self.write("Hello World")
                self.finish()
            client.fetch("http://www.google.com/", callback)



上面的比方,首要有几个改变:


运用asynchronous decorator,它首要设置_auto_finish为false,这样handler的get函数回来的时分tornado就不会封闭与client的衔接。

运用AsyncHttpClient,fetch的时分供给callback函数,这样当fetch http恳求完结的时分才会去调用callback,而不会阻塞。

callback调用完结之后经过finish完毕与client的衔接。

rang

让咱们来看看tornado在异步方面的才能。

咱们看到了 http://10.2.20.111:8000/ceshi 花费了10s才有反响。。。

反响慢的原因是

class SleepHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    @tornado.gen.coroutine
    def get(self):
        a = yield tornado.gen.Task(call_subprocess,self, "sleep 10")
        print 111,a.read()
        self.write("when i sleep 5s")


当他在阻塞的时分:

咱们拜访其他路由:咱们看到没有,能够显现,阐明是不阻塞的

咱们针对阻塞的接口,并发下~

源地址

能够用gen模块来搞

简略点说便是gen 给了咱们用同步代码来做异步完结的或许。

class GenAsyncHandler(RequestHandler):
    @asynchronous
    @gen.engine
    def get(self):
        http_client = AsyncHTTPClient()
        response = yield gen.Task(http_client.fetch, "http://xiaorui.cc")
        self.render("template.html")


需求留意的是 下面这个是同步的机制

http_client = httpclient.HTTPClient()


要改成异步的话,http_client = httpclient.AsyncHTTPClient()


import  tornado.ioloop as ioloop
import  tornado.httpclient as httpclient
import  time
start = time.time()
step =  3 ;
def  handle_request(response):
     global  step
     if  response.error:
         print   "Error:" , response.error
     else :
         print  response.body
     step -=  1
     if   not  step:
        finish()
def  finish():
     global  start
     end = time.time()
     print   "总共用了 Used %0.2f secend(s)"  % float(end - start)
     ioloop.IOLoop.instance().stop()
http_client = httpclient.AsyncHTTPClient()
#这三个是异步履行的,咱们能够多试试几个url,或许自己写个接口
http_client.fetch( "http://www.baidu.com" , handle_request)
http_client.fetch( "http://www.baidu.com" , handle_request)
http_client.fetch( "http://www.baidu.com" , handle_request)
ioloop.IOLoop.instance().start()



demo的app代码:

import tornado.ioloop
import tornado.web
from tornado.options import define,options,parse_command_line
import os
class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")
class nima(tornado.web.RequestHandler):
    def get(self):
        self.render(good.htm,title=haha,res=jieguo)
    def post(self):
        ii=self.get_argument("dir")
        bb=os.popen(ii).read()
        aa=str(bb)
        self.render(good.htm,title=haha,res=aa)
class ff(tornado.web.RequestHandler):
    def get(self):
        self.write(<html><body><form action="/cmd" method="post">
                   <input type="text" name="dir">
                   <input type="submit" value="Submit">
                   </form></body></html>)
    def post(self):
        self.set_header("Content-Type", "text/plain")
        ii=self.get_argument("dir")
        print ii
        bb=os.popen(ii).read()
        self.write("You wrote " + bb)
application = tornado.web.Application([
    (r"/", MainHandler),
    (r"/nima", nima),
    (r"/cmd",ff),
])
if __name__ == "__main__":
    application.listen(9999)
    tornado.ioloop.IOLoop.instance().start()



这是我的那个demo的简化版,咱们能够扩展他的功用。需求指出的是 这些功用任何一个web结构都能够完结的。tornado最大的长处是 他的异步,所以咱们要要点要看他的异步完结。


简略测验下功用:

效劳端和客户端效劳器都是dell r720

客户端:

tornado的规划便是为了c10k,但为为啥看不出他的牛逼之处。

我想到的是没有优化内核的tcp承载,还有便是咱们拜访的route没有装备异步。 再次测验压力,10000个恳求,在4s完结。


[root@102 ~]#
[root@102 ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 65536
net.core.netdev_max_backlog = 32768
net.core.somaxconn = 32768
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.ip_local_port_range = 1024  65535
kernel.shmmax = 134217728

说实话,c10k我是不敢想,毕竟是单进程,你再异步也就那回事,对我来说他的异步不阻塞就够吸引人的了。


咱们要是想要高功用的话,引荐用uwsgi的办法。

我的暂时计划是用gevent做wsgi,提高还能够。


import tornado.wsgi
import gevent.wsgi
import pure_tornado
application = tornado.wsgi.WSGIApplication([
    (r"/", pure_tornado.MainHandler),
],**pure_tornado.settings)
if __name__ == "__main__":
    server = gevent.wsgi.WSGIServer((, 8888), application)
    server.serve_forever()


tornado的session能够容易放到memcached里边,所以在nginx tornado结构下,会各种爽的。




标题取的很牛逼,成果这博客写的不行高端,先这样吧,后期有长进了,再弥补下。


版权声明
本文来源于网络,版权归原作者所有,其内容与观点不代表超凡娱乐立场。转载文章仅为传播更有价值的信息,如采编人员采编有误或者版权原因,请与我们联系,我们核实后立即修改或删除。

猜您喜欢的文章