ahserver/README.md
2019-07-10 18:19:57 +08:00

12 KiB

ahserver

ahserver is a http(s) server base on aiohttp asynchronous framework.

ahserver capabilities:

  • user authorization and authentication support
  • https support
  • processor for registed file type
  • pre-defined variables and function can be called by processors
  • multiple database connection and connection pool
  • a easy way to wrap SQL
  • configure data from json file stored at ./conf/config.json
  • upload file auto save under config.filesroot folder
  • i18n support
  • processors include:
    • 'dspy' file subffix by '.dspy', is process as a python script
    • 'tmpl' files subffix by '.tmpl', is process as a template
    • 'md' files subffix by '.md', is process as a markdown file
    • 'xlsxds' files subffix by '.xlsxds' is process as a data source from xlsx file
    • 'sqlds' files subffixed by '.sqlds' is process as a data source from database via a sql command

Requirements

see requirements.txt

pyutils

sqlor

How to use

see ah.py

from ahserver.configuredServer import ConfiguredServer

if __name__ == '__main__':
	server = ConfiguredServer()
	server.run()

Folder structure

  • app
  • |-ah.py
  • |--ahserver
  • |-conf
  •  |-config.json
    
  • |-i18n

Configuration file content

ahserver using json file format in its configuration, the following is a sample:

{
        "debug":true,
        "databases":{
		"aiocfae":{
                        "driver":"aiomysql",
			"async_mode":true,
                        "coding":"utf8",
                        "dbname":"cfae",
                        "kwargs":{
                                "user":"test",
                                "db":"cfae",
                                "password":"test123",
                                "host":"localhost"
                        }
		},
		"cfae":{
                        "driver":"mysql.connector",
                        "coding":"utf8",
                        "dbname":"cfae",
                        "kwargs":{
                                "user":"test",
                                "db":"cfae",
                                "password":"test123",
                                "host":"localhost"
                        }
		}
        },
        "website":{
                "paths":[
                        "/path/to/path1",
                        "/path/to/path2",
                        "/path/to/path3",
                        "/path/to/path4"
                ],
                "host":"0.0.0.0",
                "port":8080,
                "coding":"utf-8",
                "indexes":[
                        "index.html",
                        "index.tmpl",
                        "index.dspy",
                        "index.md"
                ],
                "visualcoding":{
                        "default_root":"/vc/test",
                        "userroot":{
                                "ymq":"/vc/ymq",
                                "root":"/vc/root"
                        },
                        "jrjpath":"/vc/default"
                },
                "processors":[
                        [".xlsxds","xlsxds"],
                        [".sqlds","sqlds"],
                        [".tmpl.js","tmpl"],
                        [".tmpl.css","tmpl"],
                        [".html.tmpl","tmpl"],
                        [".tmpl","tmpl"],
                        [".dspy","dspy"],
                        [".md","md"]
                ]
        },
        "langMapping":{
                        "zh-Hans-CN":"zh-cn",
                        "zh-CN":"zh-cn",
                        "en-us":"en",
                        "en-US":"en"
        }
}

run mode configuration

if want viboraserver running in debug mode, just add a key-value pairs in config.json

        "debug":true,

else just delete this key-value in config.json in conf folder

database configuration

viboraserver support database data operations, corrently, it ONLY support synchronous DBAPI2. the viboraserver using packages for database engines are:

  • oracle:cx_Oracle
  • mysql:mysql-connector
  • postgresql:psycopg2
  • sql server:pymssql

however, you can change it, but must change the "driver" value the the package name in the database connection definition.

in the databases section in config.json, you can define one or more database connection, and also, it support many database engine, just as ORACLE,mysql,postgreSQL. define a database connnect you need follow the following json format.

  • mysql or mariadb
                "metadb":{
                        "driver":"mysql.connector",
                        "coding":"utf8",
                        "dbname":"sampledb",
                        "kwargs":{
                                "user":"user1",
                                "db":"sampledb",
                                "password":"user123",
                                "host":"localhost"
                        }
                }

the dbname and "db" should the same, which is the database name in mysql database

  • Oracle
		"db_ora":{
			"driver":"cx_Oracle",
			"coding":"utf8",
			"dbname":sampledb",
			"kwargs":{
				"user":"user1",
				"host":"localhost",
				"dsn":"10.0.185.137:1521/SAMPLEDB"
			}
		}
  • SQL Server
                "db_mssql":{
                        "driver":"pymssql",
                        "coding":"utf8",
                        "dbname":"sampledb",
                        "kwargs":{
                                "user":"user1",
                                "database":"sampledb",
                                "password":"user123",
                                "server":"localhost",
                                "port":1433,
                                "charset":"utf8"
                        }
                }
  • PostgreSQL
		"db_pg":{
			"driver":"psycopg2",
			"dbname":"testdb",
			"coding":"utf8",
			"kwargs":{
				"database":"testdb",
				"user":"postgres",
				"password":"pass123",
				"host":"127.0.0.1",
				"port":"5432"
			}
		}

website configuration

paths

viboraserver can serve its contents (static file, dynamic contents render by its processors) resided on difference folders in the server file system. viboraserver finds a content identified by http url in order the of the paths specified by "paths" lists inside "website" definition of config.json file

processors

all the prcessors viboraserver using, must be listed here.

host

by defaualt, '0.0.0.0'

port

by default, 8080

coding

viboraserver recomments using 'utf-8'

indexes

default content file names, viboraserver will use first found file names in the folder

langMapping

the browsers will send 'Accept-Language' are difference even if the same language. so viboraserver using a "langMapping" definition to mapping multiple browser lang to same i18n file

international

viboraserver using MiniI18N in appPublic modules in pyutils package to implements i18n support

it will search translate text in ms* txt file in folder named by language name inside i18n folder in workdir folder, workdir is the folder where the viboraserver program resided or identified by command line paraments.

performance

You can find "Hello world" performance about viboraserver in vibora main page

viboraserver will list and performance test for its processors later

test macheine

xiaomi pro 15.6 i7 cpu, 16Gb memory,m2 256G disk os: ubuntu 18.4 vibora 0.0.7

static file

Running 30s test @ http://localhost:8080/swift/books/hub/welcome.htm
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    24.53ms   25.42ms 461.06ms   95.70%
    Req/Sec     1.50k   320.16     4.00k    69.29%
  533944 requests in 30.10s, 414.67MB read
  Socket errors: connect 0, read 177, write 0, timeout 0
  Non-2xx or 3xx responses: 162164
Requests/sec:  17740.03
Transfer/sec:     13.78MB

welcome.htm file size:

-rw-rw-r-- 1 ymq ymq   952 12月 12  2017 welcome.htm

dspy processor

Running 30s test @ http://localhost:8080/estate/functionlists.dspy
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.18ms   17.45ms 187.20ms   73.62%
    Req/Sec     0.96k   230.18     2.16k    66.82%
  343789 requests in 30.09s, 726.22MB read
Requests/sec:  11426.72
Transfer/sec:     24.14MB

functionlists.dspy

selfserv = [
	{
		"funcid":"0.1",
		"text":"用户注册"
	},
	{
		"funcid":"0.2",
		"text":"用户签到"
	},
	{
		"funcid":"0.3",
		"text":"修改密码"
	},
	{
		"funcid":"0.4",
		"text":"用户签退"
	},
]

saving = [
	{
		"funcid":"1.1",
		"text":"我的存房"
	},
	{
		"funcid":"1.2",
		"text":"存房收藏"	
	},
	{
		"funcid":"1.3",
		"text":"不动产录入"	
	}
]

rentfin = [
	{
		"funcid":"3.1",
		"text":"融房收藏"
	},
	{
		"funcid":"3.2",
		"text":"我的融房"
	},
	{
		"funcid":"3.3",
		"text":"新增融房"
	},
	{
		"funcid":"3.4",
		"text":"融房标准协议"
	},
	{
		"funcid":"3.5",
		"text":"打包资产"
	}
]
return [
	{
		"funcid":"99",
		"text":"首页"
	},
	{
		"funcid":"0",
		"text":"自服务",
		"state":"open",
		"children":selfserv
	},
	{
		"funcid":"1",
		"text":"存房",
		"state":"open",
		"children":saving
	},
	{
		"funcid":"2",
		"text":"优房"
	},
	{
		"funcid":"3",
		"text":"融房",
		"state":"open",
		"children":rentfin
	},
	{
		"funcid":"4",
		"text":"获房"
	},
	{
		"funcid":"5",
		"text":"运房"
	},
	{
		"funcid":"6",
		"text":"出房"
	}
]

Data read from database

Test performance for asynchronous dbapi2 driver, here is aiomysql

wrk -c400 -d30s -t4 http://localhost/asql.dspy
Running 30s test @ http://localhost/asql.dspy
  4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   189.76ms  137.67ms   1.82s    77.66%
    Req/Sec   401.46     63.01   660.00     72.29%
  47950 requests in 30.07s, 188.03MB read
  Non-2xx or 3xx responses: 13
Requests/sec:   1594.83
Transfer/sec:      6.25MB

asql.dspy(aiomysql)

@runSQLPaging
def sql(db,ns):
        return {

                "sql_string":"select * from product"
        }

return await sql('aiocfae',{"page":1,"rows":10,"sort":"productid"})

and synchronous dbapi2 driver(mysql-connector)

wrk -c400 -d30s -t4 http://localhost/sql.dspy
Running 30s test @ http://localhost/sql.dspy
  4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   221.07ms   80.60ms 687.16ms   72.53%
    Req/Sec   351.23    121.12   686.00     66.67%
  41997 requests in 30.06s, 164.73MB read
Requests/sec:   1397.07
Transfer/sec:      5.48MB

sql.dspy(mysql-connector)

@runSQLPaging
def sql(db,ns):
        return {

                "sql_string":"select * from product"
        }

return await sql('cfae',{"page":1,"rows":10,"sort":"productid"})

tmpl processor

Running 30s test @ http://localhost:8080/estate/saving/index.tmpl
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.53ms   14.34ms 172.99ms   73.75%
    Req/Sec     1.23k   294.38     2.57k    67.73%
  441885 requests in 30.09s, 246.53MB read
Requests/sec:  14685.00
Transfer/sec:      8.19MB

/estate/saving/index.tmpl is:

{
    "__ctmpl__":"layout",
    "data":{
        "regions":{
            "north":{
                "height":"50px",
                "remoteWidgets":[
                    "{{absurl(request,'./searchform.tmpl')}}"
                ]
            },
            "center":{
                "remoteWidgets":[
                    "{{absurl(request,'./estateView.tmpl')}}"
                ]

            }
        }
    },
    "__metadata__":{
        "__jename__":"layout"
    }
}

environment for processors

When coding in processors, viboraserver provide some environment stuff for build apllication, there are modules, functions, classes and variables

modules:

  • time
  • datetime
  • random
  • json

functions:

  • configValue
  • isNone
  • int
  • str
  • float
  • type
  • str2date
  • str2datetime
  • curDatetime
  • uuid
  • runSQL
  • runSQLPaging
  • runSQLIterator
  • runSQLResultFields
  • getTables
  • getTableFields
  • getTablePrimaryKey
  • getTableForignKeys
  • folderInfo
  • abspath
  • request2ns
  • CRUD
  • data2xlsx
  • xlsxdata
  • openfile
  • i18n
  • i18nDict
  • absurl
  • abspath
  • request2ns

variables

  • resource
  • terminalType

classes

  • ArgsConvert