项目地址：https://github.com/ray-project/ray
1、简介
Ray为构建分布式应用程序提供了一个简单、通用的API。Ray是一种分布式执行框架,便于大规模应用程序和利用先进的机器学习库。
Ray通过以下方式完成这项任务：
为构建和运行分布式应用程序提供简单的原语。
使最终用户能够并行化单个机器代码，而代码更改很少到零。
在核心Ray之上包含大量应用程序、库和工具，以支持复杂的应用程序。
2、安装
安装方式比较简单： pip install ray==1.4.1
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
[root@node2 ~]# pip install 'ray[default]'Looking in indexes: https://mirrors.aliyun.com/pypi/simple/Collecting ray[default] Downloading https://mirrors.aliyun.com/pypi/packages/13/ec/f727ddd3fbcdc6102eace62c9d5dd9d9ad8112d40eeb7de8783676aca24d/ray-1.4.1-cp36-cp36m-manylinux2014_x86_64.whl (51.6 MB) |████████████████████████████████| 51.6 MB 269 kB/s Collecting aiohttp Downloading https://mirrors.aliyun.com/pypi/packages/31/28/7b49246d7825c61e1a14306ab050e8eec48fcf5b009b19c8f6ea4e312467/aiohttp-3.7.4.post0-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB) |████████████████████████████████| 1.3 MB 121.5 MB/s Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/site-packages (from ray[default]) (0.8)Collecting protobuf>=3.15.3 Downloading https://mirrors.aliyun.com/pypi/packages/53/4e/e2db88d0bb0bda6a879eea62fddbaf813719ce3770d458bc5580512d9c95/protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) |████████████████████████████████| 1.0 MB 136.1 MB/s Collecting jsonschema Downloading https://mirrors.aliyun.com/pypi/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) |████████████████████████████████| 56 kB 7.1 MB/s Collecting aioredis Downloading https://mirrors.aliyun.com/pypi/packages/b0/64/1b1612d0a104f21f80eb4c6e1b6075f2e6aba8e228f46f229cfd3fdac859/aioredis-1.3.1-py3-none-any.whl (65 kB) |████████████████████████████████| 65 kB 6.3 MB/s Collecting pydantic>=1.8 Downloading https://mirrors.aliyun.com/pypi/packages/2b/7c/7d0b3f2d7959b7193018896db236ded165f9bca1bb75f46f4c32fa6f4f9d/pydantic-1.8.2-cp36-cp36m-manylinux2014_x86_64.whl (10.2 MB) |████████████████████████████████| 10.2 MB 420 kB/s Collecting pyyaml Downloading https://mirrors.aliyun.com/pypi/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB) |████████████████████████████████| 640 kB 19.1 MB/s Collecting numpy>=1.16 Downloading https://mirrors.aliyun.com/pypi/packages/14/32/d3fa649ad7ec0b82737b92fefd3c4dd376b0bb23730715124569f38f3a08/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB) |████████████████████████████████| 14.8 MB 296 kB/s Collecting gpustat Downloading https://mirrors.aliyun.com/pypi/packages/b4/69/d8c849715171aeabd61af7da080fdc60948b5a396d2422f1f4672e43d008/gpustat-0.6.0.tar.gz (78 kB) |████████████████████████████████| 78 kB 7.3 MB/s Collecting redis>=3.5.0 Downloading https://mirrors.aliyun.com/pypi/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl (72 kB) |████████████████████████████████| 72 kB 968 kB/s Collecting aiohttp-cors Downloading https://mirrors.aliyun.com/pypi/packages/13/e7/e436a0c0eb5127d8b491a9b83ecd2391c6ff7dcd5548dfaec2080a2340fd/aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)Collecting py-spy>=0.2.0 Downloading https://mirrors.aliyun.com/pypi/packages/9d/4d/1a9cbe9a0b543e6733cb38afe26451522a9ef8e4897b59e74cc76838f245/py_spy-0.3.7-py2.py3-none-manylinux1_x86_64.whl (3.1 MB) |████████████████████████████████| 3.1 MB 131 kB/s Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages (from ray[default]) (2.24.0)Collecting colorama Downloading https://mirrors.aliyun.com/pypi/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl (16 kB)Collecting filelock Downloading https://mirrors.aliyun.com/pypi/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl (7.6 kB)Collecting prometheus-client>=0.7.1 Downloading https://mirrors.aliyun.com/pypi/packages/09/da/4e8471ff825769581593b5b84769d32f58e5373b59fccaf355d3529ad530/prometheus_client-0.11.0-py2.py3-none-any.whl (56 kB) |████████████████████████████████| 56 kB 4.2 MB/s Collecting msgpack<2.0.0,>=1.0.0 Downloading https://mirrors.aliyun.com/pypi/packages/0c/0d/b1d9d32d03ce38ba5e2a37fbae850afd4530a14cc441e8335f1865a03705/msgpack-1.0.2-cp36-cp36m-manylinux1_x86_64.whl (272 kB) |████████████████████████████████| 272 kB 127.6 MB/s Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.6/site-packages (from ray[default]) (7.1.2)Collecting opencensus Downloading https://mirrors.aliyun.com/pypi/packages/18/59/12044123133d000f705383ad98579aeb0dd82d66b33a254a21b54bf0d6bb/opencensus-0.7.13-py2.py3-none-any.whl (127 kB) |████████████████████████████████| 127 kB 128.9 MB/s Collecting grpcio>=1.28.1 Downloading https://mirrors.aliyun.com/pypi/packages/eb/32/332a2bb0cee0d4040331952c5ef3d040d119a15b3c622b60bdcbe768bded/grpcio-1.39.0-cp36-cp36m-manylinux2014_x86_64.whl (4.3 MB) |████████████████████████████████| 4.3 MB 417 kB/s Collecting colorful Downloading https://mirrors.aliyun.com/pypi/packages/b0/8e/e386e248266952d24d73ed734c2f5513f34d9557032618c8910e605dfaf6/colorful-0.5.4-py2.py3-none-any.whl (201 kB) |████████████████████████████████| 201 kB 113.9 MB/s Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/site-packages (from grpcio>=1.28.1->ray[default]) (1.15.0)Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.6/site-packages (from pydantic>=1.8->ray[default]) (3.10.0.0)Collecting multidict<7.0,>=4.5 Downloading https://mirrors.aliyun.com/pypi/packages/a1/35/b22524d6b9cacfb4c5eff413a069bbc17c6ea628e54da5c6c989998ced5f/multidict-5.1.0-cp36-cp36m-manylinux2014_x86_64.whl (141 kB) |████████████████████████████████| 141 kB 120.6 MB/s Collecting idna-ssl>=1.0 Downloading https://mirrors.aliyun.com/pypi/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz (3.4 kB)Collecting attrs>=17.3.0 Downloading https://mirrors.aliyun.com/pypi/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl (53 kB) |████████████████████████████████| 53 kB 3.9 MB/s Requirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.6/site-packages (from aiohttp->ray[default]) (3.0.4)Collecting yarl<2.0,>=1.0 Downloading https://mirrors.aliyun.com/pypi/packages/da/08/52b26b44bce7b818b410aee37c5e424c9ea420c557bca97dc2adac29b151/yarl-1.6.3-cp36-cp36m-manylinux2014_x86_64.whl (293 kB) |████████████████████████████████| 293 kB 130.8 MB/s Collecting async-timeout<4.0,>=3.0 Downloading https://mirrors.aliyun.com/pypi/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl (8.2 kB)Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/site-packages (from idna-ssl>=1.0->aiohttp->ray[default]) (2.10)Collecting hiredis Downloading https://mirrors.aliyun.com/pypi/packages/82/72/313fb6d30b7e413c8662447f9a0abe9105494ceb9266ecb08d442c14afbb/hiredis-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (84 kB) |████████████████████████████████| 84 kB 4.8 MB/s Collecting nvidia-ml-py3>=7.352.0 Downloading https://mirrors.aliyun.com/pypi/packages/6d/64/cce82bddb80c0b0f5c703bbdafa94bfb69a1c5ad7a79cff00b482468f0d3/nvidia-ml-py3-7.352.0.tar.gz (19 kB)Collecting psutil Downloading https://mirrors.aliyun.com/pypi/packages/da/82/56cd16a4c5f53e3e5dd7b2c30d5c803e124f218ebb644ca9c30bc907eadd/psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl (291 kB) |████████████████████████████████| 291 kB 131.6 MB/s Collecting blessings>=1.6 Downloading https://mirrors.aliyun.com/pypi/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl (18 kB)Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from jsonschema->ray[default]) (39.2.0)Collecting pyrsistent>=0.14.0 Downloading https://mirrors.aliyun.com/pypi/packages/6c/19/1af501f6f388a40ede6d0185ba481bdb18ffc99deab0dd0d092b173bc0f4/pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB) |████████████████████████████████| 117 kB 138.2 MB/s Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/site-packages (from jsonschema->ray[default]) (4.3.0)Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata->jsonschema->ray[default]) (3.4.1)Collecting opencensus-context==0.1.2 Downloading https://mirrors.aliyun.com/pypi/packages/f1/33/990f1bd9e7ee770fc8d3c154fc24743a96f16a0e49e14e1b7540cc2fdd93/opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)Collecting google-api-core<2.0.0,>=1.0.0 Downloading https://mirrors.aliyun.com/pypi/packages/aa/51/629a31d4e7db2bbc0b0b789ac4034108c68647bea9cf5b54c8366df47b8e/google_api_core-1.31.0-py2.py3-none-any.whl (93 kB) |████████████████████████████████| 93 kB 1.6 MB/s Collecting contextvars Downloading https://mirrors.aliyun.com/pypi/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz (9.6 kB)Collecting google-auth<2.0dev,>=1.25.0 Downloading https://mirrors.aliyun.com/pypi/packages/d7/13/35b1e0a63e160ecad04985e60339382e48e6020d22f8328a2ab47226c910/google_auth-1.33.1-py2.py3-none-any.whl (152 kB) |████████████████████████████████| 152 kB 127.7 MB/s Collecting packaging>=14.3 Downloading https://mirrors.aliyun.com/pypi/packages/3c/77/e2362b676dc5008d81be423070dd9577fa03be5da2ba1105811900fda546/packaging-21.0-py3-none-any.whl (40 kB) |████████████████████████████████| 40 kB 6.1 MB/s Collecting googleapis-common-protos<2.0dev,>=1.6.0 Downloading https://mirrors.aliyun.com/pypi/packages/55/08/796a6bc0b550e2b7116041c953d3d5100016abea106131d71e5651826e7b/googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB) |████████████████████████████████| 198 kB 136.3 MB/s Collecting setuptools Downloading https://mirrors.aliyun.com/pypi/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl (819 kB) |████████████████████████████████| 819 kB 127.0 MB/s Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from google-api-core<2.0.0,>=1.0.0->opencensus->ray[default]) (2020.1)Collecting cachetools<5.0,>=2.0.0 Downloading https://mirrors.aliyun.com/pypi/packages/bf/28/c4f5796c67ad06bb91d98d543a5e01805c1ff065e08871f78e52d2a331ad/cachetools-4.2.2-py3-none-any.whl (11 kB)Collecting pyasn1-modules>=0.2.1 Downloading https://mirrors.aliyun.com/pypi/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) |████████████████████████████████| 155 kB 21.5 MB/s Collecting rsa<5,>=3.1.4 Downloading https://mirrors.aliyun.com/pypi/packages/e9/93/0c0f002031f18b53af7a6166103c02b9c0667be528944137cc954ec921b3/rsa-4.7.2-py3-none-any.whl (34 kB)Collecting pyparsing>=2.0.2 Downloading https://mirrors.aliyun.com/pypi/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) |████████████████████████████████| 67 kB 7.2 MB/s Collecting pyasn1<0.5.0,>=0.4.6 Downloading https://mirrors.aliyun.com/pypi/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) |████████████████████████████████| 77 kB 7.0 MB/s Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (2020.6.20)Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (1.25.10)Collecting immutables>=0.9 Downloading https://mirrors.aliyun.com/pypi/packages/4a/52/e64a14a99c509cbdfe0405e9f076aef0331cb9548a3efa1d5bacd524978a/immutables-0.15-cp36-cp36m-manylinux1_x86_64.whl (100 kB) |████████████████████████████████| 100 kB 18.2 MB/s Using legacy 'setup.py install' for idna-ssl, since package 'wheel' is not installed.Using legacy 'setup.py install' for gpustat, since package 'wheel' is not installed.Using legacy 'setup.py install' for nvidia-ml-py3, since package 'wheel' is not installed.Using legacy 'setup.py install' for contextvars, since package 'wheel' is not installed.Installing collected packages: pyasn1, setuptools, rsa, pyparsing, pyasn1-modules, protobuf, multidict, immutables, cachetools, yarl, packaging, idna-ssl, googleapis-common-protos, google-auth, contextvars, attrs, async-timeout, pyrsistent, psutil, opencensus-context, nvidia-ml-py3, hiredis, google-api-core, blessings, aiohttp, redis, pyyaml, pydantic, py-spy, prometheus-client, opencensus, numpy, msgpack, jsonschema, grpcio, gpustat, filelock, colorama, aioredis, aiohttp-cors, ray, colorful Attempting uninstall: setuptools Found existing installation: setuptools 39.2.0 Uninstalling setuptools-39.2.0: Successfully uninstalled setuptools-39.2.0 Running setup.py install for idna-ssl ... done Running setup.py install for contextvars ... done Running setup.py install for nvidia-ml-py3 ... done Running setup.py install for gpustat ... doneSuccessfully installed aiohttp-3.7.4.post0 aiohttp-cors-0.7.0 aioredis-1.3.1 async-timeout-3.0.1 attrs-21.2.0 blessings-1.7 cachetools-4.2.2 colorama-0.4.4 colorful-0.5.4 contextvars-2.4 filelock-3.0.12 google-api-core-1.31.0 google-auth-1.33.1 googleapis-common-protos-1.53.0 gpustat-0.6.0 grpcio-1.39.0 hiredis-2.0.0 idna-ssl-1.1.0 immutables-0.15 jsonschema-3.2.0 msgpack-1.0.2 multidict-5.1.0 numpy-1.19.5 nvidia-ml-py3-7.352.0 opencensus-0.7.13 opencensus-context-0.1.2 packaging-21.0 prometheus-client-0.11.0 protobuf-3.17.3 psutil-5.8.0 py-spy-0.3.7 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydantic-1.8.2 pyparsing-2.4.7 pyrsistent-0.18.0 pyyaml-5.4.1 ray-1.4.1 redis-3.5.3 rsa-4.7.2 setuptools-57.4.0 yarl-1.6.3 |
3、单机
下面是不使用分布式的代码示例：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# test_ray.pyimport rayimport cv2import numpy import time #@ray.remotedef test(img): orb = cv2.AKAZE_create() kb = orb.detect(img, None) kp, des = orb.compute(img, kp) return des # 不使用注解的方式test_remote = ray.remote(test) # 函数test 加注解时使用方式# test_remote = test.remote(test) img = cv2.imread("/tmp/test.jpg") start_time = time.time() for i in range(10): test(img) print(time.time() - start_time) |
运行：python test_ray.py
4、集群
如果需要启动集群模式，则需要先启动服务，选择一台机器作为主服务器，然后按虾米那命令启动：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
[root@node1 ~]# ray start --head --port=8888 Local node IP: 192.168.0.812021-07-28 09:58:44,051 INFO services.py:1274 -- View the Ray dashboard at http://127.0.0.1:8265 --------------------Ray runtime started.-------------------- Next steps To connect to this Ray runtime from another node, run ray start --address='192.168.0.81:8888' --redis-password='5241590000000000' Alternatively, use the following Python code: import ray ray.init(address='auto', _redis_password='5241590000000000') If connection fails, check your firewall settings and network configuration. To terminate the Ray runtime, run ray stop |
可以看到里面提示具体的用法。此时通过ray的web界面，端口8265访问，看到已有一台机器。
然后在另外两台机启动客户端节点，保证各个节点的ray版本一致,查看版本：
# pip freeze |grep ray
版本不一致的话，要升级为一致：pip install -U ray， 此处选择的版本为1.4.1
从节点启动方式(密码可以在主节点启动时显示)：
|
1
|
ray start --address='192.168.0.81:8888' --redis-password='5241590000000000' |
改写程序，用于分布式：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# test_ray.pyimport rayimport cv2import numpy import time @ray.remotedef test(img): orb = cv2.AKAZE_create() kb = orb.detect(img, None) kp, des = orb.compute(img, kp) return des # 不使用注解的方式#test_remote = ray.remote(test) # 函数test 加注解时使用方式test_remote = test.remote(test) if __name__ == "__main__": ray.init(address='auto', _redis_password='5241590000000000') start_time = time.time() img = cv2.imread("/tmp/test.jpg") futures = [test_remote.remote(img) for i in range(100)] ray.get(futures) start_time = time.time() print(time.time() - start_time) |
在主节点运行程序，程序会在3台机器上启动。停止ray服务的方式：ray stop.
到此这篇关于支持python的分布式计算框架Ray的文章就介绍到这了,更多相关python的分布式计算框架Ray内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://blog.csdn.net/Ajinnv/article/details/119153287
