update doc (3bd625b0) · Commits · Yuanle Song / reliable-download

README.md

+85 −0

Original line number	Diff line number	Diff line
		# reliable-download

		Although this project is written in haskell, it is distributed on
		[pypi](https://pypi.org/) so that user can install it more easily. Because
		python is usually bundled in the system. To install rd-api and rd, see their
		[pypi page](https://pypi.org/project/rd-api/). The same doc is in git as well,
		see `./pypi/rd-api/README.rst`.

		Here is the command help:

		```
		$ rd-api --help
		rd-api - reliable download server

		Usage: rd-api [-h\|--host HOST] [-p\|--port PORT] [--redis-host REDIS_HOST]
		[--redis-port REDIS_PORT] [-d\|--web-root DIR] [-w\|--worker INT]
		[-V\|--version]
		rd-api is an HTTP file server that provides static file hosting and reliable
		download api for rd client.

		rd-api serves files under web-root. You can use it like python3 -m http.server

		In addition, if rd command line tool is used to do the download, it will
		download in a reliable way by downloading in 2MiB blocks and verify checksum
		for each block.

		Usage:
		server side:
		$ ls
		bigfile1 bigfile2
		$ rd-api --host 0.0.0.0 --port 8082

		client side:
		$ rd http://server-ip:8082/bigfile1

		Reliable download is implemented this way:

		- user uses rd client to request a resource to download.
		- rd client requests resource block metadata via the /rd/ api. block metadata
		contains block count, block id, block byte offset, block content sha1sum.
		- rd-api calculates and serves block metadata to rd client incrementally.
		block metadata is cached in redis after calculation.
		- rd client fetches block and verifies sha1sum incrementally. When all blocks
		are downloaded and verified, combine blocks to get the final resource.
		- rd client will retry on http errors and sha1sum verification failures.
		- rd client supports continuing a partial download. You can press Ctrl-C to
		stop download anytime, and continue later by running the same command again.

		Available options:
		-h,--host HOST http listen host (default: "0.0.0.0")
		-p,--port PORT http listen port (default: 8082)
		--redis-host REDIS_HOST redis host (default: "127.0.0.1")
		--redis-port REDIS_PORT redis port (default: 6379)
		-d,--web-root DIR web root directory (default: ".")
		-w,--worker INT how many concurrent workers to calculator sha1sum for
		file (default: 2)
		-V,--version show program version and exit
		-h,--help Show this help text

		```

		```
		$ rd --help
		rd - reliable download command line tool

		Usage: rd [-r\|--block-max-retry INT] [-k\|--keep] [-d\|--temp-dir TEMP_DIR]
		[-o\|--output-dir OUTPUT_DIR] [-w\|--worker INT] [-f\|--force]
		[-v\|--verbose] [-V\|--version] [URL...]
		Download large files across slow and unstable network reliably. Requires using
		rd-api on server side. For more information, see rd-api --help

		Available options:
		-r,--block-max-retry INT max retry times for each block (default: 30)
		-k,--keep keep block data when download has finished and
		combined
		-d,--temp-dir TEMP_DIR the dir to keep block download
		data (default: ".blocks")
		-o,--output-dir OUTPUT_DIR
		the dir to keep the final combined
		file (default: ".")
		-w,--worker INT concurrent HTTP download worker (default: 5)
		-f,--force overwrite exiting target file in OUTPUT_DIR
		-v,--verbose show more debug message
		-V,--version show version number and exit
		-h,--help Show this help text
		```

operational

+9 −1

Original line number	Diff line number	Diff line
		* COMMENT -- mode: org --
		#+Date: 2018-05-04
		Time-stamp: <2018-05-11>
		Time-stamp: <2019-02-28>
		#+STARTUP: content
		* notes :entry:
		** 2018-05-09 how to release latest code on PyPI?
		@@ -499,6 +499,14 @@ only first character is in path key.

		* current :entry:
		**
		** 2019-02-28 bug: rd-api -d java/
		option -d: cannot parse value `java/'

		- stack exec rd-api -- -d t1/
		this works fine.

		probably a python wrapper issue.

		** 2018-05-10 use a proper module hierarchy.
		import RD.Utils
		import RD.Api.Lib

package.yaml

+3 −3

Original line number	Diff line number	Diff line
		@@ -2,12 +2,12 @@ name: reliable-download
		version: 1.1.1.0
		synopsis: provide reliable download service via HTTP
		description: reliable-download web application and cli tool
		homepage: "https://github.com/sylecn/reliable-download#readme"
		bug-reports: "https://github.com/sylecn/reliable-download/issues"
		homepage: "https://gitlab.emacsos.com/sylecn/reliable-download"
		bug-reports: "https://gitlab.emacsos.com/sylecn/reliable-download/issues"
		license: GPL-3
		author: Yuanle Song
		maintainer: sylecn@gmail.com
		copyright: "Copyright: (c) 2018 Yuanle Song"
		copyright: "Copyright: (c) 2018, 2019 Yuanle Song"
		category: Utilities
		extra-source-files:
		- README.md

pypi/rd-api/README.rst

+4 −4

Original line number	Diff line number	Diff line
		@@ -3,14 +3,14 @@ rd-api - reliable download server

		.. version
		.. image:: https://img.shields.io/pypi/v/rd-api.svg
		:target: https://pypi.python.org/pypi/rd-api/
		:target: https://pypi.org/project/rd-api/

		.. license
		.. image:: https://img.shields.io/pypi/l/rd-api.svg
		:target: https://pypi.python.org/pypi/rd-api/
		:target: https://pypi.org/project/rd-api/

		.. image:: https://img.shields.io/pypi/wheel/rd-api.svg
		:target: https://pypi.python.org/pypi/rd-api/
		:target: https://pypi.org/project/rd-api/

		rd-api is an HTTP file server that provides static file hosting and reliable
		download api for `rd client`_.
		@@ -31,7 +31,7 @@ To install this package:

		.. code-block:: bash

		$ sudo apt install -y redis-server # redis is used to cached block sha1sum
		$ sudo apt install -y redis-server # redis is used to cache block sha1sum
		$ pip install --user rd-api
		$ rd-api --help
		$ ~/.local/bin/rd-api --help # if ~/.local/bin/ is not in PATH

pypi/rd-client/README.rst

+3 −3

Original line number	Diff line number	Diff line
		@@ -2,13 +2,13 @@ rd - reliable download command line tool
		========================================

		.. image:: https://img.shields.io/pypi/v/rd.svg
		:target: https://pypi.python.org/pypi/rd
		:target: https://pypi.org/project/rd/

		.. image:: https://img.shields.io/pypi/l/rd.svg
		:target: https://pypi.python.org/pypi/rd
		:target: https://pypi.org/project/rd/

		.. image:: https://img.shields.io/pypi/wheel/rd.svg
		:target: https://pypi.python.org/pypi/rd
		:target: https://pypi.org/project/rd/

		Download large files across slow and unstable network reliably. Requires using
		rd-api on server side. For more information on how it works, see rd-api_.