Commit 18d63fee authored by Yuanle Song's avatar Yuanle Song
Browse files

add more doc in README.md; 

- use io-thread-pool git repo URL in stack.yaml extra-deps. no longer require
  local copy of that library.
- update docs
- add source URL in pypi metadata.
parent 7be5eb01

README.md

+60 −5
Original line number Diff line number Diff line 
# reliable-download



Although this project is written in haskell, it is distributed on

[pypi](https://pypi.org/) so that user can install it more easily. Because

python is usually bundled in the system. To install rd-api and rd, see their

[pypi page](https://pypi.org/project/rd-api/). The same doc is in git as well,

see `./pypi/rd-api/README.rst`.

Reliable Download is designed to download large files across unreliable

network via HTTP. It is especially effective when transfer files across

[the GFW](https://en.wikipedia.org/wiki/Great_Firewall). It has a server side

called rd-api, a client side called rd. To run reliable download, run rd-api

on file source node to serve the file, run rd on target node to download the

file. You can think of it as enhanced http static file server. To learn how

reliable download works, see ```rd-api --help``` below.



## Installation



Reliable download is developed and tested on linux. Windows and MacOS is not

supported. Theoretically the code can be ported to support other OS, but I

don't have the time to handle testing and distribution.



Reliable download server side and client side is both written in

Haskell. However, it is distributed on [pypi](https://pypi.org/) so that user

can install it more easily. Because python and pip is usually bundled with

linux system. To install rd-api and rd, see their

[pypi page](https://pypi.org/project/rd-api/). The same doc is kept in git as

well, see `./pypi/rd-api/README.rst`.



- [rd-api pypi page](https://pypi.org/project/rd-api/)

- [rd pypi page](https://pypi.org/project/rd/)



## Design and Command Line Help



Here is the command help:
@@ -89,3 +109,38 @@ Available options: 
  -V,--version             show version number and exit

  -h,--help                Show this help text

```



## Developer Notes, Build the Project by Yourself



see ./operational file for developer notes.



To build the project, install [stack tool](https://www.haskellstack.org/),

then run:



```

stack build --pedantic

```



Binary will be produced in ````stack path --local-install-root`/bin/``` dir.



You may also run built-in tests:



```

stack build --pedantic --test

```



## Difference With Other Similar Tools



BitTorrent can be used to transfer big files across unreliable network

reliably. I like the protocol a lot. But you need to create torrent file in

advance and either use a public tracker or run your own tracker server. It's

too much work to share a simple big file.



curl, wget, aria2 can be used to download file via HTTP. But they do not check

whether downloaded data is valid. This makes "continue downloading a partially

downloaded file" useless in unreliable network.



## License



Reliable download is released under GPLv3+. Source code can be found at

https://gitlab.emacsos.com/sylecn/reliable-download

operational

+34 −21
Original line number Diff line number Diff line
@@ -13,7 +13,9 @@ Time-stamp: <2022-03-16> 
- pypi/         for release on pypi, see pypi/Makefile



** 2018-05-09 how to release latest code on PyPI? how to make a release?

- update version number in package.yaml, src/CliVersion.hs

- update version number in

  - lib/RD/CliVersion.hs (required, used by pypi pkg)

  - package.yaml (optional)

- build binary

  stack build --test --pedantic

  stack exec hlint -- -g
@@ -561,18 +563,6 @@ I should not need to write showProgressLoop explicitly. 
/home/sylecn/persist/cache/ideaIC-2018.1.tar.gz



try use a smaller file within git tree.

** 2022-03-14 build on debian 9. push a new release to pypi.

- binary built on ryzen5 won't work because of high libc version.

  #+BEGIN_SRC sh

    root@de03:~/d# ./rd-api --version

    ./rd-api: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by ./rd-api)

    ./rd-api: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./rd-api)

  #+END_SRC



- problems

  - is readme_renderer still required? does twine require it?

    yes. it's required for "twine check" command.



** 2019-02-28 bug: rd-api -d java/

option -d: cannot parse value `java/'
@@ -581,6 +571,17 @@ option -d: cannot parse value `java/' 


  probably a python wrapper issue.



** 2018-05-09 test the app under unstable network.

I remember there are tools that can simulate packet loss.

policy in ovs can do it.



- 2018-05-11 tcp - Simulate delayed and dropped packets on Linux - Stack Overflow

  https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux



  test this in vbox VM. stretch01

  see stretch01 daylog.



* done                                                                :entry:

** 2018-05-10 use a proper module hierarchy.

import RD.Utils

import RD.Api.Lib
@@ -592,17 +593,29 @@ import RD.Client.Opts 
  hierarchy is important.

- 



** 2018-05-09 test the app under unstable network.

I remember there are tools that can simulate packet loss.

policy in ovs can do it.

** 2022-03-14 build on debian 9. push a new release to pypi.

- binary built on ryzen5 won't work because of high libc version.

  #+BEGIN_SRC sh

    root@de03:~/d# ./rd-api --version

    ./rd-api: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by ./rd-api)

    ./rd-api: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./rd-api)

  #+END_SRC



- 2018-05-11 tcp - Simulate delayed and dropped packets on Linux - Stack Overflow

  https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux

- problems

  - is readme_renderer still required? does twine require it?

    yes. it's required for "twine check" command.



  test this in vbox VM. stretch01

  see stretch01 daylog.

** 2022-03-16 io-thread-pool, github field in package.yaml is wrong.

how to use my own non-github url for source URL?



* done                                                                :entry:

- search: haskell package.yaml github field



  package.yaml is from https://github.com/sol/hpack



  use git field.



** 2022-03-16 add io-thread-pool as a git submodule. so the project can be built by other people.

just use git URL in extra-deps.

** 2018-05-07 loopUntilAllBlocksReady, how to track progress?

use a thread pool to download blocks, print overall progress when some parts

done or some time elapsed.

pypi/rd-api/README.rst

+6 −0
Original line number Diff line number Diff line
@@ -73,6 +73,12 @@ Reliable download is implemented this way: 
Reliable download is written in Haskell, binary is distributed on PyPI for

easy installation on linux system. Reliable download only runs in linux.



License

---------------



Reliable download is released under GPLv3+. Source code can be found at

https://gitlab.emacsos.com/sylecn/reliable-download



ChangeLog

---------

pypi/rd-api/setup.py

+3 −2
Original line number Diff line number Diff line
@@ -35,7 +35,8 @@ setup( 
    long_description=open('README.rst').read(),

    long_description_content_type="text/x-rst",

    license="GPLv3+",

    url="https://pypi.python.org/pypi/rd-api/",

    url="https://gitlab.emacsos.com/sylecn/reliable-download",

    download_url="https://pypi.org/project/rd-api/",

    classifiers=[

        'Development Status :: 5 - Production/Stable',

        'Intended Audience :: System Administrators',
@@ -49,6 +50,6 @@ setup( 
        'Programming Language :: Python :: 3.6',

        'Programming Language :: Python :: 3.7',

        'Programming Language :: Python :: 3.8',

        'Programming Language :: Python :: 3.9'

        'Programming Language :: Python :: 3.9',

    ]

)

pypi/rd-client/README.rst

+6 −0
Original line number Diff line number Diff line
@@ -48,6 +48,12 @@ see rd-api_. 


.. _rd-api: https://pypi.org/project/rd-api/



License

---------------



Reliable download is released under GPLv3+. Source code can be found at

https://gitlab.emacsos.com/sylecn/reliable-download



ChangeLog

---------