v1.2.0.0 rd add --rolling-combine option.

$ rd --help

rd - reliable download client

Usage: rd [-r|--block-max-retry INT] [-k|--keep] [-d|--temp-dir TEMP_DIR]

          [-o|--output-dir OUTPUT_DIR] [-w|--worker INT] [-f|--force]

          [-v|--verbose] [-V|--version] [URL...]

Usage: rd [-r|--block-max-retry INT] [-k|--keep] [-l|--rolling-combine] 

          [-d|--temp-dir TEMP_DIR] [-o|--output-dir OUTPUT_DIR] 

          [-w|--worker INT] [-f|--force] [-v|--verbose] [-V|--version] [URL...]

  Download large files across slow and unstable network reliably. Requires using

  rd-api on server side. For more information, see rd-api --help

  -r,--block-max-retry INT max retry times for each block (default: 30)

  -k,--keep                keep block data when download has finished and

                           combined

  -l,--rolling-combine     delete each block data right after combine, conflict

                           with --keep

  -d,--temp-dir TEMP_DIR   the dir to keep block download

                           data (default: ".blocks")

  -o,--output-dir OUTPUT_DIR

module CliVersion where

cliVersion :: String

cliVersion = "1.1.3.0"

cliVersion = "1.2.0.0"

      content <- LB.readFile blockFilename

      LB.appendFile targetFilename content  -- TODO how to handle error here?

                                            -- let it crash?

      when (rollingCombine (rdOptions rc)) $ do

        debugl rc $ "delete block file " <> showt blockFilename

        catchIOError (removeFile blockFilename)

                     (\e -> do

                        errorl rc $ "Remove block file " <>

                               T.pack blockFilename <> " failed: " <> showt e)

    infol rc $ "File downloaded to " <> showt targetFilename

    unless (keepBlockData opts) $ do

      let tempdir = tempDir opts </> filename

    if null $ urls opts then do

      putStrLn "No URLs given, nothing to do. See rd --help"

      exitFailure

    else

        if keepBlockData opts && rollingCombine opts then do

            putStrLn "Error: option --keep and --rolling-combine can not be used at the same time.\nSee rd --help"

            exitFailure

        else

            cliApp opts

  where

data RDOptions = RDOptions

  { blockMaxRetry :: Int

  , keepBlockData :: Bool

  , rollingCombine :: Bool

  , tempDir :: FilePath

  , outputDir :: FilePath

  , workerCount :: Int

      <> short 'k'

      <> help "keep block data when download has finished and combined"

      <> showDefault )

  <*> switch

      (  long "rolling-combine"

      <> short 'l'

      <> help "delete each block data right after combine, conflict with --keep"

      <> showDefault )

  <*> strOption

      (  long "temp-dir"

      <> short 'd'

** 2018-05-06 optparse-applicative :: Stackage Server

https://www.stackage.org/lts-10.3/package/optparse-applicative-0.14.0.0

* later                                                               :entry:

** 2022-03-14 can I make tinylog log timestamp use localtime?

- L.DateFormat

  doesn't allow IO.

  I can't use formatUnixTime from

  import Data.UnixTime (formatUnixTime)

  in logSettings.

  formatUnixTime needs to read TZ from env variable.

  well, I can read that before init the logger.

  formatUnixTime :: Format -> UnixTime -> IO ByteStringSource#

  Formatting UnixTime to ByteString in local time. This is a wrapper for

  strftime_l().

- how to create the DateFormat myself?

  formatUnixTime use C FFI.

  foreign import ccall unsafe "c_format_unix_time"

        c_format_unix_time :: CString -> CTime -> CString -> CInt -> IO CSize

- I can't do it myself. based on the code of tinylog.

  I can probably make it work with fast-logger.

  until next time.

** 2022-03-12 drop redis-server as rd-api dependency.

- use a built-in key-value db. such as Berkeley db, sqlite3, or leveldb.

  use a well known path for the db name.

* current                                                             :entry:

** 

** 2022-03-14 can I make tinylog log timestamp use localtime?

** 2019-02-28 bug: rd-api -d java/

option -d: cannot parse value `java/'

- stack exec rd-api -- -d t1/

  this works fine.

  probably a python wrapper issue.

** 2018-05-10 use a proper module hierarchy.

import RD.Utils

import RD.Api.Lib

import RD.Api.Config

import RD.Client.Lib

import RD.Client.Opts

- stack repl doesn't like duplicated Lib module. also for libs, correct mdoule

  hierarchy is important.

- 

** 2018-05-09 test the app under unstable network.

I remember there are tools that can simulate packet loss.

policy in ovs can do it.

- 2018-05-11 tcp - Simulate delayed and dropped packets on Linux - Stack Overflow

  https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux

  test this in vbox VM. stretch01

  see stretch01 daylog.

* done                                                                :entry:

** 2021-10-07 When combine blocks to final file, do not use more disk space than original file size.

   - Note taken on [2022-03-14 Mon 21:44] \\

     in 1.2.0.0 when -l is given, it will do rolling combine. won't take 2x

     file size disk space.

One way is, open target file in write mode, append block 0, remove block 0.

Append block 1, remove block 1.

...

Until last block.

But this will fail if there is a disk failure.

Maybe only do this when local disk is low on space.

Is there an API that atomically combine two files? Reuse existing disk sectors

of the two files.

- What's the current combine logic?

  client/Main.hs

  content <- LB.readFile blockFilename

  LB.appendFile targetFilename content

  which calls

  System.IO.withBinaryFile

  it already wrap code with bracket. it will not leak fd.

  // I can delete block file after append. but if combine fail, redownload

  will need to DL deleted blocks. I don't track the combine steps when

  re-download.

  just do not combine, if there is not enough disk space when combine starts.

  give an error and exit. unless --rolling-combine is given.

  --keep

  --rolling-combine

  conflict with each other.

  how to represent this in Options.Applicative? not sure.

  Parser choice?

  search: Options.Applicative choice

  I check the options myself after parsing.

  try it:

  stack exec rd -- -k -l URL

  stack exec rd -- URL

- add exception handling for all disk IO actions.

  DL file.

  combine file.

  delete file.

** 2022-03-14 release v1.1.3.0

- v1.1.3.0 changes

  - revised logging messages

  - rd-api support --verbose option. debug msg is not shown by default.

  - code ported to ghc 8.10.7

- v1.1.4.0 changes

  - TODO IO error handling when combine blocks to big file

- problems

  - DONE CliVersion module can be moved away from library.

    it is only used in rd client and rd-api executable.

    try run binary built on debian 11 in debian 10 VM.

** 2021-10-07 When combine blocks to final file, do not use more disk space than original file size.

One way is, open target file in write mode, append block 0, remove block 0.

Append block 1, remove block 1.

...

Until last block.

But this will fail if there is a disk failure.

Maybe only do this when local disk is low on space.

Is there an API that atomically combine two files? Reuse existing disk sectors

of the two files.

- What's the current combine logic?

** 2019-02-28 bug: rd-api -d java/

option -d: cannot parse value `java/'

- stack exec rd-api -- -d t1/

  this works fine.

  probably a python wrapper issue.

** 2018-05-10 use a proper module hierarchy.

import RD.Utils

import RD.Api.Lib

import RD.Api.Config

import RD.Client.Lib

import RD.Client.Opts

- stack repl doesn't like duplicated Lib module. also for libs, correct mdoule

  hierarchy is important.

- 

** 2018-05-09 test the app under unstable network.

I remember there are tools that can simulate packet loss.

policy in ovs can do it.

- 2018-05-11 tcp - Simulate delayed and dropped packets on Linux - Stack Overflow

  https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux

  test this in vbox VM. stretch01

  see stretch01 daylog.

* done                                                                :entry:

** 2022-03-12 improve logging on rd-api and rd client.

- For big files, when there is a lot of blocks, rd-api default log is too

  much.