Commit 8e3648de authored by Yuanle Song's avatar Yuanle Song
Browse files

v1.2.6.0 fix -d --web-root option 

since I used setCurrentDirectory when serving static site,
combine webroot and user requested path will result in File Not
Found error. should always just use relative path or expand it to abs path.
I choose to expand abs path. This allow skip calculate sha1sum
when the same file is accessed via different webroot dir.
parent 4836bda5

operational

+82 −21
Original line number Diff line number Diff line
@@ -480,19 +480,6 @@ via env var and command line parameter. 


* current                                                             :entry:

** 

** 2024-04-08 should I use absolute file path in redis key?

this can reduce some sha1 calculation if user run rd-api in different root dir.



e.g.



cd /foo/bar/

rd-api



cd /foo/

rd-api -d bar

# or

rd-api -d /foo/bar



** 2024-04-07 when combining blocks to create final file, don't print

"No block fetched in last 10 seconds" log any more if there is no other files

in DL list.
@@ -527,14 +514,6 @@ IO () -> IO () 


I should not need to write showProgressLoop explicitly.



** 2019-02-28 bug: rd-api -d java/

option -d: cannot parse value `java/'



- stack exec rd-api -- -d t1/

  this works fine.



  probably a python wrapper issue.



** 2018-05-09 test the app under unstable network.

I remember there are tools that can simulate packet loss.

policy in ovs can do it.
@@ -546,6 +525,88 @@ policy in ovs can do it. 
  see stretch01 daylog.



* done                                                                :entry:

** 2019-02-28 bug: rd-api -d java/

option -d: cannot parse value `java/'



- stack exec rd-api -- -d t1/

  this works fine.



  probably a python wrapper issue.



- problems

  - 2024-04-08 -d foo/ works fine.

    -d foo, -d ./foo/ also works.

  - but getFileStatus on ./test/中文.txt failed:

    ./test/中文.txt: getFileStatus: does not exist (No such file or directory)



    on s02,

    cd ~/projects/reliable-download

    rd-api -d ./test/



    on agem10,

    rd http://[240e:388:8a05:500:be24:11ff:fe06:be5f]:8082/中文.txt

  - try it on agem10,



    rd http://127.0.0.1:8082/中文.txt

    also fail.



    file name encoding issue?



    import System.FilePath ((</>))



    let filepath = webRoot (rcConfig rc) </> T.unpack reqFilePath

    getFileStatus filepath



    I see no encoding issue.

    try add some log.

    no issue.

    getFileStatus works in ghci.



    stack ghci reliable-download:exe:rd-api

    #+begin_src sh

      ghci> import System.Posix.Files (getFileStatus, fileSize)

      ghci> s1 <- getFileStatus("./test/TestApi.hs")

      ghci> fileSize s1

      7647

      ghci> s2 <- getFileStatus("./test/中文.txt")

      ghci> fileSize s2

      8

      ghci> :q

      Leaving GHCi.

    #+end_src

    so why did it fail?

    maybe it only fails when run via python?

    LANG/locale issue?



    try run via stack or just binary.

    ~/.local/pipx/venvs/rd-api/lib/python3.11/site-packages/rdapi/rd-api

    it also fail. not python wrapper issue.



  - use abs path in -d works.

    it's relative path and CWD issue.

    maybe I changed CWD somewhere. check it.



    git grep setCurrentDirectory



    yes.

    -- static app only support serving from PWD

    setCurrentDirectory (webRoot config)



    so just always use relative path, don't join with webroot dir.



** 2024-04-08 should I use absolute file path in redis key?

this can reduce some sha1 calculation if user run rd-api in different root dir.



e.g.



cd /foo/bar/

rd-api



cd /foo/

rd-api -d bar

# or

rd-api -d /foo/bar



** 2024-04-07 rd-api, rd: switch to fast-logger, use local datetime in logs, not UTC time. :logging:featurereq:

- tinylog is not maintained any more. no longer in latest stackage LTS.

- tinylog types doesn't allow use local time in logs. requires writing lots of code.

package.yaml

+1 −1
Original line number Diff line number Diff line 
name:                reliable-download

version:             1.2.5.0

version:             1.2.6.0

synopsis:            provide reliable download service via HTTP

description:         reliable-download web application and cli tool

homepage:            "https://gitlab.emacsos.com/sylecn/reliable-download"

pypi/rd-api/README.rst

+1 −0
Original line number Diff line number Diff line
@@ -85,6 +85,7 @@ ChangeLog 
* v1.5.0.0 2024-04-08

  - bugfix: properly handle unicode string in URL path

  - bugfix: use local time in log messages instead of UTC time

  - bugfix: fix -d --web-root option. this option was not working before.



* v1.4.0.0 2023-10-18

rd-api/RD/Server/App.hs

+15 −13
Original line number Diff line number Diff line
@@ -13,6 +13,7 @@ import Network.Wai (Application, pathInfo) 
import Web.Scotty

import Data.Aeson (object, (.=))

import System.FilePath ((</>))

import System.Directory (makeAbsolute)

import System.Posix.Files (getFileStatus, fileSize)

import Control.Error

import qualified Database.Redis as R
@@ -79,14 +80,15 @@ getRdHandler rc = do 
  unless (rcHasRedis rc) $

      throwE "No redis on server side, rd client support is disabled"

  req <- lift request

  let reqFilePath = T.intercalate "/" $ drop 1 $ pathInfo req

  let filepath = webRoot (rcConfig rc) </> T.unpack reqFilePath

  let relFilePath = T.intercalate "/" $ drop 1 $ pathInfo req

      filepath = webRoot (rcConfig rc) </> T.unpack relFilePath

  absFilePath <- liftIO $ makeAbsolute $ T.unpack relFilePath

  fileStatusE <- lift $ do

    liftIO $ infol rc $ "user request rd metadata for " <> T.pack filepath

    liftIO $ infol rc $ "user request rd metadata for " <> relFilePath

    liftIO $ catchIOError

      (fmap Right (getFileStatus filepath))

      (fmap Right (getFileStatus $ T.unpack relFilePath))

      (\e -> do

         let msg = "getFileStatus on " <> T.pack filepath <> " failed"

         let msg = "getFileStatus on " <> relFilePath <> " failed"

         errorl rc $ msg <> ":\n\t" <> T.pack (show e)

         return $ Left msg)

  throwOnLeft fileStatusE
@@ -97,7 +99,7 @@ getRdHandler rc = do 
        blockSizeInByte = 2097152    -- 2MiB

        blockCount = (fileSizeInByte - 1) `div` blockSizeInByte + 1

        blocks = genBlocks fileSizeInByte blockSizeInByte

        fbp = FillBlockParam { fbpFilepath=filepath

        fbp = FillBlockParam { fbpFilepath=absFilePath

                             , fbpFileSize=fileSizeInByte

                             , fbpBlockSize=blockSizeInByte

                             , fbpBlocks=blocks }
@@ -105,14 +107,14 @@ getRdHandler rc = do 
    case resultE of

      Left msg -> json $

          object ["ok" .= False

                 ,"path" .= reqFilePath

                 ,"path" .= relFilePath

                 ,"filepath" .= filepath

                 ,"msg" .= msg]

      Right _ -> do

          blocksWithSha1sum <- liftIO $ fillSha1sum rc fbp

          json RDResponse { respOk=True

                          , respMsg=""

                          , respPath=reqFilePath

                          , respPath=relFilePath

                          , respFilePath=filepath

                          , respBlockSize="2MiB"

                          , respFileSize=fileSizeInByte
@@ -136,12 +138,12 @@ mkApp rc = do 
  get (regex "^/test-rd/") $ do    -- for testing path capture

    req <- request

    let fullPath = "/" <> T.intercalate "/" (pathInfo req)

        reqFilePath = T.intercalate "/" $ drop 1 $ pathInfo req

    let filepath = webRoot (rcConfig rc) </> T.unpack reqFilePath

        relFilePath = T.intercalate "/" $ drop 1 $ pathInfo req

    let filepath = webRoot (rcConfig rc) </> T.unpack relFilePath

    json $ object ["ok" .= True

                  ,"path" .= reqFilePath

                  ,"filepath" .= filepath

                  ,"fullPath" .= fullPath]

                  ,"path" .= relFilePath -- file path in URL

                  ,"filepath" .= filepath -- file path on server side

                  ,"fullPath" .= fullPath] -- HTTP request path



  get "/debug/t1" $ do

    sha1 <- liftIO $ sha1sum "/home/sylecn/persist/cache/ideaIC-2018.1.tar.gz"