v1.3.0.0 rd-api: when a small file's content is changed on disk,

    show FileStatusDone = "done"

    show FileStatusUnknown = "unknown"

-- | convert FileStatus to ByteString

fsBytes :: FileStatus -> B.ByteString

fsBytes = Char8.pack . show

* COMMENT -*- mode: org -*-

#+Date: 2018-05-04

Time-stamp: <2024-04-25>

Time-stamp: <2024-04-26>

#+STARTUP: content

* notes                                                               :entry:

** 2022-03-14 project dir structure

  set <filepath>_blockSize_status working|done

  set <filepath>_sha1 SHA1

- worker pool is there for calculating all blocks for one file.

         fileQueue

* current                                                             :entry:

** 

** 2024-04-07 rd-api: when a file content is changed on disk, auto invalidate all

cached blocks.

** 2024-04-26 log refine.

- 2024-04-26T19:06:17  I  fileWorker done for /home/sylecn/d/t2/foo, 0.0 MiB, 1 blocks

  don't use 0.0MiB, just say <1MiB or so.

- 2024-04-26T19:12:02  I  fileWorker is waiting for jobs...

  2024-04-26T19:12:02  I  fileWorker is waiting for jobs...

- when user request rd metadata, and file's mtime changed, do a sha1sum on

  first 4M and last 4M of the file. if any of these sha1sum changes,

  invalidate cache in redis.

  fileWorker-<N> is waiting for jobs

  when calculate metadata, cache file's mtime, first 4M and last 4M of the

  file's content's sha1sum.

- this will allow DL the correct file when file content for the same file name

  is changed.

- give some log in console when file content changes.

  2024-04-26T19:12:02  I  fileWorker working on /home/sylecn/d/t2/foo

  2024-04-26T19:12:02  I  fileWorker done for /home/sylecn/d/t2/foo, 0.0 MiB, 1 blocks

  fileWorker-<N> working on ...

- 

** 2024-03-12 rd-api, if file is already transferred block by block, I can support

  see stretch01 daylog.

* done                                                                :entry:

** 2024-04-07 rd-api: when a file content is changed on disk, auto invalidate all

cached blocks.

- when user request rd metadata, and file's mtime changed, do a sha1sum on

  first 4M and last 4M of the file. if any of these sha1sum changes,

  invalidate cache in redis.

  when calculate metadata, cache file's mtime, first 4M and last 4M of the

  file's content's sha1sum.

  2024-04-25 is it easy to get last 4M? yes.

  in C, should be seek to length-4M byte and read 4M.

  from correctness point of view, this is not enough. user could modify the

  center part of the file. How about just always recalculate if mtime changed?

  how about this? just do it for small files, use full content hash as

  fingerprint. For large files, user is supposed to rename it or clear file

  metadata cache. If so, user can always rename the file before run rd again.

  just give a warning will be enough.

  design:

  auto calculate and check fingerprint for small files (<200M), skip

  fingerprint check for large files.

- this will allow DL the correct file when file content for the same file name

  is changed.

- give some log in console when file content changes.

*** auto calculate and check fingerprint for small files (<200M), 

skip fingerprint check for large files.

- implement this now.

  this change should be backward compatible to older rd clients.

  it only affect server side metadata caching.

- code in getRdHandler

  resultE <- liftIO $ runExceptT $ processNewFileAsyncMaybe rc fbp

  processNewFileAsyncMaybe :: RDRuntimeConfig -> FillBlockParam -> ExceptT T.Text IO ()

  resultE <- liftIO $ DB.insertIfNotExist rc strKey $ fsBytes FileStatusWorking

  throwOnLeft resultE

  here, if files' sha1 has changed, use DB.set instead of DB.insertIfNotExist.

- add proper logging.

- create a unit test for this?

  it's all IO actions, maybe I will test it myself.

  on agem10,

  stack exec -- rd-api -d ~/d/t2/

  create ~/d/t2/foo with some content

  stack exec -- rd http://127.0.0.1:8082/foo

- problems

  - if I send job to worker. will old redis key be rewritten?

    will it ever return expired cache data?

    it will call this:

    blocksWithSha1sum <- liftIO $ fillSha1sum rc fbp

    I need to delete the old redis key.

  - test the new code.

    when foo content changed.

    it works.

** 2024-04-25 haskell: when only executable code changes, should I in increase version in package.yaml

I think I will defer version change to avoid re-compilation of the base library.

name:                reliable-download

# do not modify version if lib/ code is not changed.

# real app version: 1.2.7.0

# real app version: 1.3.0.0

version:             1.2.6.0

synopsis:            provide reliable download service via HTTP

description:         reliable-download web application and cli tool

ChangeLog

---------

* dev

  - feature: auto detect file content changes for small files. auto invalidate metadata cache if file has changed.

* v1.5.0.0 2024-04-08

  - bugfix: properly handle unicode string in URL path

import System.IO.Error (catchIOError)

import Control.Monad (unless)

import qualified Data.Text as T

import qualified Data.ByteString as B

import qualified Data.ByteString.Lazy as LB

import Network.Wai (Application, pathInfo)

import Web.Scotty

import RD.CliVersion (cliVersion)

import RD.Types

import RD.Server.Config

import RD.Lib (sha1sum, genBlocks)

import RD.Lib (sha1sum, sha1sumOnBytes, genBlocks)

import RD.Utils

import qualified RD.Server.DB as DB

        fillBlock :: Block -> BlockWithChecksum

        fillBlock (blockId, start, end) = (blockId, start, end, decodeUtf8 $ M.lookupDefault "pending" (blockIdKey blockId) blockIdSha1sumMap)

-- | a sha1sum that has different types, used for storing sha1sum in redis.

-- only used in isNewFile.

sha1sumByteString :: FilePath -> IO B.ByteString

sha1sumByteString filename = do

  bytes <- LB.readFile filename

  return $ B.toStrict $ sha1sumOnBytes bytes

-- | try set file's working status to FileStatusWorking if it has not been set before.

trySetFileToWorkingStatus :: RDRuntimeConfig -> B.ByteString -> IO (Either T.Text Bool)

trySetFileToWorkingStatus rc strKey = do

  DB.insertIfNotExist rc strKey $ fsBytes FileStatusWorking

-- | return True if given file and block size is a new combination. if file

-- content has changed, it is always considered a new file, old cache will be

-- purged. when this function return Right True, it will set working flag in

-- redis for the file name and block size combination.

isNewFile :: RDRuntimeConfig -> FillBlockParam -> IO (Either T.Text Bool)

isNewFile rc fbp = do

  let strKey = fileStatusKey fbp

      fn = T.pack $ fbpFilepath fbp

  -- units -o %15f --terse 100MiB byte

  if fbpFileSize fbp < 104857600 then do

    cachedSha1E <- DB.get rc (fileSha1Key fbp)

    case cachedSha1E of

      Left msg -> return $ Left msg

      Right sha1Maybe -> do

        currentSha1 <- sha1sumByteString $ fbpFilepath fbp

        case sha1Maybe of

          Just cachedSha1 -> do

            if currentSha1 /= cachedSha1

              then do

                -- set progress to working

                warnl rc $ "file sha1 changed, will recalculate block hashes: " <> fn

                let hashKey = blockSha1sumHashKey fbp

                redisReply <- R.runRedis (rcRedisConn rc) $ R.del [hashKey]

                case redisReply of

                  Left reply -> do

                    errorl rc $ "invalidate blocks old cache key failed:\n\t" <> showt reply

                    return $ Left "invalidate blocks old cache key failed"

                  Right _ -> do

                    infol rc $ "invalidated blocks cache for " <> fn

                    _ <- DB.set rc (fileSha1Key fbp) currentSha1

                    resultE <- DB.set rc strKey $ fsBytes FileStatusWorking

                    case resultE of

                      Left _ -> return $ Left "set file status to working failed"

                      Right _ -> return $ Right True

              else do

                debugl rc $ "file sha1 not changed: " <> fn

                trySetFileToWorkingStatus rc strKey

          Nothing -> do

            resultE <- DB.set rc (fileSha1Key fbp) currentSha1

            case resultE of

              Left msg -> return $ Left msg

              Right _ -> trySetFileToWorkingStatus rc strKey

  else do

    debugl rc "file sha1 is not checked for large files"

    trySetFileToWorkingStatus rc strKey

-- | given a FillBlockParam, if this file is new, send job to worker and mark

-- it as working. if there is an error, return IO Left.

processNewFileAsyncMaybe :: RDRuntimeConfig -> FillBlockParam -> ExceptT T.Text IO ()

processNewFileAsyncMaybe rc fbp = do

  let strKey = fileStatusKey fbp

      filePath = fbpFilepath fbp

  resultE <- liftIO $ DB.insertIfNotExist rc strKey $ fsBytes FileStatusWorking

  resultE <- liftIO $ isNewFile rc fbp

  throwOnLeft resultE

  let insertOk = fromRight' resultE

  if insertOk then liftIO $ do