redis data schema done. sha1calculator design done

* COMMENT -*- mode: org -*-

#+Date: 2018-05-04

Time-stamp: <2018-05-05>

Time-stamp: <2018-05-06>

#+STARTUP: content

* notes                                                               :entry:

** 2018-05-05 it's impossible to do logging easily in haskell.

* current                                                             :entry:

** 

** 2018-05-05 calculate sha1sum for blocks using a thread pool.

** 2018-05-06 calculate sha1sum for blocks using a thread pool. design try 2.

- data protocol via redis.

  hset <filepath>_blockSize blockId sha1sum

  set <filepath>_blockSize_status working|done

- worker pool is there for calculating all blocks for one file.

         fileQueue

  Main ------------> fileWorker

  GET /rd/file

  if file status is None, push file to fileQueue.

  do normal logic.

  fileWorker:

  fetch file from fileQueue.

  start working on blocks one by one.

  if block already cached in redis, skip it.

  when all done, set <filepath>_blockSize_status done.

- this works and is easy to understand.

  WIP info is also kept in redis for each file's block.

- 

** 2018-05-05 write the main logic of creating block metadata.

then make it work with a thread pool with a single thread.

yes. see Route Patterns.

* wontfix                                                             :entry:

** 2018-05-05 calculate sha1sum for blocks using a thread pool. design try 1.

- data protocol via redis.

  set <filepath>_blockSize_status working|done

  hset <filepath>_blockSize blockId sha1sum

  low level, for job queue.

  lpush sha1_calculator_queue

  {filepath: xxx,

   block_size: xxx,

   block: [block_id, start, end]}

- I can get all when fill in sha1sum.

  hgetall <filepath>_blockSize

- thread communication design.

  Main

  create worker threads. worker threads wait to fetch item from jobQueue.

  create jobQueue.

  Worker

  fetch item from jobQueue, when done, write redis hash key.

  when error, put the item back. (catch exception).

  GET /rd/file

  if get <filepath>_blockSize_status is None, set status to "working", then

  add blocks data to redis list.

- should I calculate all blocks for one file in a single thread?

  or allow multi-thread? allow multi-thread use CPU better.

  use one thread allow better file cache usage.

  allow multi-thread.

// this design is too complex.

redesign.

  - formatting

  - filepath

  - unix

  - unordered-containers

library:

  source-dirs: src

import Control.Monad.IO.Class (liftIO)

import Data.Either (fromRight)

import Data.Monoid ((<>))

import Data.Text.Encoding (decodeUtf8)

import qualified Data.ByteString as B

import qualified Data.ByteString.Char8 as Char8

import qualified Data.Text as T

import qualified Data.Text.Lazy as LT

import System.FilePath (combine)

import System.Posix.Files (getFileStatus, fileSize)

import qualified Database.Redis as R

import qualified Data.HashMap.Strict as M

import Config

import RD.Lib (sha1sum)

        else

            reverse accumulator

data FillBlockParam = FillBlockParam {

      fbpFilepath :: FilePath

    , fbpBlockSize :: Integer

    , fbpFileSize :: Integer

    , fbpBlocks :: [Block]}

-- | the redis hash key used to store cached sha1sum for given FillBlockParam

blockSha1sumHashKey :: FillBlockParam -> B.ByteString

blockSha1sumHashKey fbp = Char8.pack (fbpFilepath fbp) <> "_" <> (Char8.pack . show) (fbpBlockSize fbp)

-- | the redis hash key sub key, used to store the sha1sum for that blockId.

blockIdKey :: BlockID -> B.ByteString

blockIdKey = Char8.pack . show

-- | fill block sha1sum, if sha1sum is not ready yet, put "pending" there.

fillSha1sum :: RDRuntimeConfig -> FillBlockParam -> IO [BlockWithChecksum]

fillSha1sum runtimeConfig fbp = do

  let filepath = fbpFilepath fbp

      blockSize = fbpBlockSize fbp

      hashKey = blockSha1sumHashKey fbp

  redisReply <- R.runRedis (redisConn runtimeConfig) $ R.hgetall hashKey

  case redisReply of

    Left reply -> do

      putStrLn $ "redis hgetall " <> show hashKey <> " failed: " <> show reply

      return $ map fillBlock (fbpBlocks fbp) where

        fillBlock (blockId, start, end) = (blockId, start, end, "pending")

    Right blockIdSha1sumAlist -> do

      putStrLn $ "redis hgetall " <> show hashKey <> " ok"

      return $ map fillBlock (fbpBlocks fbp) where

        blockIdSha1sumMap = M.fromList blockIdSha1sumAlist

        fillBlock :: Block -> BlockWithChecksum

        fillBlock (blockId, start, end) = (blockId, start, end, decodeUtf8 $ M.lookupDefault "pending" (blockIdKey blockId) blockIdSha1sumMap)

-- | given a redis connection pool, return a Scotty app.

mkApp :: RDRuntimeConfig -> ScottyM ()

mkApp runtimeConfig = do

        blockSizeInByte = 2097152    -- 2MiB

        blockCount = (fileSizeInByte - 1) `div` blockSizeInByte + 1

        blocks = genBlocks fileSizeInByte blockSizeInByte

    blocksWithSha1sum <- liftIO $ fillSha1sum runtimeConfig $ FillBlockParam {

                              fbpFilepath=filepath

                            , fbpFileSize=fileSizeInByte

                            , fbpBlockSize=blockSizeInByte

                            , fbpBlocks=blocks}

    json $ object [("ok" .= True)

                  ,("block_size" .= ("2MiB" :: T.Text))

                  ,("file_size" .= fileSizeInByte)

                  ,("block_count" .= blockCount)

                  ,("blocks" .= blocks)

                  ,("blocks" .= blocksWithSha1sum)

                  ,("path" .= path)

                  ,("filepath" .= filepath)

                  ]