Skip to content
operational 56.1 KiB
Newer Older
        Can't open user config file /cygdrive/c/Users/IEUser/.ssh/config: No such file or directory
        DONE do not specify config file if it doesn't exist.
        DONE fail fast if ssh key doesn't exist. mbackup will only support key based auth.
        DONE ssh options should only be added when backup to remote node.

        it works.
        PS C:\Users\IEUser> &"C:\Program Files\mbackup\publish\mbackup.exe" --target root@sylecn01.emacsos.com:/data/backup/PC-backup/IEUser --ssh-key E:\id_rsa --remote-user sylecn
      - TODO failwith should not be used in UI code. It throws unhandled exception. System.Exception.
        search: f# exit early without shifting to the right
        in haskell, I use maybeT and eitherT etc.

        in f#, you should use exception.
        Environment.Exit(exitCode) also works.

Yuanle Song's avatar
Yuanle Song committed
- problems
  - each file require it's own <Component> tag.
    rsync mingw have many files. I seems I need to generate an installer for
    rsync for windows.
  - search: wix only install file if it does not exist
    windows installer - Copy if not exist in WiX - Stack Overflow
    https://stackoverflow.com/questions/1912037/copy-if-not-exist-in-wix
    installer - What is the wix 'KeyPath' attribute? - Stack Overflow
    https://stackoverflow.com/questions/2003043/what-is-the-wix-keypath-attribute/2003366#2003366
    Component Table - Win32 apps | Microsoft Docs
    https://docs.microsoft.com/en-us/windows/win32/msi/component-table?redirectedfrom=MSDN
    How To: Check for .NET Framework Versions
    https://wixtoolset.org/documentation/manual/v3/howtos/redistributables_and_install_checks/check_for_dotnet.html
  - WONTFIX how to unpack zip file to target dir?
Yuanle Song's avatar
Yuanle Song committed
    I don't want to create a Component for every file there.

    windows installer - Wix custom action to unzip a file - Stack Overflow
    https://stackoverflow.com/questions/47691477/wix-custom-action-to-unzip-a-file

    the suggestion is to include the files in msi.
    or unzip the files at app runtime.

    I will try create a msi file for mingw64 rsync.
    then use wix bundle to install it as dependency.

    - search: wix install all files in a folder

    - Include all Files in Bin folder in Wix installer - Stack Overflow
      https://stackoverflow.com/questions/36756311/include-all-files-in-bin-folder-in-wix-installer

      create rsync-mingw64-files.wxs
      &"C:\Program Files (x86)\WiX Toolset v3.11\bin\heat.exe" dir "D:\downloads\apps\rsync-w64" -cg RsyncHeatGenerated -dr ProgramFiles64Folder -var var.RsyncSourceDir -gg -nologo -out rsync-mingw64-files.wxs -sw5150

      compile and build rsync-mingw64.msi
      &"C:\Program Files (x86)\WiX Toolset v3.11\bin\candle.exe" -dRsyncSourceDir=D:\downloads\apps\rsync-w64\  .\rsync-mingw64.wxs .\rsync-mingw64-files.wxs
      &"C:\Program Files (x86)\WiX Toolset v3.11\bin\light.exe" .\rsync-mingw64.wixobj .\rsync-mingw64-files.wixobj -o rsync-mingw64.msi

      build is getting complex. I need make and Makefile.
    - problems
      - how to pass in preprocessor variable?
        WiX undefined preprocessor variable - Stack Overflow
        https://stackoverflow.com/questions/2063630/wix-undefined-preprocessor-variable
        To pass values for preprocessor variables like $(var.MyProject.TargetDir), use options like -dMyProject.TargetDir=c:\foo.
      - why rsync-mingw64 is installed to C:\Program Files (x86)\rsync-w64? I expect C:\Program Files\rsync-w64\
        wix-users - 64 Bit program files folder
        http://windows-installer-xml-wix-toolset.687559.n2.nabble.com/64-Bit-program-files-folder-td5948080.html
      - INVALID make pattern rule doesn't work in windows
        it works. it is because my source file is missing.
      - why mbackup.exe is 32 bit component?
        try this:
        dotnet publish -c Release -r win10-x64
        still same error.
        D:\sylecn_docs\projects\mbackup-for-windows\mbackup.wxs(16) : error LGHT0204 : ICE80: This 32BitComponent mbackup.exe uses 64BitDirectory APPLICATIONROOTDIRECTORY
        search: dotnet core how to build x64 exe file
        dotnet is building x64 exe file. it's WIX issue.
        need to add Win64="yes" in <Component> attribute.

  - DONE how to run wix to create msi file?
    two command line tool
    light
    candle
    it's not added in PATH.
    C:\Program Files (x86)\WiX Toolset v3.11\bin\light.exe
    C:\Program Files (x86)\WiX Toolset v3.11\bin\candle.exe

    There is offline doc.
    C:\Program Files (x86)\WiX Toolset v3.11\doc\WiX.chm
    C:\Program Files (x86)\WiX Toolset v3.11\doc\msi.chm

    do read the WiX Tutorial.
    https://www.firegiant.com/wix/tutorial/
    https://www.firegiant.com/wix/tutorial/getting-started/

    &"C:\Program Files (x86)\WiX Toolset v3.11\bin\candle.exe" .\mbackup.wxs
    &"C:\Program Files (x86)\WiX Toolset v3.11\bin\light.exe" .\mbackup.wixobj
    it works.
    mbackup.msi file is created.
  - the mbackup.exe can't run without all those dlls.
    dotnet publish -c Release -r win10-x64 --nologo
    maybe I should build a exe that requires dotnet core 3 to run.
    I don't want to create a big msi.
    try this:
    dotnet publish -c Release --nologo
    and update wix xml files.

    need to add all files in bin\Release\netcoreapp3.0\publish\ dir.
    use heat to create another wxs file.
  - heat can't add some dll.
    "C:\Program Files (x86)\WiX Toolset v3.11\bin\heat.exe" dir bin\Release\netcoreapp3.0\publish -cg MbackupHeatGenerated -dr MBACKUP_PROGRAM_FILES -var var.MbackupPublishDir -gg -nologo -out mbackup-files.wxs -sw5150
    heat.exe : warning HEAT5151 : Could not harvest data from a file that was expected to be an assembly: D:\sylecn_docs\projects\mbackup-for-windows\bin\Release\netcoreapp3.0\publish\Argu.dll. If this file is not an assembly you can ignore this warning. Otherwise, this error detail may be helpful to diagnose the failure: 未能加载文件或程序集“FSharp.Core, Version=4.4.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”或它的某一个依赖项。系统找不到
    指定的文件。
    heat.exe : warning HEAT5151 : Could not harvest data from a file that was expected to be an assembly: D:\sylecn_docs\projects\mbackup-for-windows\bin\Release\netcoreapp3.0\publish\mbackup.dll. If this file is not an assembly 
    you can ignore this warning. Otherwise, this error detail may be helpful to diagnose the failure: 未能加载文件或程序集“System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”或它的某一个依赖项。系统
    找不到指定的文件。

    .net - Wix Toolkit: Heat balking about DLL's - Stack Overflow
    https://stackoverflow.com/questions/43992433/wix-toolkit-heat-balking-about-dlls
    com - Cannot register DLL using WiX - Stack Overflow
    https://stackoverflow.com/questions/12463256/cannot-register-dll-using-wix
  - TODO dup file?
    <Component Id="cmp981FC09307D7CB82695815FAAFCC646D" Directory="dir02A4AE56AE905AEEE7E2D4F4410A9748" Guid="{DB94490C-32CA-4B14-A96D-A95D4B557D01}">
                <File Id="fil432C46247A06CCDB891611045DD37081" KeyPath="yes" Source="$(var.MbackupPublishDir)\runtimes\win\lib\netstandard2.0\System.Security.Cryptography.ProtectedData.dll" />
    <Component Id="cmp5190411B4FE56A0E0CE782BE6C1B537E" Directory="dir1B0C588BD61288F160D459591DA01603" Guid="{F0613E24-B22B-4A58-A054-380BDF2B857C}">
                <File Id="filE0C94795A0B5F482BD659FBB774641C7" KeyPath="yes" Source="$(var.MbackupPublishDir)\System.Security.Cryptography.ProtectedData.dll" />
    why there are duplicated dll in publish dir? check publish doc.
    this is the only duplicated file. I will ignore it for now.
  - DONE ssh.exe path escape issue.
    &"C:\Program Files\mbackup\rsync-w64\usr\bin\rsync.exe" -h --stats -togr --delete --delete-excluded --ignore-missing-args --files-from=/cygdrive/c/Users/sylecn/AppData/Local/mbackup/mbackup.list --exclude-from=/cygdrive/c/ProgramData/mbackup/mbackup-default.exclude --log-file=/cygdrive/c/Users/sylecn/AppData/Local/mbackup/mbackup.log -e "/cygdrive/c/Program Files/mbackup/rsync-w64/usr/bin/ssh.exe -F /cygdrive/c/Users/sylecn/.ssh/config -i /cygdrive/c/Users/sylecn/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --remote-option=--log-file=/var/log/mbackup/B75I3.log --chown=sylecn:sylecn / root@sylecn01.emacsos.com:/data/backup/PC-backup/B75I3/
    rsync: Failed to exec /cygdrive/c/Program: No such file or directory (2)

    &"C:\Program Files\mbackup\rsync-w64\usr\bin\rsync.exe" -h --stats -togr --delete --delete-excluded --ignore-missing-args --files-from=/cygdrive/c/Users/sylecn/AppData/Local/mbackup/mbackup.list --exclude-from=/cygdrive/c/ProgramData/mbackup/mbackup-default.exclude --log-file=/cygdrive/c/Users/sylecn/AppData/Local/mbackup/mbackup.log -e "'/cygdrive/c/Program Files/mbackup/rsync-w64/usr/bin/ssh.exe' -F /cygdrive/c/Users/sylecn/.ssh/config -i /cygdrive/c/Users/sylecn/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --remote-option=--log-file=/var/log/mbackup/B75I3.log --chown=sylecn:sylecn / root@sylecn01.emacsos.com:/data/backup/PC-backup/B75I3/
** 2019-11-16 support reading remote-user=xxx option from config file.
cli arg > env var > config file.
same with target option.
Yuanle Song's avatar
Yuanle Song committed
** 2019-11-13 next todos; scheduled task command line.
- DONE fix TODOs in F# code
- DONE add rsync command and arguments for running in windows.
  i.e. -e etc.

  <Arguments>--stats -togr --chown=sylecn:sylecn --exclude-from=/cygdrive/d/sylecn_docs/texts/configs/rsync-exclude
   --files-from=/cygdrive/d/sylecn_docs/texts/configs/rsync-file-list
    --log-file=/cygdrive/d/sylecn_docs/rsync-b75i3.log
     -e ".\ssh.exe -F c:/users/sylecn/.ssh/config -i c:/Users/sylecn/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
      / sheni:/data/backup/server-backup/b75i3/</Arguments>

- DONE test param parsing is working.
  command line param > env var > mbackup default value.
  support --node-name param.
- DONE test run in console and scheduled task.
  run in console works.
  try run in scheduled task.

  SCHTASKS /Create /?
  # run mbackup 15m after user logon.
  SCHTASKS /Create /U <username> /SC ONLOGON /TN mbackup-logon /TR "\"<path\to\mbackup.exe>\" \"args\"" /DELAY 15:00
  # run mbackup at 10am and 4pm.
  SCHTASKS /Create /U <username> /SC DAILY /TN mbackup-morning /TR "\"<path\to\mbackup.exe>\" \"args\"" /ST 10:00 /ET 13:00 /K
  SCHTASKS /Create /U <username> /SC DAILY /TN mbackup-afternoon /TR "\"<path\to\mbackup.exe>\" \"args\"" /ST 16:00 /ET 19:00 /K

  # debug purpose, one time only
  SCHTASKS /Create /SC ONCE /TN mbackup-test /TR "\"<path\to\mbackup.exe>\" \"args\"" /ST 19:18

  in powershell,
  $mbackupexe = "D:\sylecn_docs\projects\mbackup-for-windows\bin\Debug\netcoreapp3.0\mbackup-for-windows.exe"
  SCHTASKS /Create /SC ONCE /TN mbackup-test /TR $mbackupexe /ST 19:21
  default run as user is current user.

  in admin powershell,
  SCHTASKS /Create /SC ONCE /TN mbackup-test /TR $mbackupexe /ST 19:32 /NP
  /NP option requires admin privilledge.
  it works.

  to delete it,
  schtasks /delete /tn mbackup-test

  - problems
    - how many scheduled task to run on a multi-user PC?
      each user have its own user-default.list expansion.
      should I iter over all users on PC?
      I think only current user can get it's profile dir and special dirs.
    - maybe config mbackup to run after user logon. with 15minute delay.
      always run as current user.
    - how to not require any param when running mbackup.exe?
      put TARGET and other option in a config file?
      define system level TARGET env variable.

      use config file is easier for user to edit and making the change effective.
      userConfigDir / mbackup.conf

      search: F# read config file
      FSharp.Configuration
      http://fsprojects.github.io/FSharp.Configuration/
    - FSharp.Configuration missing reference to System.Runtime.Caching
      search: how to reference System.Runtime.Caching for dotnet core project
      https://www.nuget.org/packages/System.Runtime.Caching/
      dotnet add package System.Runtime.Caching --version 4.6.0
      should I add 4.0.0? Can I use a higher version?

      still not compatible.
      search: use FSharp.Configuration with dotnet core 3
    - give up on FSharp.Configuration.
    - try this:
      FsConfig
      https://www.demystifyfp.com/FsConfig/
      AppSettings is only supported in V0.0.6 or below.
      try F# AppSettings directly.
      If nothing is easy to use, write my own parser.
      Support similar config format as python wells lib.
    - ConfigurationManager.AppSettings Property (System.Configuration) | Microsoft Docs
      https://docs.microsoft.com/en-us/dotnet/api/system.configuration.configurationmanager.appsettings?view=netframework-4.8
      where should I save the App.config xml file?
      search: what is app.config
      App.Config: Basics and Best Practices - SubMain Blog
      https://blog.submain.com/app-config-basics-best-practices/
      What is App.config in C#.NET? How to use it? - Stack Overflow
      https://stackoverflow.com/questions/13043530/what-is-app-config-in-c-net-how-to-use-it
      Okay. This is not what I want.
      This is for application configuration (rarely change), not for user configuration (can change any time).
      dotnet will create <yourapp>.exe.config from your App.config file.
    - how to use multiple files in F# dotnet core project?
      search: f# module and namespace
      search: f# module
      Modules - F# | Microsoft Docs
      https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/modules
    - SCHTASKS /Create /SC ONCE /TN mbackup-test /TR $mbackupexe /ST 19:21
      this runs, it will show a console window. how to not show console window?

      search: schedule tasks do not show console window

      In the “General” tab, under the “Security options” section, select the Run
      whether user is logged on or not option. (This is the option that will make
      the command window not to appear when the task runs automatically.)

      try /NP option.

- create an installer. The installer should add scheduled task on install and
  delete scheduled task on removal.

- problems
  - how to write unit test in F#?
    Unit testing F# in .NET Core with dotnet test and NUnit - .NET Core | Microsoft Docs
    https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-fsharp-with-nunit
    mkdir mbackup-tests
    cd mbackup-tests
    dotnet new nunit -lang F#
    //wait.
    dotnet test

    dotnet add reference ../mbackup-for-windows.fsproj
    dotnet build  # this will build both ref project and test project.
    dotnet test
  - how to support short option names?
    --dry-run -n
  - 
Yuanle Song's avatar
Yuanle Song committed
** 2019-11-14 support mbackup.txt config.
config file will have wells config file format.
empty lines and comment lines are ignored.
space around = is ignored.
if there are quotes on value side, it's trimmed.

target=user@host:port/path/
# target = user@host:port/path/
abc.def = some value
abc.def.ghi = 1233.45
foo = yes

env variable will be
TARGET
ABC_DEF
ABC_DEF_GHI
FOO

in F# code,
conf = WellsConfig("C:\path\to\file.txt")
conf.GetStr("target")
conf.GetFloat("abc.def.ghi")
conf.GetBool("foo")

it works.

Yuanle Song's avatar
Yuanle Song committed
** 2019-11-13 extra user default list.
Documents is replaced by real path.
Downloads
Pictures

User should always use forward slash in backup list and exclude list files.
These path are supported:
C:\Foo\Bar
C:/Foo/Bar
Pictures/Saved Pictures
Documents/

Yuanle Song's avatar
Yuanle Song committed
** 2019-11-13 build mbackup.list file from file list and exclude lists.
Yuanle Song's avatar
Yuanle Song committed
* wontfix                                                             :entry: