operational

* COMMENT -*- mode: org -*-
#+Date: 2015-11-04
Time-stamp: <2019-02-24>
#+STARTUP: content
* later								      :entry:
** 2016-01-05 "stack setup" command has a nice download progress bar.
sylecn@yygameh1:~/fromsource/qvm$ stack setup
Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
ghc-7.10.2:  372.55 KiB / 87.60 MiB (  0.42%) downloaded...

Try to add this in my image download step.

** 2016-01-04 sometimes when I boot from u1404-vm4.img, I get a read-only file system.
- not sure what's wrong.

- created a few VM on rise-16, try to reproduce it.
  I think it's a problem of the image, not qvm.
  I need to create production level u14.04 image soon.

  qvm create t1 --disk http://dfs.game.yy.com:8080/download/u1404-vm4.img --network ovsbridge:name=br0,vlan=100 --ip 192.168.100.2
  qvm create t2 --disk http://dfs.game.yy.com:8080/download/u1404-vm4.img --network ovsbridge:name=br0,vlan=100 --ip 192.168.100.3

  I see error in VNC boot screen, but it scrolls too fast.
  There is no error log in boot.log file.

- it works after VM reboot.
  maybe because of haskell laziness? between copy file and start domain.

  getDisk
  prepareMetadata
  vrish define

  They run in the same IO action. It's not about laziness.

- is it file permission problem?
  I see owner of disk image is

  libvirt-qemu:kvm

  /usr/sbin/libvirtd is run as root. I don't see a problem.

- search: KVM Read-only file system fix after reboot

** 2016-01-04 handle download errors. support continuous download.
- handle 40x error.
- handle download image is not qcow2 error.

** 2016-01-04 detect downloaded image or local image is indeed qcow2 image.
otherwise, OS boot may fail.

qemu-img info --output=json foo.img|grep qcow2
qemu-img info --output=json 9bd47715634037fcfb1ce0ae463c8e7f717ad102-cirros-0.3.4-x86_64-disk.img|jq .format

** 2015-11-04 warn when hypervisor can't start KVM type VM.
* current							      :entry:
** 
** 2016-01-06 support qvm --version, how to re-use the version in cabal file.
** 2016-01-05 support --netmask in "qvm create"
** 2016-01-04 auto run "qvm init" if it has never run.
Just detect /data/qvm/ dir.
** 2015-12-31 a new design of qvm (continue)
Here are things not implemented yet.
*** DONE qvm create
    - supported disk type

      --disk URL-to-img-file download this file. use local copy if it exists.
      --disk /path/to/foo.img
      --disk sheepdog:foo    clone from this sheepdog snapshot/disk.

      DONE First, support URL and local file.

    - DONE supported network

      --network type=bridge,bridge=br1,vlan=200,ip=192.168.1.2
      --network type=nat,bridge=br1,ip=192.168.1.2

      omit this parameter means use predefined:default network.

      --network predefined:default
      --network predefined:ovs1
      --network ovsbridge:name=br0
      --network ovsbridge:name=br0,vlan=200

*** DONE support bind IP using metadata service.
--ip 192.168.122.34
[--gateway gw]
[--dns dns]

genisoimage -input-charset utf-8 -output vm1.iso -volid cidata -joliet -rock user-data meta-data

./src/domain-templates/meta-data
./src/domain-templates/user-data

- auto fill default gw and dns if it's not given.

  - param handling.
  - defineDomain should fill in the gaps.

    domainFromOpts

- test it on rise-16

  qvm create c6 --disk http://dfs.game.yy.com:8080/download/cirros-0.3.4-x86_64-disk.img --network ovsbridge:name=br0 --ip 192.168.122.51
  qvm create c7 --disk http://dfs.game.yy.com:8080/download/cirros-0.3.4-x86_64-disk.img --network ovsbridge:name=br0 --ip 192.168.122.52
  qvm create c8 --disk http://dfs.game.yy.com:8080/download/cirros-0.3.4-x86_64-disk.img --network ovsbridge:name=br0,vlan=200 --ip 192.168.122.53
  qvm create c9 --disk http://dfs.game.yy.com:8080/download/cirros-0.3.4-x86_64-disk.img --network ovsbridge:name=br0,vlan=200 --ip 192.168.122.54

  DHCP is still enabled in cirros image, which makes boot slow.

- upload my own image on ceph-207 to dfs.

  rbd clone kvm01/vm3@t0 kvm01/vm4
  boot vm4

  - disable backup datasource.

    dpkg-reconfigure cloud-init

  - export kvm01/vm4

    time qemu-img convert -O qcow2 rbd:kvm01/vm4 u1404.img

    1m34s

    try it on qvm on ceph-207.

    qvm create vm1 --disk /data/software/u1404.img --network ovsbridge:name=br0,vlan=100 --ip 192.168.100.2

    works perfectly. boots fast.

    qvm create vm2 --disk /data/software/u1404.img --network ovsbridge:name=br0,vlan=100 --ip 192.168.100.3

    They ping each other ok.
    But there is no gw on br0. Create one:

    ovs-vsctl add-port br0 gw100 tag=100 -- set Interface gw100 type=internal
    ip a add 192.168.100.1/24 brd + dev gw100
    ip l set dev gw100 up

    works perfectly. Now VM get internet.

  - dfsupload u1404.img

    curl --upload-file u1404.img http://dfs.game.yy.com:8080/upload/u1404-vm4.img

*** support resize disk to given size.
    qemu-img resize foo.img 20G

    maybe add new option:
    --disk 20
    --image http://xxx

*** auto detect and convert raw type disk to qcow2 type.

* done								      :entry:
** DONE 2016-01-05 qvm rm: when domain is not running, do not try to destroy it.
when domain is not defined, do not try to undefine it.
just rm the instance dir if domain is already removed by user.

** 2016-01-04 create clean u1404 image.
see ~/projects/cloud-2.0/doc/cloud-image/log

This is out of the scope of qvm.

** 2016-01-05 build qvm on debian 7.
- my build runs fine on u12.04, but not on debian 7.

  libc6/glibc version
  ===============
  u12.04 2.15
  u14.04 2.19
  deb7 2.13
  deb8 2.19
  centos6.4 2.12

** 2015-11-04 how to ensure VM get same IP across reboots? use metadata.
how to list IPs for all VMs?    NA.

** 2016-01-04 etags doesn't support haskell? is there a tool for jump to def?
new haskell-mode has haskell-mode-jump-to-def-or-tag, it works well.

tags relies on hasktags, which I installed to ~/.local/bin/hasktags
** 2016-01-04 support download large image files. should not store in RAM then write.
- how to download large file in haskell without using much RAM.
  http://stackoverflow.com/questions/24718873/downloading-large-files-from-the-internet-in-haskell
  Network.HTTP.Conduit http can do it.

  https://hackage.haskell.org/package/download-curl-0.1.4/docs/Network-Curl-Download-Lazy.html
  I hope openLazyURI can do it too. Because it's easier.
  download-curl is not in lts haskell.
  add it in stack.yaml. it works.

  - download-curl still cache resp in RAM before writing to disk.
  - Conduit seems a large dependence.

    conduit works.

** 2016-01-04 import new image or make cirros work.
- does cirros work with nocloud metadata?
  yes.
  see /etc/init.d/cirros-ds-local
  cirros-ds --help
  /etc/cirros-init/config
  DATASOURCE_LIST="nocloud configdrive ec2"

  - try it.

    # cirros-ds local
    cirros-ds 'local' up at 423.39
    no results found for mode=local. up 423.41. searched: nocloud configdrive ec2

    iso didn't work there.

  - disable the ec2 ds.

- DONE config auto login on serial console.
  /etc/inittab

  -ttyS0::respawn:/sbin/getty -L 115200 ttyS0 vt100
  +ttyS0::respawn:/sbin/getty -l /bin/sh -L 115200 ttyS0 vt100

** 2015-12-31 if any of disk, domain.xml exists, just exit early.
** 2015-12-30 a new design of qvm.
- make create ubuntu VM easy. using qvm and sheepdog/nginx.
  - qvm init
    prepare env for HV.
    - install libvirt, qemu, ovs, kvm-check etc.
    - create ovs network or bridge. define libvirt network.
    - [optional] create secrets for ceph.
  - qvm create vm1 --core 2 --ram 512M --network default --disk /path/to/foo.img

    - create image for the vm.
    - define and start the vm.

    - supported disk type

      --disk /path/to/foo.img
      --disk ubuntu-14.04    find file with this name in images dir.
      --disk sheepdog:foo    clone from this sheepdog snapshot/disk.
      --disk URL-to-img-file download this file. use local copy if it exists.

      When download from URL, just save the file using the name of sha1 of
      URL, with original file suffix (for easier qemu-img operation).

      They all means make a copy of the image/snapshot and use that as first
      disk. Resize it to user requested size when cloning.

      - DONE URL type image.
	:t toLazyByteString $ byteStringHex $ SHA1.hash $ B.pack "abc"

	- how to download large file in haskell without using much RAM.
	  http://stackoverflow.com/questions/24718873/downloading-large-files-from-the-internet-in-haskell
	  Network.HTTP.Conduit http can do it.

	  https://hackage.haskell.org/package/download-curl-0.1.4/docs/Network-Curl-Download-Lazy.html
	  I hope openLazyURI can do it too. Because it's easier.
	  download-curl is not in lts haskell.
	  add it in stack.yaml. it works.

    - supported network

      --network type=bridge,bridge=br1,vlan=200,ip=192.168.1.2
      --network type=nat,bridge=br1,ip=192.168.1.2

      omit this parameter means use defined:default network.

      --network predefined:default
      --network predefined:ovs1
      --network ovsbridge:name=br0
      --network ovsbridge:name=br0,vlan=200

  - qvm rm vm1

    destroy domain, undefine domain and remove disk and xml file.

  - implementation

    - DONE base image naming schema. I want to include original name after sha1.
    - WONTFIX generate a random name if user didn't specify a name.
      error if user didn't specify a name.
      when running from cmd line, user have to specify a name.
      - check whether VM name already exists before doing anything.

    - DONE make nat and file base image work.

      - DONE where to do disk copy action?
	when you `create' the domain. not just building the domain.
      - DONE implementation renderDisk.
	should I support both qcow2 and raw? only support qcow2 for now.
	what is bochs and qed?

	- DONE can I resize when clone from qcow2?

	  Just copy it, then resize.
          qemu-img resize foo.img 20G

	- abort if disk is not in qcow2 format.
	- DONE is it easy to share a base when using qcow2? backing file?

	  yes. do not use copy file, use qemu-img:

	  qemu-img create -f qcow2 -b centos-cleaninstall.img snapshot.img

	  The new image is now a read/write snapshot of the original image --
          any changes to snapshot.img will not be reflected in
          centos-cleaninstall.img.

	  OK. I will only support qcow2 format disk.
	  Show error if given disk file is not qcow2.
          TODO I can also convert it automatically and use that as base.

    - DONE implement "qvm init"
      I don't like the way haskell handle cli arguments.
      But for type safety, they have to be more complicated.

      I want my program to be self contained. so I don't plan to run
      puppet/salt there.

      DONE qvmInit :: IO ()

      DONE support "qvm init" in cmd line handling.

    - DONE how to use relative file path in stFile?
      renderXML d = [stFile|src/domain-templates/domain.xml|]
      works. just need src/ there. build always happens in stack.yaml dir.

    - DONE use text file template to render xml.
      network and disk should be placed in as a whole.

      Which library to use?
      libraries - Which Haskell library for interpolated strings - Stack Overflow
      http://stackoverflow.com/questions/8956801/which-haskell-library-for-interpolated-strings

      try shakespeare
      http://hackage.haskell.org/package/shakespeare

      render to Text using quasiquotes works.
      #+BEGIN_SRC haskell
        renderXML :: QvmDomain -> T.Text
        renderXML d = [st|
        haha my core is #{core d}
        |]
      #+END_SRC

      TODO how to render using external file?
      I think the template file is compiled into the final binary.

      compare it with lucius.

      hamlet :: QuasiQuoter
      hamletFile :: FilePath -> Q Exp

      putStrLn $ renderHtml $ [hamlet|<body>|] render

      template = $(luciusFile "template.lucius")
      putStrLn $ renderCss $ template render

      http://stackoverflow.com/questions/12791491/use-shakespeare-text-and-external-file
      didn't mention a solution.

      - I need to understand template haskell.

        what does quasiquoter produce?
        how to use Q Exp?

	understand what st is doing, then do the similar for textFile.
	http://hackage.haskell.org/package/shakespeare-2.0.7/docs/src/Text-Shakespeare-Text.html#stext

	#+BEGIN_SRC haskell
          stext :: QuasiQuoter
          stext = 
            QuasiQuoter { quoteExp = \s -> do
              rs <- settings
              render <- [|toLazyText|]
              rendered <- shakespeareFromString rs { justVarInterpolation = True } s
              return (render `AppE` rendered)
              }

          textFile :: FilePath -> Q Exp
          textFile fp = do
              rs <- settings
              shakespeareFile rs fp
	#+END_SRC

	QuasiQuoter {quoteExp :: String -> Q Exp}

	shakespeareFromString :: ShakespeareSettings -> String -> Q Exp
	shakespeareFile :: ShakespeareSettings -> FilePath -> Q Exp

      - copied some code and it is working now.

      - shift2space for pretty xml.

        TIO.putStrLn $ shift2space $ renderNetwork $ DefaultNAT
	printXML makeDomain

    - DONE OO design or functional? I like functional.

      do
        makeDomain
        setCore 2
        setRAM "512M"
        setNetwork {type: "nat", bridge: "br1"}
        setDisk {src: "/path/to/foo.img"}
        printDomainXML

      store all useful info in data structure, then generate xml in the final
      step. do not do xml level combination/replacing.

    - DONE qvm config directory.
      /data/qvm/images/
      /data/qvm/images/cirros-0.3.4-x86_64-disk.img
      /data/qvm/instances/
      /data/qvm/instances/vm1/disk
      /data/qvm/instances/vm1/domain.xml

    - DONE network type
      predefined nat. params: network name

      #+BEGIN_SRC xml
        <interface type='network'>
          <source network='default'/>
          <model type='virtio'/>
        </interface>
      #+END_SRC

      bridge mode with vlan tag. params: bridge, vlanid
      #+BEGIN_SRC xml
        <interface type='bridge'>
          <forward mode="bridge"/>
          <source bridge="br1"/>
          <virtualport type='openvswitch'/>
          <model type='virtio'/>
          <vlan trunk='no'>
            <tag id='100'/>
          </vlan>
        </interface>
      #+END_SRC

      putStrLn $ T.unpack $ renderNetwork DefaultNAT
      putStrLn $ T.unpack $ renderNetwork $ OvsBridge "br1"
      putStrLn $ T.unpack $ renderNetwork $ OvsBridgeWithVLAN "br1" 100

* wontfix							      :entry:
** 2016-08-19 in qvm: when booting win xp vm, use clock in localtime.
<clock offset='localtime'/>

- for win xp VM, do not use virtio driver for disk or network device.
- 2019-02-24 on sheni, edit
  /data/qvm/instances/xp/domain.xml
  -<clock offset='utc'/>
  +<clock offset='localtime'/>

- win xp is a past. I won't support it.

** 2016-01-06 ubuntu 12.04 compatibility problem.
qvm init ovs-vsctl --may-exist option not supported by old ovs.
sudo ovs-vsctl --may-exist add-br br0

- 12.04 can use ubuntu cloud archive icehouse to get the newer version.
  I won't fix this.

** 2015-11-04 make this work:
qvm --core 4 --ram 2g --disk 20g

use NAT network and default disk image.

- check whether we have enough RAM for this.
- generate xml
- define domain
- start domain