* COMMENT -*- mode: org -*-
#+Date: 2014-05-09
Time-stamp: <2019-02-28>
#+STARTUP: content
* notes								      :entry:
** 2018-05-02 support test, build and deploy in gocd
- stage 1: testAndBuild
  make ci-build

  required env var:
  DOCKER_USER
  DOCKER_PASSWORD
  docker login info for private registry.

- stage 2: deploy
  make deploy

  required env var:
  KUBECTL_API
  KUBECTL_TOKEN
  kubectl auth info for running kubectl apply.

- implemented in v1.2.0

  in gocd, use the python-app pipeline template to automate stages and jobs.

** 2018-05-02 workflow for v1.2.0+
- create project on dev node.
- create gitlab repo by init-gitlab
- dev on dev node.
  make test
  make run
- when ready to push to production.
  update deb-scripts if necessary
  update ./utils/build-deb if necessary
  update app.yaml if necessary
- on gitlab, add gocd as deploy key
- on gocd, create new pipeline for the project.
  use python-app template.

  // these two step should be automated.

- on dev node, git push
  now you will see gocd start testing, building docker image and push it.
  when first stage is green, click approval to deploy to production.
- to upgrade project, make commits, and push to gitlab, the rest will follow.
  click approval to deploy to production.

** 2019-02-27 how to update virtualenv in utils/ dir
- update files
  VERSION=16.4.1
  tar tf ~/d/virtualenv-$VERSION.tar.gz
  cd src/leiningen/new/python/utils/
  tar xzf ~/d/virtualenv-$VERSION.tar.gz \
    virtualenv-$VERSION/virtualenv.py \
    virtualenv-$VERSION/virtualenv_support/ \
    virtualenv-$VERSION/LICENSE.txt
- update file list in ./src/leiningen/new/python.clj
  TODO update the code to auto include all files in utils/virtualenv-* dir.
* later								      :entry:
** 2019-02-28 about the two step CD, should I make it 3 step build?
1. test
2. build production docker image
3. on user approve, deploy to production.

currently 1 and 2 are run in make ci-build.
check the disk space usage.
sometimes, trivial doc change doesn't require a docker build.
sometimes, version change doesn't require a docker build because of new
feature is still WIP and I won't deploy it.

- check the disk space usage.
  base stretch slim is 55MiB.
  app image is additional 55MiB.
  the apt-get step 34MiB, unpack deb 21MiB.

  seems reasonable. Could save 34MiB by including python3, libpq in the base
  image.

- does kubernetes remove old images? yes.

  The default is when disk space usage is over 85%, it will clean image until
  it hits 80%.
  https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

- 2019-02-28 you can config blacklist in add/edit material.
  https://docs.gocd.org/current/configuration/admin_add_material.html#blacklist

  https://docs.gocd.org/current/configuration/configuration_reference.html#ignore

  Often you do want to specify a set of files that GoCD should ignore when it
  checks for changes. Repository changesets which contain only these files
  will not automatically trigger a pipeline. These are detailed in the ignore
  section of the configuration reference.

  ignore for this project:
  #+BEGIN_SRC conf
    .gitignore,LICENSE,README.md,operational,src/leiningen/new/python/README.rst,src/leiningen/new/python/README-dev.rst,src/leiningen/new/python/gitignore
  #+END_SRC

* current							      :entry:
** 
** 2019-02-28 add support for running application test using docker-compose.
see ~/websites/url-alias project for an example.

add docker-compose.yml
update Makfile

** 2019-02-24 fix :url in project.clj
create a blog post about this template.
** 2016-09-08 support updating site file, deb scripts etc after project is generated.
- the domain in site file could change.
- the pkg name could change.
- whether to use uwsgi could change.

./utils/update_site_file.py --domain foo.emacsos.com --uwsgi
./utils/update_site_file.py --domain foo.emacsos.com --http

** 2016-09-06 support adding celery to existing project.
- This will add a celeryconfig module and a tasks module.
  see examples in
  ~/projects/wechat-article-saver/wechat_article_saver_daemon/wechat_article_saver_daemon/celeryconfig.py
  ~/projects/wechat-article-saver/wechat_article_saver_daemon/wechat_article_saver_daemon/fetcher.py
- I don't know how to call lein from existing project yet.
  shouldn't be so hard.

** 2016-08-18 integrate tox.
let "make test" use the default python in venv, but allow "make tox" to run
unit tests against defined python in tox.ini.

** 2016-08-07 in python project template, add a project wide pylintrc file.
see example ~/.pylintrc
** 2014-05-09 add sphinx support

* done								      :entry:
** 2016-06-30 update it for use with default debian config. (non-yy env)
- update nginx config file
- add support for systemd
- create /data if it doesn't exist. auto use sudo to create the dir if it
  doesn't exist.

  make sure it works on fresh system. use a VM to test it.
** 2019-02-28 add gocd support.
// was add trivis-ci support.
- does trivis-ci send email when build failed?
- just create new project on python2.7, python3.5, pypy env.
  then run "make test"
- trivis-ci only support github now.
  GitLab integration · Issue #5931 · travis-ci/travis-ci · GitHub
  https://github.com/travis-ci/travis-ci/issues/5931
- maybe add gocd support for it instead.

** 2019-02-24 drop :repositories from project.clj
** 2019-02-27 another problem on new created project.
make test fail on new project.

futures requires Python '>=2.6, <3' but the running Python is 3.5.3

one pkg in requirements-dev.txt has bad dependencies.

pylint has problem.

#+BEGIN_SRC sh
  Collecting futures (from isort>=4.2.5->pylint)
    Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1f/9e/7b2ff7e965fc654592269f2906ade1c7d705f1bf25b7d469fa153f7d19eb/futures-3.2.0.tar.gz
  futures requires Python '>=2.6, <3' but the running Python is 3.5.3
#+END_SRC

- search: pylint futures requires Python '>=2.6, <3' but the running Python is 3.5.3
  search: isort futures broken

- try fix isort to older version.
  isort is used to sort import statements. it has no dependencies.

  what does that collect line mean?

  // isort does have dependencies. I fixed isort to 4.2.5 and it works.

  https://github.com/timothycrosley/isort/blob/4.3.9/setup.py
      install_requires=['futures; python_version < "3.2"'],
  it has required futures pkg.

  the setup.py syntax changed in v4.3.5
  v4.3.4 should work fine.

  old syntax:
  install_requires = []
  if sys.version_info.major == 2:
    install_requires = ['futures']
  ...
      install_requires=install_requires,

  new syntax:
        install_requires=['futures; python_version < "3.2"'],

  the new syntax doc:

  Building and Distributing Packages with Setuptools — setuptools 40.8.0 documentation
  https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies
  Declaring platform specific dependencies
  PEP 508 -- Dependency specification for Python Software Packages | Python.org
  https://www.python.org/dev/peps/pep-0508/
  Created:	11-Nov-2015

  which setuptools version start to support this feature?

  Why I hit this problem today? This is old stuff.
  isort 4.3.5 released on Feb 25, 2019.
  more people will be hit by this soon.

  does newer pip have the problem? yes.
  using setuptools 28.8.0 Nov 2016

  after upgrading setuptools and pip, still have problem installing latest
  isort pkg.

- reproduce
  mkdir ~/d/t3
  cd ~/d/t3
  virtualenv .venv
  .venv/bin/pip install isort
  .venv/bin/pip install --no-cache-dir isort
  .venv/bin/pip uninstall isort

  this fails on python 3.5.3 on debian stretch.
  I think the virtualenv is too old.
  python3 -m venv .venv
  works fine on stretch live distro.

  but python3 -m venv also fails on ryzen5. why?

  It may be my local environment's problem. God know which component's problem.

  try use no cache.
  no cache worked!

- how to removed the cached version of isort?

  sylecn@ryzen5:~/d/t3$ du -sh ~/.cache/pip
  239M	/home/sylecn/.cache/pip
  sylecn@ryzen5:~/d/t3$ 

- search: python pip cache and platform specific dependencies

- Because it's local cache problem. I will just clear local pip cache. and
  also upgrade local pip.

** 2019-02-27 after I upgraded lein, new lein install new version has an error.
Template resource 'leiningen/new/python/.gitignore' not found.

.gitignore file does not get included into a template · Issue #2522 · technomancy/leiningen · GitHub
https://github.com/technomancy/leiningen/issues/2522

This is a known problem.

** 2019-02-27 add document in project README on how to do CD.
added README-dev.rst

** 2016-07-27 package name should not have -, auto convert it to _.
- package name is part of module name, - is not valid there.

** 2016-09-08 when bootstrap, create /var/log/{{name}} dir using sudo and chown to current user.
this makes it easier when developing.

- just add it in bootstrap-dev.

** DONE 2016-09-08 integrate sanity-check
- update yygame-utils pkg version
- in Makefile
 check: just-test
+sanity-check:
+	$(PYTHON) wechat_article_saver_daemon/sanity_check.py
 check-coding-style: bootstrap-dev

- add the template with a dumb check.

** DONE 2016-06-30 make "lein new python bar" work, without --to-dir .
chmod -R +x utils/

- this is not run on project root dir.

  I see there is a "shell/with-sh-dir *dir*", check what is *dir*.

  search: leiningen new *dir* target dir

  leiningen.new.templates reference
  https://github.com/technomancy/leiningen/blob/master/src/leiningen/new/templates.clj
  it's either pwd or ->files file's path.

  seems it's not the target project dir.

- check how --to-dir work.
  https://github.com/technomancy/leiningen/blob/master/src/leiningen/new.clj

  I think it could be nil when --to-dir is not used.
  yes. it's null when user doesn't use --to-dir option.

** DONE 2016-06-30 use python3 in venv by default.
- make old code work with python3.
  - add future imports

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)

  - pylint fails.

- fallback to python2 if python3 is NA.
  test it on both py2 and py3.

** DONE 2016-06-30 put venv in .venv dir.
- DONE update Makefile
- DONE update ./utils/bootstrap
  use a variable in template.

  no. move bootstrap inside Makefile.

  I will create it from scratch. Just use the old files.

  a recent copy of Makefile is at 

  find /home/sylecn/python/ -name Makefile -mtime -7
  find /home/sylecn/projects/ -name Makefile -mtime -30

  /home/sylecn/projects/molecules/pbsa_jisuan/Makefile
  /home/sylecn/projects/qvm/Makefile
  /home/sylecn/projects/spider/src/python-prototyping/Makefile

  locater '\.venv$'
  DONE /home/sylecn/office/yxy/emeter-elavator-reports/.venv    // also this.
  /home/sylecn/projects/webframework-simple-benchmark/.venv
  DONE /home/sylecn/python/questionsdb/.venv    // also this.
  DONE /home/sylecn/websites/url-alias/.venv    // this is the one I'm looking for.

  - bootstrap. can I use late binding variable to make bootstrap and
    bootstrap-dev work?

    Can I just set a variable and invoke make a 2nd time on a shared target
    that uses that variable?

    well, just define a function in Makefile, pass in parameters.

  - INFO function is a very confusing concept in Makefile.

    the call function
    https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_8.html#SEC81
    This macro simply reverses its arguments:
    reverse = $(2) $(1)
    // and where is macro documented, it also looks like a function.
    Defining Canned Command Sequences
    https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_5.html#SEC55

- DONE can "python3 -m venv" replace virtualenv? no. it can only create venv
  for the python that is being run.

  just use the embedded virtualenv.

- DONE upgrade virtualenv and pip.
  virtualenv 15.0.2, pip 8.1.2 included.
  https://pypi.python.org/pypi/virtualenv
  curl -O https://pypi.python.org/packages/5c/79/5dae7494b9f5ed061cff9a8ab8d6e1f02db352f3facf907d9eb614fb80e9/virtualenv-15.0.2.tar.gz#md5=0ed59863994daf1292827ffdbba80a63

  1.8M tar.gz file.
  old dir: 1.7M
  new dir: 2.1M
  removed some files, down to 1.9M.

- test it now.
  v0.7.0 updated virtualenv, use Makefile do bootstrap, use python3 by default.

- it requires me listing all template and plain files. this is bad. should
  just generate the file list on the fly.

