operational

* COMMENT -*- mode: org -*-
#+Date: 2014-05-09
Time-stamp: <2021-12-06>
#+STARTUP: content
* notes								      :entry:
** 2021-12-06 how to create a new release?
make deploy
insert yubikey
enter yubikey OpenPGP PIN.

** 2018-05-02 support test, build and deploy in gocd
- stage 1: testAndBuild
  make ci-build

  required env var:
  DOCKER_USER
  DOCKER_PASSWORD
  docker login info for private registry.

- stage 2: deploy
  make deploy

  required env var:
  KUBECTL_API
  KUBECTL_TOKEN
  kubectl auth info for running kubectl apply.

- implemented in v1.2.0

  in gocd, use the python-app pipeline template to automate stages and jobs.

** 2018-05-02 workflow for v1.2.0+
- create project on dev node.
- create gitlab repo by init-gitlab
- dev on dev node.
  make test
  make run
- when ready to push to production.
  update deb-scripts if necessary
  update ./utils/build-deb if necessary
  update app.yaml if necessary
- on gitlab, add gocd as deploy key
- on gocd, create new pipeline for the project.
  use python-app template.

  // these two step should be automated.

- on dev node, git push
  now you will see gocd start testing, building docker image and push it.
  when first stage is green, click approval to deploy to production.
- to upgrade project, make commits, and push to gitlab, the rest will follow.
  click approval to deploy to production.

** 2020-01-29 venv and wheelhouse cache used in "make ci-build"		:doc:
- dir layout

  on go-agent node,
  ~/.cache/ci_build/$(GO_PIPELINE_NAME)/cache                            pip cache
  ~/.cache/ci_build/$(GO_PIPELINE_NAME)/venvs-$(BUILD_PY_VERSION)        venv dir
  ~/.cache/ci_build/$(GO_PIPELINE_NAME)/wheelhouses-$(BUILD_PY_VERSION)  wheelhouse dir

  on docker container,
  host dir                           container mount point
  ==========================================================
  cache                              /.cache
  venvs/venv-<hash>                  /app/.venv
  wheelhouses/wheelhouse-<hash>      /app/wheelhouse

* later								      :entry:
** 2019-02-28 about the two step CD, should I make it 3 step build?
1. test
2. build production docker image
3. on user approve, deploy to production.

currently 1 and 2 are run in make ci-build.
check the disk space usage.
sometimes, trivial doc change doesn't require a docker build.
sometimes, version change doesn't require a docker build because of new
feature is still WIP and I won't deploy it.

- check the disk space usage.
  base stretch slim is 55MiB.
  app image is additional 55MiB.
  the apt-get step 34MiB, unpack deb 21MiB.

  seems reasonable. Could save 34MiB by including python3, libpq in the base
  image.

- does kubernetes remove old images? yes.

  The default is when disk space usage is over 85%, it will clean image until
  it hits 80%.
  https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

- 2019-02-28 you can config blacklist in add/edit material.
  https://docs.gocd.org/current/configuration/admin_add_material.html#blacklist

  https://docs.gocd.org/current/configuration/configuration_reference.html#ignore

  Often you do want to specify a set of files that GoCD should ignore when it
  checks for changes. Repository changesets which contain only these files
  will not automatically trigger a pipeline. These are detailed in the ignore
  section of the configuration reference.

  ignore for this project:
  #+BEGIN_SRC conf
    .gitignore,LICENSE,README.md,operational,src/leiningen/new/python/README.rst,src/leiningen/new/python/README-dev.rst,src/leiningen/new/python/gitignore
  #+END_SRC

* current							      :entry:
** 
** 2021-08-25 some env variable should be passed down to docker container.
for example, -e SQLALCHEMY_URL is essential for yxy-wechat.

predefine PASS_DOWN_ENV_VARIABLES maybe.

** 2021-08-16 support upgrading existing application's Makefile.
- when run in existing dir, don't init git.
  just create file, let user review and do merge.
** 2021-08-25 how to make sure pip is latest without pip install -U at "make bootstrap" time?
how to not upgrade pip if it's already 19.0+?
can I use Makefile to manage the pip lib version?

** 2021-08-24 switch to a better build system.
Makefile, scons, meson, bazel.

Comparisons
https://mesonbuild.com/Comparisons.html

- features
  - easy to write build file.
    human readable.
    with proper abstraction.

    - build system must be extensible.

      user can build any kind of projects. user just need to describe how
      files are related and which command can build which file.

    - easy to update existing projects when build template (p-p-t) has updated.

      essentially there should be abstraction so that only project related
      value is listed in user build file.
      other build logic should be abstracted.

  - build fast.
    do not do unnecessary checks.
    for example, when no .py file or python venv change, pylint should not run
    if there is last run cache.
    when a library exists in venv, do not install it again.
    when all requirements have wheels, do not try to build it again.
    do not regenerate deb if nothing has changed.

    side note: I don't think I need deb when building using docker.
    install via deb just slow down the build and doesn't provide much value.

** 2019-02-28 add support for running application test using docker-compose.
see ~/websites/url-alias project for an example.

add docker-compose.yml
update Makfile

** 2016-09-06 support adding celery to existing project.
- This will add a celeryconfig module and a tasks module.
  see examples in
  ~/projects/wechat-article-saver/wechat_article_saver_daemon/wechat_article_saver_daemon/celeryconfig.py
  ~/projects/wechat-article-saver/wechat_article_saver_daemon/wechat_article_saver_daemon/fetcher.py
- I don't know how to call lein from existing project yet.
  shouldn't be so hard.

** 2016-08-18 integrate tox.
let "make test" use the default python in venv, but allow "make tox" to run
unit tests against defined python in tox.ini.

** 2016-08-07 in python project template, add a project wide pylintrc file.
see example ~/.pylintrc
** 2014-05-09 add sphinx support

* done								      :entry:
** 2021-12-06 can't release new version
- make deploy fail.

  #+BEGIN_SRC sh
    sylecn@ryzen5:~/projects/python-project-template$ make deploy
    env https_proxy=127.0.0.1:8123 lein deploy clojars
    gpg: decryption failed: No secret key
    Could not decrypt credentials from /home/sylecn/.lein/credentials.clj.gpg
    gpg: decryption failed: No secret key
  
    See `lein help gpg` for how to install gpg.
    No credentials found for clojars
    See `lein help deploying` for how to configure credentials to avoid prompts.
  #+END_SRC
  
  sylecn@ryzen5:~/projects/python-project-template$ file /home/sylecn/.lein/credentials.clj.gpg
  /home/sylecn/.lein/credentials.clj.gpg: PGP RSA encrypted session key - keyid: AEE33BDD 64D4C390 RSA (Encrypt or Sign) 2048b .

- check the key used to encrypt ~/.lein/credentials.clj.gpg

  gpg -d /home/sylecn/.lein/credentials.clj.gpg

  gpg: encrypted with 2048-bit RSA key, ID AEE33BDD64D4C390, created 2011-12-25
      "Yuanle Song <sylecn@gmail.com>"

  is my private key moved to ubikey? yes. protected via OpenPGP PIN.

** 2021-08-05 update pip/venv cache to include python version.
- see ~/projects/yxy-kucun/Makefile
  BUILD_DOCKER_IMAGE_PY_VERSION
- also integrate other changes in that Makefile.

** 2021-08-05 update build image tag and run image tag.
- DONE use debian 10 buster based images.
- DONE use system python3 to bootstrap venv.
  DONE add python3-pip etc to built deb dependencies.

** 2021-08-14 use python3 venv module instead of standalone virtualenv.
** 2019-02-24 fix :url in project.clj
create a blog post about this template.
** 2019-03-01 switch to apache license 2.0
GitHub - xsc/lein-license: Project-Level License Management
https://github.com/xsc/lein-license

lein license list
lein license update apache-2.0
lein license update

It does't update copyright notice in LICENSE file.
I will just use it once.

- problems
  - is there a clojars mirror I can use for faster download?

** 2016-06-30 update it for use with default debian config. (non-yy env)
- update nginx config file
- add support for systemd
- create /data if it doesn't exist. auto use sudo to create the dir if it
  doesn't exist.

  make sure it works on fresh system. use a VM to test it.
** 2019-02-28 add gocd support.
// was add trivis-ci support.
- does trivis-ci send email when build failed?
- just create new project on python2.7, python3.5, pypy env.
  then run "make test"
- trivis-ci only support github now.
  GitLab integration · Issue #5931 · travis-ci/travis-ci · GitHub
  https://github.com/travis-ci/travis-ci/issues/5931
- maybe add gocd support for it instead.

** 2019-02-24 drop :repositories from project.clj
** 2019-02-27 another problem on new created project.
make test fail on new project.

futures requires Python '>=2.6, <3' but the running Python is 3.5.3

one pkg in requirements-dev.txt has bad dependencies.

pylint has problem.

#+BEGIN_SRC sh
  Collecting futures (from isort>=4.2.5->pylint)
    Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1f/9e/7b2ff7e965fc654592269f2906ade1c7d705f1bf25b7d469fa153f7d19eb/futures-3.2.0.tar.gz
  futures requires Python '>=2.6, <3' but the running Python is 3.5.3
#+END_SRC

- search: pylint futures requires Python '>=2.6, <3' but the running Python is 3.5.3
  search: isort futures broken

- try fix isort to older version.
  isort is used to sort import statements. it has no dependencies.

  what does that collect line mean?

  // isort does have dependencies. I fixed isort to 4.2.5 and it works.

  https://github.com/timothycrosley/isort/blob/4.3.9/setup.py
      install_requires=['futures; python_version < "3.2"'],
  it has required futures pkg.

  the setup.py syntax changed in v4.3.5
  v4.3.4 should work fine.

  old syntax:
  install_requires = []
  if sys.version_info.major == 2:
    install_requires = ['futures']
  ...
      install_requires=install_requires,

  new syntax:
        install_requires=['futures; python_version < "3.2"'],

  the new syntax doc:

  Building and Distributing Packages with Setuptools — setuptools 40.8.0 documentation
  https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies
  Declaring platform specific dependencies
  PEP 508 -- Dependency specification for Python Software Packages | Python.org
  https://www.python.org/dev/peps/pep-0508/
  Created:	11-Nov-2015

  which setuptools version start to support this feature?

  Why I hit this problem today? This is old stuff.
  isort 4.3.5 released on Feb 25, 2019.
  more people will be hit by this soon.

  does newer pip have the problem? yes.
  using setuptools 28.8.0 Nov 2016

  after upgrading setuptools and pip, still have problem installing latest
  isort pkg.

- reproduce
  mkdir ~/d/t3
  cd ~/d/t3
  virtualenv .venv
  .venv/bin/pip install isort
  .venv/bin/pip install --no-cache-dir isort
  .venv/bin/pip uninstall isort

  this fails on python 3.5.3 on debian stretch.
  I think the virtualenv is too old.
  python3 -m venv .venv
  works fine on stretch live distro.

  but python3 -m venv also fails on ryzen5. why?

  It may be my local environment's problem. God know which component's problem.

  try use no cache.
  no cache worked!

- how to removed the cached version of isort?

  sylecn@ryzen5:~/d/t3$ du -sh ~/.cache/pip
  239M	/home/sylecn/.cache/pip
  sylecn@ryzen5:~/d/t3$ 

- search: python pip cache and platform specific dependencies

- Because it's local cache problem. I will just clear local pip cache. and
  also upgrade local pip.

** 2019-02-27 after I upgraded lein, new lein install new version has an error.
Template resource 'leiningen/new/python/.gitignore' not found.

.gitignore file does not get included into a template · Issue #2522 · technomancy/leiningen · GitHub
https://github.com/technomancy/leiningen/issues/2522

This is a known problem.

** 2019-02-27 add document in project README on how to do CD.
added README-dev.rst

** 2016-07-27 package name should not have -, auto convert it to _.
- package name is part of module name, - is not valid there.

** 2016-09-08 when bootstrap, create /var/log/{{name}} dir using sudo and chown to current user.
this makes it easier when developing.

- just add it in bootstrap-dev.

** DONE 2016-09-08 integrate sanity-check
- update yygame-utils pkg version
- in Makefile
 check: just-test
+sanity-check:
+	$(PYTHON) wechat_article_saver_daemon/sanity_check.py
 check-coding-style: bootstrap-dev

- add the template with a dumb check.

** DONE 2016-06-30 make "lein new python bar" work, without --to-dir .
chmod -R +x utils/

- this is not run on project root dir.

  I see there is a "shell/with-sh-dir *dir*", check what is *dir*.

  search: leiningen new *dir* target dir

  leiningen.new.templates reference
  https://github.com/technomancy/leiningen/blob/master/src/leiningen/new/templates.clj
  it's either pwd or ->files file's path.

  seems it's not the target project dir.

- check how --to-dir work.
  https://github.com/technomancy/leiningen/blob/master/src/leiningen/new.clj

  I think it could be nil when --to-dir is not used.
  yes. it's null when user doesn't use --to-dir option.

** DONE 2016-06-30 use python3 in venv by default.
- make old code work with python3.
  - add future imports

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)

  - pylint fails.

- fallback to python2 if python3 is NA.
  test it on both py2 and py3.

** DONE 2016-06-30 put venv in .venv dir.
- DONE update Makefile
- DONE update ./utils/bootstrap
  use a variable in template.

  no. move bootstrap inside Makefile.

  I will create it from scratch. Just use the old files.

  a recent copy of Makefile is at 

  find /home/sylecn/python/ -name Makefile -mtime -7
  find /home/sylecn/projects/ -name Makefile -mtime -30

  /home/sylecn/projects/molecules/pbsa_jisuan/Makefile
  /home/sylecn/projects/qvm/Makefile
  /home/sylecn/projects/spider/src/python-prototyping/Makefile

  locater '\.venv$'
  DONE /home/sylecn/office/yxy/emeter-elavator-reports/.venv    // also this.
  /home/sylecn/projects/webframework-simple-benchmark/.venv
  DONE /home/sylecn/python/questionsdb/.venv    // also this.
  DONE /home/sylecn/websites/url-alias/.venv    // this is the one I'm looking for.

  - bootstrap. can I use late binding variable to make bootstrap and
    bootstrap-dev work?

    Can I just set a variable and invoke make a 2nd time on a shared target
    that uses that variable?

    well, just define a function in Makefile, pass in parameters.

  - INFO function is a very confusing concept in Makefile.

    the call function
    https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_8.html#SEC81
    This macro simply reverses its arguments:
    reverse = $(2) $(1)
    // and where is macro documented, it also looks like a function.
    Defining Canned Command Sequences
    https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_5.html#SEC55

- DONE can "python3 -m venv" replace virtualenv? no. it can only create venv
  for the python that is being run.

  just use the embedded virtualenv.

- DONE upgrade virtualenv and pip.
  virtualenv 15.0.2, pip 8.1.2 included.
  https://pypi.python.org/pypi/virtualenv
  curl -O https://pypi.python.org/packages/5c/79/5dae7494b9f5ed061cff9a8ab8d6e1f02db352f3facf907d9eb614fb80e9/virtualenv-15.0.2.tar.gz#md5=0ed59863994daf1292827ffdbba80a63

  1.8M tar.gz file.
  old dir: 1.7M
  new dir: 2.1M
  removed some files, down to 1.9M.

- test it now.
  v0.7.0 updated virtualenv, use Makefile do bootstrap, use python3 by default.

- it requires me listing all template and plain files. this is bad. should
  just generate the file list on the fly.

* obsolete                                                            :entry:
** 2019-02-27 how to update virtualenv in utils/ dir
- update files
  VERSION=16.4.1
  tar tf ~/d/virtualenv-$VERSION.tar.gz
  cd src/leiningen/new/python/utils/
  tar xzf ~/d/virtualenv-$VERSION.tar.gz \
    virtualenv-$VERSION/virtualenv.py \
    virtualenv-$VERSION/virtualenv_support/ \
    virtualenv-$VERSION/LICENSE.txt
- update file list in ./src/leiningen/new/python.clj
  WONTFIX update the code to auto include all files in utils/virtualenv-* dir.
* wontfix							      :entry:
** 2016-09-08 support updating site file, deb scripts etc after project is generated.
- the domain in site file could change.
- the pkg name could change.
- whether to use uwsgi could change.

./utils/update_site_file.py --domain foo.emacsos.com --uwsgi
./utils/update_site_file.py --domain foo.emacsos.com --http

** 2020-01-02 failed to deploy new version to clojars.
first deploy has a HTTP timeout.
send deploy show redeploying non-snapshots is not allowed

#+BEGIN_SRC sh
  sylecn@ryzen5:~/projects/python-project-template$ make deploy
  lein deploy clojars
  Created /home/sylecn/projects/python-project-template/target/lein-template-1.9.1.jar
  Wrote /home/sylecn/projects/python-project-template/pom.xml
  Need to sign 2 files with GPG
  [1/2] Signing /home/sylecn/projects/python-project-template/target/lein-template-1.9.1.jar with GPG
  gpg: using "4EB55E3F" as default secret key for signing
  [2/2] Signing /home/sylecn/projects/python-project-template/pom.xml with GPG
  gpg: using "4EB55E3F" as default secret key for signing
  Sending python/lein-template/1.9.1/lein-template-1.9.1.jar (1937k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.pom (2k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.jar.asc (1k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.pom.asc (1k)
      to https://clojars.org/repo/
  Retrieving python/lein-template/maven-metadata.xml 
      from https://clojars.org/repo/
  Sending python/lein-template/maven-metadata.xml (1k)
      to https://clojars.org/repo/
  Could not transfer metadata python:lein-template/maven-metadata.xml from/to clojars (https://clojars.org/repo/): Read timed out
  Failed to deploy metadata: Could not transfer metadata python:lein-template/maven-metadata.xml from/to clojars (https://clojars.org/repo/): Read timed out
  Makefile:18: recipe for target 'deploy' failed
  make: *** [deploy] Error 1
  sylecn@ryzen5:~/projects/python-project-template$ make deploy
  lein deploy clojars
  Created /home/sylecn/projects/python-project-template/target/lein-template-1.9.1.jar
  Wrote /home/sylecn/projects/python-project-template/pom.xml
  Need to sign 2 files with GPG
  [1/2] Signing /home/sylecn/projects/python-project-template/target/lein-template-1.9.1.jar with GPG
  gpg: using "4EB55E3F" as default secret key for signing
  [2/2] Signing /home/sylecn/projects/python-project-template/pom.xml with GPG
  gpg: using "4EB55E3F" as default secret key for signing
  Sending python/lein-template/1.9.1/lein-template-1.9.1.jar (1937k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.pom (2k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.jar.asc (1k)
      to https://clojars.org/repo/
  Sending python/lein-template/1.9.1/lein-template-1.9.1.pom.asc (1k)
      to https://clojars.org/repo/
  Retrieving python/lein-template/maven-metadata.xml 
      from https://clojars.org/repo/
  Sending python/lein-template/maven-metadata.xml (1k)
      to https://clojars.org/repo/
  Could not transfer metadata python:lein-template/maven-metadata.xml from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/python/lein-template/maven-metadata.xml, ReasonPhrase: Forbidden - redeploying non-snapshots is not allowed (see https://git.io/v1IAs).
  Failed to deploy metadata: Could not transfer metadata python:lein-template/maven-metadata.xml from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/python/lein-template/maven-metadata.xml, ReasonPhrase: Forbidden - redeploying non-snapshots is not allowed (see https://git.io/v1IAs).
  Makefile:18: recipe for target 'deploy' failed
  make: *** [deploy] Error 1
  sylecn@ryzen5:~/projects/python-project-template$ 
#+END_SRC

- next time, use a https proxy when running make deploy.
- search: lein deploy clojars timeout on first try redeploying non-snapshots not allowed on second try
- search: lein deploy half-deployed because of network error
- search: lein deploy fix half deployed pom

  can't find anything.

- I will just deploy a new version when a new release is due.