Skip to content
Commits on Source (2)
Hide whitespace changes
Inline Side-by-side
.gitignore
View file @ 9a66e674
......@@ -8,4 +8,3 @@ pypi/rd-api/rdapi/rd-api
*README.rst
stack.yaml.lock
*.whl
copy-binaries.sh 0 → 100755
View file @ 9a66e674
#!/bin/sh
set -e
VERSION=`grep 'cliVersion =' ./lib/RD/CliVersion.hs | cut -d'"' -f2`
TARGET_DIR=build/reliable-download-$VERSION/
mkdir -p "$TARGET_DIR"
cp -f `stack exec -- which rd-api` "$TARGET_DIR"/
cp -f `stack exec -- which rd` "$TARGET_DIR"/
flake.nix.not-working 0 → 100644
View file @ 9a66e674
{
description = "download large files across unreliable network via HTTP";
inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
inputs.nixpkgs.follows = "haskellNix/nixpkgs-2311";
inputs.flake-utils.url = "github:numtide/flake-utils";
outputs = { self, nixpkgs, flake-utils, haskellNix }:
flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
let
overlays = [ haskellNix.overlay
(final: prev: {
# This overlay adds our project to pkgs
rdProject =
final.haskell-nix.project' {
src = ./.;
compiler-nix-name = "ghc928";
# This is used by `nix develop .` to open a shell for use with
# `cabal`, `hlint` and `haskell-language-server`
shell.tools = {
cabal = {};
hlint = {};
haskell-language-server = {};
};
# # Non-Haskell shell tools go here
shell.buildInputs = with pkgs; [
nixpkgs-fmt
];
# This adds `js-unknown-ghcjs-cabal` to the shell.
# shell.crossPlatforms = p: [p.ghcjs];
};
})
];
pkgs = import nixpkgs { inherit system overlays; inherit (haskellNix) config; };
flake = pkgs.rdProject.flake {};
in flake // {
# Built by `nix build .`
legacyPackages = pkgs;
packages.default = flake.packages."reliable-download:exe:rd-api";
});
# --- Flake Local Nix Configuration ----------------------------
nixConfig = {
# This sets the flake to use the IOG nix cache.
# Nix should ask for permission before using it,
# but remove it here if you do not want it to.
extra-substituters = ["https://cache.iog.io"];
extra-trusted-public-keys = ["hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ="];
allow-import-from-derivation = "true";
};
}
operational
View file @ 9a66e674
* COMMENT -*- mode: org -*-
#+Date: 2018-05-04
Time-stamp: <2024-04-08>
Time-stamp: <2024-04-25>
#+STARTUP: content
* notes :entry:
** 2022-03-14 project dir structure
......@@ -16,9 +16,12 @@ Time-stamp: <2024-04-08>
- update version number in
- lib/RD/CliVersion.hs (required, used by pypi pkg)
- package.yaml (optional)
- build binary
- build binary on an old OS, like debian stretch.
so the built binary can run everywhere.
stack build --test --pedantic
stack exec hlint -- -g
- update README file. add ChangeLog entry on *README.rst
./README.md
./pypi/rd-api/README.rst
......@@ -486,6 +489,108 @@ via env var and command line parameter.
* current :entry:
**
** 2024-04-24 try build this project in gocd.
- build in debian stretch agent node.
- build in docker is fine.
cp the built binary to artifacts dir.
use debian stretch base debian image.
debian stretch is no longer supported. maybe try a supported old OS.
like debian 10 buster or RHEL 7?
Red Hat Universal Base Images for Docker users | Red Hat Developer
https://developers.redhat.com/blog/2020/03/24/red-hat-universal-base-images-for-docker-users#introducing_red_hat_universal_base_images
Life-cycle Dates
https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates
RHEL 7.9 is still supported, until June 30, 2024.
I think I should just use debian 10 buster as a base.
- how to make build reuse as much cache as possible?
cache should be stored on host?
nix is indeed a good match for this scenario. too bad it can't use a
pre-built ghc.
run build on bare metal is best, everything can be debugged and purged as
will. it is not as reproducible though. To build on go-agent, I need to run
go-agent on stretch or buster VM. It is similar to manual build, just with
some automation and I can config it to always build on the same go-agent
node and it can store the built artifacts for me.
I think it's worth it. Just deploy go-agent on stretch02 VM.
- deploy go-agent on stretch02 VM.
check how to specify which files should be copied to artifacts.
I need to run go-agent as sylecn user so I can reuse stack cache and
configuration. so I decide to use the zip file based agent.
Generic Zip | GoCD User Documentation
https://docs.gocd.org/current/installation/install/agent/zip.html
Download zip
https://www.gocd.org/download/#zip
DL to /wh01/share/soft/cache/go-agent-23.5.0-18179.zip
sha256
74cfe036906b5ca246541fa3fe9c20e0ca04f80b27b5f6d03350cfbc1a2b3890
create auto reg config file.
specify go-server URL.
./config/autoregister.properties
#+begin_quote
agent.auto.register.key=67c2fdfe-a9a3-490a-9c65-190fc2258483
agent.auto.register.resources=inside-gfw,stack
agent.auto.register.hostname=stretch02
#+end_quote
./wrapper-config/wrapper-properties.conf
#+begin_quote
wrapper.app.parameter.100=-serverUrl
wrapper.app.parameter.101=https://go.emacsos.com/go
# env variables
set.AGENT_STARTUP_ARGS=-Xms256m -Xmx1024m
#+end_quote
install openjdk 17. use nix? no. just use amazon corretto deb.
./bin/go-agent start
it works perfectly. agent is detected, resources is auto added.
- now config pipeline for reliable-download.
allow go-server access my git repos.
using deploy read-only key.
what's the path of the built binary?
/home/sylecn/projects/reliable-download/.stack-work/install/x86_64-linux-tinfo6/21acf9206f416a1e336a58fea37773f98a7eb66bb7c93dff696534ca35928f26/9.2.8/bin/rd-api
/home/sylecn/projects/reliable-download/.stack-work/install/x86_64-linux-tinfo6/21acf9206f416a1e336a58fea37773f98a7eb66bb7c93dff696534ca35928f26/9.2.8/bin/rd
is this path stable? nope.
s02 path is different. it has no -tinfo and a different hash.
probably different stack version. yes. s02 stack 2.15.5, agem10 stack 2.13.1
after upgrading stack. path is still not modified.
well, I need to cp the files in gocd build stage. using scripts.
*now*
- problems
- git clone hangs.
network issue?
https://gitlab.emacsos.com/sylecn/reliable-download
this URL clone okay.
git@gitlab.emacsos.com:sylecn/reliable-download.git
this URL clone fail.
why? probably ssh config issue.
is the home dir properly mount?
go-server uses jgit?
/wh01/share/apps/go-server/home/.config/jgit/config
search: gocd jgit how to set StrictHostKeyChecking=no
** 2024-04-07 when combining blocks to create final file, don't print
"No block fetched in last 10 seconds" log any more if there is no other files
in DL list.
......@@ -2307,6 +2412,145 @@ I can use haskell fork though.
yes. see Route Patterns.
* wontfix :entry:
** 2024-04-24 try build this project using nix with flake.
- create flake.nix.
https://www.tweag.io/blog/2020-05-25-flakes/
- how to build haskell project using nix flake
Getting started with Flakes - Alternative Haskell Infrastructure for Nixpkgs
https://input-output-hk.github.io/haskell.nix/tutorials/getting-started-flakes.html
Introduction - Alternative Haskell Infrastructure for Nixpkgs
https://input-output-hk.github.io/haskell.nix/index.html
Motivation - Alternative Haskell Infrastructure for Nixpkgs
https://input-output-hk.github.io/haskell.nix/motivation.html
cable2nix
stack2nix
//motivation seems good.
nix profile install nixpkgs#nix-tools
//no package.
cabal2nix pkg is in stackage. this is not from nix-tools.
how to install nix-tools?
haskell.nix/nix-tools at master · input-output-hk/haskell.nix
https://github.com/input-output-hk/haskell.nix/tree/master/nix-tools
there is a flake.nix file in git repo.
just keep reading.
- the other solution is
Haskell Support for Nix — haskell4nix 1 documentation
https://haskell4nix.readthedocs.io/
this one has no flake support.
*** how to build my project using haskell.nix?
- Getting started with Flakes - Alternative Haskell Infrastructure for Nixpkgs
https://input-output-hk.github.io/haskell.nix/tutorials/getting-started-flakes.html
- add cache store and public key.
on agem10,
salthere nix
- TODO if this works, integrate nix build with ucompile.
when flake.nix exists, use nix build.
emacs IDE support
https://input-output-hk.github.io/haskell.nix/tutorials/development.html#emacs-ide-support
nix build .#reliable-download:test
nix build .#reliable-download:test:all-tests
2024-04-24 21:32:04 start
2024-04-24 21:33:35 it's quite fast to fail. no such target.
nix flake metadata .
this is cool.
it only defined inputs, no outputs?
nix build .
this works. it starts to fetch many files.
it tries to build ghc-8.10.7, I don't want that.
shouldn't there be a cache?
comment out compiler-nix-name didn't work.
too steep learning curve.
https://input-output-hk.github.io/haskell.nix/troubleshooting.html#why-am-i-building-ghc
okay. this is some document. good.
choose a ghc version that is tested in CI.
See ci.nix for the source of truth about what is built and tested.
for nixpkg unstable, ghc810, ghc92, ghc94, ghc96, ghc98 are all supported?
I can't understand ci.nix file.
https://github.com/input-output-hk/haskell.nix/blob/master/ci.nix
it still tries to build ghc 8.10.7.
try remove .lock file and rebuild.
nope. it still tries to build ghc 8.10.7.
try use 2311
inputs.nixpkgs.follows = "haskellNix/nixpkgs-2311";
now it tries to fetch different dependencies from store cache.
now it's fetching ghc-8.10.7 from sjtug store.
then it continues build ghc 8.10.7 again. from source. WTF.
// This will never work. I gave up.
nix build .#reliable-download:lib:reliable-download
nix build .#reliable-download:exe:rd-api
nix build .#reliable-download:exe:rd
- how to define my tests and exe in flake.nix?
- get sha256 for git dependency.
nix-prefetch-git https://gitlab.emacsos.com/sylecn/io-thread-pool.git 58041a07560383bac22f6702074242c3e2097106
nix-prefetch-git https://gitlab.emacsos.com/sylecn/tinylog.git a49750ffb2923d40e834cec77a9900b4167c2ea5
where can I find nix-prefetch-git tool?
Nix Haskell Development (2020) - Guides - NixOS Discourse
https://discourse.nixos.org/t/nix-haskell-development-2020/6170
nix profile install nixpkgs#nix-prefetch-git
it works.
- problems
- build fail.
#+begin_quote
...
warning: Git tree '/home/sylecn/projects/reliable-download' is dirty
error: flake 'git+file:///home/sylecn/projects/reliable-download' does not provide attribute 'packages.x86_64-linux.reliable-download:test', 'legacyPackages.x86_64-linux.reliable-download:test' or 'reliable-download:test'
sylecn@agem10:~/projects/reliable-download$ nix build .#reliable-download:test:all-tests
warning: Git tree '/home/sylecn/projects/reliable-download' is dirty
error: flake 'git+file:///home/sylecn/projects/reliable-download' does not provide attribute 'packages.x86_64-linux.reliable-download:test:all-tests', 'legacyPackages.x86_64-linux.reliable-download:test:all-tests' or 'reliable-download:test:all-tests'
sylecn@agem10:~/projects/reliable-download$
#+end_quote
#+begin_quote
sylecn@agem10:~/projects/reliable-download$ nix build .#reliable-download:exec:rd-api
error: flake 'git+file:///home/sylecn/projects/reliable-download' does not provide attribute 'packages.x86_64-linux.reliable-download:exec:rd-api', 'legacyPackages.x86_64-linux.reliable-download:exec:rd-api' or 'reliable-download:exec:rd-api'
#+end_quote
try find some full examples.
hello world is not enough to see what's wrong in flake.nix.
gh search: haskellNix path:flake.nix language:haskell
this search is not precise enough.
https://github.com/jgm/pandoc/blob/main/flake.nix
this is using raw nix. without much help.
you need to duplicate every dependencies already in stack package.yaml file.
- is nix good for haskell project?
What's all the hype with Nix? - Learn - Haskell Community
https://discourse.haskell.org/t/whats-all-the-hype-with-nix/2593/18
** 2022-03-14 can I make tinylog log timestamp use localtime?
- L.DateFormat
doesn't allow IO.
......
stack.yaml
View file @ 9a66e674
......@@ -28,8 +28,10 @@ packages:
extra-deps:
- git: https://gitlab.emacsos.com/sylecn/io-thread-pool.git
commit: 58041a07560383bac22f6702074242c3e2097106
# nix-sha256: 04kj3bfn17mv02bs8ldfa37s52fq2ycqwjcdg4hyns1syy1csw0l
- git: https://gitlab.emacsos.com/sylecn/tinylog.git
commit: a49750ffb2923d40e834cec77a9900b4167c2ea5
# nix-sha256: 1h21xckxd4ls8vinivsh5ibmjpccijc8nhh9zgqxlnvq56x62f7f
# Override default flag values for local packages and extra-deps
# flags: {}
......