karajo changelog

I prefer zero configuration rather that creating exclusions, like "revive.toml" file that we have earlier, even thought it will cause breaking changes to our APIs.

Breaking changes,

In the JobBase we add method Cancel to cancel running JobExec or JobHTTP.

In the HTTP server, we add endpoint "POST /karajo/api/job_exec/cancel" to cancel JobExec by its ID.

Implements: https://todo.sr.ht/~shulhan/karajo/1

This is to make the [JobBase.NextRun] always set to next interval or schedule.

Fixes: https://todo.sr.ht/~shulhan/karajo/2

Previously, the JobHttp use single file for log with limited size. The way it works is quite complex, we need to maintain one file and a buffer that able to truncate the log if its reached its max size. We also need to store the job state in separate file.

In this changes, we replace the JobHttp log mechanism to use the same as Job, where each execution will be log separately. Not only this give us less complex code, it also remove some duplicate code.

In the HTTP API we changes the path to get the JobHttp log to match with Job path.

Previously, in the HTTP endpoint for running a job, we return HTTP status code 202 (Accepted) on success, but in the WUI we check using 200 (OK).

This changes fix this by return HTTP status code 200 to make it consistent with other endpoints.

The goal is to minimize response size in API get environment. Imagine we have 10 jobs with 10 logs each, where each log may contains ~10KB in size, in total we will return 10*10*10 KB in response body.

To minimze this, return the log content only when requested through API to get Job or JobHttp Log.

The karajo status page now moved to /karajo/app/, while the old /karajo/ page is used for login.

The login page will be display only if Environment.Users is not empty, otherwise user will be redirected to app page automatically.

Sometimes the request to /karajo/api/environment does not return any result. The only explanation is something lock the resource so we cannot lock it and it will wait forever.

Previously, a job run using the following flow:

If the job triggered from HTTP request, it will run on its own goroutine.

This changes add third channel, startq, to JobBase that queue the Job. When the timer kicking in or HTTP request received in it will pushed to startq. The startq execute the job and signal the completed job using finishq.

Previously, the API path to execute Job is "/karajo/job/$job_path". This may become a conflict in the future (if we want to serve any information related to job in specific page) and inconsistent API path.

This changes the API to execute job to "/karajo/api/job/run/$job_path".

Unlike using interval, the Scheduler option is more flexible and more humanly. For example, one can run job every day at 10:00 AM using

A job can be triggered from external by sending HTTP POST request to the Job’s Path. Each request is authorized based on the AuthKind and optional Secret.

Supported AuthKind are,

Previously, all JobHttp configuration must be defined in single configuration, karajo.conf.

This changes make karajo configuration more manageable by loading JobHttp configuration from all files under directory $DirBase/etc/karajo/job_http.d as long as the file suffix is ".conf".

Previously, all job configuration must be defined in single configuration, karajo.conf.

This changes make karajo configuration more manageable by loading jobs configuration from all files under directory $DirBase/etc/karajo/job.d as long as the file suffix is ".conf".

The HTTP API have the following signature

Where id is the job ID to be resumed.

The HTTP API have the following signature

Where id is the job ID to be paused.

Previously, Hook can be triggered by sending HTTP POST request to karajo server. In most cases we create JobHttp to trigger it, so we need to define one hook and one JobHttp.

To simplify it, we add an Interval to Hook that works similar to JobHttp so now we only need to create single Hook.

Running make install will run commands to install required files to run karajo in GNU/Linux with systemd. The karajo service is installed but not enabled nor running automatically.

To uninstall run make uninstall.

This changes the package function in _AUR package to use make install instead of define each commands to minimize duplication.

Examining build.kilabit.info/karajo, both of those APIs return a large amount of data (> 400KB) which cause some delay when received on slow network.

This changes compress the returned body as gzip which decrease the size of output to 90% (40-60KB).

Now that configuration and directory structure stable, we set the default DirBase to "/".

This is also to allow packaging karajo into OS package.

There are two paths where Job.execute is called. One from handleHttp and one from Start. The one from handleHttp already check if job is paused before calling execute. If we check again inside execute then that means we doing it twice.

To fix this we move the check to Start method and set the Status as started before it.

In the Environment, we add field MaxHookRunning that defined maximum hook running at the same time.

This field is optional, default to 1.

While at it, clean up the logs format to make the console output more readable.

Without setting the PATH, any command that use sudo will return an error "command not found".

The current PATH values is derived from default PATH after bootstraping with base-devel.

Previously, hook write the HTTP response after the Call or all of the Commands are finish. If the Hook run longer than, say 5 seconds, this may cause the request that trigger the hook return with timeout.

In this changes, once we receive the request to trigger the Hook and when the signature is valid, we return with HTTP status 200 immediately and run the Hook job in the other goroutine.

The Status is set to "started" so the interface can display different color.

On Job user interface, if the NextRun is less than now, it will show text "Running…​".

On Hook, set the LastRun to zero time before running, so the WUI can show status as "Running…​".

To test it, we add random sleep on Hooks in testdata.

The Hook last running time is derived from the last log and after the Hook is finished running, either sucess or fail.

On the WUI, the last run is displayed next to the Hook name.

To minimize expanding the Job, display the next Job running time right after the Job name in the following format

"Next run in ${hours}h ${minutes}m ${seconds}s"

Previously, the job section is defined using [karajo "job"], while hook section is defined as [hook "<name>"].

The format on hook section is more friendly and short. So, to make it consistent we changes the job format to match with hook. The job section now become [job "<name>"].

Changes,

This changes affect the HTTP API for pausing and resuming the job to pass the job ID as query instead on path.

The Job log now stored under Environment.dirLogJob + job.ID.

The Job state is now split into separate struct jobState that contains last run time and status.

The Job state now saved under Environment.dirRunJob + job.ID instead of saving all jobs using gob in one file. The Job state is stored as text that can read and edited by human.

The Job IsPausing field is removed because its duplicate with Job Status.

This changes remove DirLogs and add DirBase or ini file set under karajo section with option dir_base.

The DirBase option define the base directory where configuration, job state, and log stored. This field is optional, default to current directory. The structure of directory follow the UNIX system,

Each job log stored under directory /var/log/karajo/job and the job state under directory /var/run/karajo/job.

In the Environment we add field DirPublic that define a path to serve to public.

While the WUI is served under "/karajo", a directory dir_public will be served under "/". A dir_public can contains sub directory as long as its name is not "karajo".

In the configuration file, the DirPublic is set under "karajo::dir_public" option.

The Environment now have field Secret that contains secret to check the signature from HTTP API for pausing and resuming Job.

This require adding input field on the WUI to input the secret, generate signature, and pass it on each request for Job pause and resume.

Hook is the HTTP endpoint that run a function or list of commands upon receiving request, a reverse of what a Job.

Each Hook contains Secret for authenticating request, a working directory, and a callback or list of commands to be executed when the request received.

The circle is now complete!

The Secret field (or "secret" option) define a string to sign the request query or body with HMAC+SHA-256. The signature is sent on HTTP header "x-karajo-sign" as hex string. This field is optional.

The HttpMethod field (or http_method in configuration) set the HTTP method in request. Its accept only GET, POST, PUT, or DELETE. This field is optional, default to GET if its empty.

The HttpRequestType field (or http_request_type in configuration) define the HTTP request type. Its accept only,

The type "form" and "json" only applicable if the method is POST or PUT. This field is optional, default to query.

Instead of refreshing only Jobs and its log when its opened, re-fetch the environment (that include Hooks and Jobs) and render them every 10 seconds.

Each Job execution send the parameter named _karajo_epoch with value is current server Unix time.

If the request type is query then the parameter is inside the query URL. If the request type is form then the parameter is inside the body. If the request type is json then the parameter is inside the body as JSON object, for example {"_karajo_epoch":1656750073}.

Upon started the Job log will be filled with the last logs. Currently, its read 2048 bytes from the end of log file.

The test-random hook will execute command:

Sometimes it will fail and sometimes it will success. This will allow us to check the user interface for multiple status on one hook or log.

The NormalizeForID replace white spaces non ASCII letters, digits, '-', '' with ''.

The documentation is the same with README but formatted using asciidoc.