Failed to parse, compile or execute workflow:

Hi apeerians,

I’m just trying to get started in using apeer. I successfully built a simple java/maven module. However, I fail testing it. The error log is:

[2019-08-25 10:41:11,174][INFO][buddy] Preparing workflow...
[2019-08-25 10:41:11,479][INFO][buddy] Compiling workflow...
[2019-08-25 10:41:11,488][INFO][buddy] Running workflow...
[2019-08-25 10:41:11,488][INFO][buddy] Runtime.availableProcessors: 4
[2019-08-25 10:41:12,194][INFO][buddy] Running Environment(0,Map(timeout -> 120 minutes, interactive -> false, supportsInteractive -> false),Map(moduleGlobalId -> a74544e3-a4c7-4c4d-b1a1-78b9684829b8, networkMode -> Open, agentPool -> default, flowId -> ecbd72c5-465e-1e10-5bfe-fa85b16d7c27))
[2019-08-25 10:41:12,368][INFO][buddy] Environment variables: Map(WFE_INPUT_JSON -> {"WFE_output_params_file":"wfe_module_params_0_0.json","file":"https://samples.fiji.sc/blobs.png","file_list":[]}, WFE_ENVIRONMENT -> {"tags":{"tenant_id":"1","supportsInteractive":"false","workflow_instance_id":"bc144c93-53aa-436d-b40e-f7e232e11a04","user_id":"fd84a150-2cef-405c-b10a-b4a441261f04","interactive":"false","batch_correlation_id":"00000000-0000-0000-0000-000000000000","default_result_title":"clapeer example #2","flowId":"ecbd72c5-465e-1e10-5bfe-fa85b16d7c27","workflow_id":"df7013aa-ad72-4ea1-9f97-731accc07a09"}})
[2019-08-25 10:41:17,602][INFO][buddy] Module finished with complete
[2019-08-25 10:41:17,901][INFO][buddy] Status received from execution environment: running
[2019-08-25 10:41:18,619][INFO][buddy] Webhook callback triggered with status 202 Accepted
[2019-08-25 10:41:18,624][INFO][buddy] Running Environment(1,Map(timeout -> 120 minutes, interactive -> false, supportsInteractive -> false),Map(moduleGlobalId -> 972050a8-19f4-45d0-bb3c-d36ab05bcc9a, networkMode -> Open, agentPool -> default, flowId -> 16a84d48-6c66-77fb-f5bd-4873c6e8d80b))
[2019-08-25 10:41:18,743][INFO][buddy] Environment variables: Map(WFE_INPUT_JSON -> {"WFE_output_params_file":"wfe_module_params_1_1.json","input_image":"/input/module_0_0/blobs.png"}, WFE_ENVIRONMENT -> {"tags":{"tenant_id":"1","supportsInteractive":"false","workflow_instance_id":"bc144c93-53aa-436d-b40e-f7e232e11a04","user_id":"fd84a150-2cef-405c-b10a-b4a441261f04","interactive":"false","batch_correlation_id":"00000000-0000-0000-0000-000000000000","default_result_title":"clapeer example #2","flowId":"16a84d48-6c66-77fb-f5bd-4873c6e8d80b","workflow_id":"df7013aa-ad72-4ea1-9f97-731accc07a09"}})
[2019-08-25 10:41:46,498][INFO][buddy] Module finished with failed
[2019-08-25 10:41:46,629][INFO][buddy] Finished running workflow
[2019-08-25 10:41:46,635][ERROR][buddy] Failed to parse, compile or execute workflow:
io.circe.ParsingFailure: /storage/bc144c93-53aa-436d-b40e-f7e232e11a04/module_1_1/wfe_module_params_1_1.json (No such file or directory)
[2019-08-25 10:41:46,638][INFO][buddy] Shutting down...
[2019-08-25 10:41:46,640][INFO][buddy] Executing cleanup...
[2019-08-25 10:41:46,845][INFO][buddy] Kubator result: WorkflowStatus(List(ModuleStatus(0,0,complete,{"WFE_INPUT_JSON":"{\"WFE_output_params_file\":\"wfe_module_params_0_0.json\",\"file\":\"https://samples.fiji.sc/blobs.png\",\"file_list\":[]}","WFE_ENVIRONMENT":"{\"tags\":{\"tenant_id\":\"1\",\"supportsInteractive\":\"false\",\"workflow_instance_id\":\"bc144c93-53aa-436d-b40e-f7e232e11a04\",\"user_id\":\"fd84a150-2cef-405c-b10a-b4a441261f04\",\"interactive\":\"false\",\"batch_correlation_id\":\"00000000-0000-0000-0000-000000000000\",\"default_result_title\":\"clapeer example #2\",\"flowId\":\"ecbd72c5-465e-1e10-5bfe-fa85b16d7c27\",\"workflow_id\":\"df7013aa-ad72-4ea1-9f97-731accc07a09\"}}"},{"file":"/input/module_0_0/blobs.png","file_list":[]},{"file":"https://sipwfestorageproduction.file.core.windows.net/engine/bc144c93-53aa-436d-b40e-f7e232e11a04/module_0_0/blobs.png?ss=f&sig=AN4jgrXQiK39URJI4JaSZ9Cai93CG%2Bf55iW450NAMF4%3D&se=2019-09-24T10%3A41%3A46Z&sv=2017-04-17&srt=o&sp=r&sr=o","file_list":[]},{},{"imageName":"camodules.azurecr.io/system_define_workflow_inpu_6d2cf9c4-aacd-4db4-b402-c18b1b554e73:16536","serviceName":"module--bc144c93-53aa-436d-b40e-f7e232e11a04--module-0-0","startedAt":"Sun Aug 25 10:41:12 GMT 2019","finishedAt":"Sun Aug 25 10:41:17 GMT 2019","workflow":"bc144c93-53aa-436d-b40e-f7e232e11a04","moduleId":"0","supportsInteractive":"false","interactive":"false","user-id":"1","agentPool":"default","flowId":"ecbd72c5-465e-1e10-5bfe-fa85b16d7c27","networkMode":"Open","stepId":"0","servicePrefix":"module","moduleGlobalId":"a74544e3-a4c7-4c4d-b1a1-78b9684829b8"},Sun Aug 25 10:41:13 GMT 2019,Sun Aug 25 10:41:17 GMT 2019,None,Some(false),Some(false),None,ModuleDetails(None,None)), ModuleStatus(1,1,failed,{"WFE_INPUT_JSON":"{\"WFE_output_params_file\":\"wfe_module_params_1_1.json\",\"input_image\":\"/input/module_0_0/blobs.png\"}","WFE_ENVIRONMENT":"{\"tags\":{\"tenant_id\":\"1\",\"supportsInteractive\":\"false\",\"workflow_instance_id\":\"bc144c93-53aa-436d-b40e-f7e232e11a04\",\"user_id\":\"fd84a150-2cef-405c-b10a-b4a441261f04\",\"interactive\":\"false\",\"batch_correlation_id\":\"00000000-0000-0000-0000-000000000000\",\"default_result_title\":\"clapeer example #2\",\"flowId\":\"16a84d48-6c66-77fb-f5bd-4873c6e8d80b\",\"workflow_id\":\"df7013aa-ad72-4ea1-9f97-731accc07a09\"}}"},{},{},{},{"imageName":"camodules.azurecr.io/test1_cab3f3bd-d64f-48c9-a23c-89f0120c11c8:17234","serviceName":"module--bc144c93-53aa-436d-b40e-f7e232e11a04--module-1-1","startedAt":"Sun Aug 25 10:41:18 GMT 2019","finishedAt":"Sun Aug 25 10:41:46 GMT 2019","workflow":"bc144c93-53aa-436d-b40e-f7e232e11a04","moduleId":"1","supportsInteractive":"false","interactive":"false","user-id":"1","agentPool":"default","flowId":"16a84d48-6c66-77fb-f5bd-4873c6e8d80b","networkMode":"Open","stepId":"1","servicePrefix":"module","moduleGlobalId":"972050a8-19f4-45d0-bb3c-d36ab05bcc9a"},Sun Aug 25 10:41:19 GMT 2019,Thu Jan 01 00:00:00 GMT 1970,None,Some(false),Some(false),None,ModuleDetails(None,None))),failed,running,{},Sun Aug 25 10:41:00 GMT 2019,Sun Aug 25 10:41:46 GMT 2019,WorkflowDetails(None,Some(6 milliseconds)))
[2019-08-25 10:41:46,962][INFO][buddy] Webhook callback triggered with status 202 Accepted
[INFO] [08/25/2019 10:41:47.020] [default-rediscala.rediscala-client-worker-dispatcher-22] [akka://default/user/RedisClient-$a] Connect to redis-redis.default.svc.cluster.local/10.0.9.3:6379
[INFO] [08/25/2019 10:41:47.040] [default-rediscala.rediscala-client-worker-dispatcher-22] [akka://default/user/RedisClient-$a] Connected to redis-redis.default.svc.cluster.local/10.0.9.3:6379
[INFO] [08/25/2019 10:41:47.053] [default-rediscala.rediscala-client-work

It says a json file is missing, where should i put it and what’s supposed to be in there?

io.circe.ParsingFailure: /storage/bc144c93-53aa-436d-b40e-f7e232e11a04/module_1_1/wfe_module_params_1_1.json (No such file or directory)

The correspoinding code is available online:

I’m new to apeer and docker. Thus, any hint for me getting started debugging this issue would be nice!

Thanks!

Cheers,
Robert

Hi there,

I am Oguzhan from Team APEER. Sorry to keep you waiting! This error message can occur if the module crashes unexpectedly for whatever reason. Could you please also provide the module log?

Cheers
Oguzhan

1 Like

Hi @oguzhan.angay,

thanks for the hint! The error message of the module was very helpful - opencl wasn’t installed:

[ADK] Initializing
[ADK] Found "WFE_INPUT_JSON" to be "{"WFE_output_params_file":"wfe_module_params_1_1.json","input_image":"/input/module_0_0/blobs.tif"}"
[ADK] Successfully read "WFE_INPUT_JSON". Output params will be written to "wfe_module_params_1_1.json"
clapeer
Could not load libOpenCL.so, error libOpenCL.so: cannot open shared object file: No such file or directory

Thus, I added opencl and nvidia to my dockerfile:

RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository 'deb http://deb.debian.org/debian stretch main contrib non-free'

RUN apt-get update && apt-get install -y ocl-icd-opencl-dev
RUN apt-get update && apt-get install -y clinfo nvidia-opencl-icd && clinfo

But the error message stays the same - it cannot locate libopencl. Btw. the GPU requirement is on.

Have you ever used OpenCL from docker in Apeer by chance? Again, any hint for making it work is appreciated.

Thanks!

Cheers,
Robert

Hi Robert,

this is Thomas, one of the developers of APEER. I tried reproducing your problem and created a module installing openCL + clinfo. Running clinfo gives me the following output

Number of platforms                               1
  Platform Name                                   NVIDIA CUDA
  Platform Vendor                                 NVIDIA Corporation
  Platform Version                                OpenCL 1.2 CUDA 9.0.368
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_nv_create_buffer
  Platform Extensions function suffix             NV

  Platform Name                                   NVIDIA CUDA
Number of devices                                 1
  Device Name                                     Tesla K80
  Device Vendor                                   NVIDIA Corporation
  // ...

However I don’t have any code utilizing openCL at hand so I cannot test if libOpenCL.so can be found from any code. Do you get any output when running clinfo? You can test this by running clinfo as your Dockerfile entrypoint as such

ENTRYPOINT["clinfo"]

That will fail the module because it doesn’t write any output.json file but that ok. You will still get the logs in your results.

Best,
Thomas

1 Like

Hey @thomas.irmer,

you could try my module as the code lives on github.

Alternatively, would you mind sharing your module code so that I can see what’s different?

I’ll try the clinfo thingy in the meantime :slight_smile:

Thanks for the hint!

Cheers,
Robert

Is there by the way the possibility to read the build log from successful builds?

Thanks again!

Thanks for sharing your code. I had a look at your Dockerfile and saw that you are using multi-stage docker build, which is very much appreciated since that keeps the docker images small. So thanks for that :slight_smile:

But be aware that you are installing openCL in the builder and not in the actual runtime docker file. So you should move the installation of openCL underneath the second FROM statement to have it installed in the right image.

Hope that helps!

2 Likes

Dear Robert,
I haven’t heard about CLIJ before. It sounds like an awesome idea! From your experience, what enhancement in performance is possible by using CLIJ?

Regarding your question about accessing logs from successful module builds: this is currently not possible on the platform. So far we did not have any requests for this. Thanks for the proposal - I will add it to our backlog.
There is also another way to access the logs: you could use our “module debugger tool” (https://app.gitbook.com/@apeer-microscopy/s/apeer/create-modules/module-debugging) to run and debug you module locally. In this case you can also access the build logs.

Cheers,
Bernhard

Hey @thomas.irmer,

Great - that was it! Thanks a lot. CLIJ works in your Apeer cloud! :partying_face:
So happy I made my first docker container work. Thanks! :slightly_smiling_face:

@bernhard.fichtl
CLIJ has quite some potential. Compared to ImageJ operations, we observed speedup of up to two orders of magnitude when running on powerful dedicated GPU hardware. You find insights in our preprint: https://www.biorxiv.org/content/10.1101/660704v2

No need to mention that nothing comes for free. Users need to re-implement their workflows using CLIJ. In order to simplify that for image-analysts, we wrote a FAQ outlining how to achieve optimal performance using CLIJ:

Thanks again everyone for helping me getting CLIJ running in Apeer. That very appreciated!

Cheers,
Robert

2 Likes