JS Task API Examples: accessing the internet

Introduction

In this article, we will present methods that let you access the Internet.

Outbound feature

For the requestor to be able to use the outbound feature, initiate a connection to from the provider to an external url (target_url), the following minimal conditions must be met:

The requestor must request the outbound feature in the demand and include a Computation Manifest there. The manifest must declare the target_url.
The provider must offer the outbound service at least for the target_url. (So either he needs to offer the outbound for any (unrestricted) URLs or the target_url is included in the whitelist).

You can find more information about the feature and the manifest in the following articles: Accessing the Internet and Payload Manifest.

The requestor is responsible for:

Manifest creation
Defining demand for outbound access

Both examples are accessing urls included in the default whitelist. Click here to view the whitelist.

Prerequisites

Yagna service is installed and running with try_golem app-key configured.

How to run examples

Create a project folder, initialize a Node.js project, and install libraries.

mkdir golem-example
cd golem-example
npm init
npm i @golem-sdk/task-executor
npm i @golem-sdk/pino-logger

Next, install Golem SDK CLI - a companion tool that will facilitate manifest creation.

npm install -g @golem-sdk/cli

To run the examples provided below, copy the code into the index.mjs file in the project folder and run:

node index.mjs

Basic outbound access

In this example, we will download a file from ipfs.io. The opfs.io domain is one of the domains included in the whitelist.

Manifest creation

To create a new manifest run:

golem-sdk manifest create golem/examples-outbound:latest

This will create a basic manifest.json file. You will use it to inform the provider what GVMI image we will be using. The manifest contains also your application version, application name, and description, all read from your package.json file (you can edit this information if you want).

Adding outbound configuration

The next step is to configure our manifest, so you can access a public URL. The CLI also has a handy command that will take care of that for you:

golem-sdk manifest net add-outbound https://ipfs.io

This has added 'https://ipfs.io' as the URL you want to access from the provider node. The command can be run multiple times to add more URLs or you can pass them all at once.

Your manifest is ready and stored in the manifest.json file.

Defining demand for outbound access

The example below demonstrates how to define the demand that will get access to the Internet.

Here's the manifest

{
  "version": "0.1.0",
  "createdAt": "2023-10-19T11:23:08.156+02:00",
  "expiresAt": "2024-01-17T11:23:08.156+01:00",
  "metadata": {
    "name": "outbound-docs",
    "description": "",
    "version": "1.0.0"
  },
  "payload": [
    {
      "platform": {
        "os": "linux",
        "arch": "x86_64"
      },
      "hash": "sha3:dad8f776b0eb9f37ea0d63de42757034dd085fe30cc4537c2e119d80",
      "urls": [
        "http://registry.golem.network/download/f37c8ba2b534ca631060fb8db4ac218d3199faf656aa2c92f402c2b700797c21"
      ]
    }
  ],
  "compManifest": {
    "version": "0.1.0",
    "net": {
      "inet": {
        "out": {
          "urls": ["https://github.com", "https://ipfs.io"],
          "protocols": ["https"]
        }
      }
    }
  }
}

And the requestor code:

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { readFile } from "fs/promises";

// The example is using url from domain that is included in the outbound Whitelist.
// See https://github.com/golemfactory/ya-installer-resources/tree/main/whitelist for the current default whitelist.

const url = "https://ipfs.io/ipfs/bafybeihkoviema7g3gxyt6la7vd5ho32ictqbilu3wnlo3rs7ewhnp7lly";

(async function main() {
  // Load the manifest.
  const manifest = await readFile(`./manifest.json`);

  // Create and configure a TaskExecutor instance.
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        capabilities: ["inet", "manifest-support"],
        manifest: manifest.toString("base64"),
      },
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
    },
  });

  try {
    await executor.run(async (exe) => {
      const result = await exe.run(`curl ${url} -o /golem/work/example.jpg`);

      console.log((await exe.run("ls -l")).stdout);
      if (result.result === "Ok") {
        console.log("File downloaded!");
      } else {
        console.error("Failed to download the file!", result.stderr);
      }
    });
  } catch (err) {
    console.error("The task failed due to", err);
  } finally {
    await executor.shutdown();
  }
})();

Note the most important part:

// Load the manifest file.
const manifest = await readFile(`./manifest.json`)

// Create and configure a TaskExecutor instance.
const executor = await TaskExecutor.create({
  logger: pinoPrettyLogger(),
  api: { key: 'try_golem' },
  demand: {
    workload: {
      capabilities: ['inet', 'manifest-support'],
      manifest: manifest.toString('base64'),
    },
  },
  // ... market options
})

First, in the workload object we specify additional requirements:

'inet' - indicates the requestor requires outbound service from providers,
'manifest-support' - indicates that the requestor uses a manifest to specify a demand.

Instead of providing an image tag or hash as in the quickstart example, the manifest is provided. In the manifest body listed above, you can see it contains the image hash to indicate the image to be run on providers ( in payload object ) and a list of urls of domains to which the requestor needs access (in compManifest object).

Please note the loaded manifest is encoded to base64.

api: { key: 'try_golem' } - defined the api key, to get access to the Yagna service. This particular key is available if you start the yagna according to the procedure provided in the installation example, you can also configure your own unique keys. See here for instructions.

The rest of the requestor script follows the pattern used in the quickstart example. Once you get the exeUnit instance you can initiate a connection to the external locations, that are in the domain that was declared in the manifest:

...
const url = "https://ipfs.io/ipfs/bafybeihkoviema7g3gxyt6la7vd5ho32ictqbilu3wnlo3rs7ewhnp7lly";
...

const result = await exe.run(`curl ${url} -o /golem/work/example.jpg`);

Using outbound to install node.js packages

Note: This example shows how to use the outbound whitelist to install a npm package on a provider. It is recommended to install additional packages in a directory that is defined as VOLUME in the image definition, to avoid filesystem capacity limits. If you have a large number of packages you should rather install them during the image build phase - you will avoid installing them on each provider separately and get your providers ready in a shorter time.

Here's the manifest:

{
  "version": "0.1.0",
  "createdAt": "2023-11-28T12:23:33.435+01:00",
  "expiresAt": "2024-02-26T12:23:33.435+01:00",
  "metadata": {
    "name": "npm_install",
    "description": "Example project",
    "version": "1.0.0"
  },
  "payload": [
    {
      "platform": {
        "os": "linux",
        "arch": "x86_64"
      },
      "hash": "sha3:3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd",
      "urls": [
        "http://registry.golem.network/v1/image/download?hash=3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd"
      ]
    }
  ],
  "compManifest": {
    "version": "0.1.0",
    "net": {
      "inet": {
        "out": {
          "urls": ["https://registry.npmjs.org"],
          "protocols": ["https"]
        }
      }
    }
  }
}

And the requestor code:

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { readFile } from "fs/promises";

const manifest = await readFile(`./manifest_npm_install.json`);

(async function main() {
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        manifest: manifest.toString("base64"),
        capabilities: ["inet", "manifest-support"],
      },
      expirationSec: 60 * 30, //30 min
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
    },
    // Control the execution of tasks
    task: {
      maxTaskRetries: 0,
      taskTimeout: 120 * 60 * 1000,
    },
  });

  try {
    await executor.run(async (exe) => {
      console.log("working on provider: ", exe.provider.id);

      console.log((await exe.run("npm install moment")).stdout);
      console.log((await exe.run(`cat ./package.json`)).stdout);
    });

    console.log("task completed");
  } catch (err) {
    console.error("Running the task on Golem failed due to", err);
  } finally {
    await executor.shutdown();
  }
})();