JS Task API Examples: selecting providers

Introduction

You can select providers using different criteria, i.e. defining requirements in a demand or applying filters on providers' proposals. You can:

Select a provider based on minimal requirements for remote computer CPU, disk storage, RAM.
Select a provider based on the whitelist/blacklist.
Select a provider based on the proposed costs using a custom filter.

Prerequisites

Yagna service is installed and running with the try_golem app-key configured.

How to run examples

Create a project folder, initialize a Node.js project, and install libraries.

mkdir golem-example
cd golem-example
npm init
npm install @golem-sdk/task-executor
npm install @golem-sdk/pino-logger

Copy the code into the index.mjs file in the project folder and run:

node index.mjs

Pricing filter

The market object of TaskExecutor options includes the pricing object. It is used to define the maximum price accepted in the provider's offer.

market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
 },
 ...

Note, that the total cost depends on several factors like the number of threads, the actual CPU usage, task duration, and of course unit prices.

Filtering providers based on minimal requirements:

You can define minimal requirements for an environment provided by a node by stating a minimal number of:

CPU cores minCpuCores,
RAM minMemGib,
disk space minStorageGib or
CPU threads minCpuThreads.

You can do this in the TaskExecutor options:

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";

(async function main() {
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        imageTag: "golem/node:20-alpine", //minCpuCores : 2,
        //minMemGib : 8,
        //minStorageGib: 10,
        minCpuThreads: 1,
      },
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
    },
  });

  try {
    await executor.run(async (exe) => console.log((await exe.run("echo 'Hello World'")).stdout));
  } catch (err) {
    console.error("An error occurred:", err);
  } finally {
    await executor.shutdown();
  }
})();

warning

Be careful, filtering is done internally by Yagna and if your requirements turn out to be too demanding you will not receive any proposals from providers and your requestor script will terminate after the timeout.

Job timeout log

Selecting providers based on the whitelist

In some situations, you might need your tasks to be executed on a certain provider or exclude specific providers. If you know providers' IDs or names you can use the offerProposalFilter option and use one of the predefined filters:

OfferProposalFilterFactory.disallowProvidersById(),
OfferProposalFilterFactory.disallowProvidersByName(),
OfferProposalFilterFactory.disallowProvidersByNameRegex(),
OfferProposalFilterFactory.allowProvidersById(),
OfferProposalFilterFactory.allowProvidersByName(),
OfferProposalFilterFactory.allowProvidersByNameRegex(),

The byID, and byName filters will accept an array with IDs, or respectively names of the providers, which should be accepted or excluded. The byNameRegex accepts a regex expression to filter the names of providers whose offers will be blocked or allowed.

info

For this example, you might need to update the provider's list in the whiteListsIds. Go to the Golem Network Stats and scroll the list to find a provider working on Testnet. Then click on its name and copy its ID.

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { OfferProposalFilterFactory } from "@golem-sdk/golem-js";

/**
 * Example demonstrating how to use the predefined filter `allowProvidersByName`,
 * which only allows offers from a provider whose name is in the array
 */

const whiteListNames = ["provider-2", "fractal_01_3.h", "sharkoon_379_0.h", "fractal_01_1.h", "sharkoon_379_1.h"];
console.log("Will accept only proposals from:");
for (let i = 0; i < whiteListNames.length; i++) {
  console.log(whiteListNames[i]);
}

(async function main() {
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        imageTag: "golem/node:20-alpine",
      },
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
      offerProposalFilter: OfferProposalFilterFactory.allowProvidersByName(whiteListNames),
    },
  });

  try {
    await executor.run(async (exe) =>
      console.log((await exe.run(`echo "This task is run on ${exe.provider.name}"`)).stdout),
    );
  } catch (err) {
    console.error("An error occurred:", err);
  } finally {
    await executor.shutdown();
  }
})();

info

You can read provider names from the exe exeUnit context or the proposal. We will look into proposal content in the next section.

Selecting providers based on the proposed costs using a custom filter

In this example, we will show a custom filter that can be used to select the best provider. We will use it to filter proposals based on the price, but it can be used to filter by any other attribute that is included in the provider OfferProposal.

Let's see how to do it:

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";

/**
 * Example demonstrating how to write a custom proposal filter.
 */

var costData = [];

const myFilter = (proposal) => {
  let decision = false;
  let usageVector = proposal.properties["golem.com.usage.vector"];
  let counterIdx = usageVector.findIndex((ele) => ele === "golem.usage.duration_sec");
  let proposedCost = proposal.properties["golem.com.pricing.model.linear.coeffs"][counterIdx];
  costData.push(proposedCost);
  if (costData.length < 6) return false;
  else {
    costData.shift();
    let averageProposedCost = costData.reduce((part, x) => part + x, 0) / 5;
    if (proposedCost <= 1.2 * averageProposedCost) decision = true;
    if (decision) {
      console.log(proposedCost, averageProposedCost);
    }
  }
  console.log(costData);
  console.log(proposal.properties["golem.node.id.name"], proposal.properties["golem.com.pricing.model.linear.coeffs"]);
  return decision;
};

(async function main() {
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        imageTag: "golem/node:20-alpine",
      },
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
      offerProposalFilter: myFilter,
    },
    task: {
      startupTimeout: 60_000,
    },
  });

  try {
    await executor.run(async (exe) => {
      const result = await exe.run(`echo "This task is run on ${exe.provider.id}"`);
      console.log(result.stdout, exe.provider.id);
    });
  } catch (err) {
    console.error("An error occurred:", err);
  } finally {
    await executor.shutdown();
  }
})();

Note that myFilter is a function that accepts an OfferProposal object as its parameter and should return true or false depending on the decision based on the proposal properties.

Our custom function collects pricing data until we have a set of 5 proposals. Then it accepts proposals only if the price is lower than average from the last five proposals.

Provider price is calculated as the product of prices defined per specific usage counter.

The counters are defined in the golem.com.usage.vector property and the prices are defined in golem.com.pricing.model.linear.coeffs. The last element in the price coeffs array is a fixed element of the total price (one can consider it the one-time price for deployment).

Note that the sequence of the counters is not fixed, therefore we need to find the index of the specific counter. In our examples, we take into account only the price related to the usage of the total environment (as our example task has a very short execution time).