Service Example 6: External API request

The example depicts the following features:

• Outbound Network and Computation Payload Manifest.

As with the other examples, we're assuming here you already have your yagna daemon set-up to request the test tasks and that you were able to configure your Python environment to run the examples using the latest version of yapapi. If this is your first time using Golem and yapapi, please first refer to the resources linked above.

This example involves Computation Payload Manifest.

Computation Payload Manifest making use of Outbound Network requires either:

1. Requestor certificate that's trusted by the Providers
2. an instance of a Provider with the particular domain this example uses added to its domain whitelist
3. an instance of a Provider with the requestor's self-signed Certificate imported into its keystore

The following example will show cases 2. and 3. so it will be necessary to start a local instance of a Provider.

An example app will request an external API using Provider's network and then it will print the API response to the console.

For an app to make an Outbound Network request it needs to declare which tools it will use and which URLs it will access in a Computation Payload Manifest.

Our example will make an HTTPS request using curl to a public REST API with the URL https://api.coingecko.com.

Computation Payload Manifest will need to have following objects:

• net computation constraints with URLs the app will access (https://api.coingecko.com)
• script computation constraint with commands app will execute (curl)
• payload defining Golem image containing tools used by the app (curl)

Example Computation Payload Manifest must follow a specific schema, and for our example it will take form of following manifest.json file:

{
  "version": "0.1.0",
  "createdAt": "2022-07-26T12:51:00.000000Z",
  "expiresAt": "2100-01-01T00:01:00.000000Z",
  "metadata": {
    "name": "External API call example",
    "description": "Example manifest of a service making an outbound call to the external API",
    "version": "0.1.0"
  },
  "payload": [
    {
      "platform": {
        "arch": "x86_64",
        "os": "linux"
      },
      "urls": [
        "http://yacn2.dev.golem.network:8000/docker-golem-script-curl-latest-d75268e752.gvmi"
      ],
      "hash": "sha3:e5f5ddfd649525dbe25d93d9ed51d1bdd0849933d9a5720adb4b5810"
    }
  ],
  "compManifest": {
    "version": "0.1.0",
    "script": {
      "commands": ["run .*curl.*"],
      "match": "regex"
    },
    "net": {
      "inet": {
        "out": {
          "protocols": ["https"],
          "urls": ["https://api.coingecko.com"]
        }
      }
    }
  }
}

The created file should be verified using the JSON schema.

Then it needs to be encoded in base64:

 base64 --wrap=0 manifest.json > manifest.json.base64

A base64-encoded manifest can be configured using the yapapi.payload.vm.manifest function, resulting in following external_api_request.py file:

import asyncio

from yapapi import Golem
from yapapi.services import Service
from yapapi.payload import vm

class OutboundNetworkService(Service):
    @staticmethod
    async def get_payload():
        return await vm.manifest(
            manifest = open("manifest.json.base64", "rb").read()
            # later we may add here manifest signature, digest algorithm, and app author's certificate
            min_mem_gib = 0.5,
            min_cpu_threads = 0.5,
            # capabilities used to reach Provider with a correct VM Runtime
            capabilities=["inet", "manifest-support"],
        )

    async def run(self):
        script = self._ctx.new_script()
        future_result = script.run(
            "/bin/sh",
            "-c",
            f"GOLEM_PRICE=`curl -X 'GET' \
                    'https://api.coingecko.com/api/v3/simple/price?ids=golem&vs_currencies=usd' \
                    -H 'accept: application/json' | jq .golem.usd`; \
                echo \"Golem price: $GOLEM_PRICE USD\";",
        )
        yield script

        result = (await future_result).stdout
        print(result.strip() if result else "")

async def main():
    async with Golem(budget=1.0, subnet_tag="testnet") as golem:
        await golem.run_service(OutboundNetworkService, num_instances=1)
        await asyncio.sleep(60)

Providers verify the incoming request with a Computation Payload Manifest by checking if it arrives with a signature and App author's certificate signed by a certificate they trust. If there is no signature, they verify if URLs used by Computation Payload Manifest are whitelisted.

There are two ways to make our local Provider verify the request:

• Whitelisting of the domain used by the app

  Add api.coingecko.com to Provider's domain whitelist:

  ya-provider whitelist add --patterns api.coingecko.com --type strict

• Signing manifest and adding signature with a certificate to the request

  Generate self signed certificate and then ge#nerate manifest signature.

  With a generated and base64-encoded certificate and a signature, the get_payload() function takes the following form:

  # ...
    async def get_payload():
        return await vm.manifest(
            manifest = open("manifest.json.base64", "rb").read(),
            manifest_sig = open("manifest.json.base64.sign.sha256.base64", "rb").read(),
            manifest_sig_algorithm = "sha256",
            manifest_cert = open("golem_requestor.cert.pem.base64", "rb").read(),
            min_mem_gib = 0.5,
            min_cpu_threads = 0.5,
            capabilities=["inet", "manifest-support"],
        )
  # ...

With both Requestor and Provider yagna nodes and ya-provider running in the background run:

python external_api_request.py

(keep in mind to set YAGNA_APPKEY and YAGNA_API_URL env variables pointing to the local Requestor node)