Making External API Requests with Golem Network Using Yapapi

This guide explains how to enable outbound networking for your Golem applications using Yapapi. Outbound networking allows your applications to access external APIs or services by defining specific requirements and securing trust with providers. Using the golem-cert-companion tool, you can automate the process of generating manifests, descriptors, and certificates required for this functionality.

By default, the Golem Network includes a restrictive whitelist of URLs for outbound networking. This default list ensures a high level of security, limiting internet access to a predefined set of trusted domains. You can find the current whitelist here.

However, many applications require access to APIs or other external services not included in the whitelist. To allow this, Golem enables users to define specific requirements through manifests and node descriptors. These files:

• Specify the URLs your application needs to access.
• Are signed with a certificate, which must be trusted by providers before your application can access external resources.

When you submit a task requiring outbound networking, the Golem Network looks for providers that meet the task’s requirements and trust your certificate. Without this trust:

• Providers will not accept your task: If no providers trust your certificate, the task will remain in a pending state and will not start.
• Default whitelist restrictions apply: Without trusted manifests, your application is limited to the URLs allowed by the default whitelist.

To ensure your task runs successfully:

1. Generate and sign a manifest and node descriptor.
2. Share your certificate with providers and request their trust.

Manually creating manifests and node descriptors can be a time-consuming and error-prone process. The golem-cert-companion tool streamlines this by automating the creation of:

• manifest.json: (Must be attached in your task)
• node-descriptor.signed.json: (Must be attached in your task )
• root-cert-template.signed.json: (The signed certificate that providers must trust first)

Install the tool using pip:

pip install golem-cert-companion

Run the tool:

golem-cert-companion

or its shortcut:

gcert

The tool will guide you through:

1. Image URL or SHA3 hash: These options specify the image to be downloaded and deployed on providers.

   • The image URL is a direct link to the location where the image can be downloaded.
   • The SHA3 hash is the default identifier generated by gvmkit-build, which retrieves the corresponding image from the Golem registry automatically, instead of relying on a custom URL.

   For instance, if your application uses curl for outbound API requests, you might use a Golem image URL like:

   http://yacn2.dev.golem.network:8000/docker-golem-script-curl-latest-d75268e752.gvmi

   Alternatively, you can provide the SHA3 hash generated by gvmkit-build, which automatically uploads the image to the Golem registry and makes it available for deployment.

2. Outbound URLs: Define the external APIs or URLs your application will access:

   • Specify individual URLs (e.g., https://api.coingecko.com).
   • Allow unrestricted access if your application needs to reach multiple domains.

Once you've provided this information, the tool will generate:

• manifest.json: Specifies the URLs and tools used by your application.
• node-descriptor.signed.json: Signed details about your node’s capabilities.
• root-cert-template.signed.json: Certificate for providers to trust your tasks.

Providers must trust your certificate for outbound networking to work. Share your root-cert-template.signed.json file with providers and ask them to import it using:

ya-provider rule set outbound partner import-cert root-cert-template.signed.json --mode all

You can also request trust through the Golem Discord #providers channel with a message like this:

Hi providers!
I have tasks that require outbound internet access. If you'd like to run these tasks, please trust my certificate:

1. Download the certificate: [Your Link]
2. Import it using:

ya-provider rule set outbound partner import-cert root-cert-template.signed.json --mode all

Thank you!

The following Yapapi example demonstrates fetching the current price of Golem (GLM) from the Coingecko API.

#!/usr/bin/env python3
import asyncio
import json
import pathlib
import sys
from datetime import datetime

from utils import build_parser, print_env_info, run_golem_example

from yapapi import Golem
from yapapi.payload import vm
from yapapi.services import Service

examples_dir = pathlib.Path(__file__).resolve().parent.parent
sys.path.append(str(examples_dir))


class ApiCallService(Service):
    @staticmethod
    async def get_payload():
        # Replace with manifest_whitelist.json for whitelist mode
        manifest = open("manifest_partner_unrestricted.json", "rb").read()
        node_descriptor = json.loads(open("node-descriptor.signed.json", "r").read())

        return await vm.manifest(
            manifest=manifest,
            node_descriptor=node_descriptor,
            min_mem_gib=0.5,
            min_cpu_threads=0.5,
            capabilities=[
                "inet",
            ],
        )

    async def run(self):
        script = self._ctx.new_script()
        future_result = script.run(
            "/bin/sh",
            "-c",
            "GOLEM_PRICE=`curl -X 'GET' \
                    'https://api.coingecko.com/api/v3/simple/price?ids=golem&vs_currencies=usd' \
                    -H 'accept: application/json' | jq .golem.usd`; \
                echo ---; \
                echo \"Golem price: $GOLEM_PRICE USD\"; \
                echo ---;",
        )
        yield script

        result = (await future_result).stdout
        print(result.strip() if result else "")


async def main(subnet_tag, payment_driver, payment_network):
    async with Golem(
        budget=1.0,
        subnet_tag=subnet_tag,
        payment_driver=payment_driver,
        payment_network=payment_network,
    ) as golem:
        print_env_info(golem)

        cluster = await golem.run_service(ApiCallService, num_instances=1)

        while True:
            print(cluster.instances)
            try:
                await asyncio.sleep(10)
            except (KeyboardInterrupt, asyncio.CancelledError):
                break


if __name__ == "__main__":
    parser = build_parser("External API request example")
    now = datetime.now().strftime("%Y-%m-%d_%H.%M.%S")
    parser.set_defaults(log_file=f"external-api-request-yapapi-{now}.log")
    args = parser.parse_args()

    run_golem_example(
        main(
            subnet_tag=args.subnet_tag,
            payment_driver=args.payment_driver,
            payment_network=args.payment_network,
        ),
        log_file=args.log_file,
    )

1. Manifest and Node Descriptor: These files define the application's requirements for outbound networking.
2. Capabilities: The inet capability enables internet access for the application.
3. Outbound Request: The example uses curl to fetch data from the Coingecko API.

1. Explore golem-cert-companion
   See the full documentation for the companion tool here.

2. Engage with the Community
   If you have questions or need assistance, join our Discord community.