Detect Labels

The Vision API can detect and extract information about entities in an image, across a broad group of categories.

Labels can identify general objects, locations, activities, animal species, products, and more. If you need targeted custom labels, Cloud AutoML Vision allows you to train a custom machine learning model to classify images.

Labels are returned in English only. The Cloud Translation API can translate English labels into any of a number of other languages.

Setagaya ward street image — *Image credit*: Alex Knight on Unsplash.

For example, the image above may return the following list of labels:

Description	Score
Street	0.872
Snapshot	0.852
Town	0.848
Night	0.804
Alley	0.713

Label detection requests

Set up your Google Cloud project and authentication

If you have not created a Google Cloud project, do so now. Expand this section for instructions.

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Vision API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the API

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Detect Labels in a local image

You can use the Vision API to perform feature detection on a local image file.

For REST requests, send the contents of the image file as a base64 encoded string in the body of your request.

For gcloud and client library requests, specify the path to a local image in your request.

REST

Before using any of the request data, make the following replacements:

BASE64_ENCODED_IMAGE: The base64 representation (ASCII string) of your binary image data. This string should look similar to the following string:
- /9j/4QAYRXhpZgAA...9tAVx/zDQDlGxn//2Q==
Visit the base64 encode topic for more information.
RESULTS_INT: (Optional) An integer value of results to return. If you omit the "maxResults" field and its value, the API returns the default value of 10 results. This field does not apply to the following feature types: TEXT_DETECTION, DOCUMENT_TEXT_DETECTION, or CROP_HINTS.
PROJECT_ID: Your Google Cloud project ID.

HTTP method and URL:

POST https://vision.googleapis.com/v1/images:annotate

Request JSON body:

{
  "requests": [
    {
      "image": {
        "content": "BASE64_ENCODED_IMAGE"
      },
      "features": [
        {
          "maxResults": RESULTS_INT,
          "type": "LABEL_DETECTION"
        }
      ]
    }
  ]
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: PROJECT_ID" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://vision.googleapis.com/v1/images:annotate"

PowerShell

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "PROJECT_ID" }Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://vision.googleapis.com/v1/images:annotate" | Select-Object -Expand Content

If the request is successful, the server returns a 200 OK HTTP status code and the response in JSON format.

A LABEL_DETECTION response includes the detected labels, their score, topicality, and an opaque label ID, where:

mid - if present, contains a machine-generated identifier (MID) corresponding to the entity's Google Knowledge Graph entry. Note that mid values remain unique across different languages, so you can use these values to tie entities together from different languages. To inspect MID values, refer to the Google Knowledge Graph API documentation.
description - the label description.
score - the confidence score, which ranges from 0 (no confidence) to 1 (very high confidence).
topicality - The relevancy of the ICA (Image Content Annotation) label to the image. It measures how important/central a label is to the overall context of a page.

{
  "responses": [
    {
      "labelAnnotations": [
        {
          "mid": "/m/01c8br",
          "description": "Street",
          "score": 0.87294734,
          "topicality": 0.87294734
        },
        {
          "mid": "/m/06pg22",
          "description": "Snapshot",
          "score": 0.8523099,
          "topicality": 0.8523099
        },
        {
          "mid": "/m/0dx1j",
          "description": "Town",
          "score": 0.8481104,
          "topicality": 0.8481104
        },
        {
          "mid": "/m/01d74z",
          "description": "Night",
          "score": 0.80408716,
          "topicality": 0.80408716
        },
        {
          "mid": "/m/01lwf0",
          "description": "Alley",
          "score": 0.7133322,
          "topicality": 0.7133322
        }
      ]
    }
  ]
}

Go

Before trying this sample, follow the Go setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Go API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Java

Before trying this sample, follow the Java setup instructions in the Vision API Quickstart Using Client Libraries. For more information, see the Vision API Java reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Node.js API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Python

Before trying this sample, follow the Python setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Python API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Additional languages

C#: Please follow the C# setup instructions on the client libraries page and then visit the Vision reference documentation for .NET.

PHP: Please follow the PHP setup instructions on the client libraries page and then visit the Vision reference documentation for PHP.

Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the Vision reference documentation for Ruby.

Detect Labels in a remote image

You can use the Vision API to perform feature detection on a remote image file that is located in Cloud Storage or on the Web. To send a remote file request, specify the file's Web URL or Cloud Storage URI in the request body.

REST

Before using any of the request data, make the following replacements:

CLOUD_STORAGE_IMAGE_URI: the path to a valid image file in a Cloud Storage bucket. You must at least have read privileges to the file. Example:
- ```
gs://cloud-samples-data/vision/label/setagaya.jpeg
```
RESULTS_INT: (Optional) An integer value of results to return. If you omit the "maxResults" field and its value, the API returns the default value of 10 results. This field does not apply to the following feature types: TEXT_DETECTION, DOCUMENT_TEXT_DETECTION, or CROP_HINTS.
PROJECT_ID: Your Google Cloud project ID.

HTTP method and URL:

POST https://vision.googleapis.com/v1/images:annotate

Request JSON body:

{
  "requests": [
    {
      "image": {
        "source": {
          "gcsImageUri": "CLOUD_STORAGE_IMAGE_URI"
        }
      },
      "features": [
        {
          "maxResults": RESULTS_INT,
          "type": "LABEL_DETECTION"
        },
      ]
    }
  ]
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: PROJECT_ID" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://vision.googleapis.com/v1/images:annotate"

PowerShell

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "PROJECT_ID" }Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://vision.googleapis.com/v1/images:annotate" | Select-Object -Expand Content

If the request is successful, the server returns a 200 OK HTTP status code and the response in JSON format.

A LABEL_DETECTION response includes the detected labels, their score, topicality, and an opaque label ID, where:

mid - if present, contains a machine-generated identifier (MID) corresponding to the entity's Google Knowledge Graph entry. Note that mid values remain unique across different languages, so you can use these values to tie entities together from different languages. To inspect MID values, refer to the Google Knowledge Graph API documentation.
description - the label description.
score - the confidence score, which ranges from 0 (no confidence) to 1 (very high confidence).
topicality - The relevancy of the Image Content Annotation (ICA) label to the image. It measures how important/central a label is to the overall context of a page.

{
  "responses": [
    {
      "labelAnnotations": [
        {
          "mid": "/m/01c8br",
          "description": "Street",
          "score": 0.87294734,
          "topicality": 0.87294734
        },
        {
          "mid": "/m/06pg22",
          "description": "Snapshot",
          "score": 0.8523099,
          "topicality": 0.8523099
        },
        {
          "mid": "/m/0dx1j",
          "description": "Town",
          "score": 0.8481104,
          "topicality": 0.8481104
        },
        {
          "mid": "/m/01d74z",
          "description": "Night",
          "score": 0.80408716,
          "topicality": 0.80408716
        },
        {
          "mid": "/m/01lwf0",
          "description": "Alley",
          "score": 0.7133322,
          "topicality": 0.7133322
        }
      ]
    }
  ]
}

Go

Before trying this sample, follow the Go setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Go API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Java

Before trying this sample, follow the Java setup instructions in the Vision API Quickstart Using Client Libraries. For more information, see the Vision API Java reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Node.js API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Python

Before trying this sample, follow the Python setup instructions in the Vision quickstart using client libraries. For more information, see the Vision Python API reference documentation.

To authenticate to Vision, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

gcloud

To detect labels in an image, use the gcloud ml vision detect-labels command as shown in the following example:

gcloud ml vision detect-labels gs://cloud-samples-data/vision/label/setagaya.jpeg

Additional languages

C#: Please follow the C# setup instructions on the client libraries page and then visit the Vision reference documentation for .NET.

PHP: Please follow the PHP setup instructions on the client libraries page and then visit the Vision reference documentation for PHP.

Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the Vision reference documentation for Ruby.

Try it

Try label detection below. You can use the image specified already (gs://cloud-samples-data/vision/label/setagaya.jpeg) or specify your own image in its place. Send the request by selecting Execute.

Request body:

{
  "requests": [
    {
      "features": [
        {
          "maxResults": 5,
          "type": "LABEL_DETECTION"
        }
      ],
      "image": {
        "source": {
          "imageUri": "gs://cloud-samples-data/vision/label/setagaya.jpeg"
        }
      }
    }
  ]
}