Create and manage log scopes
This document describes how you can create and manage log scopes, which you can use to help you efficiently find the log entries that you want to view or analyze. If you only want to view and analyze the log entries that originate in a project, folder, or organization, then this document isn't for you. However, if you rely on log sinks to route logs to other projects or to user-defined log buckets, or if you use log views, then the information in this document might help you efficiently find specific log entries.
This document doesn't describe how to view your logs. For information about that topic, see View logs by using the Logs Explorer.
About log scopes
Log scopes are persistent, project-level resources that list a set of resources. These resources can be projects, folders, organizations, and log views. For example, you could define a log scope that lists the projects that contain resources used for production, or one that lists the log views that include log entries for a specific resource type.
When you create a Google Cloud project, folder, or organization resource,
Logging creates a log scope named _Default.
This scope includes the project, folder, or organization that was created.
When a searched resource is a Google Cloud project, folder, or organization,
the results include the log entries that originate in the resource and
then are stored in a log bucket. The log bucket can be in any project.
When a project is searched, the results also include log entries
that are routed to the project by a sink in another project
and then stored in a log bucket.
You can create log scopes. You can also edit and delete the
log scopes that you create. However, you can't
edit or delete the log scope named _Default.
You use a log scope to control which resources the Logs Explorer page searches for log data. When you open that page and select a log scope, the page searches the resources listed in that scope and then refreshes the display.
You can also use a log scope to control which resources a logs panel searches for log data. A logs panel is a custom-dashboard widget that displays log data. Each logs panel has its own configuration, which lets you create a dashboard that contains multiple logs panels where each panel displays different log data. For more information, see Display logs and errors on a custom dashboard.
For projects, the default log scope determines the set of resources
that the Logs Explorer page searches when it opens. However,
your Identity and Access Management (IAM) roles on the searched resources and the
time-range setting determine which log entries are fetched from storage.
When projects are created, the log scope named _Default is designated
as the default log scope. You can set which log scope is the
default log scope.
How log scopes differ from centralized log storage
Both centralized log storage and log scopes provide a way for you to view log data that originates in different projects.
When you centralize your log storage, you configure the sinks in an organization or folder to route log entries to a single storage location. Centralized storage provides a single place to query for log data, which simplifies your queries when you are searching for trends or investigating issues. From a security perspective, you also have one storage location, which simplifies the tasks of your security analysts.
When a query is issued to the resources listed in a log scope, the individual query results are combined. A log scope facilitates read-time aggregation of log data which might be stored in different locations. However, a log scope can also be used to provide read access to a one or more log views on a centralized log bucket.
When the Logs Explorer page opens, it issues queries to the resources listed in the default log scope. Therefore, configure the default scope so that the page shows you the data that you usually want to view. For example, you might set the default log scope to list a log view, which when queried, returns the log data for an App Hub application.
Best practices
Because log scopes provide a way for you to define and save a configuration for future use, we recommend that you create log scopes for complex search configurations.
For example, suppose that you are troubleshooting an issue and want to view the log entries for all virtual machine (VM) instances owned by your team. To accomplish this task, you might do the following:
You determine that the log entries that you want to view are stored in multiple log buckets and in multiple projects. For most log buckets, a log view exists that includes the log entries that you want to analyze. Where a log view doesn't exist, you can create one.
You decide to create a log scope because you expect to have a similar troubleshooting task in the future.
You open the Logs Explorer page in the Google Cloud console and then use the Refine scope menu to select your new log scope.
You review the log entries and find the information you need to resolve the issue you were investigating.
After you resolve the issue, you share the failure cause with your colleagues. You also share that you expect to see similar failures in the future, so you created a log scope that will let you, or whomever is investigating the failure, quickly find relevant log entries.
App Hub applications and log scopes
Your App Hub applications might write log data to multiple projects. Your log data might be stored in the project in which it originates, or an organization administrator might have configured centralized storage. To view your application's log data, create a log scope, configure it to list the projects or log views that store your application's log data, and then configure it as the default log scope. When you complete those steps, the Logs Explorer page automatically displays the data written by your application, even when that data is stored in different projects or in a centralized log bucket.
Create the custom log scope in the project from which you will view
your log data. This project is your App Hub host project or
management project. For example, if a folder's display name is
My Folder, then the display name of the folder's management project is
My Folder-mp.
Limitations
- You can't delete or modify the log scope named
_Default. - Only Google Cloud projects support a default log scope.
- You can't add folders or organizations to a user-defined log scope.
- Log scopes are created in the
globallocation.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
-
Enable the Observability API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles. -
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
-
Enable the Observability API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles. -
To get the permissions that you need to create and view log scopes, and to set the default log scope, ask your administrator to grant you the following IAM roles on your project:
-
Logs Configuration Writer (
roles/logging.configWriter) -
Observability Editor (
roles/observability.editor)
For more information about granting roles, see Manage access to projects, folders, and organizations.
These predefined roles contain the permissions required to create and view log scopes, and to set the default log scope. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to create and view log scopes, and to set the default log scope:
-
To set the default log scope:
observability.scopes.{get, update} -
To create and manage log scopes:
logging.logScopes.{create, delete, get, list, update}
You might also be able to get these permissions with custom roles or other predefined roles.
-
Logs Configuration Writer (
-
Select the tab for how you plan to use the samples on this page:
Console
When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Terraform
To use the Terraform samples on this page in a local development environment, install and initialize the gcloud CLI, and then set up Application Default Credentials with your user credentials.
-
Install the Google Cloud CLI.
-
-
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
-
If you're using a local shell, then create local authentication credentials for your user account:
gcloud auth application-default login
You don't need to do this if you're using Cloud Shell.
If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.
For more information, see Set up ADC for a local development environment in the Google Cloud authentication documentation.