Stackvora.tech

Projects, Folders, and Organization in GCP

Effective resource management in Google Cloud Platform (GCP) is crucial for maintainability, security, and cost control. GCP's hierarchical structure,…

Effective resource management in Google Cloud Platform (GCP) is crucial for maintainability, security, and cost control. GCP's hierarchical structure, comprising Organizations, Folders, and Projects, provides the primitives for organizing resources logically and applying policies consistently. Understanding how these layers interact is fundamental to designing a scalable and secure cloud environment.

GCP Resource Hierarchy Overview

The GCP resource hierarchy mirrors a traditional organizational structure, enabling granular control and inheritance of policies. At the top is the Organization, which represents your company. Beneath it, you can create Folders to group related Projects, and within Projects, you deploy individual GCP resources like virtual machines, databases, and storage buckets.

  • Organization Node: The root node for all GCP resources. It's automatically created when you create your first Project in a new Cloud Identity or G Suite account. All resources under an Organization inherit its IAM policies and Organization Policies.
  • Folders: Optional grouping mechanism below the Organization. Folders can contain other Folders or Projects. They are ideal for segmenting resources by department, team, or environment, allowing for policy inheritance at an intermediate level.
  • Projects: The fundamental container for all billed GCP resources. Every resource you create must belong to a Project. Projects act as a boundary for billing, quotas, IAM roles, and network configurations.

Example Hierarchy Structure:

A common approach is to structure folders by department or business unit, and then by environment within those. For instance:


Organization (your-company.com)
├── Folder: HR Department
│   ├── Folder: Production
│   │   ├── Project: hr-app-prod-001 (hr-app.your-company.com)
│   │   ├── Project: hr-data-warehouse-prod
│   ├── Folder: Staging
│   │   ├── Project: hr-app-staging-001
│   │   ├── Project: hr-data-warehouse-staging
│   ├── Folder: Development
│   │   ├── Project: hr-app-dev-001
│   │   ├── Project: hr-data-warehouse-dev
├── Folder: Engineering Department
│   ├── Folder: Production
│   │   ├── Project: billing-service-prod
│   │   ├── Project: analytics-platform-prod
│   ├── Folder: Staging
│   │   ├── Project: billing-service-staging
│   │   ├── Project: analytics-platform-staging
│   ├── Folder: Development
│   │   ├── Project: billing-service-dev
│   │   ├── Project: analytics-platform-dev
├── Folder: Shared Services
│   ├── Project: networking-shared-prod
│   ├── Project: logging-monitoring-prod
│   ├── Project: security-tools-prod

Projects: The Core Unit of Work

As the primary billing and resource container, Projects are critical for isolation and management. The recommended practice is to use one project per workload plus environment. For example, a single microservice deployed in production might reside in my-service-prod-xxxxxx, while its staging counterpart lives in my-service-staging-yyyyyy.

Key Project Attributes:

  • Project Name: A human-readable identifier (e.g., "Billing Service Production").
  • Project ID: A globally unique identifier (e.g., billing-service-prod-238473). This is often used in CLI commands and API calls. It must be unique across all GCP projects.
  • Project Number: A globally unique, automatically generated numeric identifier. Used for some API calls and service accounts.

Why One Project Per Workload + Environment?

  • Billing Isolation: Each project has its own billing account association. This simplifies cost allocation and tracking.
  • IAM Scope: IAM policies are primarily scoped at the project level. This allows you to grant specific permissions to developers for their specific environments without affecting others. For example, a developer might have roles/editor on billing-service-dev but only roles/viewer on billing-service-prod.
  • Quota Management: Quotas are enforced at the project level. This prevents one application from exhausting resources or hitting API limits for another.
  • Network Isolation: While VPC networks can span regions within a project, they are isolated between projects unless explicitly connected via Shared VPC or VPC Peering.
  • Service Account Management: Service accounts are project-specific.
  • Infrastructure as Code (IaC): Tools like Terraform manage resources within a project boundary effectively.

Folders: Structuring for Scale and Policy Enforcement

Folders provide a vital intermediate layer for applying policies and organizing projects. They enable a hierarchical policy inheritance model, where policies set at a folder level apply to all projects and sub-folders within it.

When to Use Folders:

  • Departmental Grouping: Isolate resources and policies for different business units.
  • Environment Separation: Create top-level folders for Production, Staging, Development, which then contain departmental sub-folders, or vice-versa.
  • Regulatory Compliance: Apply specific Organization Policies or IAM roles to a set of projects requiring particular compliance standards.
  • Billing Segmentation: While projects are the billing unit, folders help visualize and manage groups of projects associated with specific cost centers.

Organization Policies (Org Policies) with Folders:

Organization Policies allow you to programmatically constrain the resources that can be deployed in your GCP environment. They are especially powerful when applied at the Folder level. Examples include:

  • constraints/compute.disableSerialPortAccess: Restrict serial port access on VMs for all projects within a folder.
  • constraints/gcp.resourceLocations: Enforce that all new resources within a folder can only be created in specific geographical regions (e.g., asia-east1, us-east1).
  • constraints/iam.disableServiceAccountKeyCreation: Prevent users from creating service account keys within specific environments (e.g., Production).

To list org policies for a folder:


gcloud resource-manager org-policies list --folder=<FOLDER_ID>

To set an org policy (e.g., restrict resource locations):


# Create a policy YAML file (e.g., restrict-locations.yaml)
# constrain-locations.yaml:
#   constraint: constraints/gcp.resourceLocations
#   listPolicy:
#     allowedValues:
#     - europe-west1
#     - europe-west2
#   inheritFromParent: false

gcloud resource-manager org-policies set-policy restrict-locations.yaml --folder=<FOLDER_ID>

Labels: Granular Metadata for Cost and Operations

Labels are key-value pairs that you can attach to most GCP resources (VMs, storage buckets, Pub/Sub topics, etc.). They don't enforce policies but are invaluable for metadata, cost allocation, and automation. A consistent labeling strategy is critical.

Recommended Labeling Strategy:

  • env: (e.g., prod, staging, dev, test) - Essential for distinguishing environments, especially for billing reports.
  • owner: (e.g., team-a, john-doe) - Identifies the team or individual responsible for the resource.
  • application: (e.g., billing-service, user-auth) - Links resources to a specific application or microservice.
  • cost-center: (e.g., cc-4012) - For internal accounting and chargebacks.
  • managed-by: (e.g., terraform, ansible) - Useful for indicating automation origin.

Impact on Billing and Reporting:

When you export your GCP billing data to BigQuery, labels are included. This allows you to run SQL queries to break down costs by environment, owner, application, or cost center, providing invaluable insights into your spending.


SELECT
  invoice.month AS billing_month,
  labels.key,
  labels.value,
  SUM(cost) AS total_cost
FROM
  `your-billing-project.your_billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` AS t,
  UNNEST(t.labels) AS labels
WHERE
  labels.key = 'env' AND labels.value = 'prod'
GROUP BY
  billing_month, labels.key, labels.value
ORDER BY
  billing_month, total_cost DESC;

Managing the Hierarchy: CLI and Console

You can manage Projects, Folders, and Organizations through the GCP Console, gcloud CLI, and Infrastructure as Code tools like Terraform.

Creating a New Project:

In a folder named engineering-dev with ID 123456789012:


gcloud projects create my-new-app-dev-123456 \
    --name="My New Application Dev" \
    --folder="123456789012"

Listing Projects in a Folder:


gcloud projects list --filter="parent.id=123456789012 AND parent.type=folder"

Creating a New Folder:

Under the Organization node:


gcloud resource-manager folders create --display-name="Engineering Department" --organization="your-organization-id"

Or under an existing folder (e.g., folder ID 987654321098):


gcloud resource-manager folders create --display-name="Production" --folder="987654321098"

Common Pitfalls and Troubleshooting

  • Overly Broad IAM Permissions: Granting roles/editor or roles/owner at the Folder or Organization level is highly discouraged. Use the principle of least privilege.
  • Inconsistent Project Naming/IDs: Establish clear naming conventions early. Project IDs are immutable, so choose wisely.
  • Missing Labels: Without consistent labels, cost analysis and resource identification become significantly harder. Use tools or policies to enforce label presence.
  • Organization Policies Ignored: Ensure Organization Policies are applied at the correct level (Organization, Folder) and that child resources are not overriding them incorrectly (check inheritFromParent property).
  • Default Project Quotas: Be aware that new projects come with default quotas. For production workloads, plan to request quota increases proactively.
  • No Organization Node: If you don't see an Organization node, it means your projects are not associated with a Cloud Identity or G Suite domain. This limits the use of Folders and Organization Policies. Migrate to an Organization structure for proper management.

Back to the knowledge base · Ask the AI assistant