Executing Long Running Tasks in Google App Engine — How To Do It?
Running background tasks for hours or days on Google App Engine using Task Queues — and the production scaling gotcha that catches most developers off guard.
Most developers working on Google Cloud Platform wonder: “What if I am using Google App Engine and need long running tasks that run in the background for hours or days?”
The answer involves using Task Queues, a feature of Google App Engine. However, production environments often reveal unexpected problems with long-running tasks that require specific configurations to resolve.
Google App Engine Overview
Google App Engine (GAE) is a platform for building scalable web applications and mobile backends. It offers automated security scanning, supports popular development tools like Eclipse, IntelliJ, Maven, Git, Jenkins, and PyCharm, plus features including user authentication, NoSQL Datastore, Memcache, and Task Queues.
GAE Task Queues Explained
Task queues enable asynchronous execution of background operations outside user requests. For example, a user can submit a file upload link and continue using the application while the upload processes in the background through a task queue.
The Production Problem
Applications often behave differently in production versus development environments due to different cloud resource configurations. GAE offers Frontend instances (for end-user operations) and Backend instances (for background computations).
Key Configuration Issue
By default, GAE deploys with Frontend instances using Automatic Scaling. Under this configuration, tasks have a “maximum for 10 minutes” execution window. Long-running tasks exceeding this deadline require different settings.
Solution
For tasks exceeding 10 minutes:
- Switch from Frontend to Backend instances.
- Configure Manual or Basic Scaling (not Automatic Scaling).
- Basic Scaling is recommended for cost control without complex initialization needs.
Recommendation: Test all application features on production servers before release, as behavior differs significantly from development environments.
