Bug: Environment doesn't launch. EVER!

So I was working with cloud functions and tried to run my function locally with a Python virtual environment. While installing the dependencies from the requirements.txt, the disk ran out of space and the environment setup failed.

I ran a quick lsblk command in the terminal and realized my /home disk had 10GB of space in -sdc, but I had two other disks, -sda [50GB] and -sdb [15GB or so]. So I cp -r … my project directory to /etc within -sda and tried to setup the virtual environment again, but still it crashed with a “space full” error.

First off, i don’t know how a simple Python virtual environment can fill up 23GB of available space in -sda. But the VM froze and I had to reload the browser tab. And now no matter what I do, the project won’t go passed the “building environment” step.

I’ve tried launching in recovery mode and resetting the VM. Nothing works. I have code in that project I can’t loose. If I can just recover the file system to retrieve my code I’ll be estatic.

1 Like

Understanding the Problem:

Initial Disk Full: The /home partition running out of space during pip install is the root cause. pip can download many files to a cache (often in ~/.cache/pip) and then unpack/build them, which can consume significant space, especially with complex dependencies.

cp -r … /etc: Copying your project to /etc was not a good idea. /etc is for system configuration files and is usually on the root partition, which might also be small or have restrictive permissions. This likely didn’t solve the space issue for pip and might have confused the environment.

Persistent Build Failure: The environment is now likely in a corrupted state. Either:

There are leftover incomplete installation files in your project directory or the Python environment it tries to create.

The “Building environment” step might be trying to use a specific partition for the venv that is still full or has corrupted metadata.

The configuration for your Functions Workbench instance might be pointing to a problematic state.

“Resetting the VM”: If this option truly resets the underlying virtual machine to a clean state but re-attaches your persistent project disk, then the problem lies within your project files or its immediate environment setup data, not the OS of the VM itself.

Priority #1: Recovering Your Code

This is the absolute most important step. Do not attempt too many destructive “fixes” until your code is safe.

Recovery Strategy:

Launch in Recovery Mode (Again, with a specific goal):

When you “Launch Recovery,” what exactly does it give you? Is it a terminal?

If it’s a terminal, your goal is NOT to fix the environment yet, but to COPY YOUR CODE OUT.

Navigate: Try to find your project directory. It’s often in /workspace, /home/user/project, or similar. Use ls, cd, pwd to find it.

ls /

ls /workspace

ls /home

Package your code: Once you find your project directory (let’s say it’s /workspace/my-functions-project), try to create an archive:

cd /workspace

tar -czvf my-functions-backup.tar.gz my-functions-project

Or zip -r my-functions-backup.zip my-functions-project

Exfiltrate the backup: This is the tricky part in a limited recovery environment.

Cloud Storage (Ideal): If gsutil or gcloud CLI is available in recovery mode, use it to copy the backup to a Google Cloud Storage bucket:
gsutil cp my-functions-backup.tar.gz gs://your-bucket-name/
(You’ll need a bucket you can write to).

Base64 Encode (Desperate): If no direct copy tools are available, and you can copy-paste from the terminal, you could base64 encode smaller critical files and paste them out. This is tedious and error-prone, but possible for a few key .py files if all else fails:
base64 important_file.py (then copy the output)

HTTP Upload (Less Likely): If curl is available, and you have a server you can POST to, you could try that.
curl -X POST --data-binary @my-functions-backup.tar.gz https://your-endpoint/upload

Check for UI Download: Sometimes, web-based IDEs/terminals have a hidden “download file” option if you right-click.

Check Firebase/Google Cloud Console Directly (Outside Workbench):

Firebase Functions Workbench might be built on top of other Google Cloud services like Cloud Workstations or a custom GCE setup.

Go to the Google Cloud Console (console.cloud.google.com) for the project associated with your Firebase project.

Cloud Storage: See if your project files were ever synced to a Cloud Storage bucket by default.

Cloud Workstations: If your Firebase project uses Cloud Workstations as a backend for Functions Workbench, you might see your workstation listed there. There could be options to access its persistent disk:

Take a snapshot of the persistent disk.

Attach the disk to another (working) VM to retrieve files.

This is more advanced, but a possibility.

Source Repositories: Had you connected your Functions Workbench to a Cloud Source Repository, GitHub, etc.? If so, your latest committed code is safe there.

Priority #2: Attempting to Fix the Environment (AFTER Code Backup)

Once your code is safely backed up, you can be more aggressive.

In Recovery Mode (if you have a terminal):

Identify the Problematic Python Environment: Your pip install was trying to create/populate a virtual environment. This is likely inside your project directory (e.g., .venv, venv, env) or a global cache.

Delete the Virtual Environment: If you find a venv or similar directory within your project, delete it: rm -rf /workspace/my-functions-project/.venv

Clear Pip Cache: Pip caches downloads. This cache might be full or corrupted.

rm -rf ~/.cache/pip

If running as root in recovery: rm -rf /root/.cache/pip

Delete the Erroneous Copy: If you copied your project to /etc, delete that copy: rm -rf /etc/my-functions-project (Be very careful with rm -rf in /etc!). Make sure you’re deleting the copy, not your primary project if it somehow ended up there.

Examine requirements.txt: Are there any unusually large or complex dependencies? For testing, you could try commenting out some non-essential ones to see if a minimal environment builds.

Check Disk Space Again: After deletions, use df -h to see the disk space situation on different partitions. Pay attention to /, /home, /tmp, or wherever your project/cache lives.

“Reset VM” / “Reset Workspace”:

Firebase Studio might have an option like “Reset Workspace” or “Factory Reset” for the Functions Workbench that is more thorough than just “Reset VM.” This might clear out the persistent state that’s causing the build to fail. This is a destructive option, so ensure your code is backed up first.

Understanding the 23GB “Full” Error on /sda:

Even if /sda (mounted perhaps at / or another major mount point) had 23GB, the specific directory pip was trying to write to (e.g., a subdirectory, /tmp, or ~/.cache) might have been on a different, smaller partition that was full.

Sometimes, even with space, inode exhaustion can look like a “disk full” error, though less common.

The cp -r … /etc might have filled up the root partition if your project (unlikely with just source) contained large data files or a previously built venv.

If you can’t get code out via Recovery Mode:

Contact Firebase/Google Cloud Support: This is your best bet if you cannot access the files yourself. They have backend access and might be able to retrieve the data from the persistent disk associated with your Functions Workbench instance. Explain the situation clearly, emphasizing the need to recover files from the persistent workspace.

Moving Forward (Once Resolved):

Always commit your code to a version control system (Git: GitHub, GitLab, Cloud Source Repositories) frequently.

Be mindful of disk space when installing large dependencies.

Understand where your development environment stores virtual environments and caches.

Avoid placing user projects in system directories like /etc. Standard user locations are /home/username/, or /workspace/ if provided by the IDE.

1 Like

Unfortunately Recovery Mode doesn’t work. Not even a terminal. It just fails and returns to the same screen shown above.