What fills a data scientist’s Mac: environments, caches and datasets
- Environments and data, in about equal measure. A data scientist’s Mac fills with conda and Python environments, package caches, and copies of the same dataset: the download, the unzipped folder and the cleaned version.
- What it costs you: environments and caches come back from an
environment.ymlorrequirements.txt. Datasets come back only if their source still has them. - Keep: raw data you can’t download again, notebooks, and the files that rebuild each environment.
1. What fills a data scientist’s Mac
Every experiment leaves an environment, a cache and some data behind, and each tool keeps them in its own place:
| What | Where it lives | Comes back? | More |
|---|---|---|---|
| conda’s package cache | pkgs in your conda install, such as ~/miniconda3/pkgs or ~/miniforge3/pkgs | Yes, downloaded again | conda |
| conda environments | envs in the same folder | From an environment.yml, if you kept one | conda |
| Project virtual environments | .venv or venv in each project | From requirements.txt, pyproject.toml or a lockfile | Python environments |
| pip and uv caches | ~/Library/Caches/pip, ~/.cache/uv | Yes | pip, uv |
| Hugging Face datasets | ~/.cache/huggingface/hub for downloads, ~/.cache/huggingface/datasets for Arrow files | Yes, while they’re on the Hub | Hugging Face cache |
| scikit-learn sample datasets | ~/scikit_learn_data | Downloaded again | — |
| Raw downloads and their unzipped copies | Downloads, project data folders | Only from the source | Downloads |
| Notebook checkpoints and embedded outputs | .ipynb_checkpoints, plots saved inside .ipynb files | Checkpoints yes; outputs by running again | — |
| Experiment logs and model checkpoints | mlruns, wandb, saved weights | No: they record your runs | — |
| Compiled bytecode | __pycache__ next to your modules | Yes, on the next run | pycache folders |
Measure the shared parts, biggest last, then list your conda environments:
du -sh ~/miniconda3/pkgs ~/miniforge3/pkgs ~/anaconda3/pkgs ~/Library/Caches/pip ~/.cache/uv ~/.cache/huggingface ~/scikit_learn_data 2>/dev/null | sort -h
conda env list
And find data files over 500 MB in your usual folders (add your project folders to the list):
find ~/Documents ~/Downloads -type f -size +500M 2>/dev/null
2. What’s safe to clear, and what to keep
Environments and caches
- conda: run
conda clean --all --dry-runto see what would go, then again without--dry-run. It removes the index cache, unused packages, tarballs, temporary files and logs. Before removing packages, conda checks only the environments thatconda info --envslists, so make sure ones you created elsewhere with--prefixappear there. - Old environments: run
conda env export > environment.ymlfirst if you might want one back, thenconda env remove -n <name>. - pip and uv:
pip cache purgeclears pip’s wheel and HTTP caches.uv cache pruneremoves only unused entries anduv cache cleaneverything; uv’s docs say never to change its cache by hand. - Virtual environments in finished projects can go once their requirements are written down.
Data: decide file by file
- Keep the raw data you can’t download again, and one copy of each cleaned dataset. The usual waste is the third copy: a
.zip, the folder it unzipped to, and a converted Parquet file next to both. - Hugging Face’s processed Arrow files are rebuilt from the downloads; from Python,
dataset.cleanup_cache_files()removes one dataset’s cache files. - Notebooks with many embedded plots grow large.
jupyter nbconvert --clear-output --inplace notebook.ipynbstrips the outputs; it overwrites the file, so try it on a copy first. - Experiment folders like
mlrunsandwandbare the record of every run. Archive the ones you report on before deleting.
3. How SiftMac helps a data scientist
SiftMac’s developer files list a Python environment, whatever it’s called, when it has a pyvenv.cfg and a list of its packages beside it: requirements.txt, pyproject.toml, environment.yml, uv.lock, poetry.lock, Pipfile or setup.py. The project must be unchanged for 1, 3 or 6 months, and an environment a Python process is running from is skipped. conda environments in conda’s own folder aren’t listed.
- pip’s and Poetry’s caches in
~/Library/Cachesget the verdict Rebuilds. uv’s cache in~/.cache/uvisn’t cleaned by SiftMac; the space map shows its size. - Hugging Face models are listed one by one under AI files as You decide. Datasets in the same cache aren’t, so remove those with
hf cache rm. - Duplicates finds identical files of 1 MB or more in Desktop, Documents, Downloads, Movies, Music and Pictures and keeps the best copy. It skips any folder holding a
.gitand any Python environment, so data inside a repository isn’t compared. - Large & old files lists files of 1 GB or more, and those of 200 MB or more nobody has opened for a year, in the same folders.
- Old downloads lists a
.zipin Downloads once its unzipped folder sits next to it, or once you opened it more than a week ago. - Duplicates, large files and old downloads wait on the Undo Shelf for 7 days before the space is freed.
4. A cleanup routine for a data scientist’s Mac
- Write down every environment you need:
conda env export,pip freeze > requirements.txtoruv lock. - Run
conda clean --all, after a dry run. - Remove conda environments and virtual environments of finished projects.
- Clear the pip and uv caches.
- Sort data: keep raw and final versions, delete intermediate copies and unzipped duplicates.
- Remove Hugging Face datasets you no longer load.
- Archive the experiment folders you report on to an external drive, then remove them locally.
Questions
Is it safe to delete conda’s pkgs folder by hand?
Better not. Environments can link back to packages in it, and deleting by hand can break them. conda clean --all removes only what no listed environment uses.
Can I delete .ipynb_checkpoints folders?
Yes. They’re Jupyter’s autosaved copies of your notebooks; deleting them only removes the option to revert to the last checkpoint. They’re usually small, so they’re rarely the space problem.
Why doesn’t SiftMac find duplicates in my data folders?
If the data lives inside a git repository, SiftMac skips the whole repository on purpose, because files that look identical in projects often belong there. Data in ordinary folders under Documents, Downloads or Desktop is compared.
How SiftMac handles it. SiftMac lists Python environments in projects you haven’t changed for months when the file that rebuilds them is present, lists Hugging Face models one by one, and finds duplicate and large data files in your own folders, sending those to the Undo Shelf first.
Download SiftMac · scanning is free, and so is your first 2 GB of cleaning · See how it works