Help in improving pip usage

[Brian Wawok]

I am using pip in three different steps. pip install -r requirements.txt takes almost 60 seconds to run. So I am spending several minutes installing the same requirements in three different spots.  What am I missing and how can I make this better?

Simplified workflow.. but even with the cache it is killing me

image: python:3.6.3

options:
  docker: true

pipelines:
  branches:
    master:
      - parallel:
        - step:
            name: Build Images
            caches:
              - docker
              - pip
            script:
              - pip install -r requirements.txt
              - ./build/build.sh
        - step:
            name: Test Code
            caches:
              - pip
            script:
              - pip install -r requirements.txt
              - python ./manage.py test
            services:
              - postgres
      - step:
          name: Do something else
          caches:
            - pip
          script:
            - pip install -r requirements.txt
            - python ./do_something_else

definitions:
  services:
    postgres:
      image: postgres:10.3

Most of the pip time is spent downloading.. can I also cache that somehow?

Collecting sqlparse>=0.2.0 (from django-debug-toolbar==1.9.1->-r requirements.txt (line 17))  Downloading https://files.pythonhosted.org/packages/65/85/20bdd72f4537cf2c4d5d005368d502b2f464ede22982e724a82c86268eda/sqlparse-0.2.4-py2.py3-none-any.whl
Collecting python-memcached (from djcacheutils==3.0.0->-r requirements.txt (line 18))  Downloading https://files.pythonhosted.org/packages/f5/90/19d3908048f70c120ec66a39e61b92c253e834e6e895cd104ce5e46cbe53/python_memcached-1.59-py2.py3-none-any.whl
Collecting django-appconf (from django-celery-email==2.0.0->-r requirements.txt (line 20))  Downloading https://files.pythonhosted.org/packages/5b/78/726cdf3e04660560cf25f9def95b8f2736310c581dabed9adfe60154a9f8/django_appconf-1.0.2-py2.py3-none-any.whl
Collecting py-moneyed>=0.7 (from django-money==0.12.3->-r requirements.txt (line 21))  Downloading https://files.pythonhosted.org/packages/4c/2c/3c57645cddb0e64aac10749829a843eb5e3aab8e067e2c870d98c38a7947/py_moneyed-0.7.0-py3-none-any.whl
(212kB)Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage==1.8.0->-r requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/e3/39/b082c5bd0095d2baf6638338b66c5b6d2e8e372f605a1a85f33c0abe5533/google_api_core-1.1.1-py2.py3-none-any.whl 

[Brian Wawok]

For future me..

This is what I did and it helped a fair bit

I defined a new cache of the install directory for python3.6

definitions:
  caches:
    python-packages: /usr/local/lib/python3.6/site-packages/

And then I added this cache in addition to the pip cache to my steps

name: Test Code
caches:
  - pip
  - python-packages
script:
  - pip install -r requirements.txt
  - pip install -r requirements-test.txt
  - python ./manage.py test

That took my total pip down (including downloading caches) down from:

75 seconds (no caches)

60 seconds (pre-defined pip cache)

10 seconds (adding in python-packages cache)