Project

General

Profile

Feature #10704

Improve documentation describing development deployments

Added by Evgeny Novikov 21 days ago. Updated 11 days ago.

Status:
Closed
Priority:
High
Category:
Documentation
Target version:
Start date:
02/08/2021
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

Pavel reported that he had an extremely slow update process of Klever in the development mode after size of the Bridge media directory became ~3 GB. It can take about an hour to update Klever! I don't see such the behavior with large directory build bases (~10 GB) on our CI when using pip 20.1. Since the development mode is rather important we need to deal with this issue.

History

#1

Updated by Ilya Shchepetkov 21 days ago

Storing large amount of data in the Klever repository is strongly discouraged. If it is necessary, however, then the only possible way to avoid copying of said data is to use "editable" mode during pip install (-e option). This solution is currently recommended by pip developers.

pip 20.1, that you use in CI, contained another, experimental solution to this problem: building of local Python packages in place, without copying them to the temporary folder. Unfortunately, this experimental solution had a number of issues and was quickly reverted in pip 20.1.1. These issues are still not solved 8 months later, so it is probably a very bad idea to keep using pip 20.1.

#2

Updated by Evgeny Novikov 21 days ago

We experienced strange issues multiple times with "-e" option, so, it is not a good idea to force users to leverage it. Perhaps, we should think about avoiding storing large amount of data in the Klever repository. I guess that at the moment there are just 2 typical sources of it:
  • The first one is build bases. To avoid it I think that we can add a special option, say, "--klever-data-directory" to klever-deploy-local, so that users will be able to provide custom locations of build bases rather than to use deployment scripts to install them. In fact, I guess that this is relevant just for CI.
  • The Bridge media directory. This stems from the fact that in the development mode Bridge is configured and launched directly from sources. I think that here it is possible to follow almost the same way as for the production mode but instead of using "/var/www" deployment scripts should use the Klever deployment directory.
#3

Updated by Evgeny Novikov 19 days ago

  • Category changed from Deployment to Documentation
After all, I suggest to improve the documentation a bit:
  • Specify how to upgrade Klever Python package in the development mode.
  • Do not require using dangerous option "--allow-allow-symbolic-links" (completely remove section "Deployment for Development Purposes").
  • Specify how to deal with possible failures - likely to remove a virtual environment and to create it from scratch. I hope that this will not be required often.
#4

Updated by Evgeny Novikov 18 days ago

BTW, note "This additional command, however, should be skipped if Klever Python package was installed in editable mode (with -e flag)." is not correct when it is necessary to update dependencies (Python packages) of Klever.

#5

Updated by Evgeny Novikov 17 days ago

  • Subject changed from Speed up deployment with large size of repository to Improve documentation describing development deployments
#6

Updated by Evgeny Novikov 17 days ago

  • Assignee changed from Ilya Shchepetkov to Evgeny Novikov

I will do this myself.

#7

Updated by Evgeny Novikov 11 days ago

  • Status changed from New to Closed

I did this in branch fix-dev-deploy-doc and immediately merged this branch to master in 242f5f2d9 since it touches just documentation and does not require testing.

Also available in: Atom PDF