Feature #10704
closed
Improve documentation describing development deployments
Added by Evgeny Novikov almost 4 years ago.
Updated almost 4 years ago.
Description
Pavel reported that he had an extremely slow update process of Klever in the development mode after size of the Bridge media directory became ~3 GB. It can take about an hour to update Klever! I don't see such the behavior with large directory build bases (~10 GB) on our CI when using pip 20.1. Since the development mode is rather important we need to deal with this issue.
Storing large amount of data in the Klever repository is strongly discouraged. If it is necessary, however, then the only possible way to avoid copying of said data is to use "editable" mode during pip install (-e option). This solution is currently recommended by pip developers.
pip 20.1, that you use in CI, contained another, experimental solution to this problem: building of local Python packages in place, without copying them to the temporary folder. Unfortunately, this experimental solution had a number of issues and was quickly reverted in pip 20.1.1. These issues are still not solved 8 months later, so it is probably a very bad idea to keep using pip 20.1.
We experienced strange issues multiple times with "-e" option, so, it is not a good idea to force users to leverage it. Perhaps, we should think about avoiding storing large amount of data in the Klever repository. I guess that at the moment there are just 2 typical sources of it:
- The first one is build bases. To avoid it I think that we can add a special option, say, "--klever-data-directory" to klever-deploy-local, so that users will be able to provide custom locations of build bases rather than to use deployment scripts to install them. In fact, I guess that this is relevant just for CI.
- The Bridge media directory. This stems from the fact that in the development mode Bridge is configured and launched directly from sources. I think that here it is possible to follow almost the same way as for the production mode but instead of using "/var/www" deployment scripts should use the Klever deployment directory.
- Category changed from Deployment to Documentation
After all, I suggest to improve the documentation a bit:
- Specify how to upgrade Klever Python package in the development mode.
- Do not require using dangerous option "--allow-allow-symbolic-links" (completely remove section "Deployment for Development Purposes").
- Specify how to deal with possible failures - likely to remove a virtual environment and to create it from scratch. I hope that this will not be required often.
BTW, note "This additional command, however, should be skipped if Klever Python package was installed in editable mode (with -e flag)." is not correct when it is necessary to update dependencies (Python packages) of Klever.
- Subject changed from Speed up deployment with large size of repository to Improve documentation describing development deployments
- Assignee changed from Ilya Shchepetkov to Evgeny Novikov
- Status changed from New to Closed
I did this in branch fix-dev-deploy-doc and immediately merged this branch to master in 242f5f2d9 since it touches just documentation and does not require testing.
Also available in: Atom
PDF