Project

General

Profile

Feature #9356

Do not deploy the same build bases

Added by Ilja Zakharov almost 2 years ago. Updated about 1 month ago.

Status:
Closed
Priority:
Urgent
Category:
Deployment
Target version:
Start date:
10/29/2018
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

Deployment scripts in the current version copy build bases each time. It is better to compare hashes for build bases to speedup update procedure.


Related issues

Related to Klever - Feature #9414: Reuse archive with source files between different error traces on report uploadingClosed01/25/201907/02/2019

Actions
Has duplicate Klever - Bug #9696: Klever deployment script removes build basesRejected06/03/2019

Actions
Blocks Klever - Feature #10101: Deploy addons and build bases within base imagesNew02/05/2020

Actions

History

#1

Updated by Evgeny Novikov almost 2 years ago

  • Related to Feature #9414: Reuse archive with source files between different error traces on report uploading added
#2

Updated by Evgeny Novikov almost 2 years ago

  • Description updated (diff)
  • Subject changed from Support hashing build bases to avoid copying them each time at OpenStack deployment to Do not deploy the same build bases
#3

Updated by Evgeny Novikov over 1 year ago

  • Assignee set to Evgeny Novikov
#4

Updated by Evgeny Novikov over 1 year ago

With https://github.com/17451k/clade/issues/41 we can distinguish build bases by their unique identifier.

#5

Updated by Evgeny Novikov over 1 year ago

  • Target version deleted (3.0)
  • Priority changed from Urgent to High

We have no time to implement these tasks in Klever 3.0.

#6

Updated by Evgeny Novikov 9 months ago

  • Blocks Feature #10101: Deploy addons and build bases within base images added
#7

Updated by Evgeny Novikov 8 months ago

  • Has duplicate Bug #9696: Klever deployment script removes build bases added
#8

Updated by Ilya Shchepetkov 4 months ago

  • Assignee changed from Evgeny Novikov to Ilya Shchepetkov
  • Status changed from New to Resolved

Implemented in the improve-openstack-deployment branch using rsync. Now Klever sources, addons and build bases are deployed only if there are some changes in them. As a consequence, deployment speed during "update" action is considerably faster then before.

#9

Updated by Evgeny Novikov 3 months ago

  • Target version set to 3.0
  • Priority changed from High to Urgent

Let's include this into 3.0.

#10

Updated by Evgeny Novikov 3 months ago

  • Target version deleted (3.0)
  • Priority changed from Urgent to High
  • Assignee deleted (Ilya Shchepetkov)
  • Status changed from Resolved to Open

Branch improve-openstack-deployment avoids copying the same local build bases to instances, but it does not solve the problem in general. In general we need some means like for addons which are not updated (nothing is downloaded, extracted and so on) until their versions change within a deployment configuration file.

#11

Updated by Ilya Shchepetkov about 2 months ago

  • Assignee set to Ilya Shchepetkov

Implemented in the deploy-build-bases branch, which also contains other improvements regarding deployment of build bases:

  • Now deployment scripts automatically extract build bases during deployment (previously they were simply copied)
  • Now build bases are deployed during 'update' action only if they are changed
  • Now deployment of build bases and Klever Addons uses the same code, which should simplify further improvements
#12

Updated by Evgeny Novikov about 2 months ago

  • Target version set to 3.0
  • Priority changed from High to Urgent

Since it is solved, let's include it within Klever 3.0.

#13

Updated by Evgeny Novikov about 2 months ago

  • Status changed from Open to Resolved

Likely resolved. Needs testing.

#14

Updated by Evgeny Novikov about 2 months ago

  • Status changed from Resolved to Open

It works, but due to ill-formed archives (e.g. https://forge.ispras.ru/attachments/7328) users may meet very strange issues because of different build bases can be intermixed in the unpredictable way within the same directory.

I suggest to check before installation to the final destination the following:
  1. A build base is placed within at least one top-level directory within an archive.
  2. There is not most low-level directory that contains the build base at the deployment directory already.

Perhaps, some assistance from Clade will be necessary, since just Clade knows what is a build base.

#15

Updated by Ilya Shchepetkov about 1 month ago

  • Status changed from Open to Resolved

I have added the following changes to the same deploy-build-bases branch:

  • Now data transferred by rsync is compressed (unless the data itself is an archive)
  • Now paths with spaces are handled correctly by rsync
  • Now there is an optional ability to manually specify versions of build bases (but by default versions are calculated automatically)
  • Now each build base has a unique name specified in the klever.json file. This name can be used in the job.json file to refer to the required build base.
#16

Updated by Evgeny Novikov about 1 month ago

  • Status changed from Resolved to Closed

We tested and fixed some additional related issues, after all everything seems to work perfect. I merged the branch to master in 4a52fad75. Everybody is welcome to deploy build bases within their Klever instances!

Important remark. Existent Klever instances deployed by means of scripts need some manual migrations, e.g. OpenStack needs the following (I hope that local instances can be easily reinstalled or you may try steps like below):
  1. Edit /etc/default/klever. Change KLEVER_DATA_DIR="/home/debian/klever-inst/klever/build bases" to KLEVER_DATA_DIR="/home/debian/klever-inst/build bases".
  2. Move directory /home/debian/klever-inst/klever/build bases to /home/debian/klever-inst/build bases.
  3. Restart the scheduler: "sudo systemctl restart klever-native-scheduler.service".

Also available in: Atom PDF