Project

General

Profile

Actions

Bug #5691

closed

Error: Your application used more memory than the safety cap of 1024M.

Added by Alexander Kamkin over 9 years ago. Updated over 9 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Andrei Tatarnikov
Category:
-
Target version:
Start date:
03/06/2015
Due date:
% Done:

100%

Estimated time:
Detected in build:
svn
Platform:
Published in build:
150324

Description

Template: random.rb (miniMIPS)

1000000.times { ... }
Actions #1

Updated by Alexander Kamkin over 9 years ago

It seems that the generator stores all generated test cases in memory. It should be implemented in a stream manner.

Actions #2

Updated by Andrei Tatarnikov over 9 years ago

  • Assignee changed from Andrei Tatarnikov to Alexander Kamkin
  • Priority changed from Normal to Urgent

Stream manner will be useful. However, these is another problem. In my opinion, the "1000000.times { ... }" Ruby construct is not a proper way to repeat instructions in a test template.

Here is what causes the error: The N.times Ruby feature causes the same Ruby code being executed N times to create the same Java objects (N of them). When N is significantly high, memory problems arise. A solution could be the following: as soon as Ruby code has created a Java representation of a single test (or initializating/finalizing block), this represenation must be immediately processed (sequence generation, data generation, similation, printing) and abandoned (left for the garbage collector to release from memory). This is what you call the stream manner. However, this will work only when the N.times consruct is applied to a top-level block. E.g like this:

  def run
    int32_dist = dist(range(:value => 0,                      :bias => 25),  # Zero
                      range(:value => 1..2,                   :bias => 25),  # Small
                      range(:value => 0xffffFFFE..0xffffFFFF, :bias => 50))  # Large

    100.times {
      atomic {
        add t0, t1, t2 do situation('random_biased', :size => 32,
          :dist => dist(range(:value=> int32_dist,              :bias => 80),  # Simple
                        range(:value=> [0xDEADBEEF, 0xBADF00D], :bias => 20))) # Magic
        end
      }
    }
end

In this case, the code inside the N.times block can be considered a separate test and processed individually (I can implement this). However, if you do like this:

  def run
    int32_dist = dist(range(:value => 0,                      :bias => 25),  # Zero
                      range(:value => 1..2,                   :bias => 25),  # Small
                      range(:value => 0xffffFFFE..0xffffFFFF, :bias => 50))  # Large

    block {
      100.times {
        atomic {
          add t0, t1, t2 do situation('random_biased', :size => 32,
            :dist => dist(range(:value=> int32_dist,              :bias => 80),  # Simple
                          range(:value=> [0xDEADBEEF, 0xBADF00D], :bias => 20))) # Magic
          end
        }
      }
    }
end

the code inside in the N.times block is part of a bigger test and cannot be processed independently. So the same problem will occur.

I see the solution as supporting constructs like this:

atomic (:repeat => 1000000) {
  ...
}

In this case, only one instance of instructions inside the block will be created, but it will be processed 1000000 times. This is much better from the performance point of view and it will work in any place in a test template.

Summary:

N.times is bad because:
  • The same code (Ruby and Java interaction is executed) N times to create the same objects. This will cause a performance penalty.
  • Multiple copies of the same objects are created. This cases memory erasing. In some cases, the scheme "create" -> "process" -> "throw way" -> "take another" will not work (the repeated fragment must be an independent test case, which is not always true).

atomic (:repeat => 1000000) is good because:

  • Only one copy which is processed N times (the iterator returns the same sequence N times).
  • No processor time is spent to create N identical objects (no time-consuming interactions between Ruby and Java).
  • No memory is wasted to store N identical objects.
  • This will work in any place in a test template (not only for top-level blocks)

TODO

  1. Design and implements the atomic (:repeat => 1000000) feature
  2. Consider processing in a stream manner as further optimization

Any objections?

Actions #3

Updated by Alexander Kamkin over 9 years ago

  • Category set to 43
  • Assignee changed from Alexander Kamkin to Andrei Tatarnikov

A solution could be the following: as soon as Ruby code has created a Java representation of a single test (or initializating/finalizing block), this represenation must be immediately processed (sequence generation, data generation, similation, printing) and abandoned (left for the garbage collector to release from memory). This is what you call the stream manner.

Yes, that's what I mean. Let's do it.

However, this will work only when the N.times consruct is applied to a top-level block.

That's right. Using N.times construct inside blocks is possible but not recommended (the precise semantics should be described in the manual).

N.times is bad because:

No, N.times is not bad. It can be used at the top level, but we don't recommend using it inside blocks.

atomic (:repeat => 1000000) is good because:

Such construct should be be implemented, but don't do it in a hurry - we need to discuss the issue.

Actions #4

Updated by Alexander Kamkin over 9 years ago

  • Subject changed from [generator] Error: Your application used more memory than the safety cap of 1024M. to Error: Your application used more memory than the safety cap of 1024M.
Actions #5

Updated by Andrei Tatarnikov over 9 years ago

  • Status changed from New to Open
  • % Done changed from 0 to 50

A solution could be the following: as soon as Ruby code has created a Java representation of a single test (or initializating/finalizing block), this represenation must be immediately processed (sequence generation, data generation, similation, printing) and abandoned (left for the garbage collector to release from memory). This is what you call the stream manner.

Yes, that's what I mean. Let's do it.

Done in r3304.

Actions #6

Updated by Andrei Tatarnikov over 9 years ago

atomic (:repeat => 1000000) is good because:

Such construct should be be implemented, but don't do it in a hurry - we need to discuss the issue.

Task #5716

Actions #7

Updated by Andrei Tatarnikov over 9 years ago

  • Status changed from Open to Resolved
  • % Done changed from 50 to 100

This bug can be considered resolved.

Actions #8

Updated by Andrei Tatarnikov over 9 years ago

  • Status changed from Resolved to Closed
  • Published in build set to 150324
Actions

Also available in: Atom PDF