Klever

A side note.
Should we generate code like below at all?

int ldv_EMGentry_init_chd_dec_module_init_29_10(int (*arg0)(void)) {
  /* Callback init_chd_dec_module_init */
  return (chd_dec_module_init)();
}

Alexey, all callbacks should be called in the files where corresponding functions are declared and your example contains such a function which is called at a control function with process/automaton logic. So the answer is yes, we always should generate them.

We decided that comparison of model call forests starting from callback actions (including pre- and post-calls) non depending on the order of callback actions looks to be a quite good criterion to be used by default. I suggest to use the following formula: similarity = the number of completely equal model call forests / the maximum number of model call forests. If there is no call forests in both error traces than they can be considered as equal.

Evgeny Novikov wrote:

similarity = the number of completely equal model call forests / the maximum number of model call forests.

What is model call forests? Is it a set of trees, where tree is internal actions inside a model call?

Alexey Khoroshilov wrote:

Evgeny Novikov wrote:

similarity = the number of completely equal model call forests / the maximum number of model call forests.

What is model call forests? Is it a set of trees, where tree is internal actions inside a model call?

Not a model but a callback that has some model call at some level of its stack. So it may be something like that:

pre_probe()
  pre_probe_model()
probe()
  ...
  func1_model()
  ...
  func2_model()
  ...
post_probe()
  post_probe_model()

This issue may be resolved in manual filtering (implemented in branch new_format_of_strategies) by removing from compared error trace all functions, which correspond to EMG. Thus, it is possible to create one mark, which will be relevant for different versions of EMG.

Vitaly Mordan wrote:

This issue may be resolved in manual filtering (implemented in branch new_format_of_strategies) by removing from compared error trace all functions, which correspond to EMG. Thus, it is possible to create one mark, which will be relevant for different versions of EMG.

This issue is about automatic error traces processing.

Evgeny Novikov wrote:

Everybody are so tired to re-create marks due to continuous changes in EMG that actually doesn't influence on found Unsafes.

But marks are always created manually. Because of this issue we did not use marks mechanism at all during previous evaluation of results.
After mark has been created, it will be automatically compared with new error traces.

Vitaly Mordan wrote:

Evgeny Novikov wrote:

Everybody are so tired to re-create marks due to continuous changes in EMG that actually doesn't influence on found Unsafes.

But marks are always created manually. Because of this issue we did not use marks mechanism at all during previous evaluation of results.

It does differ if one needs to push one or two buttons or to make some additional markup to create a new mark. Here no additional manual markup is assumed.
And this wasn't the only reason why we didn't use marks. Until all specifications will be established well there is actually very little sense to create marks.

Evgeny Novikov wrote:

I tried it for the error trace from the attached job and observed some issues when see on the converted error trace:

1. By some reason usb_serial_module_init() is at the end of the list of functions although it should be at the beginning. Although this likely doesn't matter much since comparison shouldn't respect the order of forests, but it would be nice to see the correct order.
2. It looks like all functions from execution context starting from callback actions are added to forests, that is model function calls aren't treated at all. We need to add to forest trees just function stacks finishing with model function calls.
3. The string format to represent trees is quite hard to read and to edit. I suggest to use something more user friendly. This doesn't belong just to this concrete feature request but this can be done here.

One more issue is that file forests.json is created in directory bridge.

Evgeny Novikov wrote:

I tried it for the error trace from the attached job and observed some issues when see on the converted error trace:

1. By some reason usb_serial_module_init() is at the end of the list of functions although it should be at the beginning. Although this likely doesn't matter much since comparison shouldn't respect the order of forests, but it would be nice to see the correct order.

The only order I can do now: all forests of the first found thread, then forests of second thread and so on.

1. It looks like all functions from execution context starting from callback actions are added to forests, that is model function calls aren't treated at all. We need to add to forest trees just function stacks finishing with model function calls.

Ok, I'll do it.

1. The string format to represent trees is quite hard to read and to edit. I suggest to use something more user friendly. This doesn't belong just to this concrete feature request but this can be done here.

I have already added 2 formats, the one you saw is human-readable. I can add spaces to it and line breakes. But this will increase volume of converted traces and time for comparing it. But with this format I can't exclude functions for your second issue.

The second format is like:
[forest1, forest2, ...] where forest is: [function_name, [child1, child2, ...]] if function_name has children or [function_name] in other cases. The child has the same format as parent.
The first option is to add to final list of forests json.dumps(forest). I think it is faster to compare strings, not the full structures.
The second is to save and compare full structure of forests. It is not so optimal but in visualization it will be nice.
Which option do you want?

Update 1: for the second issue I have to rewrite the whole converter. So you can offer any converted error traces format.

Update 2: Ok, I've rewrite converter so there are 3 supported formats for each forest:
1) "function1{...}..."
2) [ [function1, [child1, ...] ], ...]
3) [{function1: [child1, ...]}, ...] where children have the same format as parents.

2nd and 3d formats are easy-to-read, but there will be (2*N*f) calls of json.dumps() after each mark is created, where N - number of error traces with the same attribute values which are compared in mark, f - the average number of forests in each unsafe. I had to sacrifice performance so that you can read error traces because all converted error traces should have the same format: json.dumps(converted, indent=2). This format is nively visualized. Visulazition should not depend on convertion function.

Vladimir Gratinskiy wrote:

Evgeny Novikov wrote:

I tried it for the error trace from the attached job and observed some issues when see on the converted error trace:
By some reason usb_serial_module_init() is at the end of the list of functions although it should be at the beginning. Although this likely doesn't matter much since comparison shouldn't respect the order of forests, but it would be nice to see the correct order.

The only order I can do now: all forests of the first found thread, then forests of second thread and so on.

So, I don't understand how forests of a thread intended for module initialization are put after forests for all other threads.

The string format to represent trees is quite hard to read and to edit. I suggest to use something more user friendly. This doesn't belong just to this concrete feature request but this can be done here.

I have already added 2 formats, the one you saw is human-readable. I can add spaces to it and line breakes. But this will increase volume of converted traces and time for comparing it. But with this format I can't exclude functions for your second issue.

The second format is like:
[forest1, forest2, ...] where forest is: [function_name, [child1, child2, ...]] if function_name has children or [function_name] in other cases. The child has the same format as parent.
The first option is to add to final list of forests json.dumps(forest). I think it is faster to compare strings, not the full structures.
The second is to save and compare full structure of forests. It is not so optimal but in visualization it will be nice.
Which option do you want?

Update 1: for the second issue I have to rewrite the whole converter. So you can offer any converted error traces format.

Update 2: Ok, I've rewrite converter so there are 3 supported formats for each forest:
1) "function1{...}..."
2) [ [function1, [child1, ...] ], ...]
3) [{function1: [child1, ...]}, ...] where children have the same format as parents.

2nd and 3d formats are easy-to-read, but there will be (2*N*f) calls of json.dumps() after each mark is created, where N - number of error traces with the same attribute values which are compared in mark, f - the average number of forests in each unsafe. I had to sacrifice performance so that you can read error traces because all converted error traces should have the same format: json.dumps(converted, indent=2). This format is nively visualized. Visulazition should not depend on convertion function.

JSON is a very good format since it allows to represent both simple (call stacks, call forests, etc.) and complex (everything imaginable) converted error traces, it is easy to read, to edit and to parse.

As for performance, I am not sure that this will really matter much. Converted error traces will be most likely quite small, and usually there won't be very many similar error traces by corresponding attributes.

Evgeny Novikov wrote:

Looks great! Let's consider performance issues after all, especially because of there shouldn't be considerable ones here.

The only bad thing was done is modification of bridge/.idea/misc.xml. Was it intentional? If not, please, revert it.

It has been changed after pycharm update. Let's use the last versions of pycharm files?