A Mechanically Checked Generation of Correlating Programs Directed by Structured Syntactic Differences

This page presents the correlating_program tool implementing correlating program generation, as described in our paper, along with detailed experimental results on a series of examples.

Scope of the tool

As described in our paper, the correlating_program tool only covers the generation of a correlating program, that is, a static interleaving of two programs' instructions in a way that preserves their semantics.

It does not cover the statical analysis of the resulting correlated program. In our experiments, we performed this analysis through abstract interpretation using a slightly patched version of Eran and Yahav's dizy tool.

Changes with regards to the paper

The correlating_program tool handles an extension of the language described in our paper, in which various arithmetic operators have been added, as well as array values. The approach presented in our paper naturally extends to those changes, and both the algorithm and the proofs have been updated accordingly.

Note on languages

We have chosen to represent the input language using a concrete syntax close to that of the C language. Although its semantics do not exactly match C's, we believe it to be equivalent to the subset of C corresponding to the concrete syntax of our language.

Likewise, our syntactic diff language is designed to resemble the widely-used unified diff format, in order to be immediately readable by those familiar with the latter. Those two languages, however, are fundamentally incompatible, as one is syntactic while the other is purely textual.

Experimental results

While the soundness of our correlating_program is proven in the Coq proof assistant, we evaluate the quality of its output by comparing the results of its analysis by dizy.

Protocol

Our comparison is done on a selection of difference examples presented as pairs of programs in our source language. For each of those pairs:

  1. A representation Δ of the syntactic differences between the two programs of the pair is computed using a simple recursive algorithm.
  2. The two input programs are translated to actual C for ccc.
  3. A first correlating program is generated from the two C programs using ccc.
  4. A second correlating program is generated from Δ using correlating_program.
  5. Dizy is used to analyze both correlating programs, and its output is used to tell if the return value of each considered function may have changed (✗) or not (✓).

All of this is automated by a Python (run_tests.py) script that takes a file listing the tests along with the expected result (equivalent or non-equivalent), and an optional counter-example to the correction of the correlating program.

Results

Example Expected result ccc + dizy correlating_program + dizy
00-identity
[patched] [syntactic diff]

(equivalent)

(0.08 seconds)

(0.06 seconds)
01-rename
[patched] [syntactic diff]

(equivalent)

(0.05 seconds)

(0.01 seconds)
02-split
[patched] [syntactic diff]

(equivalent)

(0.09 seconds)

(0.06 seconds)
03-move
[patched] [syntactic diff]

(equivalent)

(0.08 seconds)

(0.06 seconds)
04-fibo
[patched] [syntactic diff]

(equivalent)

(0.62 seconds)

(0.60 seconds)
05-move
[patched] [syntactic diff]

(equivalent)

(0.70 seconds)

(0.41 seconds)
06-split
[patched] [syntactic diff]

(equivalent)

(0.74 seconds)

(0.58 seconds)
07-bounds
[patched] [syntactic diff]

(equivalent)

(1.39 seconds)

(1.56 seconds)
08-compose
[patched] [syntactic diff]

(equivalent)

(1.55 seconds)

(1.39 seconds)
09-gcd
[patched] [syntactic diff]

(equivalent)

(0.04 seconds)

(0.16 seconds)
10-equiv-expr
[patched] [syntactic diff]

(equivalent)

(0.13 seconds)

(0.20 seconds)
11-identity-nonlinear
[patched] [syntactic diff]

(equivalent)

(0.13 seconds)

(0.10 seconds)
12-square-root
[patched] [syntactic diff]

(equivalent)

(6.59 seconds)

(5.07 seconds)
13-looptwice
[patched] [syntactic diff]

(equivalent)

(17.75 seconds)

(93.40 seconds)
14-move
[patched] [syntactic diff]

(equivalent)

(0.83 seconds)

(0.56 seconds)
15-two-loops
[patched] [syntactic diff]

(equivalent)

(1200.00 seconds)

(1200.00 seconds)
16-break
[patched] [syntactic diff]

(nonequivalent)

(0.10 seconds)

(0.12 seconds)
17-break-move
[patched] [syntactic diff]

(equivalent)

(0.37 seconds)

(0.48 seconds)
18-subst-nonequivalent
[patched] [syntactic diff]

(nonequivalent)

(0.07 seconds)

(0.07 seconds)
19-move-nonequivalent
[patched] [syntactic diff]

(nonequivalent)

(0.93 seconds)

(0.67 seconds)
20-sum
[patched] [syntactic diff]

(equivalent)

(0.12 seconds)

(0.06 seconds)
21-search
[patched] [syntactic diff]

(equivalent)

(5.53 seconds)

(3.93 seconds)
22-counter-machine
[patched] [syntactic diff]

(equivalent)

(1200.00 seconds)

(1200.00 seconds)
23-parity_step_id
[patched] [syntactic diff]

(equivalent)

(1200.00 seconds)

(38.84 seconds)