Skip to content

Commit

Permalink
Upgrade .NET to 8.0 and update build scripts. (#22)
Browse files Browse the repository at this point in the history
  • Loading branch information
m-carrasco authored Feb 3, 2024
1 parent 9e9dbb7 commit 9ae091f
Show file tree
Hide file tree
Showing 22 changed files with 165 additions and 111 deletions.
7 changes: 7 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.dockerignore

ci/docker/Dockerfile
ci/docker/Dockerfile.base
ci/build-image.sh

integration-test/output
28 changes: 10 additions & 18 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
- name: Setup .NET
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.0.x'
dotnet-version: '8.0.x'
- name: Install Souffle
run: brew install --HEAD souffle-lang/souffle/souffle
- name: Build Souffle program
Expand Down Expand Up @@ -55,14 +55,14 @@ jobs:
- name: Setup .NET
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.0.x'
dotnet-version: '8.0.x'
- name: Build dotnet
run: |
dotnet build
dotnet test --verbosity normal
build-linux:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2
with:
Expand All @@ -85,37 +85,29 @@ jobs:
- name: Setup .NET
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.0.x'
dotnet-version: '8.0.x'
- name: Setup cmake
uses: jwlawson/[email protected]
with:
cmake-version: '3.21.x'
- name: Setup dependencies
run: |
sudo chmod +x ci/install-lit.sh && sudo ./ci/install-lit.sh
sudo chmod +x ci/install-llvm.sh && sudo ./ci/install-llvm.sh
sudo chmod +x ci/install-mono.sh && sudo ./ci/install-mono.sh
sudo ./ci/install-lit.sh
sudo ./ci/install-llvm.sh
sudo ./ci/install-mono.sh
- name: Build Souffle programs from scratch
run: |
cd $GITHUB_WORKSPACE/souffle && rm -r bin/ && ./build-all-with-docker.sh
- name: Build dotnet
run: |
dotnet build
dotnet test --verbosity normal
- name: Configure LIT test cases
run: |
mkdir build
cd build
cmake ..
- name: Integration test
run: |
cd build
lit integration-test -v
release:
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
needs: [build-macos, build-windows, build-linux]
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2
with:
Expand All @@ -138,7 +130,7 @@ jobs:
- name: Setup .NET
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.0.x'
dotnet-version: '8.0.x'
- name: Build Souffle programs from scratch
run: |
cd $GITHUB_WORKSPACE/souffle && rm -r bin/ && ./build-all-with-docker.sh
Expand All @@ -159,7 +151,7 @@ jobs:
mirror:
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
needs: [build-macos, build-windows, build-linux]
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-ubuntu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ jobs:
- name: Build the Docker image
# On purpose Souffle programs aren't compiled from scratch.
# This is helpful if we forget to update the ones shipped in the repository.
run: cd $GITHUB_WORKSPACE && docker build . --file Dockerfile --tag net-ssa:net-ssa
run: cd $GITHUB_WORKSPACE && ./ci/build-image.sh
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Custom entries

build/
integration-test/output/

## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
Expand Down
21 changes: 0 additions & 21 deletions Dockerfile

This file was deleted.

2 changes: 1 addition & 1 deletion Dockerfile-souffle
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:20.04
FROM ubuntu:22.04

COPY . /net-ssa

Expand Down
73 changes: 38 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,35 @@
# net-ssa [![.NET (build, test and release if necessary)](https://github.com/m-carrasco/net-ssa/actions/workflows/build.yml/badge.svg?branch=main&event=push)](https://github.com/m-carrasco/net-ssa/actions/workflows/build.yml) [![Docker image](https://github.com/m-carrasco/net-ssa/actions/workflows/docker-ubuntu.yml/badge.svg?branch=main&event=push)](https://github.com/m-carrasco/net-ssa/actions/workflows/docker-ubuntu.yml) ![Nuget](https://img.shields.io/nuget/v/net-ssa-lib)

Microsoft's high-level programming languages, such as C#, are compiled to CIL bytecode. The instruction set architecture of CIL operates on a stack virtual machine with local variables. CIL instruction's operands are implicit because they are elements in the stack. `net-ssa` provides a register-based intermediate representation for CIL where operands become explicit.

Using CIL properties, it is possible to know for every instruction which slots of the stack it consumes. Similarly, it is possible to know how many elements it pushes into the stack. `net-ssa` computes its initial representation promoting stack slots into registers. In this form, a stack slot promoted to register can be defined more than once. Local variables are accessed through store and load instructions (like LLVM-IR).
Microsoft's high-level programming languages, such as C#, are compiled into Common Intermediate Language (CIL) bytecode. The CIL instruction set operates on a stack virtual machine with implicit operands, as they are elements within the stack. `net-ssa` introduces a register-based intermediate representation for CIL, making operands explicit.

The initial register-based representation can be transformed into SSA form. SSA guarantees that every register is only defined once, and its unique definition dominates its uses. This transformation is based on dominance frontiers and partially implemented in Datalog.
CIL properties allow `net-ssa` to determine the stack slots consumed and the pushed elements amount for each instruction. The initial representation involves promoting stack slots into registers using this information. In this stage, a stack slot promoted to a register may have multiple definitions. Local variables are accessed through store and load instructions, similar to LLVM-IR.

`net-ssa` can be either used as a library `net-ssa-lib` or as command-line application `net-ssa-cli`.
The initial register-based representation can undergo transformation into Static Single Assignment (SSA) form. SSA ensures that each register is defined only once, and its unique definition dominates its uses. This transformation relies on dominance frontiers and is partially implemented in Datalog.

If you have any questions or suggestions, feel free to open an issue to discuss it.
`net-ssa` can be employed either as a library (`net-ssa-lib`) or as a command-line application (`net-ssa-cli`).

Feel free to open an issue to discuss any questions or suggestions you may have.

### Table of Contents

* [Quick setup](#quick-setup)
* [Build from sources](#build-from-sources)
* [Build native dependencies](#build-native-dependencies)
* [Example: net-ssa-cli](#disassembling-with-net-ssa-cli)
* [Example: net-ssa-lib](#disassembling-with-net-ssa-lib)
* [Type inference analysis](#type-inference-analysis)
* [Contributing](#contributing)
* [Acknowledgements](#acknowledgements)
- [net-ssa ](#net-ssa---)
- [Table of Contents](#table-of-contents)
- [Mirror](#mirror)
- [Quick setup](#quick-setup)
- [Build from sources](#build-from-sources)
- [Ubuntu 22.04](#ubuntu-2204)
- [Windows and macOS](#windows-and-macos)
- [Build native dependencies](#build-native-dependencies)
- [Examples](#examples)
- [Disassembling with net-ssa-cli](#disassembling-with-net-ssa-cli)
- [Disassembling with net-ssa-lib](#disassembling-with-net-ssa-lib)
- [Type inference analysis](#type-inference-analysis)
- [Simple type inference analysis](#simple-type-inference-analysis)
- [Precise type inference analysis](#precise-type-inference-analysis)
- [Contributing](#contributing)
- [Acknowledgements](#acknowledgements)
- [License](#license)

## Mirror

Expand All @@ -29,43 +39,36 @@ This is caused by Github's Git LFS bandwidth, which is 1GB per month.
## Quick setup

It is possible to develop and test `net-ssa` without installing any dependency in your system but [Docker](https://docs.docker.com/get-docker/).
However, it is adviced to compile the project at least once in the host system. This is mainly for downloading dependencies and correctly setting up any IDE.

1. `git clone [email protected]:m-carrasco/net-ssa.git`
2. `cd net-ssa`
3. `git lfs checkout`
* Install [git lfs](https://git-lfs.github.com/)
4. `dotnet build && dotnet test`
* This is optional, it requires installing dotnet.
5. `docker build -t net-ssa/net-ssa .`
6. `docker run --name dev -v $(pwd):/net-ssa -ti net-ssa/net-ssa`
* This is now an interactive container. `$(pwd)` of the host is shared with the container as `net-ssa` source code folder.
7. Introduce changes in the source code using your IDE as usual.
8. Build and test in the container, execute these commands in the container terminal:
* `cd build`
* `(cd /net-ssa && dotnet build)`
* `lit integration-test/ -vvv`
* `exit # once you finish working`
9. `docker start -i dev # to resume the container terminal`

4. `./scripts/build-image.sh`
5. `./scripts/tmp-container.sh`
* This is now an interactive and temporary container.
* The host's folder containing the repository is shared with the container. In the container, this is located at `/home/ubuntu/net-ssa/`.
6. Introduce changes in the source code using your IDE as usual.
7. Build and test in the container, execute these commands in the container terminal:
* `dotnet build`
* `dotnet test`
* `lit integration-test/ -vvv`

## Build from sources

### Ubuntu 20.04
### Ubuntu 22.04

1. `cd net-ssa`
2. `git lfs checkout`
* Install [git lfs](https://git-lfs.github.com/)
3. `dotnet build`
4. `dotnet test`
5. `mkdir build`
6. `cd build && cmake ..`
7. `lit integration-test/ -vvv`
5. `lit integration-test/ -vvv`

To know the required dependencies for the integration tests (`cmake` and `lit` step), please check the [Dockerfile](https://github.com/m-carrasco/net-ssa/blob/main/Dockerfile).
Please check the [Dockerfile](https://github.com/m-carrasco/net-ssa/blob/main/Dockerfile) to know which dependencies must be installed.
The Dockerfile executes the shell scripts under the [ci](https://github.com/m-carrasco/net-ssa/tree/main/ci) folder. You would just need to execute them once in your system.

### Windows and MacOS
### Windows and macOS

The steps are the same as in Ubuntu. The project building and unit testing is done in the CI. Yet, the integration tests aren't configured.
Anyway, the dependencies should be the same as in Ubuntu. If you encounter any problem while trying this, please open an issue.
Expand All @@ -75,9 +78,9 @@ Anyway, the dependencies should be the same as in Ubuntu. If you encounter any p
`net-ssa` has native dependencies, which are shipped in the project already. You shouldn't need to build them. Usually, this is only required for development. The supported systems are:
* Linux - x86-64
* Windows - x86-64
* MacOS - x86-64 and arm64
* macOS - x86-64 and arm64

In case they must be re-built, [`./net-ssa/souffle/build-all-with-docker.sh`](https://github.com/m-carrasco/net-ssa/blob/main/souffle/build-all-with-docker.sh) is available. The script compiles these dependencies from scratch using the source code in your repository. Under the hood, the script isolates this process using Docker. This only builds the Linux and Windows dependencies. Cross-compilation for MacOS is incredible difficult. If you are a MacOS user, check the CI to figure out the required dependencies and execute `build-souffle-macos-x86-64-arm64.sh`.
In case they must be re-built, [`./net-ssa/souffle/build-all-with-docker.sh`](https://github.com/m-carrasco/net-ssa/blob/main/souffle/build-all-with-docker.sh) is available. The script compiles these dependencies from scratch using the source code in your repository. Under the hood, the script isolates this process using Docker. This only builds the Linux and Windows dependencies. Cross-compilation for macOS is incredible difficult. If you are a macOS user, check the CI to figure out the required dependencies and execute `build-souffle-macos-x86-64-arm64.sh`.

## Examples

Expand Down
12 changes: 12 additions & 0 deletions ci/build-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash
set -e

CURRENT_DIR=$(dirname "$(readlink -f "$0")")

pushd $CURRENT_DIR/..

BASE_IMAGE=net-ssa-base:latest
docker build --file $CURRENT_DIR/docker/Dockerfile.base --build-arg USER_ID=$(id -u) --build-arg GROUP_ID=$(id -g) -t $BASE_IMAGE .
docker build --file $CURRENT_DIR/docker/Dockerfile --build-arg BASE_IMAGE=$BASE_IMAGE -t net-ssa:latest .

popd
8 changes: 8 additions & 0 deletions ci/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
ARG BASE_IMAGE
FROM ${BASE_IMAGE}

COPY --chown=ubuntu:mygroup . ${NET_SSA_SRC_DIR}

RUN dotnet build && \
dotnet test --verbosity normal && \
lit ./integration-test -vv
28 changes: 28 additions & 0 deletions ci/docker/Dockerfile.base
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
ARG REPO=mcr.microsoft.com/dotnet/sdk
FROM $REPO:8.0-jammy

ARG USER_ID
ARG GROUP_ID

# Create a new group and user with the specified IDs
# Set up passwordless sudo access for the non-root user
RUN groupadd -g $GROUP_ID mygroup && \
useradd -u $USER_ID -g $GROUP_ID ubuntu && \
apt-get update && \
apt-get install --no-install-recommends -y sudo && \
echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# Switch to the new user
USER ubuntu
RUN sudo mkdir -p /home/ubuntu && sudo chown -R ubuntu:mygroup /home/ubuntu

ENV NET_SSA_SRC_DIR=/home/ubuntu/net-ssa/
COPY --chown=ubuntu:mygroup ./ci/ ${NET_SSA_SRC_DIR}/ci

RUN sudo ${NET_SSA_SRC_DIR}/ci/install-souffle.sh && \
sudo ${NET_SSA_SRC_DIR}/ci/install-lit.sh && \
sudo ${NET_SSA_SRC_DIR}/ci/install-llvm.sh && \
sudo ${NET_SSA_SRC_DIR}/ci/install-mono.sh


WORKDIR ${NET_SSA_SRC_DIR}
3 changes: 0 additions & 3 deletions ci/install-cmake.sh

This file was deleted.

Empty file modified ci/install-lit.sh
100644 → 100755
Empty file.
2 changes: 1 addition & 1 deletion ci/install-llvm.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#!/bin/bash
set -e
apt-get install -y clang++-12 llvm-12 llvm-12-tools && ln -s /usr/bin/llvm-config-12 /usr/bin/llvm-config
apt-get install -y clang++-12 llvm-12 llvm-12-tools
Empty file modified ci/install-mingw32.sh
100644 → 100755
Empty file.
Empty file modified ci/install-mono.sh
100644 → 100755
Empty file.
Empty file modified ci/install-souffle.sh
100644 → 100755
Empty file.
8 changes: 8 additions & 0 deletions ci/tmp-container.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash
set -e

CURRENT_DIR=$(dirname "$(readlink -f "$0")")

pushd $CURRENT_DIR/..
docker run --rm -it -v $(pwd):/home/ubuntu/net-ssa net-ssa:latest
popd
Loading

0 comments on commit 9ae091f

Please sign in to comment.