Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upgrade bazel buildfarm to v2.7.0 #12

Draft
wants to merge 59 commits into
base: bazel-ios-fork
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
ec4b82c
Manipulate worker set directly in RSB
werkt Sep 26, 2023
c48d600
Remove publishTtlMetric option
werkt Sep 26, 2023
2abad58
Config-compatible behavior for publishTtlMetric
werkt Oct 4, 2023
10189d6
Correct logging advisements for current Java
werkt Sep 26, 2023
0c30cb0
Rename GracefulShutdownTest
werkt Oct 3, 2023
5a0d8d6
Remove WebController
werkt Oct 3, 2023
fa8dd6f
Interrupt+Join operationQueuer/dispatchMonitor
werkt Oct 3, 2023
60093db
Present operationNames by stage
werkt Oct 3, 2023
9ff8e64
Remove subscribeToBackplane, adjust failsafe op
werkt Sep 26, 2023
b21ab2e
Removing AWS/GCP Metrics and Admin controls
werkt Sep 28, 2023
f97f048
Remove unused setOnCancelHandler
werkt Oct 3, 2023
c2c1544
Update BWoB docs for ensureOutputsPresent
werkt Oct 4, 2023
09015c9
Disable Bzlmod explicitly in .bazelrc
meteorcloudy Oct 6, 2023
1041b57
Log write errors with worker address
amishra-u Sep 29, 2023
30412ad
Revert "Use integer ids for Sqlite bidirectional index"
80degreeswest Oct 8, 2023
8379068
Common String.format for PipelineStage
werkt Oct 3, 2023
fd20d13
Cleanup matched logic in SWC listener
werkt Oct 3, 2023
632fceb
Refactor SWC matcher and clarify Nullable
werkt Oct 3, 2023
7385c74
Interrupt matchStage to induce prepare shutdown
werkt Oct 3, 2023
79cf648
Specify example config with grpc target
werkt Oct 3, 2023
d73628b
Remove SpringBoot usage
werkt Oct 3, 2023
6f88e34
Enable graceful shutdown for server (#1490)
80degreeswest Oct 10, 2023
81122e7
refactor: code cleanup
jasonschroeder-sfdc Oct 6, 2023
ad44854
Log paths created on putDirectory
werkt Jun 22, 2023
b31ffed
Permit regex realInputDirectories
werkt Jul 7, 2023
1dec684
Log execPath rather than the cache dir path
werkt Jul 7, 2023
6af857a
Shore up OutputDirectory for silence on duplicates
werkt Jul 7, 2023
71a928c
Trigger realInputDirectories to have empty files
werkt Jul 7, 2023
f8017bd
Switch to positive check for linkInputDirectories
werkt Jul 8, 2023
5d88e54
docs(configuration): document --prometheus_port CLI argument
jasonschroeder-sfdc Oct 12, 2023
21448c3
docs(configuration): readability and typos
jasonschroeder-sfdc Oct 12, 2023
461cc5a
style(configuration.md): table formatting
jasonschroeder-sfdc Oct 12, 2023
b74531c
feat: support --redis_uri command line option
jasonschroeder-sfdc Oct 6, 2023
d860b27
docs(configuration): document the --redis_uri command line options
jasonschroeder-sfdc Oct 12, 2023
10bf139
Example should use `container_image` instead of `java_image`
sousajo-cc Oct 11, 2023
9d4c86c
chore: bump rules_jvm_external
jasonschroeder-sfdc Sep 22, 2023
5bc4564
chore: bump rules_cc
jasonschroeder-sfdc Sep 22, 2023
edf2211
Implement local resources for workers (#1282)
luxe Oct 17, 2023
97e3b90
build: override grpc dependencies with our dependencies
jasonschroeder-sfdc Oct 16, 2023
96f239d
chore(deps): bump protobuf runtime to 3.19.1
jasonschroeder-sfdc Oct 16, 2023
af3f34e
chore(deps) add transitive dependencies
jasonschroeder-sfdc Oct 16, 2023
380f8a1
feat: add Proto reflection service to shard worker
jasonschroeder-sfdc Oct 16, 2023
7e7979d
fixup! build: override grpc dependencies with our dependencies
jasonschroeder-sfdc Oct 16, 2023
1f9d01f
fixup! chore(deps) add transitive dependencies
jasonschroeder-sfdc Oct 16, 2023
578589f
Bug: Fix Blocked thread in WriteStreamObserver Caused by CASFile Writ…
amishra-u Oct 24, 2023
dfa5937
Pin the Java toolchain to `remotejdk_17` (#1509)
stefanobaghino Oct 24, 2023
f6459d1
docs: add markdown language specifiers for code blocks
jasonschroeder-sfdc Oct 18, 2023
018e177
Support OutputPaths in OutputDirectory
werkt Oct 25, 2023
8b37013
Permit Absolute Symlink Targets with configuration
werkt Oct 26, 2023
df9ce1d
chore: update bazel to 6.4.0 (#1513)
jasonschroeder-sfdc Oct 29, 2023
bd740c9
Rename instance types (#1514)
luxe Oct 30, 2023
2a61f77
Create SymlinkNode outputs during upload (#1515)
werkt Oct 30, 2023
76c2657
feat: Implement CAS lease extension (#1455)
amishra-u Oct 31, 2023
cfa2e18
Bump org.json:json from 20230227 to 20231013 in /admin/main (#1516)
dependabot[bot] Oct 31, 2023
3167fc5
Remove custom Spring exit
chenj-hub Sep 16, 2024
2b9b79b
Merge commit 'cfa2e18724914034eb19c4c01f293117610fa0b7' into jackies/…
chenj-hub Sep 23, 2024
7225f81
Pass failing tests
chenj-hub Sep 23, 2024
5a507ce
Temporarily make worker public
chenj-hub Sep 26, 2024
cd5591a
Re-add missing graceful shutdown functionality (#1520)
80degreeswest Nov 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: add markdown language specifiers for code blocks
jasonschroeder-sfdc authored and werkt committed Oct 24, 2023
commit f6459d199d1a40bdd8b31a75d7c19da7a1af69f4
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -19,17 +19,17 @@ All commandline options override corresponding config settings.

Run via

```
docker run -d --rm --name buildfarm-redis -p 6379:6379 redis:5.0.9
```shell
$ docker run -d --rm --name buildfarm-redis -p 6379:6379 redis:5.0.9
redis-cli config set stop-writes-on-bgsave-error no
```

### Bazel Buildfarm Server

Run via

```
bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- <logfile> <configfile>
```shell
$ bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- <logfile> <configfile>

Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```
@@ -40,8 +40,8 @@ Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=

Run via

```
bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- <logfile> <configfile>
```shell
$ bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- <logfile> <configfile>

Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml

@@ -53,9 +53,9 @@ Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm

To use the example configured buildfarm with bazel (version 1.0 or higher), you can configure your `.bazelrc` as follows:

```
```shell
$ cat .bazelrc
build --remote_executor=grpc://localhost:8980
$ build --remote_executor=grpc://localhost:8980
```

Then run your build as you would normally do.
@@ -67,20 +67,20 @@ Buildfarm uses [Java's Logging framework](https://docs.oracle.com/javase/10/core
You can use typical Java logging configuration to filter these results and observe the flow of executions through your running services.
An example `logging.properties` file has been provided at [examples/logging.properties](examples/logging.properties) for use as follows:

```
bazel run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```

and

```
bazel run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
``` shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```

To attach a remote debugger, run the executable with the `--debug=<PORT>` flag. For example:

```
bazel run //src/main/java/build/buildfarm:buildfarm-server -- --debug=5005 $PWD/examples/config.minimal.yml
```shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-server -- --debug=5005 $PWD/examples/config.minimal.yml
```


32 changes: 16 additions & 16 deletions _site/docs/configuration/configuration.md
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ has_children: true

Minimal required:

```
```yaml
backplane:
redisUri: "redis://localhost:6379"
queues:
@@ -38,7 +38,7 @@ For an example configuration containing all of the configuration values, see `ex

Example:

```
```yaml
digestFunction: SHA1
defaultActionTimeout: 1800
maximumActionTimeout: 1800
@@ -79,7 +79,7 @@ worker:

Example:

```
```yaml
server:
instanceType: SHARD
name: shard
@@ -96,7 +96,7 @@ server:

Example:

```
```yaml
server:
grpcMetrics:
enabled: false
@@ -114,7 +114,7 @@ server:

Example:

```
```yaml
server:
caches:
directoryCacheMaxEntries: 10000
@@ -132,7 +132,7 @@ server:

Example:

```
```yaml
server:
admin:
deploymentEnvironment: AWS
@@ -151,14 +151,14 @@ server:

Example:

```
```yaml
server:
metrics:
publisher: log
logLevel: INFO
```

```
```yaml
server:
metrics:
publisher: aws
@@ -207,7 +207,7 @@ server:

Example:

```
```yaml
backplane:
type: SHARD
redisUri: "redis://localhost:6379"
@@ -224,7 +224,7 @@ backplane:

Example:

```
```yaml
backplane:
type: SHARD
redisUri: "redis://localhost:6379"
@@ -262,7 +262,7 @@ backplane:
| realInputDirectories | List of Strings, _external_ | | A list of paths that will not be subject to the effects of linkInputDirectories setting, may also be used to provide writable directories as input roots for actions which expect to be able to write to an input location and will fail if they cannot |
| gracefulShutdownSeconds | Integer, 0 | | Time in seconds to allow for operations in flight to finish when shutdown signal is received |

```
```yaml
worker:
port: 8981
publicName: "localhost:8981"
@@ -279,7 +279,7 @@ worker:

Example:

```
```yaml
worker:
capabilities:
cas: true
@@ -296,7 +296,7 @@ worker:

Example:

```
```yaml
worker:
sandboxSettings:
alwaysUse: true
@@ -313,7 +313,7 @@ worker:

Example:

```
```yaml
worker:
dequeueMatchSettings:
acceptEverything: true
@@ -333,7 +333,7 @@ worker:

Example:

```
```yaml
worker:
storages:
- type: FILESYSTEM
@@ -361,7 +361,7 @@ worker:

Example:

```
```yaml
worker:
executionPolicies:
- name: test
26 changes: 17 additions & 9 deletions _site/docs/execution/execution_policies.md
Original file line number Diff line number Diff line change
@@ -17,7 +17,7 @@ This policy type specifies that a worker should prepend a single path, and a num

This example will use the buildfarm-provided executable `as-nobody`, which will upon execution demote itself to a `nobody` effective process owner uid, and perform an `execvp(2)` with the remaining provided program arguments, which will subsequently execute as a user that no longer matches the worker process.

```
```yaml
# default wrapper policy application
worker:
executionPolicies:
@@ -50,32 +50,37 @@ These wrappers are used for detecting actions that rely on time. Below is a dem
This addresses two problems in regards to an action's dependence on time. The 1st problem is when an action takes longer than it should because it's sleeping unnecessarily. The 2nd problem is when an action relies on time which causes it to eventually be broken on master despite the code not changing. Both problems are expressed below as unit tests. We demonstrate a time-spoofing mechanism (the re-writing of syscalls) which allows us to detect these problems generically over any action. The objective is to analyze builds for performance inefficiency and discover future instabilities before they occur.

### Issue 1 (slow test)
```

```bash
#!/bin/bash
set -euo pipefail

echo -n "testing... "
sleep 10;
echo "done"
```

The test takes 10 seconds to run on average.
```
bazel test --runs_per_test=10 --config=remote //cloud/buildfarm:sleep_test

```shell
$ bazel test --runs_per_test=10 --config=remote //cloud/buildfarm:sleep_test
//cloud/buildfarm:sleep_test PASSED in 10.2s
Stats over 10 runs: max = 10.2s, min = 10.1s, avg = 10.2s, dev = 0.0s
```

We can check for performance improvements by using the `skip-sleep` option.
```
bazel test --runs_per_test=10 --config=remote --remote_default_exec_properties='skip-sleep=true' //cloud/buildfarm:sleep_test

```shell
$ bazel test --runs_per_test=10 --config=remote --remote_default_exec_properties='skip-sleep=true' //cloud/buildfarm:sleep_test
//cloud/buildfarm:sleep_test PASSED in 1.0s
Stats over 10 runs: max = 1.0s, min = 0.9s, avg = 1.0s, dev = 0.0s
```

Now the test is 10x faster. If skipping sleep makes an action perform significantly faster without affecting its success rate, that would warrant further investigation into the action's implementation.

### Issue 2 (future failing test)
```

```bash
#!/bin/bash
set -euo pipefail

@@ -89,12 +94,15 @@ echo "Times change."
date
exit -1;
```

The test passes today, but will it pass tomorrow? Will it pass a year from now? We can find out by using the `time-shift` option.
```
bazel test --test_output=streamed --remote_default_exec_properties='time-shift=31556952' --config=remote //cloud/buildfarm:future_fail

```shell
$ bazel test --test_output=streamed --remote_default_exec_properties='time-shift=31556952' --config=remote //cloud/buildfarm:future_fail
INFO: Found 1 test target...
Times change.
Mon Sep 25 18:31:09 UTC 2023
//cloud/buildfarm:future_fail FAILED in 18.0s
```

Time is shifted to the year 2023 and the test now fails. We can fix the problem before others see it.
25 changes: 15 additions & 10 deletions _site/docs/execution/execution_properties.md
Original file line number Diff line number Diff line change
@@ -76,37 +76,42 @@ Despite being given 1 core, they see all of the cpus and decide to spawn that ma

**Standard Example:**
This test will succeed when env var TESTVAR is foobar, and fail otherwise.
```

```shell
#!/bin/bash
[ "$TESTVAR" = "foobar" ]
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
FAIL
```

```
./bazel test --remote_default_exec_properties='env-vars={"TESTVAR": "foobar"}' \
```shell
$ ./bazel test --remote_default_exec_properties='env-vars={"TESTVAR": "foobar"}' \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
PASS
```

**Template Example:**
If you give a range of cores, buildfarm has the authority to decide how many your operation actually claims. You can let buildfarm resolve this value for you (via [mustache](https://mustache.github.io/)).
```
```bash
#!/bin/bash
[ "$MKL_NUM_THREADS" = "1" ]
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
FAIL
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_default_exec_properties='env-vars="MKL_NUM_THREADS": "{{limits.cpu.claimed}}"' \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
25 changes: 15 additions & 10 deletions _site/docs/quick_start.md
Original file line number Diff line number Diff line change
@@ -25,7 +25,8 @@ Let's start with a bazel workspace with a single file to compile into an executa
Create a new directory for our workspace and add the following files:

`main.cc`:
```

```c
#include <iostream>

int main( int argc, char *argv[] )
@@ -35,7 +36,8 @@ int main( int argc, char *argv[] )
```

`BUILD`:
```

```starlark
cc_binary(
name = "main",
srcs = ["main.cc"],
@@ -118,15 +120,18 @@ That `2 remote` indicates that your compile and link ran remotely. Congratulatio
## Container Quick Start

To bring up a minimal buildfarm cluster, you can run:

```shell
$ ./examples/bf-run start
```
./examples/bf-run start
```

This will start all of the necessary containers at the latest version.
Once the containers are up, you can build with `bazel run --remote_executor=grpc://localhost:8980 :main`.

To stop the containers, run:
```
./examples/bf-run stop

```shell
$ ./examples/bf-run stop
```

## Next Steps
@@ -137,8 +142,8 @@ We've started our worker on the same host as our server, and also the same host

You can now easily launch a new Buildfarm cluster locally or in AWS using an open sourced [Buildfarm Manager](https://github.com/80degreeswest/bfmgr).

```
wget https://github.com/80degreeswest/bfmgr/releases/download/1.0.7/bfmgr-1.0.7.jar
java -jar bfmgr-1.0.7.jar
Navigate to http://localhost
```shell
$ wget https://github.com/80degreeswest/bfmgr/releases/download/1.0.7/bfmgr-1.0.7.jar
$ java -jar bfmgr-1.0.7.jar
$ open http://localhost
```