Skip to content

Commit

Permalink
Merge branch 'develop' into feature/GH-118-AWS_virtual_network
Browse files Browse the repository at this point in the history
  • Loading branch information
HildericSB committed Sep 10, 2020
2 parents f4cf22b + f7dbca0 commit c014b18
Show file tree
Hide file tree
Showing 192 changed files with 7,640 additions and 1,198 deletions.
1 change: 1 addition & 0 deletions .github/workflows/actions-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: actions-test

on: [repository_dispatch]


jobs:
build:

Expand Down
48 changes: 48 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,54 @@

## UNRELEASED

### FEATURES

* Added an ElasticSearch store for events and logs ([GH-658](https://github.com/ystia/yorc/issues/658))
* [Slurm] Expose Slurm scontrol show job results as job attributes ([GH-664](https://github.com/ystia/yorc/issues/664))

### SECURITY FIXES

* Fix [vulnerability in golang.org/x/crypto/ssh](https://snyk.io/vuln/SNYK-GOLANG-GOLANGORGXCRYPTOSSH-551923) by upgrading dependency

### ENHANCEMENTS

* Alien4Cloud download URL change ([GH-637](https://github.com/ystia/yorc/issues/637))
* Enhance logs and events long-polling performances on file storage ([GH-654](https://github.com/ystia/yorc/issues/654))

### BUG FIXES

* Yorc generates forcePurge tasks on list deployments API endpoint ([GH-674](https://github.com/ystia/yorc/issues/674))
* Yorc is getting slow when there is a lot of tasks ([GH-671](https://github.com/ystia/yorc/issues/671))
* Yorc does not build on Go1.15 ([GH-665](https://github.com/ystia/yorc/issues/665))
* Concurrency issue in workflows execution may lead to a task never reaching a terminal status ([GH-659](https://github.com/ystia/yorc/issues/659))
* Bootstrap on Centos 7 on GCP fails ([GH-649](https://github.com/ystia/yorc/issues/649))
* Kubernetes Client uses deprecated apis removed on recent versions of K8S (v1.17+) ([GH-645](https://github.com/ystia/yorc/issues/645))
* Bootstrap may failed with a nil pointer error if download of a component fails ([GH-634](https://github.com/ystia/yorc/issues/634))
* Missing concurrency limit during data migration for logs and events file storage ([GH-640](https://github.com/ystia/yorc/issues/640))
* Unable to undeploy a deployment in progress from Alien4Cloud ([GH-630](https://github.com/ystia/yorc/issues/630))

## 4.0.0 (April 17, 2020)

### BUG FIXES

* Location is not listed if its properties are missing ([GH-625](https://github.com/ystia/yorc/issues/625))
* Sometimes Yorc bootstrap fails to start local Yorc instance because Consul is not properly started ([GH-623](https://github.com/ystia/yorc/issues/623))

## 4.0.0-rc.1 (March 30, 2020)

### FEATURES

* Support TOSCA 1.3 inputs/outputs for workflows ([GH-556](https://github.com/ystia/yorc/issues/556))

### ENHANCEMENTS

* Yorc Tasks should have an error message when appropriate ([GH-613](https://github.com/ystia/yorc/issues/613))

### BUG FIXES

* CLI yorc locations list doesn't return HostsPool information ([GH-615](https://github.com/ystia/yorc/issues/615))
* Check for location type mismatch ([GH-550](https://github.com/ystia/yorc/issues/550))

## 4.0.0-M10 (March 10, 2020)

### FEATURES
Expand Down
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Ystia Orchestrator

[![Download](https://api.bintray.com/packages/ystia/yorc-engine/distributions/images/download.svg?version=4.0.0-M10)](https://bintray.com/ystia/yorc-engine/distributions/4.0.0-M10/link) [![Build Status](https://travis-ci.org/ystia/yorc.svg?branch=develop)](https://travis-ci.org/ystia/yorc) [![Documentation Status](https://readthedocs.org/projects/yorc/badge/?version=latest)](http://yorc.readthedocs.io/en/latest/?badge=latest) [![Go Report Card](https://goreportcard.com/badge/github.com/ystia/yorc)](https://goreportcard.com/report/github.com/ystia/yorc) [![license](https://img.shields.io/github/license/ystia/yorc.svg)](https://github.com/ystia/yorc/blob/develop/LICENSE) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) [![Docker Pulls](https://img.shields.io/docker/pulls/ystia/yorc.svg?style=flat)](https://hub.docker.com/r/ystia/yorc) [![Join the chat at https://gitter.im/ystia/yorc](https://badges.gitter.im/ystia/yorc.svg)](https://gitter.im/ystia/yorc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Download](https://api.bintray.com/packages/ystia/yorc-engine/distributions/images/download.svg?version=4.0.0)](https://bintray.com/ystia/yorc-engine/distributions/4.0.0/link) [![Build Status](https://travis-ci.org/ystia/yorc.svg?branch=release/4.0)](https://travis-ci.org/ystia/yorc) [![Documentation Status](https://readthedocs.org/projects/yorc/badge/?version=latest)](http://yorc.readthedocs.io/en/latest/?badge=latest) [![Go Report Card](https://goreportcard.com/badge/github.com/ystia/yorc)](https://goreportcard.com/report/github.com/ystia/yorc) [![license](https://img.shields.io/github/license/ystia/yorc.svg)](https://github.com/ystia/yorc/blob/develop/LICENSE) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) [![Docker Pulls](https://img.shields.io/docker/pulls/ystia/yorc.svg?style=flat)](https://hub.docker.com/r/ystia/yorc) [![Join the chat at https://gitter.im/ystia/yorc](https://badges.gitter.im/ystia/yorc.svg)](https://gitter.im/ystia/yorc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

<p align="center">
<img src="https://raw.githubusercontent.com/ystia/yorc/develop/doc/_static/logo/github-repository-open-graph-template.png" width="50%" height="50%" title="Yorc Logo">
</p>

---

Yorc is an hybrid cloud/HPC [TOSCA](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.2/TOSCA-Simple-Profile-YAML-v1.2.html) orchestrator.

Expand Down
6 changes: 5 additions & 1 deletion commands/bootstrap/inputs.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ var (
description: "Prefer OpenSSH over Paramiko, python implementation of SSH",
value: false,
},
"ansible.host_operations_allowed": defaultInputType{
description: "Allow to execute operations on orchestrator host",
value: false,
},
}

yorcDefaultInputs = map[string]defaultInputType{
Expand Down Expand Up @@ -115,7 +119,7 @@ var (
"alien4cloud.download_url": defaultInputType{
description: "Alien4Cloud download URL",
value: fmt.Sprintf(
"https://fastconnect.org/maven/content/repositories/opensource/alien4cloud/alien4cloud-dist/%s/alien4cloud-dist-%s-dist.tar.gz",
"https://www.portaildulibre.fr/nexus/repository/opensource-releases/alien4cloud/alien4cloud-premium-dist/%s/alien4cloud-premium-dist-%s-dist.tar.gz",
alien4cloudVersion, alien4cloudVersion),
},
"alien4cloud.port": defaultInputType{
Expand Down
1 change: 1 addition & 0 deletions commands/bootstrap/resources/topology/topology_ha.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,7 @@ topology_template:
properties:
use_openssh: {{.Ansible.UseOpenSSH}}
inventory: {{formatAsJSON .Ansible.Inventory}}
host_operations_allowed: {{.Ansible.HostOperationsAllowed}}
requirements:
- ansibleYorcServer:
type_requirement: host
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ topology_template:
properties:
use_openssh: {{.Ansible.UseOpenSSH}}
inventory: {{formatAsJSON .Ansible.Inventory}}
host_operations_allowed: {{.Ansible.HostOperationsAllowed}}
requirements:
- ansibleYorcServer:
type_requirement: host
Expand Down
Binary file modified commands/bootstrap/resources/topology/tosca_types.zip
Binary file not shown.
69 changes: 37 additions & 32 deletions commands/bootstrap/setup.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
package bootstrap

import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
Expand Down Expand Up @@ -114,6 +114,7 @@ func setupYorcServer(workingDirectoryPath string) error {
KeepGeneratedRecipes: true,
UseOpenSSH: inputValues.Ansible.UseOpenSSH,
Inventory: inputValues.Ansible.Inventory,
HostedOperations: config.HostedOperations{UnsandboxedOperationsAllowed: inputValues.Ansible.HostOperationsAllowed},
},
}

Expand Down Expand Up @@ -360,51 +361,56 @@ func startConsul(workingDirectoryPath string) (*exec.Cmd, error) {
fmt.Println("Consul is already running")
return nil, nil
}

fmt.Println("Starting Consul...")
consulLogPath := filepath.Join(workingDirectoryPath, "consul.log")
consulLogFile, err := os.OpenFile(consulLogPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
return nil, err
}
fmt.Printf("Starting Consul (logs: %s)", consulLogPath)
// Consul startup options
// Specifying a bind address or it would fail on a setup with multiple
// IPv4 addresses configured
cmdArgs := "agent -server -bootstrap-expect 1 -bind 127.0.0.1 -data-dir " + dataDir
cmd = exec.Command(executable, strings.Split(cmdArgs, " ")...)
output, _ := cmd.StdoutPipe()
cmd.Stdout = consulLogFile
err = cmd.Start()
if err != nil {
return nil, err
}

// Wait for a new leader to be elected, else Yorc could try to access Consul
// when it is not yet ready
err = waitForOutput(output, "New leader elected", 60*time.Second)
if err != nil {
cleanBootstrapSetup(workingDirectoryPath)
return nil, err
}

fmt.Println("...Consul started")
waitForConsulReadiness("http://127.0.0.1:8500")
fmt.Println(" Consul started!")
return cmd, err
}

func waitForOutput(output io.ReadCloser, expected string, timeout time.Duration) error {

reader := bufio.NewReader(output)
timer := time.NewTimer(timeout)
defer timer.Stop()
func waitForConsulReadiness(consulHTTPEndpoint string) {
for {
select {
case <-timer.C:
return fmt.Errorf("Timeout waiting for %s", expected)
default:
line, err := reader.ReadString('\n')
if err != nil {
return err
}
if strings.Contains(line, expected) {
return nil
}
fmt.Print(".")
leader, _ := getConsulLeader(consulHTTPEndpoint)
if leader != "" {
return
}
<-time.After(2 * time.Second)
}
}

func getConsulLeader(consulHTTPEndpoint string) (string, error) {
resp, err := http.Get(fmt.Sprintf("%s/v1/status/leader", consulHTTPEndpoint))
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", errors.New(resp.Status)
}
bodyBytes, err := ioutil.ReadAll(resp.Body)
matches := regexp.MustCompile(`"(.*)"`).FindStringSubmatch(string(bodyBytes))
if len(matches) < 2 {
return "", nil
}
return matches[1], nil
}

// downloadDependencies downloads Yorc Server dependencies
Expand Down Expand Up @@ -481,17 +487,16 @@ func download(url, fileName, destinationPath string, overwrite bool) (string, er
defer outputFile.Close()

response, err := http.Get(url)

//bar
bar := pb.Full.Start(int(response.ContentLength))
barReader := bar.NewProxyReader(response.Body)

if err != nil {
outputFile.Close()
return filePath, err
}
defer response.Body.Close()

//bar
bar := pb.Full.Start(int(response.ContentLength))
barReader := bar.NewProxyReader(response.Body)

_, err = io.Copy(outputFile, barReader)
outputFile.Close()
bar.Finish()
Expand Down
111 changes: 111 additions & 0 deletions commands/bootstrap/setup_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
// Copyright 2018 Bull S.A.S. Atos Technologies - Bull, Rue Jean Jaures, B.P.68, 78340, Les Clayes-sous-Bois, France.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package bootstrap

import (
"net/http"
"net/http/httptest"
"testing"
"time"

"gotest.tools/v3/assert"
)

func Test_waitForConsulReadiness(t *testing.T) {
type endpoint struct {
started bool
httpStatus int
text string
}
tests := []struct {
name string
endpoint endpoint
shouldSucceed bool
}{
{"NormalStartup", endpoint{true, 200, `"127.0.0.1:8300"`}, true},
{"ConsulStartedButNotReady", endpoint{true, 200, `""`}, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {

ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(tt.endpoint.httpStatus)
w.Write([]byte(tt.endpoint.text))
}))
if !tt.endpoint.started {
// close now!
ts.Close()
} else {
defer ts.Close()
}
c := make(chan struct{}, 0)
go func() {
waitForConsulReadiness(ts.URL)
close(c)
}()
var timedOut bool
select {
case <-c:
case <-time.After(6 * time.Second):
timedOut = true
}
if tt.shouldSucceed == timedOut {
t.Errorf("expecting waitForConsulReadiness() to succeed but it timedout")
} else if !tt.shouldSucceed && !timedOut {
t.Errorf("expecting waitForConsulReadiness() to timeout but it succeeded")
}
})
}
}

func Test_getConsulLeader(t *testing.T) {
type endpoint struct {
started bool
httpStatus int
text string
}
tests := []struct {
name string
endpoint endpoint
want string
wantErr bool
}{
{"NormalStartup", endpoint{true, 200, `"127.0.0.1:8300"`}, "127.0.0.1:8300", false},
{"ConsulStartedButNotReady", endpoint{true, 200, `""`}, "", false},
{"ConsulStartedButBadResponse", endpoint{true, 200, `qsdhjgy@àà*ùù$ugfbdnflv`}, "", false},
{"ConsulStartedButBadStatus", endpoint{true, 500, ``}, "", true},
{"ConsulNotStarted", endpoint{false, 0, ``}, "", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(tt.endpoint.httpStatus)
w.Write([]byte(tt.endpoint.text))
}))
defer ts.Close()
got, err := getConsulLeader(ts.URL)
if (err != nil) != tt.wantErr {
t.Errorf("getConsulLeader() error = %v, wantErr %v", err, tt.wantErr)
return
}
if err != nil && tt.endpoint.httpStatus == 500 {
assert.ErrorContains(t, err, "500")
}
if got != tt.want {
t.Errorf("getConsulLeader() = %v, want %v", got, tt.want)
}
})
}
}
9 changes: 5 additions & 4 deletions commands/bootstrap/topology.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,11 @@ import (

// AnsibleConfiguration provides Ansible user-defined settings
type AnsibleConfiguration struct {
Version string
PackageRepositoryURL string `yaml:"extra_package_repository_url" mapstructure:"extra_package_repository_url"`
UseOpenSSH bool `yaml:"use_openssh,omitempty" mapstructure:"use_openssh" json:"use_open_ssh,omitempty"`
Inventory map[string][]string `yaml:"inventory,omitempty" mapstructure:"inventory"`
Version string
PackageRepositoryURL string `yaml:"extra_package_repository_url" mapstructure:"extra_package_repository_url"`
UseOpenSSH bool `yaml:"use_openssh,omitempty" mapstructure:"use_openssh" json:"use_open_ssh,omitempty"`
Inventory map[string][]string `yaml:"inventory,omitempty" mapstructure:"inventory"`
HostOperationsAllowed bool `yaml:"host_operations_allowed" mapstructure:"host_operations_allowed"`
}

const (
Expand Down
Loading

0 comments on commit c014b18

Please sign in to comment.