Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(upstream): Port upstream changes from Netflix #2067

Merged
merged 70 commits into from
Sep 21, 2023
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
c1ea933
fix(auth): instantiate filter correctly
robfletcher Sep 16, 2021
9adc94d
Merge pull request #571 in SPKR/keel-nflx from fix-java-toolchain to …
robfletcher Sep 17, 2021
4b54863
feat(scheduler): tuneable batch size for checking
emjburns Sep 17, 2021
e8e2648
fix(notifications): use the correct branch for notifications
Sep 17, 2021
21ea3a4
feat(dgs): added a new boolean to the schema that indicates if previe…
Sep 21, 2021
0922394
feat(imdsv2): controlled and instumented rollout of IMDSv2
robfletcher Aug 12, 2021
5a08398
fix(imdsv2): bad column reference in query
robfletcher Sep 21, 2021
de782e0
fix(imdsv2): actually translate IMDSv2 setting in spec to desired low…
robfletcher Sep 21, 2021
db018a9
fix(slack): Fix Slack callback handling
luispollo Sep 21, 2021
80eca72
chore(cluster): abstract construction of upsert stages
emjburns Sep 21, 2021
3581888
fix(imdsv2): get current state of IMDSv2 flag
robfletcher Sep 21, 2021
dec4851
feat(preview-envs): Allow users to override FP email via notifications
luispollo Sep 21, 2021
8bd8769
fix(dgs): improve error message when git repo details are incorrect
Sep 22, 2021
52bddd9
fix(database): standardize table name
robfletcher Sep 23, 2021
e756e0c
fix(delete): Fix pipeline behavior for multi-stage delete tasks
luispollo Sep 24, 2021
ad7cda5
chore(imdsv2): added a missing test case (to reassure myself it worked)
robfletcher Sep 24, 2021
ff9b6b9
fix(preview-envs): Fix moniker in destroyServerGroup
luispollo Sep 24, 2021
2b34930
feat(resolvers): extracted a base class for rollout-capable resolvers
robfletcher Sep 27, 2021
ad17f0e
feat(resolvers): add column to capture rollout state
robfletcher Sep 28, 2021
48e51e3
fix(resolvers): fixed typo in migration
robfletcher Sep 29, 2021
abe027d
feat(reports): Allow loading app cache from disk for local testing
luispollo Sep 29, 2021
8a0bfb0
feat(imdsv2): IMDSv2 resolver for Titus clusters
robfletcher Sep 29, 2021
6d187ff
fix(import): invalidate the cache before fetching the app by name
Sep 30, 2021
481392a
refactor(resolvers): Moved some of the Arrow Optics declarations into…
robfletcher Sep 30, 2021
955c3fb
fix(imdsv2): Wire up the Titus IMDSv2 resolver
robfletcher Sep 30, 2021
575ae54
fix(imdsv2): Fix bean name collision in IMDSv2 resolvers
robfletcher Sep 30, 2021
2de6c73
fix(notifications): explicit specify the notification branch
Sep 30, 2021
b5b0bed
Merge pull request #598 in SPKR/keel-nflx from unpinMessage to master
gal-yardeni Sep 30, 2021
ea4b046
fix(actuation): Properly trigger recheck of resources when a version …
Sep 30, 2021
6c3eeec
feat(imdsv2): Removed the fast property that controls retries as it's…
robfletcher Sep 30, 2021
bb4bbf0
fix(imdsv2): If user fixes IMDSv2 rollout after it failed, we should …
robfletcher Oct 1, 2021
a534415
fix(dgs): move resources to be under the artifact directly
Oct 1, 2021
f5143b9
fix(preview-envs): Revisit resource renaming
luispollo Oct 1, 2021
9290f76
Revert "fix(preview-envs): Revisit resource renaming"
luispollo Oct 5, 2021
85e1b38
fix(build): make build compatible with Java 8 tooling
robfletcher Oct 6, 2021
f8585ba
chore(logs): Fix logging of authenticated user
luispollo Oct 6, 2021
50c028c
fix(build): put io.spinnaker.package plugin back in to see if that's …
robfletcher Oct 6, 2021
8b9b22f
fix(auth): desperately hoping this fixes auth
robfletcher Oct 6, 2021
6f7a545
chore(build): Align kork/fiat coordinates on com.spinnaker.netflix.{}…
luispollo Oct 7, 2021
84726c4
chore(build): see if we can get the build buildable on Java 8
robfletcher Oct 7, 2021
7f8093c
fix(preview-envs): Revisit resource renaming
luispollo Oct 7, 2021
9679b32
chore(logs): Add more debug logs around delivery config import
luispollo Oct 7, 2021
89470b9
fix(resourceTasks): load tasks from task tracking rep
emjburns Oct 7, 2021
93d62f7
feat(constraint): restart constraint evaluation
Oct 8, 2021
3a9958b
Merge pull request #617 in SPKR/keel-nflx from currentVersion to master
gal-yardeni Oct 8, 2021
9f0d2b4
fix(tasks): order sql query before fetching
emjburns Oct 8, 2021
e090ba2
feat(artifacts): specify which artifact is it on MJ notification
gal-yardeni Oct 11, 2021
df2d26a
feat(ops): cancel relevant in flight tasks pin/veto
emjburns Oct 11, 2021
7d8272c
fix(ALBs): ensure listener actions are sorted consistently to avoid f…
robfletcher Oct 11, 2021
133b492
chore(verification): add logs and make batch size fast prop-able
emjburns Oct 12, 2021
ea9569d
chore(batch-size): increase default batch size to 5, fast prop post d…
emjburns Oct 12, 2021
6a860c2
feat(compareLink): show which changeset were deployed as a part of th…
gal-yardeni Oct 12, 2021
9cdec8d
feat(rollout): add managed rollout option for clusters
emjburns Oct 13, 2021
f0bab83
chore(test): fix test for managed rollout
emjburns Oct 13, 2021
dd7b04a
feat(export): First stab at exporting pipelines
luispollo Oct 14, 2021
3502e96
feat(tasks): get only one batch of tasks
emjburns Oct 14, 2021
d2de401
fix(ALBs): nested rule actions need to be serialized the same as defa…
robfletcher Oct 14, 2021
86b01fe
feat(preview-envs): Create preview environment artifacts automatically
luispollo Oct 17, 2021
06988b8
fix(task): load only 30s batch for all task
emjburns Oct 18, 2021
4f50de7
chore(dgs): Migrate to match the reggie schema format
Oct 18, 2021
298be85
feat(rollback): rollback to a server group if one exists with no diff
emjburns Oct 19, 2021
b4ccbdd
fix(build) Remove buoy dependency
osoriano Sep 2, 2023
afcc6f6
Revert "chore(build): Align kork/fiat coordinates on com.spinnaker.ne…
osoriano Sep 2, 2023
9926d0e
Revert "Merge pull request #571 in SPKR/keel-nflx from fix-java-toolc…
osoriano Sep 2, 2023
15e65fa
Revert "fix(auth): instantiate filter correctly"
osoriano Sep 2, 2023
aebc4b9
fix(dependencies) Restore spinnakerGradleVersion
osoriano Sep 2, 2023
6805681
Revert "chore(build): see if we can get the build buildable on Java 8"
osoriano Sep 2, 2023
5d2a4f3
fix(exception): Move from UnsupportedOperationException to TODO
osoriano Sep 20, 2023
7ec9211
Revert "feat(preview-envs): Allow users to override FP email via noti…
osoriano Sep 20, 2023
fee0d81
Revert "fix(slack): Fix Slack callback handling"
osoriano Sep 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ org.gradle.parallel=true
testContainersVersion=1.15.3
okHttpVersion=4.5.0
resilience4jVersion=1.5.0
spinnakerGradleVersion=8.26.0
spinnakerGradleVersion=8.23.0
slackSdkVersion=1.12.1

# Used to control whether to spin up docker to run liquibase before jooq
buildingInDocker=false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ abstract class ClusterDeployStrategy {

companion object {
val DEFAULT_WAIT_FOR_INSTANCES_UP: Duration = Duration.ofMinutes(30)
const val RED_BLACK_STRATEGY = "red-black"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, red-black has been.. "deprecated" in favor of standard terms "blue-green"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing lots of existing "red-black" terms in this repo. Going to check if they can be migrated and follow up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOT an immediate need - spinnaker should auto translate those.

const val HIGHLANDER_STRATEGY = "highlander"
const val NONE_STRATEGY = "none"
const val ROLLING_PUSH_STRATEGY = "rolling-push"
}
}

Expand All @@ -28,7 +32,7 @@ data class RedBlack(
// The order of this list is important for pauseTime based staggers
override val stagger: List<StaggeredRegion> = emptyList()
) : ClusterDeployStrategy() {
override val strategy = "red-black"
override val strategy = RED_BLACK_STRATEGY

override val isStaggered: Boolean
get() = stagger.isNotEmpty()
Expand All @@ -37,13 +41,13 @@ data class RedBlack(
data class Highlander(
override val health: DeployHealth = AUTO
) : ClusterDeployStrategy() {
override val strategy = "highlander"
override val strategy = HIGHLANDER_STRATEGY
}

data class NoStrategy(
override val health: DeployHealth = AUTO
): ClusterDeployStrategy() {
override val strategy = "none"
override val strategy = NONE_STRATEGY
}

data class RollingPush(
Expand All @@ -54,7 +58,7 @@ data class RollingPush(
val totalRelaunches: Int? = null,
val terminationOrder: TerminationOrder? = null
): ClusterDeployStrategy() {
override val strategy = "rolling-push"
override val strategy = ROLLING_PUSH_STRATEGY
}

enum class TerminationOrder {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package com.netflix.spinnaker.keel.api


/**
* When managed rollout is enabled, we will deploy with a ManagedRollout stage instead of
* the normal deploy stage.
*/
data class ManagedRolloutConfig(
val enabled: Boolean = false,
val selectionStrategy: SelectionStrategy? = null
)

// duplication of com.netflix.buoy.sdk.model.SelectionStrategy
// so that we don't add another dependency into this module
enum class SelectionStrategy {
ALPHABETICAL, OFF_PEAK
}
15 changes: 15 additions & 0 deletions keel-api/src/main/kotlin/com/netflix/spinnaker/keel/api/Moniker.kt
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,19 @@ data class Moniker(
detail == null -> "$app-$stack"
else -> "$app-${stack.orEmpty()}-$detail"
}

/**
* @return The [Moniker] with an updated [Moniker.detail] field containing as much of the specified
* [suffix] as possible while respecting max length constraints on resource names.
*/
fun withSuffix(suffix: String, maxNameLength: Int = 32): Moniker {
// calculates the truncation point in the detail field based on how many characters are left of the
// max name length after removing the current detail and accounting for empty stack and detail (which
// cause extra dashes to be added to the name)
var truncateAt = (maxNameLength - toName().length - suffix.length - 1)
if (stack == null) --truncateAt
if (detail == null) --truncateAt else truncateAt += detail!!.length
val updatedDetail = listOfNotNull(detail?.take(truncateAt), suffix).joinToString("-")
return copy(detail = updatedDetail)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,21 @@ data class Resource<out T : ResourceSpec>(
else -> null
}

/**
* Adds the specified [suffix] to the resource [id] and all properties of the [spec] derived from it.
*/
fun deepRename(suffix: String): Resource<T> {
val updatedSpec = spec.deepRename(suffix)
return copy(
spec = updatedSpec as T,
metadata = metadata + mapOf(
// this is so the resource ID is updated with the new name (which is in the spec)
"id" to updatedSpec.id,
"application" to application
)
)
}

// TODO: this is kinda dirty, but because we add uid to the metadata when persisting we don't really want to consider it in equality checks
override fun equals(other: Any?): Boolean {
if (this === other) return true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ interface ResourceSpec {
* form the fully-qualified resource id.
*
* This can be a property that is part of the spec, or derived from other properties. If the
* latter remember to annotate the overridden property with
* [com.fasterxml.jackson.annotation.JsonIgnore].
* latter, remember to annotate the overridden property with [com.fasterxml.jackson.annotation.JsonIgnore].
*/
val id: String

Expand All @@ -30,4 +29,14 @@ interface ResourceSpec {
* other fields.
*/
val displayName: String

/**
* Applies the given [suffix] to the resource [id], and to all aggregate properties of the spec
* whose names are derived from the [id].
*
* @return a copy of the original [ResourceSpec] with the modified identifiers.
*/
@JvmDefault
fun deepRename(suffix: String): ResourceSpec =
throw UnsupportedOperationException("Not implemented")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional unsupported I'd guess for now?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kotlin nit: should use TODO()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found this deepRename call is used for Preview / PR environments. I haven't used this feature yet though

I was tempted to remove the default UnsupportedOperationException here, but it could cause plugins to break if it's not implemented, so I moved to TODO() 👍

}
Original file line number Diff line number Diff line change
Expand Up @@ -16,21 +16,24 @@ interface TaskLauncher {
resource = resource,
description = description,
correlationId = correlationId,
stages = listOf(job)
stages = listOf(job),
artifactVersion = null
)

suspend fun submitJob(
resource: Resource<*>,
description: String,
correlationId: String,
stages: List<Job>
stages: List<Job>,
artifactVersion: String? = null
): Task

fun submitJobAsync(
resource: Resource<*>,
description: String,
correlationId: String,
stages: List<Map<String, Any?>>
stages: List<Map<String, Any?>>,
artifactVersion: String? = null
): CompletableFuture<Task>

suspend fun submitJob(
Expand All @@ -43,7 +46,8 @@ interface TaskLauncher {
correlationId: String? = null,
stages: List<Job>,
artifacts: List<Map<String, Any?>> = emptyList(),
parameters: Map<String, Any> = emptyMap()
parameters: Map<String, Any> = emptyMap(),
artifactVersion: String? = null
): Task =
submitJob(
user = user,
Expand All @@ -56,7 +60,8 @@ interface TaskLauncher {
stages = stages,
type = SubjectType.CONSTRAINT,
artifacts = artifacts,
parameters = parameters
parameters = parameters,
artifactVersion = artifactVersion
)

/**
Expand All @@ -76,7 +81,8 @@ interface TaskLauncher {
stages: List<Job>,
type: SubjectType,
artifacts: List<Map<String, Any?>> = emptyList(),
parameters: Map<String, Any> = emptyMap()
parameters: Map<String, Any> = emptyMap(),
artifactVersion: String? = null
): Task

suspend fun correlatedTasksRunning(correlationId: String): Boolean
Expand All @@ -85,4 +91,9 @@ interface TaskLauncher {
* @return The [TaskExecution] matching the [taskId].
*/
suspend fun getTaskExecution(taskId: String): TaskExecution

/**
* Cancels the given tasks as the provided user identity
*/
suspend fun cancelTasks(taskIds: List<String>, user: String)
}
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ abstract class DeliveryArtifact {
/** Filters for the artifact origin in source control. */
open val from: ArtifactOriginFilter? = null

/** Whether this artifact was created for a preview environment. */
open val isPreview: Boolean = false

@get:ExcludedFromDiff
val filteredByBranch: Boolean
get() = from?.branch != null
Expand Down Expand Up @@ -155,5 +158,14 @@ abstract class DeliveryArtifact {
.map { (it.spec as? ArtifactReferenceProvider)?.artifactReference }
.contains(reference)

/**
* returns the resource ids using the artifact in the environment
*/
fun resourcesUsing(environment: Environment) =
environment
.resources
.filter { reference == (it.spec as? ArtifactReferenceProvider)?.artifactReference }
.map { it.id }

override fun toString() = "${type.toUpperCase()} artifact $name (ref: $reference)"
}
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ data class PublishedArtifact(
name: String,
type: String,
version: String,
reference: String? = null,
status: ArtifactStatus? = null,
createdAt: Instant? = null,
gitMetadata: GitMetadata? = null,
Expand All @@ -43,7 +44,7 @@ data class PublishedArtifact(
) : this(
name = name,
type = type.toLowerCase(),
reference = name,
reference = reference?: name,
version = version,
metadata = (metadata ?: emptyMap()) + mapOf(
"releaseStatus" to status?.name,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ interface ConstraintRepository {

fun getConstraintStateById(uid: UID): ConstraintState?

fun deleteConstraintState(deliveryConfigName: String, environmentName: String, type: String)
fun deleteConstraintState(deliveryConfigName: String, environmentName: String, reference: String, version: String, type: String): Int

fun constraintStateFor(application: String): List<ConstraintState>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ class ArtifactListener(
launch {
val lastStoredVersions = repository.artifactVersions(artifact, artifactRefreshConfig.limit)
val currentVersions = lastStoredVersions.map { it.version }
log.debug("Last recorded versions of $artifact: $currentVersions")
log.debug("Last recorded versions of $artifact: $currentVersions")

val artifactSupplier = artifactSuppliers.supporting(artifact.type)
val latestAvailableVersions = artifactSupplier.getLatestArtifacts(artifact.deliveryConfig, artifact, artifactRefreshConfig.limit)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,8 @@ data class LaunchTemplateData(
val keyName: String,
val iamInstanceProfile: IamInstanceProfile,
val monitoring: InstanceMonitoring,
val ramDiskId: String?
val ramDiskId: String?,
val metadataOptions: Map<String, String> = emptyMap()
)

data class IamInstanceProfile(
Expand Down
Loading