Add go wrapper around git diff-tree --raw -r -M

[McRaeAlex]

• Implemented calling git diff-tree
• Ensures wrapper function is called with valid arguments
• Parses output into go struct, using strong typing when possible

[McRaeAlex]

Context for this PR: This was split out of a larger PR I hope to upstream to gitea for showing all files in a file tree at once rather than having a show more button at the bottom.

Note that this wouldn't change the behaviour of the files list to the right of the file tree. (Lets have that discussion in the PR which modifies the frontend)

[lunny]

And if it's for the left tree of git comparing, I don't think the structure is enough. Maybe it should like

type TreeViewNode struct {
	Name         string          `json:"name"`
	Type         string          `json:"type"`
	Path         string          `json:"path"`
	SubModuleURL string          `json:"sub_module_url,omitempty"`
	Children     []*TreeViewNode `json:"children,omitempty"`
}

This will be introduced in https://github.com/go-gitea/gitea/pull/32721/files#diff-ab9647487f3912999527f743b0fcada9b0bf1858b95e0bb7a684cc9fa383b157

[McRaeAlex]

And if it's for the left tree of git comparing, I don't think the structure is enough. Maybe it should like

type TreeViewNode struct {
	Name         string          `json:"name"`
	Type         string          `json:"type"`
	Path         string          `json:"path"`
	SubModuleURL string          `json:"sub_module_url,omitempty"`
	Children     []*TreeViewNode `json:"children,omitempty"`
}

This will be introduced in https://github.com/go-gitea/gitea/pull/32721/files#diff-ab9647487f3912999527f743b0fcada9b0bf1858b95e0bb7a684cc9fa383b157

The way I did this for allspice was the same structure as this with a function for transforming from output of git diff-tree to something more consumable by the UI.

Heres the function we are using in allspice.

type FileDiffFile struct {
	Name        string
	NameHash    string
	IsSubmodule bool
	IsBinary    bool
	IsViewed    bool
	Status      string
}

// transformDiffTreeForUI transforms a DiffTree into a slice of FileDiffFile for UI rendering
// it also takes a map of file names to their viewed state, which is used to mark files as viewed
func transformDiffTreeForUI(diffTree *gitdiff.DiffTree, filesViewedState map[string]pull_model.ViewedState) []FileDiffFile {
	files := make([]FileDiffFile, 0, len(diffTree.Files))

	for _, file := range diffTree.Files {
		nameHash := git.HashFilePathForWebUI(file.HeadPath)
		isSubmodule := file.HeadMode == git.EntryModeCommit
		isBinary := file.HeadMode == git.EntryModeExec

		isViewed := false
		if fileViewedState, ok := filesViewedState[file.Path()]; ok {
			isViewed = (fileViewedState == pull_model.Viewed)
		}

		files = append(files, FileDiffFile{
			Name:        file.HeadPath,
			NameHash:    nameHash,
			IsSubmodule: isSubmodule,
			IsBinary:    isBinary,
			IsViewed:    isViewed,
			Status:      file.Status,
		})
	}

	return files
}

This allows it to be used in box.tmpl

gitea/templates/repo/diff/box.tmpl

Line 61 in f88dbf8

const diffDataFiles = [{{range $i, $file := .Diff.Files}}{Name:"{{$file.Name}}",NameHash:"{{$file.NameHash}}",Type:{{$file.Type}},IsBin:{{$file.IsBin}},IsSubmodule:{{$file.IsSubmodule}},Addition:{{$file.Addition}},Deletion:{{$file.Deletion}},IsViewed:{{$file.IsViewed}}},{{end}}];

with some other minor modifications (like removing the code which relies on Added and Deleted).

[lunny]

And if it's for the left tree of git comparing, I don't think the structure is enough. Maybe it should like

type TreeViewNode struct {
	Name         string          `json:"name"`
	Type         string          `json:"type"`
	Path         string          `json:"path"`
	SubModuleURL string          `json:"sub_module_url,omitempty"`
	Children     []*TreeViewNode `json:"children,omitempty"`
}

This will be introduced in #32721 (files)

The way I did this for allspice was the same structure as this with a function for transforming from output of git diff-tree to something more consumable by the UI.

Heres the function we are using in allspice.

type FileDiffFile struct {
	Name        string
	NameHash    string
	IsSubmodule bool
	IsBinary    bool
	IsViewed    bool
	Status      string
}

// transformDiffTreeForUI transforms a DiffTree into a slice of FileDiffFile for UI rendering
// it also takes a map of file names to their viewed state, which is used to mark files as viewed
func transformDiffTreeForUI(diffTree *gitdiff.DiffTree, filesViewedState map[string]pull_model.ViewedState) []FileDiffFile {
	files := make([]FileDiffFile, 0, len(diffTree.Files))

	for _, file := range diffTree.Files {
		nameHash := git.HashFilePathForWebUI(file.HeadPath)
		isSubmodule := file.HeadMode == git.EntryModeCommit
		isBinary := file.HeadMode == git.EntryModeExec

		isViewed := false
		if fileViewedState, ok := filesViewedState[file.Path()]; ok {
			isViewed = (fileViewedState == pull_model.Viewed)
		}

		files = append(files, FileDiffFile{
			Name:        file.HeadPath,
			NameHash:    nameHash,
			IsSubmodule: isSubmodule,
			IsBinary:    isBinary,
			IsViewed:    isViewed,
			Status:      file.Status,
		})
	}

	return files
}

This allows it to be used in box.tmpl

gitea/templates/repo/diff/box.tmpl

Line 61 in f88dbf8

const diffDataFiles = [{{range $i, $file := .Diff.Files}}{Name:"{{$file.Name}}",NameHash:"{{$file.NameHash}}",Type:{{$file.Type}},IsBin:{{$file.IsBin}},IsSubmodule:{{$file.IsSubmodule}},Addition:{{$file.Addition}},Deletion:{{$file.Deletion}},IsViewed:{{$file.IsViewed}}},{{end}}];

with some other minor modifications (like removing the code which relies on Added and Deleted).

Since we are rewriting the logic, maybe we can consider the performance more. Perhaps we can avoid traversing twice. Using a callback in the runGitDiffTree function might be more efficient, eliminating the need to create a slice of DiffTreeRecord before transforming it into another slice.

[McRaeAlex]

And if it's for the left tree of git comparing, I don't think the structure is enough. Maybe it should like

type TreeViewNode struct {
	Name         string          `json:"name"`
	Type         string          `json:"type"`
	Path         string          `json:"path"`
	SubModuleURL string          `json:"sub_module_url,omitempty"`
	Children     []*TreeViewNode `json:"children,omitempty"`
}

This will be introduced in #32721 (files)

The way I did this for allspice was the same structure as this with a function for transforming from output of git diff-tree to something more consumable by the UI.
Heres the function we are using in allspice.

type FileDiffFile struct {
	Name        string
	NameHash    string
	IsSubmodule bool
	IsBinary    bool
	IsViewed    bool
	Status      string
}

// transformDiffTreeForUI transforms a DiffTree into a slice of FileDiffFile for UI rendering
// it also takes a map of file names to their viewed state, which is used to mark files as viewed
func transformDiffTreeForUI(diffTree *gitdiff.DiffTree, filesViewedState map[string]pull_model.ViewedState) []FileDiffFile {
	files := make([]FileDiffFile, 0, len(diffTree.Files))

	for _, file := range diffTree.Files {
		nameHash := git.HashFilePathForWebUI(file.HeadPath)
		isSubmodule := file.HeadMode == git.EntryModeCommit
		isBinary := file.HeadMode == git.EntryModeExec

		isViewed := false
		if fileViewedState, ok := filesViewedState[file.Path()]; ok {
			isViewed = (fileViewedState == pull_model.Viewed)
		}

		files = append(files, FileDiffFile{
			Name:        file.HeadPath,
			NameHash:    nameHash,
			IsSubmodule: isSubmodule,
			IsBinary:    isBinary,
			IsViewed:    isViewed,
			Status:      file.Status,
		})
	}

	return files
}

This allows it to be used in box.tmpl

gitea/templates/repo/diff/box.tmpl

Line 61 in f88dbf8

const diffDataFiles = [{{range $i, $file := .Diff.Files}}{Name:"{{$file.Name}}",NameHash:"{{$file.NameHash}}",Type:{{$file.Type}},IsBin:{{$file.IsBin}},IsSubmodule:{{$file.IsSubmodule}},Addition:{{$file.Addition}},Deletion:{{$file.Deletion}},IsViewed:{{$file.IsViewed}}},{{end}}];

with some other minor modifications (like removing the code which relies on Added and Deleted).

Since we are rewriting the logic, maybe we can consider the performance more. Perhaps we can avoid traversing twice. Using a callback in the runGitDiffTree function might be more efficient, eliminating the need to create a slice of DiffTreeRecord before transforming it into another slice.

If we want to avoid iterating over it twice, once when transforming and once when rendering I can modify transformDiffTreeForUI to take in a single record and transform it. Then it can be used in the go template.

However I think this conversation belongs in the next PR. If during the next PR we find the performance of this PRs code unacceptable for whatever reason I am happy to come back and find ways of improving it.

[delvh]

Also, keep in mind that the two queries serve two different purposes:
The first one is supposed to be unpaginated, the second one is still intended to be paginated.
As such, it might even be more performant to keep them separate.

[McRaeAlex]

Also, keep in mind that the two queries serve two different purposes: The first one is supposed to be unpaginated, the second one is still intended to be paginated. As such, it might even be more performant to keep them separate.

With the proposed UI changes we can completely remove this script tag

gitea/templates/repo/diff/box.tmpl

Line 60 in ed84f37

<script id="diff-data-script" type="module">

when file-only=true which will make the "show more" button faster and means we don't have to recompute the file tree on the frontend after every new page.

[McRaeAlex]

@lunny @wxiaoguang Are there any other changes you want to this PR?

I can allocate some time this afternoon to address the testing but will only do so if we feel confident there is nothing else we want addressed.

[lunny]

Since #33507 modified the test data, some tests might need to be updated as well. However, I’m not sure which ones should be changed since this update doesn’t affect any existing code. Do you plan to submit additional PRs to utilize the newly introduced code? If so, we could improve the code in the following PRs.

[McRaeAlex]

Since #33507 modified the test data, some tests might need to be updated as well. However, I’m not sure which ones should be changed since this update doesn’t affect any existing code. Do you plan to submit additional PRs to utilize the newly introduced code? If so, we could improve the code in the following PRs.

I will update all the tests in git_diff_tree_tests.go to use to other test fixtures (like was done in https://github.com/go-gitea/gitea/pull/33507/files). I will post this patch and another stacked PR for the UI changes this afternoon.

Oops, still one pending: #33369 (comment)