cli/pkg/build/archive.go
Snider 5e2765fd5f feat: wire release command, add tar.xz support, unified installers (#277)
* feat(cli): wire release command and add installer scripts

- Wire up `core build release` subcommand (was orphaned)
- Wire up `core monitor` command (missing import in full variant)
- Add installer scripts for Unix (.sh) and Windows (.bat)
  - setup: Interactive with variant selection
  - ci: Minimal for CI/CD environments
  - dev: Full development variant
  - go/php/agent: Targeted development variants
- All scripts include security hardening:
  - Secure temp directories (mktemp -d)
  - Architecture validation
  - Version validation after GitHub API call
  - Proper cleanup on exit
  - PowerShell PATH updates on Windows (avoids setx truncation)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(build): add tar.xz support and unified installer scripts

- Add tar.xz archive support using Borg's compress package
  - ArchiveXZ() and ArchiveWithFormat() for configurable compression
  - Better compression ratio than gzip for release artifacts
- Consolidate 12 installer scripts into 2 unified scripts
  - install.sh and install.bat with BunnyCDN edge variable support
  - Subdomains: setup.core.help, ci.core.help, dev.core.help, etc.
  - MODE and VARIANT transformed at edge based on subdomain
- Installers prefer tar.xz with automatic fallback to tar.gz
- Fixed CodeRabbit issues: HTTP status patterns, tar error handling,
  verify_install params, VARIANT validation, CI PATH persistence

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add build and release config files

- .core/build.yaml - cross-platform build configuration
- .core/release.yaml - release workflow configuration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: move plans from docs/ to tasks/

Consolidate planning documents in tasks/plans/ directory.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(install): address CodeRabbit review feedback

- Add curl timeout (--max-time) to prevent hanging on slow networks
- Rename TMPDIR to WORK_DIR to avoid clobbering system env var
- Add chmod +x to ensure binary has execute permissions
- Add error propagation after subroutine calls in batch file
- Remove System32 install attempt in CI mode (use consistent INSTALL_DIR)
- Fix HTTP status regex for HTTP/2 compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(rag): add Go RAG implementation with Qdrant + Ollama

Add RAG (Retrieval Augmented Generation) tools for storing documentation
in Qdrant vector database and querying with semantic search. This replaces
the Python tools/rag implementation with a native Go solution.

New commands:
- core rag ingest [directory] - Ingest markdown files into Qdrant
- core rag query [question] - Query vector database with semantic search
- core rag collections - List and manage Qdrant collections

Features:
- Markdown chunking by sections and paragraphs with overlap
- UTF-8 safe text handling for international content
- Automatic category detection from file paths
- Multiple output formats: text, JSON, LLM context injection
- Environment variable support for host configuration

Dependencies:
- github.com/qdrant/go-client (gRPC client)
- github.com/ollama/ollama/api (embeddings API)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(deploy): add pure-Go Ansible executor and Coolify API integration

Implement infrastructure deployment system with:

- pkg/ansible: Pure Go Ansible executor
  - Playbook/inventory parsing (types.go, parser.go)
  - Full execution engine with variable templating, loops, blocks,
    conditionals, handlers, and fact gathering (executor.go)
  - SSH client with key/password auth and privilege escalation (ssh.go)
  - 35+ module implementations: shell, command, copy, template, file,
    apt, service, systemd, user, group, git, docker_compose, etc. (modules.go)

- pkg/deploy/coolify: Coolify API client wrapping Python swagger client
  - List/get servers, projects, applications, databases, services
  - Generic Call() for any OpenAPI operation

- pkg/deploy/python: Embedded Python runtime for swagger client integration

- internal/cmd/deploy: CLI commands
  - core deploy servers/projects/apps/databases/services/team
  - core deploy call <operation> [params-json]

This enables Docker-free infrastructure deployment with Ansible-compatible
playbooks executed natively in Go.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(deploy): address linter warnings and build errors

- Fix fmt.Sprintf format verb error in ssh.go (remove unused stat command)
- Fix errcheck warnings by explicitly ignoring best-effort operations
- Fix ineffassign warning in cmd_ansible.go

All golangci-lint checks now pass for deploy packages.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style(deploy): fix gofmt formatting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(deploy): use known_hosts for SSH host key verification

Address CodeQL security alert by using the user's known_hosts file
for SSH host key verification when available. Falls back to accepting
any key only when known_hosts doesn't exist (common in containerized
or ephemeral environments).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(ai,security,ide): add agentic MVP, security jobs, and Core IDE desktop app

Wire up AI infrastructure with unified pkg/ai package (metrics JSONL,
RAG integration), move RAG under `core ai rag`, add `core ai metrics`
command, and enrich task context with Qdrant documentation.

Add `--target` flag to all security commands for external repo scanning,
`core security jobs` for distributing findings as GitHub Issues, and
consistent error logging across scan/deps/alerts/secrets commands.

Add Core IDE Wails v3 desktop app with Angular 20 frontend, MCP bridge
(loopback-only HTTP server), WebSocket hub, and Claude Code bridge.
Production-ready with Lethean CIC branding, macOS code signing support,
and security hardening (origin validation, body size limits, URL scheme
checks, memory leak prevention, XSS mitigation).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address PR review comments from CodeRabbit, Copilot, and Gemini

Fixes across 25 files addressing 46+ review comments:

- pkg/ai/metrics.go: handle error from Close() on writable file handle
- pkg/ansible: restore loop vars after loop, restore become settings,
  fix Upload with become=true and no password (use sudo -n), honour
  SSH timeout config, use E() helper for contextual errors, quote git
  refs in checkout commands
- pkg/rag: validate chunk config, guard negative-to-uint64 conversion,
  use E() helper for errors, add context timeout to Ollama HTTP calls
- pkg/deploy/python: fix exec.ExitError type assertion (was os.PathError),
  handle os.UserHomeDir() error
- pkg/build/buildcmd: use cmd.Context() instead of context.Background()
  for proper Ctrl+C cancellation
- install.bat: add curl timeouts, CRLF line endings, use --connect-timeout
  for archive downloads
- install.sh: use absolute path for version check in CI mode
- tools/rag: fix broken ingest.py function def, escape HTML in query.py,
  pin qdrant-client version, add markdown code block languages
- internal/cmd/rag: add chunk size validation, env override handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(build): make release dry-run by default and remove darwin/amd64 target

Replace --dry-run (default false) with --we-are-go-for-launch (default
false) so `core build release` is safe by default. Remove darwin/amd64
from default build targets (arm64 only for macOS). Fix cmd_project.go
to use command context instead of context.Background().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-04 00:49:57 +00:00

291 lines
8.4 KiB
Go

// Package build provides project type detection and cross-compilation for the Core build system.
package build
import (
"archive/tar"
"archive/zip"
"bytes"
"compress/gzip"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"github.com/Snider/Borg/pkg/compress"
)
// ArchiveFormat specifies the compression format for archives.
type ArchiveFormat string
const (
// ArchiveFormatGzip uses tar.gz (gzip compression) - widely compatible.
ArchiveFormatGzip ArchiveFormat = "gz"
// ArchiveFormatXZ uses tar.xz (xz/LZMA2 compression) - better compression ratio.
ArchiveFormatXZ ArchiveFormat = "xz"
// ArchiveFormatZip uses zip - for Windows.
ArchiveFormatZip ArchiveFormat = "zip"
)
// Archive creates an archive for a single artifact using gzip compression.
// Uses tar.gz for linux/darwin and zip for windows.
// The archive is created alongside the binary (e.g., dist/myapp_linux_amd64.tar.gz).
// Returns a new Artifact with Path pointing to the archive.
func Archive(artifact Artifact) (Artifact, error) {
return ArchiveWithFormat(artifact, ArchiveFormatGzip)
}
// ArchiveXZ creates an archive for a single artifact using xz compression.
// Uses tar.xz for linux/darwin and zip for windows.
// Returns a new Artifact with Path pointing to the archive.
func ArchiveXZ(artifact Artifact) (Artifact, error) {
return ArchiveWithFormat(artifact, ArchiveFormatXZ)
}
// ArchiveWithFormat creates an archive for a single artifact with the specified format.
// Uses tar.gz or tar.xz for linux/darwin and zip for windows.
// The archive is created alongside the binary (e.g., dist/myapp_linux_amd64.tar.xz).
// Returns a new Artifact with Path pointing to the archive.
func ArchiveWithFormat(artifact Artifact, format ArchiveFormat) (Artifact, error) {
if artifact.Path == "" {
return Artifact{}, fmt.Errorf("build.Archive: artifact path is empty")
}
// Verify the source file exists
info, err := os.Stat(artifact.Path)
if err != nil {
return Artifact{}, fmt.Errorf("build.Archive: source file not found: %w", err)
}
if info.IsDir() {
return Artifact{}, fmt.Errorf("build.Archive: source path is a directory, expected file")
}
// Determine archive type based on OS and format
var archivePath string
var archiveFunc func(src, dst string) error
if artifact.OS == "windows" {
archivePath = archiveFilename(artifact, ".zip")
archiveFunc = createZipArchive
} else {
switch format {
case ArchiveFormatXZ:
archivePath = archiveFilename(artifact, ".tar.xz")
archiveFunc = createTarXzArchive
default:
archivePath = archiveFilename(artifact, ".tar.gz")
archiveFunc = createTarGzArchive
}
}
// Create the archive
if err := archiveFunc(artifact.Path, archivePath); err != nil {
return Artifact{}, fmt.Errorf("build.Archive: failed to create archive: %w", err)
}
return Artifact{
Path: archivePath,
OS: artifact.OS,
Arch: artifact.Arch,
Checksum: artifact.Checksum,
}, nil
}
// ArchiveAll archives all artifacts using gzip compression.
// Returns a slice of new artifacts pointing to the archives.
func ArchiveAll(artifacts []Artifact) ([]Artifact, error) {
return ArchiveAllWithFormat(artifacts, ArchiveFormatGzip)
}
// ArchiveAllXZ archives all artifacts using xz compression.
// Returns a slice of new artifacts pointing to the archives.
func ArchiveAllXZ(artifacts []Artifact) ([]Artifact, error) {
return ArchiveAllWithFormat(artifacts, ArchiveFormatXZ)
}
// ArchiveAllWithFormat archives all artifacts with the specified format.
// Returns a slice of new artifacts pointing to the archives.
func ArchiveAllWithFormat(artifacts []Artifact, format ArchiveFormat) ([]Artifact, error) {
if len(artifacts) == 0 {
return nil, nil
}
var archived []Artifact
for _, artifact := range artifacts {
arch, err := ArchiveWithFormat(artifact, format)
if err != nil {
return archived, fmt.Errorf("build.ArchiveAll: failed to archive %s: %w", artifact.Path, err)
}
archived = append(archived, arch)
}
return archived, nil
}
// archiveFilename generates the archive filename based on the artifact and extension.
// Format: dist/myapp_linux_amd64.tar.gz (binary name taken from artifact path).
func archiveFilename(artifact Artifact, ext string) string {
// Get the directory containing the binary (e.g., dist/linux_amd64)
dir := filepath.Dir(artifact.Path)
// Go up one level to the output directory (e.g., dist)
outputDir := filepath.Dir(dir)
// Get the binary name without extension
binaryName := filepath.Base(artifact.Path)
binaryName = strings.TrimSuffix(binaryName, ".exe")
// Construct archive name: myapp_linux_amd64.tar.gz
archiveName := fmt.Sprintf("%s_%s_%s%s", binaryName, artifact.OS, artifact.Arch, ext)
return filepath.Join(outputDir, archiveName)
}
// createTarXzArchive creates a tar.xz archive containing a single file.
// Uses Borg's compress package for xz compression.
func createTarXzArchive(src, dst string) error {
// Open the source file
srcFile, err := os.Open(src)
if err != nil {
return fmt.Errorf("failed to open source file: %w", err)
}
defer srcFile.Close()
srcInfo, err := srcFile.Stat()
if err != nil {
return fmt.Errorf("failed to stat source file: %w", err)
}
// Create tar archive in memory
var tarBuf bytes.Buffer
tarWriter := tar.NewWriter(&tarBuf)
// Create tar header
header, err := tar.FileInfoHeader(srcInfo, "")
if err != nil {
return fmt.Errorf("failed to create tar header: %w", err)
}
header.Name = filepath.Base(src)
if err := tarWriter.WriteHeader(header); err != nil {
return fmt.Errorf("failed to write tar header: %w", err)
}
if _, err := io.Copy(tarWriter, srcFile); err != nil {
return fmt.Errorf("failed to write file content to tar: %w", err)
}
if err := tarWriter.Close(); err != nil {
return fmt.Errorf("failed to close tar writer: %w", err)
}
// Compress with xz using Borg
xzData, err := compress.Compress(tarBuf.Bytes(), "xz")
if err != nil {
return fmt.Errorf("failed to compress with xz: %w", err)
}
// Write to destination file
if err := os.WriteFile(dst, xzData, 0644); err != nil {
return fmt.Errorf("failed to write archive file: %w", err)
}
return nil
}
// createTarGzArchive creates a tar.gz archive containing a single file.
func createTarGzArchive(src, dst string) error {
// Open the source file
srcFile, err := os.Open(src)
if err != nil {
return fmt.Errorf("failed to open source file: %w", err)
}
defer srcFile.Close()
srcInfo, err := srcFile.Stat()
if err != nil {
return fmt.Errorf("failed to stat source file: %w", err)
}
// Create the destination file
dstFile, err := os.Create(dst)
if err != nil {
return fmt.Errorf("failed to create archive file: %w", err)
}
defer dstFile.Close()
// Create gzip writer
gzWriter := gzip.NewWriter(dstFile)
defer gzWriter.Close()
// Create tar writer
tarWriter := tar.NewWriter(gzWriter)
defer tarWriter.Close()
// Create tar header
header, err := tar.FileInfoHeader(srcInfo, "")
if err != nil {
return fmt.Errorf("failed to create tar header: %w", err)
}
// Use just the filename, not the full path
header.Name = filepath.Base(src)
// Write header
if err := tarWriter.WriteHeader(header); err != nil {
return fmt.Errorf("failed to write tar header: %w", err)
}
// Write file content
if _, err := io.Copy(tarWriter, srcFile); err != nil {
return fmt.Errorf("failed to write file content to tar: %w", err)
}
return nil
}
// createZipArchive creates a zip archive containing a single file.
func createZipArchive(src, dst string) error {
// Open the source file
srcFile, err := os.Open(src)
if err != nil {
return fmt.Errorf("failed to open source file: %w", err)
}
defer srcFile.Close()
srcInfo, err := srcFile.Stat()
if err != nil {
return fmt.Errorf("failed to stat source file: %w", err)
}
// Create the destination file
dstFile, err := os.Create(dst)
if err != nil {
return fmt.Errorf("failed to create archive file: %w", err)
}
defer dstFile.Close()
// Create zip writer
zipWriter := zip.NewWriter(dstFile)
defer zipWriter.Close()
// Create zip header
header, err := zip.FileInfoHeader(srcInfo)
if err != nil {
return fmt.Errorf("failed to create zip header: %w", err)
}
// Use just the filename, not the full path
header.Name = filepath.Base(src)
header.Method = zip.Deflate
// Create file in archive
writer, err := zipWriter.CreateHeader(header)
if err != nil {
return fmt.Errorf("failed to create zip entry: %w", err)
}
// Write file content
if _, err := io.Copy(writer, srcFile); err != nil {
return fmt.Errorf("failed to write file content to zip: %w", err)
}
return nil
}